Topic: e621 Pool Downloader

Posted under e621 Tools and Applications

I has created a e621 pool downloader:
https://github.com/toboe-wolf/e926PoolDownloader/zipball/master

I has made it for YOU! =3

( e621 pool downloader is a modification of:
http://sourceforge.net/projects/danboorupooldl/ )

EDIT (23/10/17):
I have updated the pool downloader with some bug fixes. In particular, it now works with unicode pool names, and will no longer sometimes fail to download certain posts.

Please note that as of right now, it still only works with Python 2.7 (not Python 3)

You can download it for windows/mac/linux here:
https://github.com/toboe-wolf/e926PoolDownloader/zipball/master

Updated by Sevrin

Zenti said:
Not working. :/

it's for windows only. you need python. then put the pool id in the .bat

Updated by anonymous

toboe said:
it's for windows only. you need python. then put the pool id in the .bat

I have windows vista, i have python, and the problem is that the CMD window close as soon as i execute the .bat

Updated by anonymous

Still don't work. So i modified the.bat file and cleans all the syntaxes error in the pool.py file.

Now i have another problem: the downloader.py try to import urllib2 but urllib and urllib2 are merged since python 3.0.0 .... :\

Updated by anonymous

ippiki_ookami said:
Hmm. Yes... I know some of these words.

And we aren't using difficult words for now :o)

Updated by anonymous

It works for me on windows 7 and python 2.7.2, and I love it. Thanks.

Couldn't it be made sticky or something? It would be shame if it drown in flood of other posts.

EDIT:
I've said that it works too soon. It downloaded pool #1920, but at pool #1445 I've got error

http://e621.net/pool/show.xml?id=1445&page=1
Traceback (most recent call last):

(...)

self.downloadPosts()
File "MYDIR\e621boorupooldl_v0_5\_e621boorupooldl_v0_5\util\Downloader.py", line 152, in download
Posts
os.mkdir(poolFolder)
WindowsError: [Error 267] Folder name is incorrect*: u'MYDIR\e621boorupooldl_v0_5\outp
ut_pools\MLP:_Dragon_You_Over!'

*Or something like that - I've translated it from my language.

You should handle, that Windows forbids ':' in folder names.

Adding:

os.chdir(destinationFolder)
        >>self.poolName = self.poolName.replace(":", "")
        poolFolder = os.path.join(os.getcwd(),self.poolName)

in Downloader.py solved this for me, so adding few lines like this (or is there one function for this, I don't know python that good), should fix it.

EDIT 2:
There is another, more important bug. I've started dowloading one pool, and that happened

Downloading /data/3c/c8/3cc8645743303d67ab2e8fda987e7c7f.png (521KB)...
Saving to MYDIR\Downloads\e621boorupooldl_v0_5\output_pools\Love_can_be_different_[colored]\8_22747_questiona
ble.png
Finished.
File is corruptdownloading again.Downloading /data/3c/c8/3cc8645743303d67ab2e8fda987e7c7f.png (521KB)...
Saving to MYDIR\Downloads\e621boorupooldl_v0_5\output_pools\Love_can_be_different_[colored]\9_22747_questiona
ble.png
Finished.
downloading again.Downloading /data/3c/c8/3cc8645743303d67ab2e8fda987e7c7f.png (521KB)...
Saving to MYDIR\Downloads\e621boorupooldl_v0_5\output_pools\Love_can_be_different_[colored]\10_22747_question
able.png
Finished.
downloading again.Downloading /data/3c/c8/3cc8645743303d67ab2e8fda987e7c7f.png (521KB)...
Saving to MYDIR\Downloads\e621boorupooldl_v0_5\output_pools\Love_can_be_different_[colored]\11_22747_question
able.png
Finished.

After getting corrupted file with prefix 8, it re-download it with prefix 9, then re-download it with prefix 10, and again, and again, until I've stopped it.

Updated by anonymous

i apologize.

CHRISTMAS IS CANCELLED.

I will try to fix, yes no?

Updated by anonymous

*TELEGRAPH SOUNDS*
BREAKING NEWS!:

Allied NATO forces have reached a peace agreement with the North Pole. CHRISTMAS IS UN-CANCELLED.

Horribly named pools will no longer cause the downloader to become emo.
The downloader will no longer go around changing filename prefixes for re-attempts.

DOWNLOAD V 0.6 HERE: https://github.com/toboe-wolf/e926PoolDownloader/zipball/master

Please let me know if you run into any more problems. ;3

Updated by anonymous

toboe said:
I CAN HAS STICKY?

No, you can't. It stopped working after recent update of site.

http://e621.net/pool/show.xml?id=2861&page=1
old filename: Walking_the_Neighbor's_Dog
new filename: Walking_the_Neighbor's_Dog
Downloading http://static1.e621.net/data/95/87/95875f1793f3796afd972e8f7fcd978f.
jpg (3688KB)...
Saving to \output_pools\Walking_the_
Neighbor's_Dog\1_356286_safe.jpg
Traceback (most recent call last):
File "_e621boorupooldl_v0_6\pool.
py", line 272, in <module>
main()
File "6boorupooldl_v0_6\_e621boorupooldl_v0_6\pool.
py", line 258, in main
downloader.getPosts()
File "6boorupooldl_v0_6\_e621boorupooldl_v0_6\util\
Downloader.py", line 152, in getPosts
self.downloadPosts()
File "\6boorupooldl_v0_6\_e621boorupooldl_v0_6\util\
Downloader.py", line 187, in downloadPosts
self.getFile(post)
File "\6boorupooldl_v0_6\_e621boorupooldl_v0_6\util\
Downloader.py", line 250, in getFile
file = urllib2.urlopen("http://e621.net" + url)
File "Python27\lib\urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "Python27\lib\urllib2.py", line 394, in open
response = self._open(req, data)
File "Python27\lib\urllib2.py", line 412, in _open
'_open', req)
File "Python27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "Python27\lib\urllib2.py", line 1199, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "Python27\lib\urllib2.py", line 1140, in do_open
h = http_class(host, timeout=req.timeout) # will parse host:port
File "Python27\lib\httplib.py", line 693, in __init__
self._set_hostport(host, port)
File "Python27\lib\httplib.py", line 718, in _set_hostport
raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
httplib.InvalidURL: nonnumeric port: ''

Updated by anonymous

Gilda_The_Gryphon said:
No, you can't. It stopped working after recent update of site.

http://e621.net/pool/show.xml?id=2861&page=1
old filename: Walking_the_Neighbor's_Dog
new filename: Walking_the_Neighbor's_Dog
Downloading http://static1.e621.net/data/95/87/95875f1793f3796afd972e8f7fcd978f.
jpg (3688KB)...
Saving to \output_pools\Walking_the_
Neighbor's_Dog\1_356286_safe.jpg
Traceback (most recent call last):
File "_e621boorupooldl_v0_6\pool.
py", line 272, in <module>
main()
File "6boorupooldl_v0_6\_e621boorupooldl_v0_6\pool.
py", line 258, in main
downloader.getPosts()
File "6boorupooldl_v0_6\_e621boorupooldl_v0_6\util\
Downloader.py", line 152, in getPosts
self.downloadPosts()
File "\6boorupooldl_v0_6\_e621boorupooldl_v0_6\util\
Downloader.py", line 187, in downloadPosts
self.getFile(post)
File "\6boorupooldl_v0_6\_e621boorupooldl_v0_6\util\
Downloader.py", line 250, in getFile
file = urllib2.urlopen("http://e621.net" + url)
File "Python27\lib\urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "Python27\lib\urllib2.py", line 394, in open
response = self._open(req, data)
File "Python27\lib\urllib2.py", line 412, in _open
'_open', req)
File "Python27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "Python27\lib\urllib2.py", line 1199, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "Python27\lib\urllib2.py", line 1140, in do_open
h = http_class(host, timeout=req.timeout) # will parse host:port
File "Python27\lib\httplib.py", line 693, in __init__
self._set_hostport(host, port)
File "Python27\lib\httplib.py", line 718, in _set_hostport
raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
httplib.InvalidURL: nonnumeric port: ''

go here
\6boorupooldl_v0_6\_e621boorupooldl_v0_6\util\Downloader.py
go to line 250, in getFile, find this,

file = urllib2.urlopen("http://e621.net" + url) 

replace with

file = urllib2.urlopen(url)

Updated by anonymous

Gilda_The_Gryphon said:
No, you can't. It stopped working after recent update of site.

http://e621.net/pool/show.xml?id=2861&page=1
old filename: Walking_the_Neighbor's_Dog
new filename: Walking_the_Neighbor's_Dog
Downloading http://static1.e621.net/data/95/87/95875f1793f3796afd972e8f7fcd978f.
jpg (3688KB)...
Saving to \output_pools\Walking_the_
Neighbor's_Dog\1_356286_safe.jpg
Traceback (most recent call last):
File "_e621boorupooldl_v0_6\pool.
py", line 272, in <module>
main()
File "6boorupooldl_v0_6\_e621boorupooldl_v0_6\pool.
py", line 258, in main
downloader.getPosts()
File "6boorupooldl_v0_6\_e621boorupooldl_v0_6\util\
Downloader.py", line 152, in getPosts
self.downloadPosts()
File "\6boorupooldl_v0_6\_e621boorupooldl_v0_6\util\
Downloader.py", line 187, in downloadPosts
self.getFile(post)
File "\6boorupooldl_v0_6\_e621boorupooldl_v0_6\util\
Downloader.py", line 250, in getFile
file = urllib2.urlopen("http://e621.net" + url)
File "Python27\lib\urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "Python27\lib\urllib2.py", line 394, in open
response = self._open(req, data)
File "Python27\lib\urllib2.py", line 412, in _open
'_open', req)
File "Python27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "Python27\lib\urllib2.py", line 1199, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "Python27\lib\urllib2.py", line 1140, in do_open
h = http_class(host, timeout=req.timeout) # will parse host:port
File "Python27\lib\httplib.py", line 693, in __init__
self._set_hostport(host, port)
File "Python27\lib\httplib.py", line 718, in _set_hostport
raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
httplib.InvalidURL: nonnumeric port: ''

Congrats for bumping a 9 month old thread...

Updated by anonymous

But it was a legitimate bump. No point in spreading this topic across multiple threads.

Updated by anonymous

Conker said:
Congrats for bumping a 9 month old thread...

Well it stopped working now, not 9 months ago.

BTW can't we make sticky thread with links to forum thread with useful tools like this thread, forum #70380, forum #48817, or some more I don't remember?

@Aurali
It worked, thanks.

Updated by anonymous

Conker said:
Congrats for bumping a 9 month old thread...

Congrats on missing the point.

Updated by anonymous

Gilda_The_Gryphon said:
Well it stopped working now, not 9 months ago.

BTW can't we make sticky thread with links to forum thread with useful tools like this thread, forum #70380, forum #48817, or some more I don't remember?

@Aurali
It worked, thanks.

No prob, I can read error codes like a bawss <3

Updated by anonymous

Aurali said:
go here
\6boorupooldl_v0_6\_e621boorupooldl_v0_6\util\Downloader.py
go to line 250, in getFile, find this,

file = urllib2.urlopen("http://e621.net" + url) 

replace with

file = urllib2.urlopen(url)

Thanks for helping to fix it. I hadn't used it lately, and I don't know how to watch forum threads; I didn't know it was broken

Updated by anonymous

Might be a "bit" of a bump, but I'm having an issue with your downloader.. First off, love the app, best pool downloader I've found so far! Just recently, in the past couple months, I've been getting this error:

-OUTPUT OMITTED-

C:\Users\jimmychanga\Desktop\New folder (2)\image grabbing programs\6boorupooldl_v0_6_
2>set POOL_ID_LIST=(2988)

-OUTPUT OMITTED-

C:\Users\jimmychanga\Desktop\New folder (2)\image grabbing programs\6boorupooldl_v0_6_
2>.\_e621boorupooldl_v0_6\pool.py 2988 --folder=.\output_pools\ --corrupt_retrie
s=3 --name=${pos}_${id}_${rating}
postCount: 153.0 , totalPages: 7.0
153 posts found in pool "Ehs".

http://e621.net/pool/show.xml?id=2988&page=1
old filename: Ehs
new filename: Ehs
Traceback (most recent call last):
File "C:\Users\jimmychanga\Desktop\New folder (2)\image grabbing programs\6boorupool
dl_v0_6_2\_e621boorupooldl_v0_6\pool.py", line 272, in <module>
main()
File "C:\Users\jimmychanga\Desktop\New folder (2)\image grabbing programs\6boorupool
dl_v0_6_2\_e621boorupooldl_v0_6\pool.py", line 258, in main
downloader.getPosts()
File "C:\Users\jimmychanga\Desktop\New folder (2)\image grabbing programs\6boorupool
dl_v0_6_2\_e621boorupooldl_v0_6\util\Downloader.py", line 152, in getPosts
self.downloadPosts()
File "C:\Users\jimmychanga\Desktop\New folder (2)\image grabbing programs\6boorupool
dl_v0_6_2\_e621boorupooldl_v0_6\util\Downloader.py", line 187, in downloadPosts
self.getFile(post)
File "C:\Users\jimmychanga\Desktop\New folder (2)\image grabbing programs\6boorupool
dl_v0_6_2\_e621boorupooldl_v0_6\util\Downloader.py", line 225, in getFile
extension = os.path.splitext(url)[1]
File "C:\Python27\lib\ntpath.py", line 199, in splitext
return genericpath._splitext(p, sep, altsep, extsep)
File "C:\Python27\lib\genericpath.py", line 91, in _splitext
sepIndex = p.rfind(sep)
AttributeError: 'NoneType' object has no attribute 'rfind'

I have limited coding experience, and can't seem to find what the issue is and why it can't find an "rfind" attribute..

Updated by anonymous

toboe

Privileged

JimmyChanga said:
Might be a "bit" of a bump, but I'm having an issue with your downloader.. First off, love the app, best pool downloader I've found so far! Just recently, in the past couple months, I've been getting this error:

-OUTPUT OMITTED-

I have limited coding experience, and can't seem to find what the issue is and why it can't find an "rfind" attribute..

I've fixed the pool downloader.

You can find the latest version here: https://github.com/toboe-wolf/e926PoolDownloader/zipball/master

Updated by anonymous

I have a problem.I keep getting this error :

C:\Users\*****\Desktop\e621 Image Downloader>REM edit e621.bat, and on the line "
set POOL_ID_LIST" put the id # of each pool inside the parenthesis, seperated by
a space

C:\Users\*****\Desktop\e621 Image Downloader>REM EXAMPLE: set POOL_ID_LIST=(1188
803 1819 1274)

C:\Users\*****\Desktop\e621 Image Downloader>set POOL_ID_LIST=(1167 1874 5417)

C:\Users\*****\Desktop\e621 Image Downloader>for %i in (1167 1874 5417) do .\pool.p
y %i --folder=.\output_pools\ --corrupt_retries=3 --name=${pos}_${id}_${rating}

C:\Users\*****\Desktop\e621 Image Downloader>.\pool.py 1167 --folder=.\output_pools
\ --corrupt_retries=3 --name=${pos}_${id}_${rating}
File "C:\Users\*****\Desktop\e621 Image Downloader\pool.py", line 70
print '\n--Login--'
^
SyntaxError: Missing parentheses in call to 'print'

C:\Users\*****\Desktop\e621 Image Downloader>.\pool.py 1874 --folder=.\output_pools
\ --corrupt_retries=3 --name=${pos}_${id}_${rating}
File "C:\Users\*****\Desktop\e621 Image Downloader\pool.py", line 70
print '\n--Login--'
^
SyntaxError: Missing parentheses in call to 'print'

C:\Users\*****\Desktop\e621 Image Downloader>.\pool.py 5417 --folder=.\output_pools
\ --corrupt_retries=3 --name=${pos}_${id}_${rating}
File "C:\Users\*****\Desktop\e621 Image Downloader\pool.py", line 70
print '\n--Login--'
^
SyntaxError: Missing parentheses in call to 'print'

C:\Users\*****\Desktop\e621 Image Downloader>PAUSE

I hid username for reasons.

Updated by anonymous

Serkan said:
I have a problem.I keep getting this error :

C:\Users\*****\Desktop\e621 Image Downloader>REM edit e621.bat, and on the line "
set POOL_ID_LIST" put the id # of each pool inside the parenthesis, seperated by
a space

C:\Users\*****\Desktop\e621 Image Downloader>REM EXAMPLE: set POOL_ID_LIST=(1188
803 1819 1274)

C:\Users\*****\Desktop\e621 Image Downloader>set POOL_ID_LIST=(1167 1874 5417)

C:\Users\*****\Desktop\e621 Image Downloader>for %i in (1167 1874 5417) do .\pool.p
y %i --folder=.\output_pools\ --corrupt_retries=3 --name=${pos}_${id}_${rating}

C:\Users\*****\Desktop\e621 Image Downloader>.\pool.py 1167 --folder=.\output_pools
\ --corrupt_retries=3 --name=${pos}_${id}_${rating}
File "C:\Users\*****\Desktop\e621 Image Downloader\pool.py", line 70
print '\n--Login--'
^
SyntaxError: Missing parentheses in call to 'print'

C:\Users\*****\Desktop\e621 Image Downloader>.\pool.py 1874 --folder=.\output_pools
\ --corrupt_retries=3 --name=${pos}_${id}_${rating}
File "C:\Users\*****\Desktop\e621 Image Downloader\pool.py", line 70
print '\n--Login--'
^
SyntaxError: Missing parentheses in call to 'print'

C:\Users\*****\Desktop\e621 Image Downloader>.\pool.py 5417 --folder=.\output_pools
\ --corrupt_retries=3 --name=${pos}_${id}_${rating}
File "C:\Users\*****\Desktop\e621 Image Downloader\pool.py", line 70
print '\n--Login--'
^
SyntaxError: Missing parentheses in call to 'print'

C:\Users\*****\Desktop\e621 Image Downloader>PAUSE

I hid username for reasons.

You have python 3.x installed and it is written for python 2.7. In 3.x you need to call print as a function instead of as a keyword.

Updated by anonymous

OP: if you from __future__ import print_function (and adjust the usage of print accordingly, ie. to the py3 style), it should work on both Py2 and Py3. The 2to3 tool can probably do this for you.

You can see here that print_function has been available since alpha2 of 2.6.0 . util/Downloader.py actually already uses print function, it's only pool.py that doesn't.

Updated by anonymous

toboe

Privileged

I have updated the pool downloader with some bug fixes. In particular, it now works with unicode pool names, and will no longer sometimes fail to download certain posts.

Please note that as of right now, it still only works with Python 2.7 (not Python 3)

You can download it for windows/mac/linux here:
https://github.com/toboe-wolf/e926PoolDownloader/zipball/master

Updated by anonymous

Not all heroes wear capes. You my fellow internet user (and all the other furries who helped debug the script) are a hero.

Updated by anonymous

Toboe not sure if you saw this, didn't see a response to this post. But why not make this a browser script? Tampermonkey is one I've seen most commonly used and it's easier to keep up to date

Updated by anonymous

I don't think I understand how to use this. How do I get the pool ID? the only pool ID I know of is something like 12345 but the script asks for a pool ID like 1188 803 1819 1274 apparently. It downloads the default thing so I know the script works, I just don't know how to change it?

Updated by anonymous

  • 1