I'm a little addicted to tagging my porn, and after a few months of tagging my yiff folder contents i've realised i can make my job easier.
This program is made to search e621.net for tags and add them to the images it founds.
Be prepared for a storm of trash (1) tags and typos. On the other hand, you can use the results to see badly tagged images and fix them here on e621.

https://github.com/AyoKeito/e621updater-python

What can my tool do:
Add tags from e621 to your saved pictures (as EXIF metadata, compatible with a lot of things), or as sidecar txt files.

What my tool could do (and will hopefully can in future):
Download favorites for any user (by username)
Download updated pictures if they were replaced there on e621
Find and favorite e621 posts of your locally saved pictures

--------------------
DISCLAMER: The following procedure is highly stupid. I'm not accountable for any injuries. Impalement or circumcision may occur.
If you're running this for the first time, i would advise you to make a copy of your gallery first.
Never not backup your shit.
--------------------

It uses 1 additional open source program: exiftool.exe
--------------------
How to use (Windows):
1) Download latest source from Releases, unzip and place all files into (any) desired directory.
2) Run start.bat and wait for it to download 2 files: artists.parquet and posts.parquet
4) When files are downloaded, tagging part will launch itself. Select a folder with your pictures.
5) Go to sleep
--------------------

It will run through all the PNG and JP(E)G images it founds and try to tag them.
It should take less than 2 seconds to tag one image.

Please have mercy on my tool and remove any occurrences of multiple extensions (e.g. ass.jpg.png).

I haven't actually tested this, but I wanted to mention a couple things.

Keito said:

IT ONLY WORKS FOR IMAGES THAT ARE NAMED AFTER THEIR MD5 CHECKSUM. NO TAGS FOR YOUR "yomamma.jpg" IMAGES!

The md5 is hard to rely on for this (particularly for images that have added metadata), but why not attempt to calculate md5 when the filename match fails?

I know certUtil -hashfile somefilename MD5 works for a single file, but you might be able to push your input through that and get something meaningful out of it (I haven't tried doing this). Else I'm sure there's a freeware utility out there that can do this for you.

Secondly, I highly recommend using our API to retrieve the tag information. It should make queries run noticeably faster but mostly it will be much much easier for you to parse and less likely to break than trying to scan the page for strings.

I prefer JSON myself, but we support XML as well.

JSON: https://e621.net/post/tags.json?md5=b727f5f7982e8d6d1c6e01754562c075
XML: https://e621.net/post/tags.xml?md5=b727f5f7982e8d6d1c6e01754562c075

Just note that the tag types are actually written incorrectly on the API help page (I haven't gotten around to updating it yet). These are the correct ones:

"General"   => 0

"Artist"    => 1

"Copyright" => 3

"Character" => 4

"Species"   => 5

Keito said:
If you can tell me how to implement API, i'd do that.

I don't personally tag my local files, but I figured I'd chime in on this.

If you can find yourself a JSON parser or XML parser to replace your HTML parser, using the API is quite simple. Just replace your request URL https://e621.net/post/index/1/md5: with https://e621.net/post/tags.json?md5= or https://e621.net/post/tags.xml?md5= and make whatever small changes are needed to tell the parser to give you just the tag names from that.

Here's an example of the output:
https://e621.net/post/tags.json?md5=466A53E868164275F9386F0A90A3FE8F
https://e621.net/post/tags.xml?md5=466A53E868164275F9386F0A90A3FE8F

JSON is slightly more efficient than XML, but either should give your program a significant speed boost.

Here's a link to the full API documentation if you wish to learn more: e621:api

Cheers. :)

edit: whoops, parasprite beat me to it.

Thank you for the ideas.
I will switch to API later then, since xidel supports XML (maybe JSON too?). It'll be quite hard for me since API is giving a lot of useless (for what i'm doing) info. I literally only need tags as a plain text, delimited by ",". I'm not even parsing the actual post page, but the search results page tooltips, because they are closer to what i want there. :)
See: http://c2n.me/3nX66OM.png
Parsing that XML page actually look like a lot of more work from the first look. Okay it's actually not that bad after i'll figure out "properties" i think...
But i like the fact that i can export artist tag into appropriate XMP tag. Without tag types, that was impossible.

Regarding filenames, i'm not sure what else can i rely on to automate the process. MD5 sounds like a best shot.
And if the md5 has changed on the server side, there is no way to find a new image automatically AFAIK.

Not sure why would anyone rename his files, but i'll consider adding MD5 checks later.

Keito said:
Not sure why would anyone rename his files, but i'll consider adding MD5 checks later.

Well, I like the filename to be actually meaningful -- much better than '6a7931cc3867dbaced4c032b4af79888', and calculating md5sum only takes a few seconds per 10mb of input, which is completely trivial for 99% of files.

I suppose I should publish my own tool to do this.

BTW, if md5 has changed on the server side, that means that the post you are looking for still exists, as a deleted post (probably with a link to the revised post in the deletion reason.).

Funny thing it indeed exists. I was wondering how did i manage to break "if file exist on server" checks and just now realised he has found all the files when using JSON.
I will contact you later if i fail to found a tool for that.
There is a workaround to first get the original post id, then go to the actual page and parse HTML... But that's not worth it since most of the results i've checked are deleted permanently (takedown requests and stuff).
I wonder if we can add reason to the API?

Updated to v2.

• Reworked the batch file to use JSON.
• Now 2x faster!
• Getting tags for pictures that was deleted.
• Fixed possible PNG bug (not writing anything to PNG)
• Updated exiftool to the most recent version. Just in case.

Forgot to update readme before updating. *sigh*

I don't like internal custom metadata that is not based on intrinsic properties because I hate multiple file versions. So if you use this tool, please do not redistribute your files, but who am I to tell you what to do with them... *grumbles*

Otherwise, go crazy! :)

Chessax said:

Yep, i agree with that. If you guys use this tool, you better don't upload your files anywhere, since they are not original images anymore.
They will differ at least by size and md5 checksum, so we may end up with 10 similar pictures with different tags.

Ah, now I understand why the files must be named in accordance with md5sum. My tool avoids that problem by not modifying the file at all, using TMSU to tag it instead. This has corresponding tradeoffs, of course (if your file manager isn't TMSU aware, renaming or duplicating the file after you've tagged it is not a good idea. On the other hand, you can tag any file type, not just PNGs or JPGs.)

However, previously I had some files which included both a normal name and md5sum string, separated by a comma. That is an option you could consider supporting.

We're version 3 now (and a quick 3.1 update!)
I had little time to test it, but it should work okay.

3.1 Filenames and authors again

• "conditional_dnp" is no longer an artist.
• If you encounter any more trash author tags, let me know.
• All invalid filenames should now be ignored.

3.0 Filenames and authors

• Moved off from "md5 only"
• Tool will remove all dots and underlines from all filenames in directory. Not able to remove other invalid sybols yet.
• Tool will now calculate md5 if file is not named after the checksum and try to use it for search.
• Added PULLER.bat and md5.exe to implement that. Still not a virus.
• Will now save author to the "creator" (author in windows) field as well as into tags!
• Keeping it in tags just in case someone's software only works with one field, not with both tags and authors. For now.
• Will save author to xmp-dc:creator
• Only 1 artist per picture at the moment.
• Removed excessive -api PNGEarlyXMP when calling exiftool. We don't need that for JPG files. We still do need that for PNGs, for some reason.

A little 3.2 update is up.

3.2 Authors! Again.

• I've moved code that is getting artists from e621 to the external file "ARTISTS.bat"
• Now getting all the artists! For some reason, Adobe recognises n-1 artists (not less than 1 tho). Fuck Adobe!
• "unknown_artist" is no longer an artist.
• "avoid_posting" is no longer an artist.

I've honestly tried to make Adobe recognise all the artists, but failed miserably.
All the other software including Windows Explorer gets all the tags correctly. What a shame...

The only improvement that comes to mind is a "hey bro there's updated\biggger\better version on e621 already!" notification, but i'll need some changes on the server side for that (they are planned already).
If you have anything else to suggest, make it rain.

I've added the updater component mentioned in previous post. To use it, simply run updater.bat

UPDATE: It seems that updater might be very slow at the beginning. It depends on amount of files in the directory, and if it's taking more than one second to proceed each filename (1st stage), please tell me about that.
If it's renaming files for more than 5 minutes, you need a fix.
Temp fix: remove lines number 9-46
UPDATE; Fixed in 4.1

4 Updater

• Updater is now included. You may think that it updates this tool, but it will download new pictures instead of the old ones saved locally.
• Many pictures may have new, higher resolution, not watermarked, not fucked up versions uploaded to e621. Updater will find all the new pictures and download them, moving your old files to "backup" folder.
• Still requires testing, since technically it will save parent of deleted picture. It may be the same picture, or it may be edited one (still i guess it should be better if the old one got removed from there)
• Only works with md5 filenames at the moment.
• Possible fix to authors: it should now get authors for not-md5 filenames too.

Great tool! Would you mind adding support to tag the date as well?

To be honest, i have no idea how date works in JSON.
I mean, as an example:
"created_at":{"n":236133000,"s":1445010595,"json_class":"Time"}
How does this translates into 2015-10-16 is a mystery.
If i'll have some spare time to make this work, i will. Otherwise you'll have to live with year tags in "tags".
I can copy year (only) from "tags" to the "date" if you want.

Keito said:
To be honest, i have no idea how date works in JSON.
I mean, as an example:
"created_at":{"n":236133000,"s":1445010595,"json_class":"Time"}
How does this translates into 2015-10-16 is a mystery.
If i'll have some spare time to make this work, i will. Otherwise you'll have to live with year tags in "tags".
I can copy year (only) from "tags" to the "date" if you want.

s is seconds since epoch (Jan 1, 1970 AFAIK), n is nanoseconds (so, this is the fractional part; usually can be ignored)

That said, date doesn't work in JSON, there is no date type. Different applications simply use different hacks to store time in JSON, either as strings or as dictionaries. There are at least 3 common ways.

savageorange said:
That said, date doesn't work in JSON, there is no date type. Different applications simply use different hacks to store time in JSON, either as strings or as dictionaries. There are at least 3 common ways.

Making a date object from a string is dead easy... wouldn't call it a hack.

Lance_Armstrong said:
Making a date object from a string is dead easy... wouldn't call it a hack.

They're all hacks because there is no JSON date type and no 'one single obvious representation method'. Something like RFC 2822 or ISO 8601 is just less of a hack because it has some official weight.

Keito said:
To be honest, i have no idea how date works in JSON.
I mean, as an example:
"created_at":{"n":236133000,"s":1445010595,"json_class":"Time"}
How does this translates into 2015-10-16 is a mystery.
If i'll have some spare time to make this work, i will. Otherwise you'll have to live with year tags in "tags".
I can copy year (only) from "tags" to the "date" if you want.

created_at.s is what's called a Unix timestamp and is a simple integer representation of the time (number of seconds since January 1, 1970). Almost every programming language on earth has a function to convert it to any date format you need. Just Google "<language name> timestamp to date".

4.1 Bugfixes

• Removed filenames checks completely, since it works way too slow for me (and probably for someone\everyone else).
• I don't expect everything to stop working because of that, but feel free to report any bugs.
• Added some more comments to both updater and tagger.
• No longer deletes old log after launching the tagger.

4.2 Updater filenames

• Updater will now calculate md5 if file is not named after the checksum and try to use it for search. (Same thing tagger does, now for updater)

Tool is expected to be not working because of implemented CAPTCHA.
I have not yet tried it out but it shouldn't work if e621\cloudflare is asking you to enter captcha.

Keito said:
Tool is expected to be not working because of implemented CAPTCHA.
I have not yet tried it out but it shouldn't work if e621\cloudflare is asking you to enter captcha.

I'm forwarding this to Dari, our sysadmin, to see if there's a way to fix this.

Edit: The captchas should be less aggressive now. Please test it and let me know if it is working properly.

I'm getting an error 403 (Forbidden) when trying to use this script now.

Vayln said:
I'm getting an error 403 (Forbidden) when trying to use this script now.

It's because the API has changes.

5 GUI

• GUI v1
• Batch script is recoded into a proper EXE with GUI. It's slightly less miserable now!
• Still not a virus.
• Tagger and updater consolidated. Use the program and choose what you want to do.
• xidel is removed, wget added instead.

Screenshot

Feel free to report any bugs and broken images! ;)

5.1 Bugfixes

• "Close app" question at the end of the proccess is fixed:
• Window no longer duplicate if answered "No" after "Update" sequence
• App will correctly restart if answered "No" after "Tag" sequence

If you have any feature requests please post them here.
I would also like to remind all of you that you should avoid sharing your tagged images.

Bringing the heat again!

6 GUI v2

• WGET and it's libraries are removed.
• Internet Explorer 3 or greater must be installed for this program to work. Firewalls or the presence of multiple network adapters may cause this program to fail. You can use WGET edition (see bottom of first post) if you have problems with that.
• You can now use proxies! This program will use a proxy server to access the Internet if such a proxy has been configured in Microsoft Internet Explorer's settings.
• 70-90% faster.
• No longer outputs -user-agent=Keito/Tagtool1.0 for e621 admins. Sorry guys. Can't figure out how to restore this without WGET.
• "Shutdown after" checkbox added. Will shutdown your PC when proccess finishes. Useful!
• "Force remove old tags" checkbox is no longer checked by default. Should make the whole proccess faster. (1-2 seconds per image)
• "Time elapsed" and "time left" counters added. Time left is very inaccurate as expected. They update after every image processed (not realtime).
• ImagesList file is now correctly deleted when program closes.
• Creating after_log.txt file after every launch. Might be useful for everyone who used "Shutdown after" and wants to know how proccess went after.
• Only basic statistics for now. Time start, time end, files processed, files skipped.
• Minor GUI changes.

Screenshot

I've also noticed that there are some invalid tags showing up, e.g. "\u00e9". These are incorrectly parsed special symbols. Pokémon and it's é is a good example of such tag.
Please let me know if you encounter any and PM me with MD5 checksum or ID of that file to fix it. Thanks!

How did you get around the restriction they put in on only being able to search for a single MD5? just spamming the server with requests? I once made an offline image tagger (it's glorious days are over, nobody gave a shit about it and I was sick paying for the server for hosting it) and made a plugin for it to grab the tags from e621:
http://i.imgur.com/36qdSf6.png
but ever since they removed that feature and only allowed searches for single tags, it kinda just lays around on my hard drive, I'm too lazy to tag everything manually. If searching for multiple md5 tags works some other way, I'd probably use it again :)
If you're just spamming server requests, then that's not working for me, a proper pause between requests would make it have ages and not putting in a pause would not sit well with me, I don't want to punish the poor server giving me the infos :)

Also, out of curiosity, where do you put the tags? In the exif data? PNG does not have exif but metadata chunks, but gif have none, so can they not be tagged? or do you put it in an alternate data stream? I toyed with that idea, but it only works on NTFS file systems and all the tags are lost when you copy/move the images to any other, like to a USB stick. Or does it use an internal database so the tags are only visible when you open your program? (that's the way I went back then with my tool)

EDIT: and why did e621 deem it neccessary to remove all spaces after my punctuation marks? lol
EDIT2: and here they are again. what.

Klaue said:
How did you get around the restriction they put in on only being able to search for a single MD5? just spamming the server with requests?

Yep, pretty much. Select file, get it's MD5, request JSON url (https://e621.net/post/show.json?md5=%filename%) using name without extension, if result is empty, request it again using MD5 i've got before, if it's empty again just skip the file. It's not really spamming since it takes 2 seconds or so for each file, but technically it is. :)
I doubt it's hard for server to catch up with this tool working, i guess it's like 3-4 people browsing e621 at the same time...

I'm not really sure about your problem with multiple tags, since i've never ever touched tags.json
I get just enough info from show.json to make things work.

Klaue said:
Also, out of curiosity, where do you put the tags? In the exif data?

Yes, it's actually mentioned in the OP post. I'll just quote it :)
Using XMP dc Subject for tags and XMP dc Creator for artist. It is compatible with Adobe, Windows and Picasa. Probably with all the others.
You are correct, i can't tag GIFs or SWFs. I have A LOT of pictures stored and GIFs are less than 1% of those so i can't really care about them. Plus, i don't really like GIFs as a format and I feel like they are obsolete.
There's no database or alternate data streams. I'll add something like that if i'll be bored enough, but i'm happy with this tool as it is now (well except TODO features in OP post...)

GIFs would be obsolete if there was a decent replacement.
SWFs were supposed to replace them, but they're going nowhere fast, heh.
WEBMs satisfy some of the same need, but not all.
Same applies to APNG (covering basically all the use cases WEBM is bad at).
Once we actually have reliable support for both APNG and WEBM across all major browsers, as well as in image viewers in the case of APNG, GIF will finally be obsolete.
(then we need to work on replacing SWF with SVG.)

@Klaue:
That's a preview-only bug, ignore it.

PNG exif: there are actually multiple ways to do exif in PNG. None of them are official, unfortunately.

Keito said:
Yep, pretty much. Select file, get it's MD5, request JSON url (https://e621.net/post/show.json?md5=%filename%) using name without extension, if result is empty, request it again using MD5 i've got before, if it's empty again just skip the file. It's not really spamming since it takes 2 seconds or so for each file, but technically it is. :)
I doubt it's hard for server to catch up with this tool working, i guess it's like 3-4 people browsing e621 at the same time...

Oh, well, if it takes two seconds, then it doesn't matter too much. Question is why it takes so long though, jou only take the filename, right? even calculating the hash anew takes like 4 ms per image here.
2 secs per file would mean it would take multiple days for my (kinda massive, granted, been collecting for years) local library

with multiple tags I mean that something like that used to work before, now a year past:
show.json?md5=%hash%,%hash2%,%hash3%
(I don't remember how it was exactly, I just know I could check a hundred or so hashes with one server call)

Yes, it's actually mentioned in the OP post. I'll just quote it :)

Ah, my bad :) didn't read the readme, sorry

For your TODO, I fear they're not really possible or rather, make not much sense. All the date e621 has is the upload date and, some times, if it is tagged, a tag for the year. I've seen years old pics uploaded and given a recent date. As for alternative sources, weasyl API has no way to get by hash. FN API is not out yet. FA has none. So you'd be out of luck there. Best bet would be to call a reverse image search, but TinEye wants $200 for even their lowest tier (500 searches) and google has no access to boot and likes to use captchas for sources it deems questionable. InadescentAPI, another one, wants 75 pounds for their lowest tier and claims to use, beside others, google, so my scam alert is tingling. I don't think you can really do that. At least not if the file comes from e621 and hasn't the ID of, say, FA in it ;)

well thanks for the answers :)

and thanks savageorange
personally I'm a fan of APNG.. full color instead of just the 156/frame of gifs, true transparency instead of "transparent or not" and its files are smaller than gifs. Too bad the support for it is abysmal

Klaue said:
Oh, well, if it takes two seconds, then it doesn't matter too much. Question is why it takes so long though, jou only take the filename, right? even calculating the hash anew takes like 4 ms per image here.
2 secs per file would mean it would take multiple days for my (kinda massive, granted, been collecting for years) local library

Honestly i've never bothered with searching for bottlenecks. I just take it as granted - it takes 2-6 seconds to tag one image. I can definately improve speed by making network checks optional. Will do this tomorrow and update you with the new speed charts :P
It took me 24 hours or so to tag all my pictures (~20000), and it was a previous version of my tool with WGET, which made the whole proccess 2 times slower.

Regarding alternative sources, i thought about making some kind of "instructions database" for my updater. E.G. if it's a FA link do this to parse the page, get the image link and download it. Same for all the popular resources.
If the link is bad, e.g simple FA mainpage link or some weird site, just skip it. That could help a lot with artists that decided to go DNP but still had source links in their descriptions.

As for tagging the date - it was requested there (the only thing requested ever!) and i too don't see any real use for it. But still decided to save it to TODO in case i'll be very-very bored.

7 Need for Speed - v3

• Performance improved
• 2 features added: "Fast mode" and "Disable network checks"
• "Fast mode" disables waiting for program to finish all in-app calls (for all exiftool and some wget calls, e.g call exiftool to tag picture). Should not cause problems. Experimental, default on. ~50% performance increase.
• "Disable network checks" disables... Network checks. All of them. Don't use it if your connection is unstable (VPN, proxy, shitty network, etc). ~50% performance increase.
• Use both for maximum speed! Almost 3x times faster.
• Small GUI changes
• Error in log file for updater is fixed

Speed comparison
Seconds spent to tag one image. Lower is better. Orange is default version without WGET. Blue is WGET version.

Oops! My bad!
I've tried to make my code more readable and broken mode selection.
It's fixed now.

8 Favorites

• Favorites downloader added!
• Just select needed mode, enter your username and it will get all the favs you have.
• Ratings selection. Able to download any combinations of ratings (e.g. only E and Q, without safe images).
• Existing files (that you already have in target folder) are skipped.
• Connection checks are not supported at the moment. Don't use it for shitty network.
• Now calculating MD5 inside the tool itself.
• Removed MD5.exe
• GUI changed.
• "More information" checkbox added for all modes.
• Information field on bottom of the tool is replaced with list box. Tool will probably work too fast for you to get any info from there.
• "Tag after updating" option added for favorites and updater modes.
• "Fast mode" is now always on, option removed. No drawbacks from using it.
• Optimized JSON parsing.
• Corrected mistakes in log file again.

I've failed to reproduce the bug i've had with this tool tagging pictures with the same artist and tag after some number of properly tagged images.
I'm reuploading it and reverting exiftool to previous version we had just in case.
Tagging mode is working fine after some checks i've done. Maybe it's another mode that's tagging images wrong, but i hardly imagine how can any other mode overwrite all the images you have.
Please tell me if you run into this problem yourself. You can always use v7 WGET before i fix the possible bug.

On a side note, updater mode progressbar is bugged and shows nothing... Maybe?
I'll look into that later.

Anonyderps said:
For some reason its not tagging PNG files for me.
Is this a bug?

My guess is that you're trying to see those tags in WIndows Explorer. PNG's tags are not visible in it, because ~standards~.
You should use something like Picasa or Adobe Photoshop\Lightroom to see them. Alternatively, you can convert all your PNGs to JPGs (keeping correct MD5 in filenames). I can't recommend that but it's a workaround if you really need those Windows tags.
See: https://dl.dropboxusercontent.com/u/21291309/3A3phoT.png
There's no universal, "official" way to apply EXIF tags to PNGs. Therefore, Microsoft decided not to bother and is not trying to get tags from PNG files, since there is a ton of ways to tag them, yet none of these methods is recommended.
See: http://stackoverflow.com/questions/8113314/does-png-support-metadata-fields-like-author-camera-model-etc

There's also another problem with PNGs: i'm writing XMPs using PNGEarlyXMP flag, because i had some problems with tagging in early versions of this tool. So we're one more step away from any standards (shouldn't cause problems tho).
You can find more technical information about it here

Just tell me if you won't find them in Photoshop too.

Erm, It's not allowing me to even open the app. It says that this app is not compatible with my PC. I'm running windows 10 on a 64-bit dual-core 1.6 Ghz processor.

Can the opposite of the "get favorites" feature be implemented? Check a specified folder for images present on e621 and add them to a specified user's favorites? That's what I would use.

Faux-Pa said:
Erm, It's not allowing me to even open the app. It says that this app is not compatible with my PC. I'm running windows 10 on a 64-bit dual-core 1.6 Ghz processor.

I'm using the same system and it's working fine. I've just tried downloading it to a fresh system and it's working there too. Something is wrong on your end. Try to download it again and extract to your desktop, try running from there. A few more things to check out: anti-virus, EMET or something that acts like it.
Also try to give me a screenshot or a precise quote of error message.

abadbird said:
Can the opposite of the "get favorites" feature be implemented? Check a specified folder for images present on e621 and add them to a specified user's favorites? That's what I would use.

It's definately possible to code and i will add it to planned improvements list. Not sure when i'll do that tho.

Okay, so i've managed to code what you've asked for.
It's called sync mode and requires you to put a curl.exe next to the main executable to work. Pretty much any curl will work, but you better use mine.

It was surprisingly fast so expect bugs (in other modes too...)
Also this exact mode is extremely stupid and won't catch errors or posts that don't exist: it will just bruteforce through your images folder.

9 Sync

• Sync mode added!
• Enter your username into the "name" field (get it from https://e621.net/user/home to be sure)
• Generate an API key and enter it into the "API key" field (get it from https://e621.net/user/api_key )
• Press start and all local images will have their E621 posts favorited.
• No checks are coded: make sure there are no mistakes and spaces in both fields or you'll simply waste your time.
• Sync mode requires curl.exe to work. Proxy is not supported.
• Small GUI changes
• Updater mode backup is fixed

I don't have time to check if other modes are working correctly so if something is broken, use the old version

My soul is bared. That's something I sort of wanted to do for a few years. Now, if I ragequit furry/porn or my storage/computer dies again, I can pick up where I left off. Thanks, and thanks for working so quickly.

There's some uses your program doesn't handle:

• skips animations. Can you add support to sync mode for the other extensions supported by e621? Tagging isn't needed, just checking MD5 and favoriting.
• fails filenames that contain more or less than just their MD5. Can you create a "second pass" option that checks e621 for the MD5 hashes of local files that were "not found" (but doesn't rename them)?
• it seems like a version of this idea is already (partially?) built into the program with the Extracted MD5 field, which does work, but the result is always Not Found and such files aren't added to my favorites. I tested with a file that I downloaded from e621 that I don't have favorited, renamed it, ran it through your program, and got the Status "Not Found!" and it wasn't added to my favorites.
• scope creep: if some files still aren't found on e621, it would conceivably be handy to uploaders for your program to produce a list of the files that truly are not on e621.
• checks entire contents of previously sync'd folder again if sync'ing is attempted a second time. Would it be possible to download some sort of list of a user's current favorites and check local files against that first? If that's possible, it seems like it would be faster? Maybe you already do that. I wouldn't know. It's just that something like FreeFileSync compares the contents of local folders nearly instantaneously (for my purposes), so I assume something similar should be possible with a local list of a user's favorites.
• program doesn't remember anything, namely previously entered username, API key, folder of local files, and window position
• after completing a sync of a folder and answering the prompt that I don't want to close the program yet, the program effectively relaunches itself. Compounds difficulty of use with the previous issue because I have to re-enter everything.
• Choose Another Folder also effectively relaunches the program
• doesn't check subfolders. This would be a nice option, and an actual checklist, hierarchy, or tree for manual selection of individual folders/subfolders would be even better. Compounds difficulty of use with the previous two issues, meaning a user needs to re-enter everything to sync the contents of each folder.
• Pause buttons functions as a Pause/Resume feature, but there is no proper cancel/abort mechanism except for closing the program.

I'm not demanding this be done tomorrow or anything, just providing feedback on the "holes" of your program as I see it. I'm appreciative for whatever you implement, whenever you implement it.

abadbird said:

• skips animations. Can you add support to sync mode for the other extensions supported by e621?

Sure, that happened because i've used code from another mode and it only supports images. I can fix that easily a bit later.

fails filenames that contain more or less than just their MD5.

• it seems like a version of this idea is already (partially?) built into the program

I'll see what's wrong but it uses exactly the same code as the updater. Maybe i've managed to remove something important from it when i've coded sync mode...

checks entire contents of previously sync'd folder again if sync'ing is attempted a second time.

Similar thing is already on todo list.

program doesn't remember anything, namely previously entered username, API key, folder of local files, and window position

I have no idea how to save window position but i already thought about adding config file functionality (save settings to file, load them from file). Added to todo.

after completing a sync of a folder and answering the prompt that I don't want to close the program yet, the program effectively relaunches itself.

Yep, it was much easier to code that :P
I did that to make sure nothing bad happens when you change modes because all variables are cleared and other garbage is removed when program restarts.

doesn't check subfolders.

That's intentional because program creates working folders such as Backup and NotFound and moves some files to them. I guess i can ignore them tho...

9.1 Sync and config

• Sync mode updated:
• No longer limited to images: webm, gif and swf support added.
• Will calcualte MD5 for all incorrectly named files.
• Config file added (e621updater.ini):
• Settings saved for now: last used directory, username and API key, "network checks" and "backup" checkboxes state.
• Updater mode backup is fixed

Good news:

• The program seemed to correctly hash and favorite a few pictures that didn't have MD5 filenames, as from tumblr, inkbunny, hentai-foundary, and so on.

• GIFs were also favorited.

• Username and API key saved properly. That's two less repetitive steps per folder.

• WebMs and SWFs appear in local files list.

Bad news:

• SWFs only found and favorited if the filename was an MD5 hash. Other filenames produced a -1 MD5 Extracted value and Status "Not found". Old code?

• WebMs not found and favorited regardless of filename.

I tested one SWF and one WebM once with MD5-only filenames and again with extra text in their filenames. I hashed both local files using an online MD5 hasher and confirmed that they were unmodified, and I searched their hashes on e621 and found the expected posts.

Other stuff:

• With no mode selected, I can press Start and the program seems to hang. Start button should be grayed out until a mode is selected?

• When I Choose Another Folder, the last used directory is not brought up. That's what I really wanted, so I wouldn't have to navigate 4 or 5 folders to my base furry repository folder every time I wanted to switch folders. Currently, copy and paste folder path from Explorer is faster.

• Add a quick link to e621's API key generation page for user-friendliness?

Thanks again.

Keito said:
I'm using the same system and it's working fine. I've just tried downloading it to a fresh system and it's working there too. Something is wrong on your end. Try to download it again and extract to your desktop, try running from there. A few more things to check out: anti-virus, EMET or something that acts like it.
Also try to give me a screenshot or a precise quote of error message.

http://d621.net/static/error1.1.png : Here's that screenshot you asked for.

I assume you've tried running this app with admin rights? Also try to set UAC to "Never notify me" option.
I can't find any reasons for this app to throw this message. Most of problems that cause this are dumb, something like "you are running zero-length exe file" or "you are running 16 bit app" none of which apply. Damn, i even have the same system (10 x64)...

Keito said:
I assume you've tried running this app with admin rights? Also try to set UAC to "Never notify me" option.
I can't find any reasons for this app to throw this message. Most of problems that cause this are dumb, something like "you are running zero-length exe file" or "you are running 16 bit app" none of which apply. Damn, i even have the same system (10 x64)...

Heheh... I guess I read my computer's arch type wrong. Apparently when I was reinstalling Windows 10, the install media decided to install the 32-bit verion of Windows 10 Home Edition as opposed to the 64-bit version.

Will you be releasing a 32-bit version of this program, or am I out of luck?

Hah, i actually never wanted it to be x64. I'll make sure all next updates will be x32.
For now, use this link to download x32.

Keito said:
Hah, i actually never wanted it to be x64. I'll make sure all next updates will be x32.
For now, use this link to download x32.

Works flawlessly. I see the tags in the file details, but now I need a Adobe Lightroom kind of tool to organize the pictures. Any recommendations?

Edit:I know that DigiKam is one that I've used that was great, but I didn't know if it was for Linux only.

Try using Picasa. AFAIK Google got rid of it but it still works as a desktop app. You can also use simple Windows search if you want to find pictures with tags\tag combinations you want.
If none of them are good for you, there's nothing better than Lightroom imo.

9.2 QoL

• App is x32 now, should work for everyone.
• Sync mode bugfixes:
• SWFs and WEBMs are correctly favorited.
• MD5 is now correctly found for SWFs and WEBMs.
• GUI bugfixes.
• Choose Another Folder remembers your last folder and starts from there.
• "Get API" button added for sync mode.
• DLLCall added to prevent PC from sleeping while app is working.
• Can be disabled by modifying config file: manually set PCAwake to 0.

Oh and by the way i'm now compressing the EXE with MPRESS, so some antivirus software may "find" something in it. Fear not!

Keito said:
Try using Picasa. AFAIK Google got rid of it but it still works as a desktop app. You can also use simple Windows search if you want to find pictures with tags\tag combinations you want.
If none of them are good for you, there's nothing better than Lightroom imo.

Picasa works beautifully for me. DigiKam was making me want to bash my face into brick walls just for fun.

Thanks for the update! Everything seems to be working. E621 Updater now meets my current needs.

If a suggestion would be okay, would it not be possible to take advantage of NTFS's support for storing property data in an annex to any file, thereby enabling proper metadata for the formats not currently taggable, mainly gifs, a real pain tagging all these gifs one at a time.

Windows doesn't normally allow for this except for a few file formats because the feature was pretty much only 99% completed, but there's a free program that finishes the feature thereby allowing all files to have metadata on them that can be searched through windows search and explorer like such~ http://i.imgur.com/HBzI9Wn.png

Hope this is an okay suggestion~ ='w'=

NekoSnowy said:
but there's a free program that finishes the feature

How is it called? Any documentation on it?
I will definately add some way to save tags without touching the file in future.

Keito said:
How is it called? Any documentation on it?
I will definately add some way to save tags without touching the file in future.

It's this thing over there, I'm just an intermediate level programmer, more of a web and graphical designer, so sorry if I can't be of any more help outside of whatever documentation is inside this link~

https://filemeta.codeplex.com/

For those first discovering this and wishing to use Picasa, you can use this link: https://dl.google.com/picasa/picasa39-setup.exe

Google has shut down picasa but the desktop application should still work fine. However, you have to go to the wayback machine to find the link as it's no longer public facing.

Yep, desktop app works if you manage to find it.
I've recently switched to just searching the folder with explorer and launching a slideshow. Works just fine. ;)

RE: alternative data streams, i'll look into it in the next update. Not sure what method\tool to use yet, but i'll see what i can do.

NekoSnowy said:
If a suggestion would be okay, would it not be possible to take advantage of NTFS's support for storing property data in an annex to any file, thereby enabling proper metadata for the formats not currently taggable, mainly gifs, a real pain tagging all these gifs one at a time.

Windows doesn't normally allow for this except for a few file formats because the feature was pretty much only 99% completed, but there's a free program that finishes the feature thereby allowing all files to have metadata on them that can be searched through windows search and explorer like such~ http://i.imgur.com/HBzI9Wn.png

This sounds like basically the same idea as Ext4 xattrs. The principle problem with storing important data in xattrs is revision and replacement. For example, you load an image in, edit something, and save. There are two possible ways a program would implement that:

• Truncate the file and rewrite it from the start
• Write a new file, and when it's finished writing, remove the old file. Finally, move the new file to the same location as the old file.

The former approach would preserve xattrs, but it is not safe (an error during writing could result in a completely broken file). The latter approach is safe, and thus by far the most common approach used. But it only copies xattrs if the program explicitly transfers them over too. I would be very surprised if this is any different for NTFS properties.

The other main concern being if you zip/rar/7z them up, are the properties preserved? Again, this depends on the individual archiver to 'do the right thing', otherwise you just won't get this data on the extracted files. If you change filesystems, you may not get the data even if the utility did store it. Or you might get just part of the data, as different filesystems have different restrictions on attribute size.

All of the above is by way of saying you need to be very cautious about storing data in anything that isn't strictly an ordinary file.

Extended attributes / properties are, AFAIK, in practice mainly used for the purpose of caching expensive calculations. Search tools extracting key information from files is the classic example of this.

EDIT: OP, you might want to look at this page , which basically takes the premise 'We want tagging via extended attributes.' and really looks into what is involved. Then it backtracks a bit and looks at the question of "what can we currently get?". You might not want to read -all- of it, but it gives some solid context.

It is a concern when programs, rather than update the file in place, save updates using a write new file, delete original file, rename new to original method. The former would leave the metadata in alternate data streams intact as far as I'm aware, while the latter wipes it out.

For archiving, you have to store the files in a WIM with the store NTFS alternate data streams enabled, and then compress that, which will preserve the metadata that's inside the NTFS ADS. As a backup, you can have filemeta, the program I mentioned earlier, export the metadata from the NTFS ADS to an XML file, and then filemeta can import it back to the file.

Honestly none of those looks good for me. Most of them are tied to using some kind of database or attributes and 3rd party software that may or may not work for some people. Supporting that kind of stuff looks painful.
I guess i'll put alternative tagging methods to the end of todo for now... Maybe i'll find some decent method later.

Keito said:
Honestly none of those looks good for me. Most of them are tied to using some kind of database or attributes and 3rd party software that may or may not work for some people. Supporting that kind of stuff looks painful.
I guess i'll put alternative tagging methods to the end of todo for now... Maybe i'll find some decent method later.

Yeah, it's pretty painful, and nothing is great. You could maybe try making something that just downloads the tags into XML files that filemeta can recognize so it can import the tags to the files for you, I don't know, that's all I have. I wish you luck in figuring this out, not an easy task~

I'm not familiar with how to view tags or how to take advantage of using them. Are the tags only available through certain programs? Do they have to be enabled? It doesn't appear like they can be searchable but I don't actually know. What are some of your tagging workflows? What do you use them for?

Artifice781 said:
I'm not familiar with how to view tags or how to take advantage of using them. Are the tags only available through certain programs?

Mostly, but systems like TMSU that employ VFS enable you to eg. search by tags via browsing the filesystem.

Do they have to be enabled?

I have no idea what that would mean, unless you mean in the context of the options of a particular program.

It doesn't appear like they can be searchable but I don't actually know.

Yeah, they are searchable. Would be no point otherwise IMO. Although the embedded-tags approach used by e621 updater can be a bit slow to search (data must be extracted from each image in order to determine whether it matches the search)

What are some of your tagging workflows? What do you use them for?

For images (downloaded images, my art), I have it set up so that the tags of the currently viewed image is displayed in sxiv's status bar. When I type an alphabetic character, it opens an interactive tagger (add tags by typing, with autocompletion etc).
If I want to find files by tag, I either mount the TMSU VFS, eg tmsu mount /tmp/tfs and browse that, or just do the search on the commandline, eg tmsu files apple pear not banana.

In my file browser (SpaceFM), I have it set up so that Shift+Space triggers the same interactive tagger on the active/selected files. I also have a script bound to Shift+Keypad_0 which simply displays the tags of the selected files in a basic UI.

Overall I really use it for.. any and all tracking. Since I can in effect associate arbitrary text data with any file, it's easy to use this to:

• rate quality of things
• Track admin work (things that need organization, deletion, etc)
• Track reading progress (eg. via a tag+value pair: 'p=50')
• Add crossreferences to other files by referencing their fingerprint hash
• Arbitrarily group project files without needing to have them all in the exact same directory.
• yeah.. any and all bookkeeping that needs to be searchable, basically.

If you are asking specifically about this app, then...

Artifice781 said:
Are the tags only available through certain programs?

They are available through the most of file managers and image viewers. For Windows i use explorer and Picasa or Adobe Lightroom. For Android i use F-Stop. They should be visible in most of the programs you may use for viewing images tho.

Artifice781 said:
Do they have to be enabled?

Nope, if your file\image manager supports tags (XMP in particular), they should be visible. Well maybe they are hidden somewhere in your software's interface.
In Windows Explorer for example you can enable "Tags" column in details layout. That's not very useful but you can see that your images are tagged at least.
To make use of them there, see the next quote.

Artifice781 said:
What are some of your tagging workflows? What do you use them for?

The easy way (i use in W10): just open the folder with (tagged) images, go to search panel -> other properties -> tags. It basically just adds "tags:" to the search field. Then after that i type, for example, "female domination". Then after a minute or two i can launch a slideshow and enjoy some femdom action, lol.
For Adobe Lightroom it's easier: there's a list of ALL tags from imported folder, so i just click checkboxes near tags i'm interested in and others disappear. Then i can launch the slideshow.

Quick tip: If exiftool isn't writing tags to your files, make sure your working path/directory doesn't have any spaces in it. No folders with spaces. "D:\this folder won't work\0C19A2AA4533C71961ED6E468E0A42C4.jpg" won't be tagged.

[Edit] I got jpgs to work, but I can't seem to get exiftool to tag png format. This seems to be a known issue.

Artifice781 said:
Quick tip: If exiftool isn't writing tags to your files, make sure your working path/directory doesn't have any spaces in it. No folders with spaces. "D:\this folder won't work\0C19A2AA4533C71961ED6E468E0A42C4.jpg" won't be tagged.

This is a quoting bug, and almost certainly e6updater's responsibility. If you are using exiftool independently on the command-line, you'll need to doublequote such paths (as you did in your post)

[Edit] I got jpgs to work, but I can't seem to get exiftool to tag png format. This seems to be a known issue.

Works OK for me. AFAIK the bug is in Explorer (or more properly, the bug is in the PNG spec not standardizing this stuff..)

Artifice781 said:
[Edit] I got jpgs to work, but I can't seem to get exiftool to tag png format. This seems to be a known issue.

PNG's should work if JPGs work for you. But they are not displayed in many programs. See post #200333
Try file properties in photoshop.

downloading a program from an image booru without source code published is begging for something bad happening to your machine - virus scans or otherwise.

fewrahuxo said:
downloading a program from an image booru without source code published is begging for something bad happening to your machine - virus scans or otherwise.

Yeah, whatever. Not going to bother with other files in package. Feel free to scan them. :)
Or you can just manually do what this program does - i don't mind.

Keito said:
Yeah, whatever. Not going to bother with other files in package. Feel free to scan them. :)
Or you can just manually do what this program does - i don't mind.

there are things that a virus scanner won't pick up - such as accessing your files, getting metadata about your computer, and sending information about such things - which won't be detected, and never will be due to the source not being released.

even if a program doesn't do these things, the idea that a program has the capability to is cause for concern.

all it takes is a GitHub profile with the source released to make a trustworthy program. surely you've heard of GitHub?

fewrahuxo said:
even if a program doesn't do these things, the idea that a program has the capability to is cause for concern.

I'm pretty sure that most of the people who downloaded this don't have these concerns. You can also check if this program connects to anything that's not e621.net: it's not exactly hard to do. If you find any addresses that are not affiliated with e621, i'll release the sources right away. :)

fewrahuxo said:
all it takes is a GitHub profile with the source released to make a trustworthy program. surely you've heard of GitHub?

I've heard about github but i don't want to bother with uploading anything as long as i keep updating this program.
First of all, my code is so shitty even i blush when i look at it. I'm not prepared for 10000 optimizations it needs. I also don't want anyone to fix them for me because it will be hard to realize what was changed and why - i won't be able to add functionality because i won't know what's where and what does it do.
As long as it's my hobby and the first proper GUI program i've coded i'll prefer not to share the sources. But yeah, if i abandon developing it, i'll release the sources: it's mentioned in the OP post.

(btw it's quite easy to get the source code from exe: it's an AHK script packed into EXE with AHK's built-in tool and compressed with MPRESS. If you really want the code, it won't take more than 10 minutes to get it)

Artifice781 said:
"D:\this folder won't work\0C19A2AA4533C71961ED6E468E0A42C4.jpg" won't be tagged.

I see the problem in my code, but i'm not sure how to fix it yet. :)
I'll look into it next time i update something.