Topic: e621 updater - tag local files!

I'm a little addicted to tagging my porn, and after a few months of tagging my yiff folder contents i've realised i can make my job easier.
This program is made to search e621.net for tags and add them to the images it founds.
Be prepared for a storm of trash (1) tags and typos. On the other hand, you can use the results to see badly tagged images and fix them here on e621.

https://github.com/AyoKeito/e621updater-python

What can my tool do:
Add tags from e621 to your saved pictures (as EXIF metadata, compatible with a lot of things), or as sidecar txt files.

What my tool could do (and will hopefully can in future):
Download favorites for any user (by username)
Download updated pictures if they were replaced there on e621
Find and favorite e621 posts of your locally saved pictures

--------------------
DISCLAMER: The following procedure is highly stupid. I'm not accountable for any injuries. Impalement or circumcision may occur.
If you're running this for the first time, i would advise you to make a copy of your gallery first.
Never not backup your shit.
--------------------

It uses 1 additional open source program: exiftool.exe
--------------------
How to use (Windows):
1) Download latest source from Releases, unzip and place all files into (any) desired directory.
2) Run start.bat and wait for it to download 2 files: artists.parquet and posts.parquet
4) When files are downloaded, tagging part will launch itself. Select a folder with your pictures.
5) Go to sleep
--------------------

It will run through all the PNG and JP(E)G images it founds and try to tag them.
It should take less than 2 seconds to tag one image.

Please have mercy on my tool and remove any occurrences of multiple extensions (e.g. ass.jpg.png).

Questions and Answers

Q: Will it tag everything?
A: Everything that is found on e621 by searching md5 hash. No, i doubt it will tag ALL of your files.

Q: Will it break everything?
A: Probably not. In the worst scenario, it will remove present metadata and waste your time.

Q: Will it tag GIFs?
A: It won't. They are bad with metadata. Only PNG, JPG and JPEG files are supported.

Q: What if %filename% is not found on e621?
A: Nothing will happen, that file will be moved to "NotFound" directory.

Q: How much files can i tag?
A: The number of files is unlimited. Tag all the things.

Q: How exactly is it tagging my images?
A: Using XMP dc Subject for tags and XMP dc Creator for artist. It is compatible with Adobe, Windows and Picasa. Probably with all the others.

Q: Some of my images are already tagged. What's going to happen?
A: All of your tags are going to be erased. Sorry.

Q: Can i use that with proxy?
A: You don't need a proxy to tag images. You may need one to download the database, tho. You can use --proxy flag to specify proxy.

Q: Are you using API?
A: Don't need it at the moment. Using database export that is updated daily.

Q: Is it a virus?
A: No. Current version is not a virus. Source code is open on Github.

Changelog

Python V1:

  • Rewritten in Python via ChatGPT.
    • Only tagger mode is working in this version.

9.5 Functionality restored

  • All modes are re-enabled after limited testing.
  • Update fully done by @Graiden

9.4 IE Doomsday

  • Requests from old IE are now denied by e621.net
    • Had to change all requests to use curl and send custom useragent to get API data.
  • Here's what this means for you:
    • MD5 mode is disabled (can't get MD5 from files properly, program only works with MD5 in filenames, at least for now). It was turned off by default, but if you've used it, that's bad news for you.
    • Tagger tested on limited amount of picture and works OK
    • Updater tested on even less amount of picture and works OK (but that's not 100%)
    • Sync and Downloader modes are completely disabled for now. There's a lot of code to rewrite to make them work, i'm not ready to do that now.
    • Overall speed is reduced.

9.3 Updater fix

  • Updater is working properly now.
    • Updater mode is now sending "Keito\e621updater" as a user-agent.

9.2 QoL

  • App is x32 now, should work for everyone.
  • Sync mode bugfixes:
    • SWFs and WEBMs are correctly favorited.
    • MD5 is now correctly found for SWFs and WEBMs.
  • GUI bugfixes.
  • Choose Another Folder remembers your last folder and starts from there.
  • "Get API" button added for sync mode.
  • DLLCall added to prevent PC from sleeping while app is working.
    • Can be disabled by modifying config file: manually set PCAwake to 0.

9.1 Sync and config

  • Sync mode updated:
    • No longer limited to images: webm, gif and swf support added.
    • Will calcualte MD5 for all incorrectly named files.
  • Config file added (e621updater.ini):
    • Settings saved for now: last used working directory, username and API key, "network checks" and "backup" checkboxes state.
  • Updater mode backup is fixed

9 Sync

  • Sync mode added:
    • Enter your username into the "name" field (get it from https://e621.net/user/home to be sure)
    • Generate an API key and enter it into the "API key" field (get it from https://e621.net/user/api_key )
    • Press start and all local images will have their E621 posts favorited.
    • No checks are coded: make sure there are no mistakes and spaces in both fields or you'll simply waste your time.
    • Sync mode requires curl.exe to work. Proxy is not supported.
  • Small GUI changes
  • Updater mode backup is fixed

8 Favorites

  • Favorites downloader added:
    • Just select needed mode, enter your username and it will get all the favs you have.
    • Ratings selection. Able to download any combinations of ratings (e.g. only E and Q, without safe images).
    • Existing files (that you already have in target folder) are skipped.
    • Connection checks are not supported at the moment. Don't use it for shitty network.
  • Now calculating MD5 inside the tool itself.
    • Removed MD5.exe
  • GUI changed.
    • "More information" checkbox added for all modes.
    • Information field on bottom of the tool is replaced with list box. Tool will probably work too fast for you to get any info from there.
  • "Tag after updating" option added for favorites and updater modes.
  • "Fast mode" is now always on, option removed. No drawbacks from using it.
  • Optimized JSON parsing.
  • Corrected mistakes in log file again.

7 Need for Speed - v3

  • Performance improved
  • 2 features added: "Fast mode" and "Disable network checks"
    • "Fast mode" disables waiting for program to finish to in-app calls (for all exif and some wget calls). Should not cause problems. Experimental, default on. ~50% performance increase.
    • "Disable network checks" disables... Network checks. All of them. Don't use it if your connection is unstable (VPN, proxy, shitty network, etc). ~50% performance increase.
    • Use both for maximum speed! Almost 3x times faster.
  • Small GUI changes
  • Error in log file for updater is fixed

6 GUI v2

  • WGET and it's libraries are removed.
    • Internet Explorer 3 or greater must be installed for this program to work. Firewalls or the presence of multiple network adapters may cause this program to fail. You can use WGET edition (see bottom of this post) if you have problems with that.
    • You can now use proxies! This program will use a proxy server to access the Internet if such a proxy has been configured in Microsoft Internet Explorer's settings.
    • 70-90% faster.
    • No longer outputs -user-agent=Keito/Tagtool1.0 for e621 admins. Sorry guys. Can't figure out how to restore this without WGET.
  • Added "Shutdown after" checkbox. Will shutdown your PC when proccess finishes. Useful!
  • "Force remove old tags" checkbox is no longer checked by default. Should make the whole proccess faster. (1-2 seconds per image)
  • Added "time elapsed" and "time left" counters. Time left is very inaccurate as expected. They update after every image processed (not realtime).
  • ImagesList file is now correctly deleted when program closes.
  • Creating after_log.txt file after every launch. Might be useful for everyone who used "Shutdown after" and wants to know how proccess went after.
    • Only basic statistics for now. Time start, time end, files processed, files skipped.
  • Minor GUI changes.

5.1 Bugfixes

  • "Close app" question at the end of the proccess is fixed:
    • Window no longer duplicate if answered "No" after "Update" sequence
    • App will correctly restart if answered "No" after "Tag" sequence

5 GUI

  • GUI v1
  • Batch script is recoded into a proper EXE with GUI. It's slightly less miserable now!
    • Still not a virus.
  • Tagger and updater consolidated. Use the program and choose what you want to do.
  • xidel is removed, wget added instead.

4.2 Updater filenames

  • Updater will now calculate md5 if file is not named after the checksum and try to use it for search.

4.1 Bugfixes

  • Removed filenames checks completely, since it works way too slow for me (and probably for someone\everyone else).
    • I don't expect everything to stop working because of that, but feel free to report any bugs.
  • Added some more comments to both updater and tagger.
  • No longer deletes old log after launching the tagger.

4 Updater

  • Updater is now included. You may think that it updates this tool, but it will download new pictures instead of the old ones saved locally.
    • Many pictures may have new higher resolution, not watermarked, not fucked up version uploaded to e621. This tool will find all the new pictures and download them, moving your old files to "backup" folder.
    • Still requires testing, since technically it will save parent of deleted picture. It may be the same picture, or it may be edited one (still i guess it should be better if the old one got removed)
    • Only works with md5 filenames at the moment.
  • Possible fix to authors: it should now get authors for not-md5 filenames too.

3.2 Authors! Again.

  • I've moved code that is getting artists from e621 to the external file "ARTISTS.bat"
    • Now getting all the artists! For some reason, Adobe recognises n-1 artists (not less than 1 tho). Fuck Adobe!
  • "unknown_artist" is no longer an artist.
  • "avoid_posting" is no longer an artist.

3.1 Filenames and authors again

  • "conditional_dnp" is no longer an artist.
    • If you encounter any more trash author tags, let me know.
  • All invalid filenames should now be ignored.

3.0 Filenames and authors

  • Moved off from "md5 only"
    • Tool will remove all dots and underlines from all filenames in directory. Not able to remove other invalid sybols yet.
    • Tool will now calculate md5 if file is not named after the checksum and try to use it for search.
    • Added PULLER.bat and md5.exe to implement that. Still not a virus.
  • Will now save author to the "creator" field as well as into tags!
    • Keeping it in tags just in case someone's software only works with one field, not with both tags and authors. For now.
    • Will save author to xmp-dc:creator
  • Removed excessive -api PNGEarlyXMP when calling exiftool. We don't need that for JPG files. We still do need that for PNGs, for some reason.

2.0 API

  • Reworked the batch file to use JSON.
    • Now 2x faster!
    • Getting tags for pictures that was deleted.
  • Fixed possible PNG bug (not writing anything to PNG)
  • Updated exiftool to the most recent version. Just in case.

1.0 Release

Other versions

Latest archived AHK version that doesn't work anymore:
Github: https://github.com/AyoKeito/e621updater
Current version is v9.5 (v5 GUI) (10.18.2020)

Previous (v8) version:
https://dl.dropboxusercontent.com/u/21291309/updaterv8.zip
https://mega.nz/#!PZsSyJrT!EMihNFssojXmJmC_AtvkUkbq82r7aahfDW3CUcCYcKM

Obsolete WGET edition:
WGET edition is no longer updated.
It's at least 2 times slower than the default version on top of this page. I'm too lazy to code a favorites downloader for it.

Current version is v7 (v3 GUI) (06.10.2016)
Dropbox: https://dl.dropboxusercontent.com/u/21291309/updaterv7wget.zip
MEGA: https://mega.nz/#!CYtH2ajR!77cCXCxOxxj54Z84xJhrow_5ad3oZ1GnENnlXmJRjIs

Obsolete batch:
These batch files are not working and i'm not updating them anymore.
Last time i've looked through them they needed user agent parameter added to all xidel.exe calls to get them working.
Have fun!
Dropbox: https://dl.dropboxusercontent.com/u/21291309/taggerv4p2.zip
MEGA: https://mega.nz/#!jd0ySKDa!TB-00GGFmqHkKHYgqBLBSEi16bE1BlgVbNyoI2P7Pgc

Updated