EsixSync is a tool (currently in command line) aims to provide an easy way to sync posts between local folder and sever.

• Functions ?

- You can stop during any progress of this tool without suffering download progress loss.
- You can sync between local folders and server. The program will determined which post was in download queue or already downloaded by itself.
- EsixSync support sync function with "repository". A repository is a folder itself.
- Will show dynamic progress bar and internet speed when you sync posts/pools in command line.

• How to use it ?

- First, use "esync add" command to add a repository.
- Second, use "esync update" command to update repository summary file.
- Thrid, use "esync sync" command to sync with e621 server.

• Is it safe to use ?

- It is a open source project hosted on github: https://github.com/FillanFaolan/EsixSync.
- You can review the code and pack it all by yourself.

• Where to download ?

- Github release page: https://github.com/FillanFaolan/EsixSync/releases.

• Anyting else ?

- If you like this project, you can give me some suggest to make the project better!
- Feel free to modify these code. But please, make sure to keep it open source. (Thank you!)
- Please help me to pack esync into more platform if you willing to. :)
- Happy if you like it! OwO

• What's next ?

- Add GUI to EsixSync (May use Qt 5 Community).
- Make the program run faster.
- Simplify the code.

• Support commands:

Usage: esync <command>
Available commands: about | add | info | login | ls | plsearch | rebuild | rm | sync | updated

Usage: esync about
Print some information about this command line tool.

Usage: esync update [--repo $REPO_NAME]
Update summary file of a repository, leave blank for sync all repositories.

Usage: esync sync [--repo $REPO_NAME]
Sync file of a repository, leave blank for sync all repositories.

Usage: esync ls
List all repositories.

Usage: esync add --name $REPO_NAME --id $IDENTIFIER --type pool|posts --folder $FOLDER_PATH
Add a repository

Usage: esync remove $REPO_NAME
Remove a repository

Usage: esync rebuild --repo $REPO_NAME
If case you .summary file is broken, use rebuild command to make a valid .summary file.
But you have to update & sync the repo again from begging

Usage: esync info $REPO_NAME
List summary of a repository

Usage: esync plsearch --title $SEARCH_TITLE
Search pools with the title

Usage: esync login --uid $USER_ID --key $API_KEY [--logout]
Use login command will give you the ability to use your e621 account's blacklist settings and so on.
You can log out anytime with --logout option

Usage: esync rename --name $OLD_NAME --newname $NEW_NAME
Rename repo from name to newname

Usage: esync chdir --name $REPO_NAME --newfolder $NEW_FOLDER
Move repo storage folder to newfolder

Usage: esync help
List all commands you could use

a few pertinent questions come to mind, which i can't easily answer by reading the code or documentation …

• what are its limitations? for example, if i create a 'posts' repo with --id mammal, what should i expect?

• what happens when posts are removed from a pool, or when tags are removed from downloaded posts?

• how does it handle pagination of API results? (e.g. going beyond page=750)

bipface said:
a few pertinent questions come to mind, which i can't easily answer by reading the code or documentation …

• what are its limitations? for example, if i create a 'posts' repo with --id mammal, what should i expect?

• what happens when posts are removed from a pool, or when tags are removed from downloaded posts?

• how does it handle pagination of API results? (e.g. going beyond page=750)

• For posts, limitations are the same with the search keywords when you browse e621. For pool, id should be a integer which stands for post id.
• If the posts are already downloaded (sync complete), it will always be there.
• I manually set 100 posts per page. I works like:

while (STILL_NEED_NEXT_PAGE) {
new_posts += FetchPosts(index)
index += 1
}
The STILL_NEED_NEXT_PAGE flag is generated by comparing md5 in new_posts with local md5 cache (in .summary.json => fileMetadata).

faolanwolf said:

• For posts, limitations are the same with the search keywords when you browse e621. For pool, id should be a integer which stands for post id.

what i meant was, mammal has over 1.5 million posts — is the synchronisation able to operate at that scale? would all the files be placed in a single directory? would they be indexed using a single json file?

if it's using basic pagination (which, judging by the ListPosts function, it is), hitting the API for mammal would eventually reach an error at
page 751 — what should the user expect to happen in such cases?

bipface said:
what i meant was, mammal has over 1.5 million posts — is the synchronisation able to operate at that scale? would all the files be placed in a single directory? would they be indexed using a single json file?

if it's using basic pagination (which, judging by the ListPosts function, it is), hitting the API for mammal would eventually reach an error at
page 751 — what should the user expect to happen in such cases?

- As far as I test e621 API will return an empty json result when page>750, so the program will stop at page 750. And I really do not suggest using it to sync any post set, not only cause it's slow but also stress e621 server.
- The program will not check all posts ever time you use update command. Eg. You add a pool/posts repo and use update command. It will check all pages( assume that it's 10 page).Next update will work on the basis of last update result. So update will be quick after the first update procedure.

faolanwolf said:
…

okay; just keep in mind that knowing these details is very important to the types of users who would consider such a tool