Topic: Bash e621 downloader and dumper

Posted under e621 Tools and Applications

Bash e621 downloader v1.7

It can download and dump pools, sets, tags with selective method or command line. With only one script (e621downloader and -cl) or separate scripts (Pool,Set and tagdownloader)
I am new at coding and after updating api I can't bulk download posts. I determined to create this script. You can download pools, sets and tags.
Download link:Github.com
Source code:Yusifx1/e621-scripts

Changelog

+ Added url support
+ fixed some problems
+ cleaned script from useless code

To-Do

Make user friendly interface to main scrip (e621-downloader and -cl)
Clean script from useless part and duplicate
Port script to js (userscript) (I know that Re621 can download all you want but I want learn javascript)

Userscript Project

Good news. I finished writing userscript and it work on pc and mobile Firefox. Now I want fix scrapper function that work for only one page. The link to this script

Send issue to my github.
PLEASE SEND ME SCRIPT PROBLEMS.
Write what you want see in this script. I will try make it.
Big thanks to savageorange for reply.
Need testers because i can't find error 503 problem. If post not downloding (not download part of posts) say me and i will add request limit. Also termux on my phone sometime stuck running script where parsing url. If url file is zero byte then run script again.

Updated

PLEASE SEND ME SCRIPT PROBLEMS

OK.

  • Lots of code duplication, the only line that differs seems to be the one which fetches the json.
  • .. And, you force the user to input exactly one id, rather than allowing them to pass pool/set ids on the commandline.
  • -> the combination of these two things makes me think: Make it into single function e6dl {...} whose first argument ($1) specifies whether you want set or pool (or tag..), and all args after that specify individual set or pool ids. If no ids are passed ([[ -z $2 ]] returns 0), THEN prompt the user to input one.
  • (I see in your readme you say "Add function to use both: name and id.", but I can't tell whether you mean function in the informal sense (capability) or in the "bash keyword" sense.)
  • Lack of indentation makes the code hard to read and probably to maintain, but I get that would be a pain to do on a phone.
  • Command substitution: `command` is deprecated in favor of $(command). This is not a pure matter of taste, $(command) actually works more consistently (for example, if you need to nest command substitutions and you're using ``, have fun getting the escaping correct!). I see you use $() in a few places, AFAIK you can easily switch to using it everywhere.
  • the whole 'checking for a valid extension' looks like it could be done with only a few lines, since the pattern is a regexp .. uh, e=$(if [[ $l =~ ^.*[.](gif|jpg|png|webm) ]] ;then echo "${l##*.}"; else true; fi) .. [[ -z $v ]] && echo "Corrupted link!!!"
  • "I didn't add response limit" -- do you mean rate limit? It seems like you could do that with a sleep 1, say, at the end of each loop which invokes curl.
  • Putting tmp files in the current directory is janky. I don't know Android details, though, so maybe you can't put them in /tmp/ like normal. In that case making them dotfiles (.url rather than url) would be a compromise that would still reduce the risk of overwriting a legitimate non-temporary file.

EDIT: fixed some extra quotes

Updated

Thank you for reply. I will fix all this. I am new at making scripts and stackoverflow don't help me. I can't install linux on my PC and emulators not work. So I write all this on my phone. I am planing to all this to js. It will be better than bash script. And sometime $() not work on Termux.
Sorry for my english

Updated

Fair enough, it's definitely easier to develop shell scripts on Linux or OSX. However, did you know that you can install bash, curl, and jq on Windows?

That termux $() issue is really weird and suggests that `` and $() actually have different internal implementations in Bash.

I guess doing this in Javascript would be quite different, but I can see why it would be better.

Your grammar is not quite complete, but you communicate clearly, so I'd say your English is OK.

I can't install linux on my PC and emulators not work. So I write all this on my phone. I am planing to all this to js. It will be better than bash script. And sometime $() not work on Termux.
Sorry for my english

albert12 said:
I can't install linux on my PC and emulators not work. So I write all this on my phone. I am planing to all this to js. It will be better than bash script. And sometime $() not work on Termux.
Sorry for my english

What is this?

  • 1