Topic: RE621 [v.1.5.68] Feature-packed toolkit and mass downloader for e621

Posted under e621 Tools and Applications

silent_strider said:
So far I got FOUR TIMES the same zip archive with pictures 123 to 182. And it just started to download the same pictures a fifth time.

What do I wrong?

Sorry, did not mean to ignore you there.
Weird issue, I'm seeing it too. I'll see what I can do.

silent_strider said:
Also: is it possible to browse only the new posts since last visit? If not, could this be taken as a feature request?

No, not really.
I'll think about how to best implement something like this, but no promises.

I'll think about how to best implement something like this, but no promises.

RE621 could save the PostID which was the most current one from the last viewing and maybe make a thick line in the list of thumbnails when this PostID comes up again. Not sure if the search function could be used to find PostIDs newer/higher than X.

The vote state seems not maintained after a page refresh, e.g. click upvote on a post, refresh the page, then hover on the post, the thumb up/down icon color no longer indicates the upvote/downvote state.

Is it expected behavior?

The environment is chrome + tampermonkey.

watchdog22 said:
The vote state seems not maintained after a page refresh, e.g. click upvote on a post, refresh the page, then hover on the post, the thumb up/down icon color no longer indicates the upvote/downvote state.

Is it expected behavior?

The environment is chrome + tampermonkey.

Yes, unfortunately.

There is no practical way to get how you voted on specific posts from the API.
The best method available to me would require up to 8 requests per search page, which is not acceptable.

winterrr said:
Would it be possible to implement sound whenever a video is being displayed with hover zoom?

If I remember correctly, I've done it like that because Chrome would not auto-play non-muted videos that loaded without an explicit user request (aka on hover, without clicking).
I'll take a look if that changed, but no promises.

Hello again! Hope all is well with you. Just wanted to say I'm still rooting for the features you added to your ToDo list last we spoke!

Basically these:

Blacklisted Thumbnail Issues
- The 'Skip Blacklisted' option in the HoverZoom section doesn't work for Blacklisted white thumbnails and still previews them in Pools pages, Tag info pages, Wiki pages, Artist pages, Comments, and with Forum avatars. Everywhere else seems to be fine because those thumbnails get blackholed entirely (love that feature). It would be awesome if either Skip Blacklisted worked everywhere or blacklisted stuff just got blackholed everywhere. The latter makes everything very clean and tidy though!

Faster Page Leaping
- On any Posts page, a 'Jump to Page...' option at the bottom, like where the ellipses is between page numbers would be great. Or, an option to adjust how wide the range of page numbers are available to click down there. Even if the option is an absurd list of tiny numbers to fit 100 page links, that'd still be some fast and specific leaping!

Sorry if your plate is already full! I was just curious enough to mention them. Thanks again for everything you do :]

Updated

bitWolfy

Former Staff

Just a quick announcement.

Some of you may have already noticed that the update link for the script changed.
It's now pointing to the new code repository: https://github.com/re621/re621.Legacy

This is perfectly fine. Just keep using the script as normal, there is no need to update manually.
The script at the other repository (https://github.com/re621/re621) is an Alpha build of version 2.0, and is not ready for a public release yet.

bitWolfy

Former Staff

silent_strider said:
Compressing pools with PNGs still take ages. It's around 16 seconds for 1 PNG file to be compressed. Sounds terrible kinda.

Downloading the pool https://e621.net/pools/30076 currently. JPGs on the other hand are compressed very quickly.

Nothing really changed since last time.

I don't know why this happens, but it seems to be something the library I use for compressing files does.
Unfortunately, there aren't many libraries for this sort of thing out there. The only prominent one is JSZip, which I already use.
Dropping the library altogether and just downloading files one at a time isn't really an option either, since it will cause problems of its own, ranging from having to confirm dozens of file downloads to browser seeing this as malicious and blocking them altogether.

bitwolfy said:
Dropping the library altogether and just downloading files one at a time isn't really an option either

but isn't it what the script is doing right now? In my %temp% directory, I see a mozilla-temp-files directory with names like mozilla-temp-24464. And if adding the extension PNG I see, those are the images. But I guess since Mozilla took away the possibility to write files anywhere... though wouldn't it be possible to create the picture files in the default download directory?

bitWolfy

Former Staff

silent_strider said:
but isn't it what the script is doing right now? In my %temp% directory, I see a mozilla-temp-files directory with names like mozilla-temp-24464. And if adding the extension PNG I see, those are the images. But I guess since Mozilla took away the possibility to write files anywhere... though wouldn't it be possible to create the picture files in the default download directory?

Filesystem API is not supported by Firefox, sadly. https://caniuse.com/filesystem
I'm not 100% sure how Firefox is handling it. Chrome downloads the files into memory, I believe.

Downloading many loose files at once is something browsers really do not want to allow.
It makes sense, of course: imagine a malicious script filling up your downloads folder with thousands of junk files.
But it is fairly inconvenient in this case.

I'm still trying to download the pool from my first post today. It's the second time now the status goes to

Download
Compressing . . . 100.00%

And then nothing happens. Book 2 and 3 I could download, but Book 1 doesn't seem to like me

Bug report: If you click the refresh button while viewing an image, Search: (none) will appear.
Temporarily disabling re621 does not cause any problems, so it seems to be a problem on this side.

drag007 said:
Bug report: If you click the refresh button while viewing an image, Search: (none) will appear.
Temporarily disabling re621 does not cause any problems, so it seems to be a problem on this side.

I can't replicate this, every time I refresh a post page it just puts whatever's in the "?q=" bit of the url into the navigation buttons on the post

bitWolfy

Former Staff

drag007 said:
Bug report: If you click the refresh button while viewing an image, Search: (none) will appear.
Temporarily disabling re621 does not cause any problems, so it seems to be a problem on this side.

Go to script settings -> "General" tab -> "Post Page" section, and turn off "Trim query Parameter".
This feature makes the URLs neater... but it does screw with the search parameters if you reload the page.

Would it be possible to add something similar to the recent tags field, but instead of listing what tags you have recently added to images, it lists a set of tags you can configure manually? Something like a painter's palette but for tags. There may already be a feature like this already that I don't know about, I'm still pretty new.

IQDB: Internal Error 403 whenever I upload an image from my files, retry doesn't help

Also it would be nice to have a code block button for writing in the forum

bitWolfy

Former Staff

oozeenthusiast said:
Would it be possible to add something similar to the recent tags field, but instead of listing what tags you have recently added to images, it lists a set of tags you can configure manually? Something like a painter's palette but for tags. There may already be a feature like this already that I don't know about, I'm still pretty new.

Sorry for the late response.
A feature like this exists on e621 natively, I believe.
"Settings" -> "Advanced" -> "Frequent Tags".
Don't ask me how it works, I've never used it.

gattonero2001 said:
IQDB: Internal Error 403 whenever I upload an image from my files, retry doesn't help

Local files are not supported, only URLs. Sorry.

gattonero2001 said:
Also it would be nice to have a code block button for writing in the forum

Buttons are fully customizable.
Click the button that looks like three horizontal lines in the top right corner of the input form.
Then, click the plus button to create a new one.

The name and icon can be anything you want.
The "Content" section should look like this:
[code]
%selection%
[/code]

Is there a way to add a custom flag for favorites of another user? I was gonna flag this account's favorites on an alt b/c I'm at the fav limit on this one, but the custom flag doesn't mark them for some reason. I know I can add -fav:JoyfulDragon1652 as a persistent tag, but I didn't wanna have them blacklisted or filtered out completely though.

bitWolfy

Former Staff

joyfuldragon1652 said:
Is there a way to add a custom flag for favorites of another user? I was gonna flag this account's favorites on an alt b/c I'm at the fav limit on this one, but the custom flag doesn't mark them for some reason. I know I can add -fav:JoyfulDragon1652 as a persistent tag, but I didn't wanna have them blacklisted or filtered out completely though.

The script does not have access to this information, unfortunately.
It just has the post's tags. Custom flags won't work with pools or posts sets either.

bitwolfy said:
The script does not have access to this information, unfortunately.
It just has the post's tags. Custom flags won't work with pools or posts sets either.

Dang, well that’s too bad

how do I get the subscription manager and easy blacklisting features to work? currently I don't even see any tabs for notifications as displayed on the download page, and I don't see any shortcuts or buttons to be able to add an artist to said favorites page
The blacklisting option shows up in the mode switcher on the home page, but seemingly does nothing when enabled (the download page also mentions something about an "x" icon appearing over tags, which also is not showing up whatsoever)

I'm on the latest version as of posting (2.0.7) and I have checked for updates via tampermonkey, so I have to assume its ignorance on my part.
what am I missing here?

bitWolfy

Former Staff

egg_head said:
how do I get the subscription manager and easy blacklisting features to work? currently I don't even see any tabs for notifications as displayed on the download page, and I don't see any shortcuts or buttons to be able to add an artist to said favorites page
The blacklisting option shows up in the mode switcher on the home page, but seemingly does nothing when enabled (the download page also mentions something about an "x" icon appearing over tags, which also is not showing up whatsoever)

I'm on the latest version as of posting (2.0.7) and I have checked for updates via tampermonkey, so I have to assume its ignorance on my part.
what am I missing here?

Try the version linked in the first message of this thread.
https://e621.net/forum_topics/25872
Stable v.1.5.48.

Version 2.0 is not ready for public release yet.
Where did you even get it?

bitwolfy said:

Features

Visit re621.bitwolfy.com for a full feature overview.

on the features page there is a download for version 2.0.7
I instinctively assumed this was the correct version without reading the rest of the message I guess
whoops!

bitWolfy

Former Staff

egg_head said:
on the features page there is a download for version 2.0.7
I instinctively assumed this was the correct version without reading the rest of the message I guess
whoops!

Ah, shoot. I should probably fix it.

Feature suggestion, because I'm bored, but I'm also lazy and like saving time so... feature suggestion.

Add an option where when up/down voting a post from the tag results screen, the image disappears from the roster and everything moves up one slot, possibly with a user-defined time delay in case of misclicks. Might not do much for specific searches, but when you're looking more generally and dis/liking tens, maybe a hundred results at a time then it could prove useful. Might even save my wrist a bit with the scrolling since I've set five per line.

Also, when's that 2.0 coming out. Sounds like it must be a big feature update, what's it got?

So I figured out the issue with the re621 script after e621 migrated to cloudflare. The script sets it's own useragent, which triggers Cloudflare's browser check. The fix is to have the script use the browser's useragent.

Example edit to make it work.
window.re621 = new function() { this.name = "re621-legacy"; this.displayName = "re621 - e621 Reimagined"; this.version = "1.5.48"; this.build = "220918:2148"; this.type = "script"; this.links = { website: "https://e621.net/forum_topics/25872", repository: "https://github.com/re621/re621.Legacy", issues: "https://github.com/re621/re621.Legacy/issues/", releases: "https://github.com/re621/re621.Legacy/releases", forum: "https://e621.net/forum_topics/25872", }; this.useragent = window.navigator.userAgent; this.privacy = false; this.debug = true; };

I'd file a github issue or a PR, but I don't have a github account that's not tied to my work. I'll maybe setup one tomorrow if I have the time.

bitWolfy

Former Staff

ottah said:
So I figured out the issue with the re621 script after e621 migrated to cloudflare. The script sets it's own useragent, which triggers Cloudflare's browser check. The fix is to have the script use the browser's useragent.
I'd file a github issue or a PR, but I don't have a github account that's not tied to my work. I'll maybe setup one tomorrow if I have the time.

Spoofing the browser's user-agent is explicitly prohibited.
What exactly is the issue here? The script seems to be working fine for me with the default user-agent.

I was having the same issue.

bitwolfy said:
Spoofing the browser's user-agent is explicitly prohibited.
What exactly is the issue here? The script seems to be working fine for me with the default user-agent.

Upon checking, mine didn't say anything about a default agent; instead it had "re621-legacy:script/1.5" listed. Not sure why, or even what this stuff means, but it was definitely causing issues which are now fixed after making the edit that ottah mentioned.

bitWolfy

Former Staff

lloxie said:
I was having the same issue.

Upon checking, mine didn't say anything about a default agent; instead it had "re621-legacy:script/1.5" listed. Not sure why, or even what this stuff means, but it was definitely causing issues which are now fixed after making the edit that ottah mentioned.

What browser are you using?

Do not edit the user-agent. Spoofing the browser user-agent is explicitly against the rules, and can result in you getting blocked.
... plus, you won't be able to update the script automatically anymore.

bitwolfy said:
What browser are you using?

Do not edit the user-agent. Spoofing the browser user-agent is explicitly against the rules, and can result in you getting blocked.
... plus, you won't be able to update the script automatically anymore.

Firefox. Or, well, Waterfox, but it's basically the same thing with some minor differences that shouldn't be relevant to this site.

I'm also not real clear on what a user-agent even is, tbh. I was just having problems with the site, saw someone post about a relevant fix in a script I was using, and tweaked it, heh.

bitWolfy

Former Staff

lloxie said:
Firefox. Or, well, Waterfox, but it's basically the same thing with some minor differences that shouldn't be relevant to this site.

I'm also not real clear on what a user-agent even is, tbh. I was just having problems with the site, saw someone post about a relevant fix in a script I was using, and tweaked it, heh.

Yeah, I can confirm that the new cloudflare protection seems to have broken the API somewhat.
I've poked Kira, e621's lead developer, about it.

bitwolfy said:
Yeah, I can confirm that the new cloudflare protection seems to have broken the API somewhat.
I've poked Kira, e621's lead developer, about it.

So can you explain why the default script had that "re621-legacy:script/1.5" line? If anything, ottah's tweak looks more like it's restoring the default, while the thing being changed is putting up some kind of spoofing. That's reading it as a complete layperson on this, of course, since it's all kind of alien to me, heh.

bitWolfy

Former Staff

lloxie said:
So can you explain why the default script had that "re621-legacy:script/1.5" line? If anything, ottah's tweak looks more like it's restoring the default, while the thing being changed is putting up some kind of spoofing. That's reading it as a complete layperson on this, of course, since it's all kind of alien to me, heh.

It's a bit complicated, I'm afraid =P
Essentially, a user-agent is a line of text that is sent alongside every API request in order to identify the application making that request.
So, "re621-legacy:script/1.5" tells the server that the request is coming from "re621", script version (as opposed to an extension), version 1.5.

If you set the user-agent to what ottah said, you are essentially telling re621 to pretend to be a browser.
There are a number of reasons why that is considered to malicious, the main one is making it harder to police which apps are allowed to make requests to the API, and which are not.
Thus, attempts to do so usually result in a block.

bitwolfy said:
It's a bit complicated, I'm afraid =P
Essentially, a user-agent is a line of text that is sent alongside every API request in order to identify the application making that request.
So, "re621-legacy:script/1.5" tells the server that the request is coming from "re621", script version (as opposed to an extension), version 1.5.

If you set the user-agent to what ottah said, you are essentially telling re621 to pretend to be a browser.
There are a number of reasons why that is considered to malicious, the main one is making it harder to police which apps are allowed to make requests to the API, and which are not.
Thus, attempts to do so usually result in a block.

I see. Well, thanks for answering. Here's to hoping the issue gets fixed soon, heh.

Hmm... I don't get thumbnails at all. They seem to be infinitely loading. I'd like to add I AM using a VPN, though.

moximoore said:
Hmm... I don't get thumbnails at all. They seem to be infinitely loading. I'd like to add I AM using a VPN, though.

It's not your VPN. I'm on one too, and tried without it. Something on e621's end isn't playing nice with re621 for whatever reason. From what I gather, it's affected some other things using the API as well.

bitwolfy said:
It's a bit complicated, I'm afraid =P
Essentially, a user-agent is a line of text that is sent alongside every API request in order to identify the application making that request.
So, "re621-legacy:script/1.5" tells the server that the request is coming from "re621", script version (as opposed to an extension), version 1.5.

If you set the user-agent to what ottah said, you are essentially telling re621 to pretend to be a browser.
There are a number of reasons why that is considered to malicious, the main one is making it harder to police which apps are allowed to make requests to the API, and which are not.
Thus, attempts to do so usually result in a block.

I would argue that there's not much of a difference between a userscript and the browser it's running in. I certainly wouldn't call it "spoofing" the agent.

Besided, the useragent is a really week security tool from the api side of things. I mean, that's why you'd migrate to cloudflare in the first place wouldn't it? I'd rely on cloudflare to help mitigate API abuse, instead of trying to implement something off of client headers like the user agent string.

Stopped working for me completely just stays at "connection needs checking" screen and nothing, was a good tool before update

The site update broke older versions it seems. Updating to the latest 2.0.7 version allowed the site to load.

However, I no longer have a subscriptions tab on the right. How does one open and configure subs on the newest version?

Thanks!

bitWolfy

Former Staff

yosys said:
The site update broke older versions it seems. Updating to the latest 2.0.7 version allowed the site to load.

However, I no longer have a subscriptions tab on the right. How does one open and configure subs on the newest version?

Thanks!

2.0 is still in development. Roll back to version 1.5, whichever the latest patch on that one is.
The reason 2.0 works is because it uses a different method of fetching post data. But I assure you that it's as broken as 1.5 due to site issues.

Nothing either of us can do about it until the site stops blocking API calls.

bitwolfy said:
2.0 is still in development. Roll back to version 1.5, whichever the latest patch on that one is.
The reason 2.0 works is because it uses a different method of fetching post data. But I assure you that it's as broken as 1.5 due to site issues.

Nothing either of us can do about it until the site stops blocking API calls.

Ah, alright. I'll miss the sub pane dearly but at least we know what's causing it now.

yosys said:
Ah, alright. I'll miss the sub pane dearly but at least we know what's causing it now.

1.5.48 seems to be working fine on my end since the update. Haven't noticed a thing except for the my app 'the wolfs stash' which needed to reaffirm its api request when I used it last night.

Of course, this might change the next time I restart my computer and reopen my browser, we'll have to see.

mdf said:
1.5.48 seems to be working fine on my end since the update. Haven't noticed a thing except for the my app 'the wolfs stash' which needed to reaffirm its api request when I used it last night.

Of course, this might change the next time I restart my computer and reopen my browser, we'll have to see.

I can confirm 1.5.48 seems to be working now.

Everything seems to be back online now and mobile apps are also working

dont know if this is a common issue, but my shortcuts/hotkeys stopped working? havent found a setting in the mod that would do that

Switched to tag script #1. To switch tag scripts, use the number keys.

This is text visible to all people even if they're not using the extension, but re621 should probably modify this text since, by default, numbers navigate through the tabs on the top menu unless you're holding ALT.

Sorry if this has been asked already in this very long thread, but are there any plans to incorporate the features of RE621 into the main site? Since the author is an admin, this would make a lot of sense. It may also prevent future website updates from breaking functionality as those features would be part of your official QA process.

The reason I ask is because of my bug report in topic #36377 (notification emails never arrive on some occasions).

bitWolfy

Former Staff

restrainedraptor said:
Sorry if this has been asked already in this very long thread, but are there any plans to incorporate the features of RE621 into the main site? Since the author is an admin, this would make a lot of sense. It may also prevent future website updates from breaking functionality as those features would be part of your official QA process.

The reason I ask is because of my bug report in topic #36377 (notification emails never arrive on some occasions).

It's not impossible, of course.
I started this project before joining the staff team. Since then, some of the features and fixes that once were a part of re621 were ported to the site itself.
Some of it was done by me, some by Earlopain.

For example, the formatting buttons did not exist on e621 two years ago.

However, some features are not going to be ported. At least, not in their current form.
Notifications are one of those. They have been discussed, and were judged to be too unwieldy and difficult to maintain.
RE621's notifications break at times too, mind you =P

Just curious to see if everything is working correctly for everybody as not long ago the app seems to be flying over blacklisted tags and just downloads everything

Something I'd love to see in v2: popup a confirmation alert if I'm editing or posting with invalid tags: "There are invalid tags: %invalidTagList%. Do you wish to continue)? Y/N"

If I want to clear a hotkey (empty it, to have no binding) what do I press while it's recording?

I want to remove left and right arrow from cycling posts, without adding a new key

bitWolfy

Former Staff

sulmarobar said:
If I want to clear a hotkey (empty it, to have no binding) what do I press while it's recording?

I want to remove left and right arrow from cycling posts, without adding a new key

Press the Escape key.

Don't know if I requested this before, but as long as its on my mind I'll risk repeating it.

When downloading images, require all single-click links be right-clicked and opened in a new tab.

You learn not to, but I always eye-roll when the process gets interrupted simply because I accidentally clicked on something.

Am I understanding correctly that 2.0.7 does not support many features (subscriptions, mass downloads) yet? Been trying the last hour to get it to work until I found this thread.
It's the version it tells you to install on the website (https://re621.bitwolfy.com/) so I figured it's the stable version...

bitWolfy

Former Staff

cloakedbanshee said:
Am I understanding correctly that 2.0.7 does not support many features (subscriptions, mass downloads) yet? Been trying the last hour to get it to work until I found this thread.
It's the version it tells you to install on the website (https://re621.bitwolfy.com/) so I figured it's the stable version...

I have fixed that link. Sorry about that.
1.5.48 is the most recent stable version of the script.

the whitelisting feature does not seem to work as expected for me

on whitelist
insomniacovrlrd
on blacklist
~vore

images that where supposed to be whitelisted show up as usual on the preview page, but get listed as blacklisted once clicked on, making the feature essentially useless
im not sure what causes this, or if its just on my end, but it is incredibly annoying since the blacklist's lack of compatibility with artist tags was the one feature I absolutely could not stand being the default

egg_head said:
the whitelisting feature does not seem to work as expected for me

on whitelist
insomniacovrlrd
on blacklist
~vore

images that where supposed to be whitelisted show up as usual on the preview page, but get listed as blacklisted once clicked on, making the feature essentially useless
im not sure what causes this, or if its just on my end, but it is incredibly annoying since the blacklist's lack of compatibility with artist tags was the one feature I absolutely could not stand being the default

I have managed to fix this using the default blacklist, after realizing ~'s are broken for whatever reason, that might be what was causing the whitelist to freak out as well but I haven't tested that yet
I do still think this is a bug that should be fixed, I doubt the whitelist working this way ever was intentional

Would it be possible to give the option to enable the "tag suggestion" feature while editing already uploaded posts?

It's a handy feature when uploading directly, but it's not something I can have in third-party uploaders like reBot afaik, so I think it'd be very useful to use a bulk-upload, and fill in tags I may have forgotten with re621 later.

And if possible, also allow it to work with the site's quick-edit.

Great tool by the way, can't use e621 without it now ^^

by default the CHARS flag should include -trio, even if it's a relatively rare tag, it is useful in some cases.

egg_head said:
by default the CHARS flag should include -trio, even if it's a relatively rare tag, it is useful in some cases.

I don't really use the custom flags feature so I'm not for certain how it works, but wouldn't it be redundant to -group anyway since trio implies group?

egg_head said:
by default the CHARS flag should include -trio, even if it's a relatively rare tag, it is useful in some cases.

Oh damn, I forgot about custom flags, they work perfectly for my need.

Would it be possible to make them update live as I add the tags while editing? Currently I need to fill in the missing tags that are shown on my flags, but then I would have to edit again for new next flags to show up based on previous tags, which kinda feels like edit spamming.

Example: I have a flag for character coung. "-solo -duo -trio -group"
and then I have another flag for position "solo standing sitting"

The position flag would only appear after my first edit that added the character count was sent.

Anyone else been having trouble downloading only webm's and getting empty folders?

When you fav/sub to a tag ( Artist/copyright/character/species/general tag/meta ) but go into the settings to look at all the tags you've favorited. There seems to be no color indication that seperates which tag is which.
There is no color orientation to help tell the difference between the tags. They all have the same general color.

It makes it hard to pinpoint which tag I'm looking for since there's no order or at least any color difference.

Is it possible in future updates we may be able to go into our settings and see that the tags are color coded just like they are in any post? It'd really help.

When getting a notification about a new image in a pool, RE621 will always lead to the last image in the pool. But when multiple images have been released since I last was on e621 I wouldn't even know I missed a page or even a few if not always looking at the previous image in the pool myself.

First of all, creating and maintaining this script is greatly appreciated, thanks! One thing I would like to point out is that on the Popular page, the overlay on thumbnails that shows if you've favorited a post or up/downvoted it is absent. It appears as it should though on pool pages or search pages for example so it was just odd to me that it's missing on the popular page. I'm on v1.5.50, and it's been this way for as long as I can remember using the script.
I'm not sure if this is the right place for this, but I'd like to say it'd be nice if there was a mass favorite/upvote button or hotkey option like there is for downloads. Mainly for pools so one doesn't have to individually favorite each post in it.
Thanks for reading! :)
edit: was wrong about pool pages, the favorite star and vote thumbs aren't showing up there for me

Updated