My question is, what is e621's policy regarding mass downloads (using an addon or something like that to download several hundred images in a short time)? I'm trying to get used to this site before I upload or download anything, and while I've already downloaded quite a few pictures using right click save, I want to know what e621's policies are regarding mass downloads, as I'm afraid I will spam the server and screw it up or something like that. I'm kind of nervous about being seen as a spammer or something, especially in the wake of all the DDOS attacks these past couple of days.
Thanks for your help and time!

This is not at all official, but considering what various apps do, I think the stance is 'just be reasonable about it'. For example, queuing downloads up in a download manager like DownThemAll, setting max download bandwidth / sec to something conservative like 10-20k, and number of splits to 1; this is probably less load on e621 than an ordinary user browsing it.

savageorange said:
This is not at all official, but considering what various apps do, I think the stance is 'just be reasonable about it'. For example, queuing downloads up in a download manager like DownThemAll, setting max download bandwidth / sec to something conservative like 10-20k, and number of splits to 1; this is probably less load on e621 than an ordinary user browsing it.

I think I get what you are saying (sorry, I'm awful at computers). So there is no problem with mass downloading, as long as it's under a certain limit. That's really helpful, thanks! I have a few other questions though, so please bear with me. :D

1. This is probably going to sound stupid, but is it possible to unintentionally spam the server by opening up a lot of new pages in e621? I ask because I sometimes open up images in new tabs really fast.

2. In a similar vein to the original post, does anyone know what FA's policy regarding mass downloads is? I'm asking because a while back I went and got an FA downloader add-on and like a knucklehead I went and tried it out without reading any of the instructions, and one of them was that the download rate was based on your internet speed. Since I have a fairly fast internet (1Gbps), it downloaded dozens of pictures in seconds, maybe more. Now I'm kind of nervous that they think I'm a spammer.

3. Just out of curiousity, in case of an actual spammer, what sort of action is taken against them (on both FA and e621)?

Kataphraktarii said:
I think I get what you are saying (sorry, I'm awful at computers). So there is no problem with mass downloading, as long as it's under a certain limit. That's really helpful, thanks! I have a few other questions though, so please bear with me. :D

1. This is probably going to sound stupid, but is it possible to unintentionally spam the server by opening up a lot of new pages in e621? I ask because I sometimes open up images in new tabs really fast.

Could do if the images are large (in filesize, not in pixel dimensions), there are many of them, and you're not using the 'show reduced samples' option. But in most cases no, this is not possible, especially given that e621's content is hosted on cloudfront which has reasonable DoS protection.

BTW I think the term 'spam' you are using is probably incorrect. Supposing your scenario above actually played out - your browser clogged e621 with requests and e621 fell over - then what has occurred is a unintentional, though otherwise textbook, Denial of Service attack. Spam typically refers to repeated posting of irrelevant content (eg. spam emails, forum posts, SMSes)

You can pretty much download as much as you want, all files are mirrored by Cloudflare so we only get a very, very tiny amount of additional load from every page load.

If you want to stay safe stick to 1 image per second.

As for FA, no idea, they seem to have more trouble with their servers than we do, but I'd guess even they won't really care if you stick to 1 image per second.

savageorange said:
Could do if the images are large (in filesize, not in pixel dimensions), there are many of them, and you're not using the 'show reduced samples' option. But in most cases no, this is not possible, especially given that e621's content is hosted on cloudfront which has reasonable DoS protection.

BTW I think the term 'spam' you are using is probably incorrect. Supposing your scenario above actually played out - your browser clogged e621 with requests and e621 fell over - then what has occurred is a unintentional, though otherwise textbook, Denial of Service attack. Spam typically refers to repeated posting of irrelevant content (eg. spam emails, forum posts, SMSes)

Ah, that clears quite a bit up. I see what you mean by the incorrect term, sorry about that (although I'm not quite sure what other term to use in it's place to refer to filling a server up with requests. Where I live spam is basically an all-encompassing term for these kinds of things haha).

Also, now that it has been mentioned, I am interested in what services Cloudflare (I think you meant that instead of Cloudfront? If not, then my mistake.) provides. I've been hearing about it a lot this past week, but I'm not sure what it actually does or how it works (you mentioned DoS protection?).

In addition, I'm still quite concerned about FA and it's mass download policy. I hope I didn't mess anything up with them. I hope someone can let me know more.

Also, thanks savage for taking the time to explain all this to me, I really appreciate it. All this internet stuff makes me nervous because of how little I know.

NotMeNotYou said:
You can pretty much download as much as you want, all files are mirrored by Cloudflare so we only get a very, very tiny amount of additional load from every page load.

If you want to stay safe stick to 1 image per second.

As for FA, no idea, they seem to have more trouble with their servers than we do, but I'd guess even they won't really care if you stick to 1 image per second.

Ah, sorry, you posted while I was typing out my last response. Thanks for letting me know the download rate limit, so that I don't accidentally mess things up and create more problems for the admins if I do decide to mass download in the future :)

Kataphraktarii said:
1. This is probably going to sound stupid, but is it possible to unintentionally spam the server by opening up a lot of new pages in e621? I ask because I sometimes open up images in new tabs really fast.

"spamming" a server with requests from one host (e.g. one user) in the sense of a DOS-attack is often a very minor problem since that specific user can be identified and blocked or throttled fairly easily. Also note that there is a big difference between bandwidth and number of requests. e621 has a lot of bandwidth thanks to CloudFlare (as mentioned) but the number of requests it can process is probably much lower. As long as bandwidth is high enough, spamming a server as a human requesting mostly heavy files (images) is rarely a problem, your own computer will probably suffer a lot more than the server handling the requests.

The problem comes when many hosts work together in botnets to launch DDoS attacks (i.e. Distributed-DoS), these are much harder to block. Without proper tools and techniques such an attack will make just about any web server stumble to the point of almost basically standing still and unable to handle any requests at all, hence "Denial of Service".

Kataphraktarii said:
Also, now that it has been mentioned, I am interested in what services Cloudflare (I think you meant that instead of Cloudfront? If not, then my mistake.) provides. I've been hearing about it a lot this past week, but I'm not sure what it actually does or how it works (you mentioned DoS protection?).

CloudFlare is primarily a content delivery network, relieving strain on the main servers which instead can focus on other things. They also provide a variety of support services associated with content delivery and web services, one of those things is DDoS mitigation, but also other security aspects, DNS, and analysis tools etc. You can read more about this on the CloudFlare website, they got fairly detailed and straight forward explanations of what services they provide.

Kataphraktarii said:
2. In a similar vein to the original post, does anyone know what FA's policy regarding mass downloads is? I'm asking because a while back I went and got an FA downloader add-on and like a knucklehead I went and tried it out without reading any of the instructions, and one of them was that the download rate was based on your internet speed. Since I have a fairly fast internet (1Gbps), it downloaded dozens of pictures in seconds, maybe more. Now I'm kind of nervous that they think I'm a spammer.

I would suggest doing sequential downloads, but that's maybe not possible using third party software. So, mentioned one image/request per second is probably a good choice, if you want to be extra kind and have the time, increase it as you see fit.

Ideally you should also use any API that is available, rather than crawling the html, as using the API will most likely limit any strain on servers, but that most likely require you to build your own tool, or use specialized tools for specific sites.

Kataphraktarii said:
3. Just out of curiousity, in case of an actual spammer, what sort of action is taken against them (on both FA and e621)?

Probably the absolutely worst thing that will happen is that you will be temporarily or permanently blocked. Something you will hopefully be able to rectify by contacting an admin and getting yelled at. Then hopefully getting the block lifted if you cooperate and show that you're willing to work with them and not looking to hurt them.

Kataphraktarii said:
My question is, what is e621's policy regarding mass downloads (using an addon or something like that to download several hundred images in a short time)? I'm trying to get used to this site before I upload or download anything, and while I've already downloaded quite a few pictures using right click save, I want to know what e621's policies are regarding mass downloads, as I'm afraid I will spam the server and screw it up or something like that. I'm kind of nervous about being seen as a spammer or something, especially in the wake of all the DDOS attacks these past couple of days.
Thanks for your help and time!

They're completely fine with it, and a minimum setting is ideal.

I use Bulk Image Downloader and set it to one download at a time, with the tagging specifications kept strict for both ease of download and for future sorting.

I used to use Python, but BID is universal.

Kataphraktarii said:
Ah, that clears quite a bit up. I see what you mean by the incorrect term, sorry about that (although I'm not quite sure what other term to use in it's place to refer to filling a server up with requests. Where I live spam is basically an all-encompassing term for these kinds of things haha).

Like I tried to say, 'filling a server up' with requests is an absolutely textbook DoS attack.

Also, now that it has been mentioned, I am interested in what services Cloudflare (I think you meant that instead of Cloudfront? If not, then my mistake.)

Yeah, that's my mistake. Too many weather reports.

GameManiac said:
They're completely fine with it, and a minimum setting is ideal.

I use Bulk Image Downloader and set it to one download at a time, with the tagging specifications kept strict for both ease of download and for future sorting.

I used to use Python, but BID is universal.

Hm?... No, it's only for Windows.
DownThemAll is much closer to universal, since it runs entirely in your browser.

Thanks everyone for all the responses, they were all really helpful! I'll be sure to set my download rate low to be on the safe side if I do decide to use a mass downloader in the future.