Topic: Random newb's proposal to save bandwidth: local caching/forwarding proxies

Posted under General

Sorry to drop in unannounced, but I read you guys had funding problems. Presuming a large part of the bandwidth is used for images, I had an idea that may help reduce the load somewhat. Basically, it involves a two-part system: interested users who want to help could install a proxy that filters e621 requests and sends a special response to the e621 server, indicating that it has kept a local copy of the image in question and is willing to share it with, say, ten other users. Then, when another user requests that image, one of two things may happen: 1) the other user also uses the proxy, in which case it could initiate a multi-source download for best speed. Or 2), the other user doesn't use a proxy in which case the server would HTTP forward the request to the proxy with the highest capacity (or take it itself if none is found). (1. also works to reward people who help out with better speeds, without unduly penalizing people who don't)

Presuming that a decent majority of the bandwidth is spent on a relatively small amount of images, this could reduce the server load, and bandwidth costs in the process.

Does that sound feasible? Can it work? If not, why not?

Thanks for feedback.

Updated by cetetic

Such a system would be difficult to set up, and there is a lot of potential to break such a system (things like replacing all the images with goatse, and flash with rickroll).

I myself run behind a caching proxy, so my affect on the bandwidth is lowered.

Updated by anonymous

Goatse/Flash exploitery could be reduced by checksumming. When a proxied client gets an invalid segment from another proxied client, it could tell the server to drop that client from its redirect list.

Also, most of the setup cost would be server side - once the proxy works, installing it can be made arbitrarily simple, and even if somebody doesn't use it himself he could still use other proxies in "forwarding server mode" and thus benefit.

Updated by anonymous

FeepingCreature said:
Goatse/Flash exploitery could be reduced by checksumming. When a proxied client gets an invalid segment from another proxied client, it could tell the server to drop that client from its redirect list.

That doesn't stop the client that provides the post to the end user from modifying the content.

Updated by anonymous

Kitsu: That's true, but I'm afraid there is no effective, foolproof way of preventing that while using HTTP.

cetetic: coral might work, but I can't find what their policy on adult content is. Any idea?

[edit] Another way to cut down on imagespam is to include in the site a link along the lines of "Click here if this is not the image you expected to see. ", then count strikes against each proxy provider and kick them off when it becomes too high.

Updated by anonymous

coral might work, but I can't find what their policy on adult content is. Any idea?

I don't think they have any content restriction: their policy is that of a carrier not responsible for the nature of what's served through them.

The only content-related mention in their FAQ is that DMCA notices should be sent to the source website, not Coral. If they don't care about illegal content, i doubt they'll give a hoot about some drawings of dubious prudence.

Updated by anonymous

  • 1