Forum - Tips loading sample images

Chessax

Privileged

I'm looking for any tips of why I can't load sample images in my little web app. Loading the full image always seem to work, but the sample almost never works. All I get is a redirect to the post page (but don't even get any body since the last redirect doesn't fire). I'm most likely missing out on something basic that I've overlooked.

I get sample_url directly from the /post/index API and the thing I'm struggling with comprehending is why visiting the sample url directly through the browser or other tool works, e.g.:

curl -I "https://static1.e621.net/data/sample/14/29/1429b7511cf149c5df236595056c7fb6.jpg"

But when I make a very simple request of the image through JavaScript it takes a completely different route, sample code below:

img = document.createElement("img");
document.getElementsByTagName('body')[0].appendChild(img); //Might not be necessary...
sample_url = "https://static1.e621.net/data/sample/14/29/1429b7511cf149c5df236595056c7fb6.jpg";
img.src = sample_url;

What's even stranger is that querying the sample url in the browser with javascript, then with curl, and again with javascript and it works, querying twice with javascript doesn't work. Additionally, running the script on https://e621.net/ works as expected, which leads me to believe that there's a header problem and e6 thinks that I'm some kind of criminal, but I have been unable to isolate the issue and is a bit confused.

This is what I get using the JavaScript above:

Request 1 (302 redirect with https demotion)

General:
Remote Address:104.25.118.23:443
Request URL:https://static1.e621.net/data/sample/14/29/1429b7511cf149c5df236595056c7fb6.jpg
Request Method:GET
Status Code:302 Moved Temporarily

Response Headers:
cache-control:no-cache
cache-control:max-age=2678400
cf-cache-status:MISS
cf-ray:2331ecd407670a5a-ARN
content-type:text/html
date:Sat, 10 Oct 2015 11:18:32 GMT
expires:Tue, 10 Nov 2015 11:18:32 GMT
location:http://e621.net/post/md5redirect?id=1429b7511cf149c5df236595056c7fb6
server:cloudflare-nginx
status:302 Moved Temporarily
vary:Accept-Encoding
version:HTTP/1.1

Request Headers:
:host:static1.e621.net
:method:GET
:path:/data/sample/14/29/1429b7511cf149c5df236595056c7fb6.jpg
:scheme:https
:version:HTTP/1.1
accept:image/webp,image/*,*/*;q=0.8
accept-encoding:gzip, deflate, sdch
accept-language:sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4
cookie:__cfduid=debb112b87e492af2b3e5bbade76d3ecb1417642208; _gat=1; __utmt=1; __utma=71838535.1756148194.1417642210.1444315026.1444475695.570; __utmb=71838535.1.10.1444475695; __utmc=71838535; __utmz=71838535.1432074241.268.3.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); _ga=GA1.2.1756148194.1417642210; _gali=p173795
referer:https://xxx/
user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36

Request 2 (301 redirect, https re-promotion)

General:
Remote Address:104.25.118.23:80
Request URL:http://e621.net/post/md5redirect?id=1429b7511cf149c5df236595056c7fb6
Request Method:GET
Status Code:301 Moved Permanently

Response Headers:
CF-RAY:2331ecd54d2816a6-ARN
Connection:keep-alive
Content-Type:text/html
Date:Sat, 10 Oct 2015 11:18:32 GMT
Location:https://e621.net/post/md5redirect?id=1429b7511cf149c5df236595056c7fb6
Server:cloudflare-nginx
Transfer-Encoding:chunked

Request Headers:
Accept:image/webp,image/*,*/*;q=0.8
Accept-Encoding:gzip, deflate, sdch
Accept-Language:sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Cookie:__cfduid=debb112b87e492af2b3e5bbade76d3ecb1417642208; history-columns=true%2Ctrue%2Ctrue%2Ctrue%2Ctrue%2Ctrue%2Ctrue%2Ctrue%2Ctrue; hidden_categories=; css=hexagon%3Boverrides%2Fchristmas%3B1412208000; login=Chessax; hide_news_notice=1442348112; cf_use_ob=0; _gat=1; __utmt=1; __utma=71838535.1756148194.1417642210.1444315026.1444475695.570; __utmb=71838535.1.10.1444475695; __utmc=71838535; __utmz=71838535.1432074241.268.3.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); mode=view; chessax_tags=female; chessax_page=1; blacklisted_tags=; blacklist_avatars=false; blacklist_users=false; e621=xxx; _ga=GA1.2.1756148194.1417642210; _gali=p173795
Host:e621.net
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36

Query String Parameters:
id:1429b7511cf149c5df236595056c7fb6

Request 3 (200 OK, but not so OK...)

General:
Remote Address:104.25.118.23:443
Request URL:https://e621.net/post/md5redirect?id=1429b7511cf149c5df236595056c7fb6
Request Method:GET
Status Code:200 OK

Response Headers:
cache-control:private, max-age=0, must-revalidate
cf-ray:2331ecd7b7a30a5a-ARN
content-encoding:gzip
content-type:text/html; charset=utf-8
date:Sat, 10 Oct 2015 11:18:33 GMT
etag:W/"36aeec37f05ae52088625faea0e07920"
server:cloudflare-nginx
set-cookie:e621=xxx; path=/; expires=Sat, 17-Oct-2015 11:18:33 GMT; HttpOnly
set-cookie:blacklisted_tags=; path=/
set-cookie:blacklist_avatars=false; path=/
set-cookie:blacklist_users=false; path=/
status:200 OK
status:200 OK
version:HTTP/1.1
x-powered-by:Phusion Passenger 4.0.33
x-runtime:4

Request Headers:
:host:e621.net
:method:GET
:path:/post/md5redirect?id=1429b7511cf149c5df236595056c7fb6
:scheme:https
:version:HTTP/1.1
accept:image/webp,image/*,*/*;q=0.8
accept-encoding:gzip, deflate, sdch
accept-language:sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4
cookie:__cfduid=debb112b87e492af2b3e5bbade76d3ecb1417642208; history-columns=true%2Ctrue%2Ctrue%2Ctrue%2Ctrue%2Ctrue%2Ctrue%2Ctrue%2Ctrue; hidden_categories=; css=hexagon%3Boverrides%2Fchristmas%3B1412208000; login=Chessax; hide_news_notice=1442348112; cf_use_ob=0; _gat=1; __utmt=1; __utma=71838535.1756148194.1417642210.1444315026.1444475695.570; __utmb=71838535.1.10.1444475695; __utmc=71838535; __utmz=71838535.1432074241.268.3.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); mode=view; chessax_tags=female; chessax_page=1; blacklisted_tags=; blacklist_avatars=false; blacklist_users=false; e621=xxx; _ga=GA1.2.1756148194.1417642210; _gali=p173795
if-none-match:W/"36aeec37f05ae52088625faea0e07920"
user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36

Query String Parameters:
id:1429b7511cf149c5df236595056c7fb6

And when visiting visiting above Request URL directly in browser:

Non-redirected Request 4 (200 OK, but not an image!)

General:
Remote Address:104.25.118.23:443
Request URL:https://e621.net/post/show?md5=1429b7511cf149c5df236595056c7fb6
Request Method:GET
Status Code:200 OK

Response Headers:
cache-control:private, max-age=0, must-revalidate
cf-ray:23324dab56120a4e-ARN
content-encoding:gzip
content-type:text/html; charset=utf-8
date:Sat, 10 Oct 2015 12:24:39 GMT
etag:W/"42ba15c18fd26c46646356d7acda534e"
server:cloudflare-nginx
set-cookie:e621=xxx; path=/; expires=Sat, 17-Oct-2015 12:24:39 GMT; HttpOnly
set-cookie:blacklisted_tags=; path=/
set-cookie:blacklist_avatars=false; path=/
set-cookie:blacklist_users=false; path=/
status:200 OK
status:200 OK
version:HTTP/1.1
x-powered-by:Phusion Passenger 4.0.33
x-runtime:119

Request Headers:
:host:e621.net
:method:GET
:path:/post/show?md5=1429b7511cf149c5df236595056c7fb6
:scheme:https
:version:HTTP/1.1
accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
accept-encoding:gzip, deflate, sdch
accept-language:sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4
cookie:__cfduid=debb112b87e492af2b3e5bbade76d3ecb1417642208; history-columns=true%2Ctrue%2Ctrue%2Ctrue%2Ctrue%2Ctrue%2Ctrue%2Ctrue%2Ctrue; hidden_categories=; css=hexagon%3Boverrides%2Fchristmas%3B1412208000; login=Chessax; hide_news_notice=1442348112; cf_use_ob=0; __utma=71838535.1756148194.1417642210.1444315026.1444475695.570; __utmc=71838535; __utmz=71838535.1432074241.268.3.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); mode=view; chessax_tags=female; chessax_page=1; _gat=1; _ga=GA1.2.1756148194.1417642210; blacklisted_tags=; blacklist_avatars=false; blacklist_users=false; e621=xxx
if-none-match:W/"6d6dbd783c4a759e8aa3817b677a1990"
referer:https://e621.net/post/md5redirect?id=1429b7511cf149c5df236595056c7fb6
upgrade-insecure-requests:1
user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36

Query String Parameters:
md5:1429b7511cf149c5df236595056c7fb6

Updated over 8 years ago

user 59725

Member

over 8 years ago

Yeah, the site forces https for everything except static content (i.e., manually edit the url on a direct image link), so no surprises there. Any http links will just result in a redirect.

I'm terrible with anything related to the api, and even worse with Javascript, but it might be worth trying to rule out the cache as a source of your problems. Try appending ? followed by a few random characters to the url (e.g., ....jpg?123). This will essentially tell Cloudflare to not hit the cache.

Also, since the images are rarely accessed directly, setting the referer to something meaningful might help too.

Updated by anonymous over 8 years ago

Chessax

Privileged

over 8 years ago

parasprite said:
Yeah, the site forces https for everything except static content (i.e., manually edit the url on a direct image link), so no surprises there. Any http links will just result in a redirect.

Force HTTPS on all but non-static content = Sure
Downgrade from HTTPS to HTTP on a 302 redirect = BAD!!! >:[
(But since I suck at cryptography I have no idea if that redirect is a security hole...)

parasprite said:
I'm terrible with anything related to the api, and even worse with Javascript, but it might be worth trying to rule out the cache as a source of your problems. Try appending ? followed by a few random characters to the url (e.g., ....jpg?123). This will essentially tell Cloudflare to not hit the cache.

Made zero difference, both adding query string and disabling local cache.

parasprite said:
Also, since the images are rarely accessed directly, setting the referer to something meaningful might help too.

asdgahsd... Was sure I had tried setting referer... Okay, either way, this doesn't make sense:

Referer: (empty header) => Not OK
Referer: https://www.google.com/ => Not OK
No header => OK (I guess, though it's not better than empty...)
Referer: https://e621.net/ => OK

E.g.:

curl -I -e "" "https://static1.e621.net/data/sample/1f/2a/1f2a36da5d79a20478a40af20fb3f90b.jpg" # => 302 => 301 => 200 (but not really OK)
curl -I -e "https://www.google.com/" "https://static1.e621.net/data/sample/1f/2a/1f2a36da5d79a20478a40af20fb3f90b.jpg" # => 302 => 301 => 200 (but not really OK)
curl -I "https://static1.e621.net/data/sample/1f/2a/1f2a36da5d79a20478a40af20fb3f90b.jpg" # => 200
curl -I -e "https://e621.net/" "https://static1.e621.net/data/sample/1f/2a/1f2a36da5d79a20478a40af20fb3f90b.jpg" # => 200

However making a new unsuccessful request after a successful one and the referer header doesn't seem to make any difference, i.e. all the above requests return the wanted file. For me that's still a big mystery :|

I just wanted to avoid having to load that 80 MB mlp post we got on here just 'cause I want a bigger version than the thumbnail.

Unfortunately I don't know how to solve this in a good way at the moment. I can't do a standard AJAX request since you're not allowed to change the Referer header (and I don't like to spoof anyway...), and I have no idea if e621 allows the use of CORS in any way that could be utilized here...

I also don't really see the point why it even works in this strange way, why allow full access to full sizes but limited access to sample sizes?

Either way, thanks for trying to help, very much appreciated :)

Updated by anonymous over 8 years ago

News - Apr 02, 2024 (about 1 month ago) Click to show.

Topic: Tips loading sample images

Chessax

user 59725

Chessax

Login to respond »

Over 18?