Ponyponypony said:
Just got a Error 500 while doing a search for 'clopper-dude'.

lol me too but searching for a different artist.

Varka said:
Hey folks,

This thread is for the ongoing discussion of e621's performance - how fast it loads, how often it gives a 500 error, and so on. If you have a comment or question on the site's performance, here's the place to post it!

This will also be used by me to ask questions and get feedback. My first one is... Does the site feel noticeably faster today? I have put the images on a seperate machine now, and would like to know if it's improved things at all.

Thanks!

Varka

shadowwolf said:

ive cleard my chases and i still cant load flash file on here there blank

I can't view images at all on the 'safe' version of this site, e926

The preview is just the tags, as is the image itself. Clicking on 'download' does nothing, either.

EDIT: On firefox, tried it with IE and Chrome, no go.

For some reason I'm unable to view Download posts on e621, but only after I log in. If I log in and log out, I still can't view them.

I'm using Chrome on Windows 7. I tried clearing the browser cache and flushing the DNS, but I'm still having this problem.

Varka said:
This thread is for the ongoing discussion of e621's performance - how fast it loads, how often it gives a 500 error, and so on. If you have a comment or question on the site's performance, here's the place to post it!

Please detail exactly what sort (eg make/model) of hardware e621 is running on. How many physical machines? What sort of processor / RAM / disk combos? How is the database query load distributed? What sort of back end props things up?

chicory said:
For some reason I'm unable to view Download posts on e621, but only after I log in. If I log in and log out, I still can't view them.

I'm using Chrome on Windows 7. I tried clearing the browser cache and flushing the DNS, but I'm still having this problem.

shadowwolf said:
ive cleard my chases and i still cant load flash file on here there blank

clear your cookies, if it still doesn't work then you gotta trim your blacklist.

We will get this fixed as soon as possible.

Well I've been getting 500's in this hour, it stopped for a time and thought it was gone but it's happening again to me. All I've been doing is browsing posts.

Azazial said:
Please detail exactly what sort (eg make/model) of hardware e621 is running on. How many physical machines? What sort of processor / RAM / disk combos? How is the database query load distributed? What sort of back end props things up?

2 physical machines, one machine for application, one machine for db and static materials (one virtual machine each). first machine has raid of hdds, the second has ssds. in terms of ram and processor, let's say it's a shit ton.

Butterscotch said:
Well I've been getting 500's in this hour, it stopped for a time and thought it was gone but it's happening again to me. All I've been doing is browsing posts.

Aurali said:
2 physical machines, one machine for application, one machine for db and static materials (one virtual machine each). first machine has raid of hdds, the second has ssds. in terms of ram and processor, let's say it's a shit ton.

Offering performance suggestions when you don't know anything about hotspots or bottlenecks or load is usually less than helpful, but if you don't currently have a caching layer, it might really help out with the slow performance the site has at times (like now).

Something like a reverse proxy cache in front plus memcached for the database might improve things quite a bit. Doing caching right can be hard, but is essentially always worthwhile. Just a thought :)

Super slow today too. =~.~=

I have a problem with searching myself in comments. When I click on it, it starts loading, and still says it is, but it doesn't go anywhere. Even if you refresh the page it doesn't do anything; it just loads for hours and doesn't move.

xTheWolf94x said:
I have a problem with searching myself in comments. When I click on it, it starts loading, and still says it is, but it doesn't go anywhere. Even if you refresh the page it doesn't do anything; it just loads for hours and doesn't move.

Noted.
Edit: Aaaand Optimized.

DrNick said:
Offering performance suggestions when you don't know anything about hotspots or bottlenecks or load is usually less than helpful, but if you don't currently have a caching layer, it might really help out with the slow performance the site has at times (like now).

Something like a reverse proxy cache in front plus memcached for the database might improve things quite a bit. Doing caching right can be hard, but is essentially always worthwhile. Just a thought :)

I custom built a caching layer for the system, just gotta wait for the go ahead from Varka, who is afraid anything new will interfere with getting accurate numbers.

As for memcached, We have a memcached server, doesn't really help when the database has gotten outrageously big in the recent months. Been rewriting one query after another from the original code to improve performance.

Aurali said:
2 physical machines, one machine for application, one machine for db and static materials (one virtual machine each). first machine has raid of hdds, the second has ssds. in terms of ram and processor, let's say it's a shit ton.

Could you be a little more specific than "a shit ton"? Are we talking quad socket motherboards with hundreds of GB of RAM here?

Azazial said:
Could you be a little more specific than "a shit ton"? Are we talking quad socket motherboards with hundreds of GB of RAM here?

I am contractually bound not to go into any more details.

aka she cant remember

for some reason I started to have some [X] on this web...
f5 fixed it but sometimes it still happen

I just had 500 error that lasted more than one refresh try. I hoped that these are at least gone.

the site seems to be slower to load at the moment

edit: the site was unavailable for 5 minutes

Aurali said:
I am contractually bound not to go into any more details.

Alright. Thanks for the information.

We know what the issue is and will be working on it today.

https://e621.net/user?name=&level=-1&order=tagedits
That page is unreachable. It takes a good 20 mins to load and after all that, it ends with a 500 error

TheHuskyK9 said:
https://e621.net/user?name=&level=-1&order=tagedits
That page is unreachable. It takes a good 20 mins to load and after all that, it ends with a 500 error

It actually takes the same time for all levels in tagedits order, I was trying to see that about an hour ago.

Butterscotch said:
It actually takes the same time for all levels in tagedits order, I was trying to see that about an hour ago.

Were there 500 errors in the others?

I actually did not get 500 in any (including the one you mentioned) of them, but it really takes a good amount of time, about 20 mins as you said. The 500s are happening to me sometimes when I'm either searching or trying to edit tags in a post.

Butterscotch said:
It actually takes the same time for all levels in tagedits order, I was trying to see that about an hour ago.

tony when you get a chance can you swap the hard look up with the user_profile values?

May I ask if it is normal for the site to be slow right now? (just asking), it's just that I'm getting 500's frequently and uploading and editing tags take a good amount of time to load >.<

So many 500s. So many. Haven't hit one on searching, but loading posts and editing posts and loading forum threads have all 500d on me in the past few minutes.

Azazial said:
Could you be a little more specific than "a shit ton"? Are we talking quad socket motherboards with hundreds of GB of RAM here?

200GB would just barely be able to cache the entire image collection (and not for very much longer), 64GB is fairly typical for an enterprise-level database server nowadays, so somewhere in between there.

I wouldn't bother to think "shit ton" until, say, a TB memcached cluster.

So. Perhaps it's time for a CDN? One with TLS/HTTPS, so I can keep that nice green/gold padlock.

ThenIThought said:
200GB would just barely be able to cache the entire image collection (and not for very much longer), 64GB is fairly typical for an enterprise-level database server nowadays, so somewhere in between there.

I wouldn't bother to think "shit ton" until, say, a TB memcached cluster.

So. Perhaps it's time for a CDN? One with TLS/HTTPS, so I can keep that nice green/gold padlock.

We don't need that.

Yeah, my desktop has 32GB of RAM in it so I don't really start thinking "shit ton" until the hundreds at least. Bandwidth isn't really their problem at the moment so a CDN is rather pointless. It's raw power needed to handle the database which might not fit entirely in memory I'm going to guess. The stuff that tends to take forever are are obscure or complex queries pointing again to a query or database issue as best I can guess.

Given what Aurali said, Varka apparently made her (and probably Tony and Char and whoever the new head admin is going to be) sign some sort of confidentiality agreement about certain parts of the inner workings of the site, so it's reasonable to guess that we'll never quite get the full story on this subject.

Maybe you ought to try talking to the Danbooru people seeing as how they have over 4 times as many posts as e621 but I'm unable to find a query that bogs down Danbooru for more than 6-8 seconds. They even have numerous single tags which have a higher count than the entire number of pictures on e621. Ouroboros is a fork of the Danbooru code base anyways, IIRC.

Azazial said:
Yeah, my desktop has 32GB of RAM in it so I don't really start thinking "shit ton" until the hundreds at least. Bandwidth isn't really their problem at the moment so a CDN is rather pointless. It's raw power needed to handle the database which might not fit entirely in memory I'm going to guess. The stuff that tends to take forever are are obscure or complex queries pointing again to a query or database issue as best I can guess.

Given what Aurali said, Varka apparently made her (and probably Tony and Char and whoever the new head admin is going to be) sign some sort of confidentiality agreement about certain parts of the inner workings of the site, so it's reasonable to guess that we'll never quite get the full story on this subject.

Maybe you ought to try talking to the Danbooru people seeing as how they have over 4 times as many posts as e621 but I'm unable to find a query that bogs down Danbooru for more than 6-8 seconds. They even have numerous single tags which have a higher count than the entire number of pictures on e621. Ouroboros is a fork of the Danbooru code base anyways, IIRC.

I'm not sure as I haven't saved the picture but I believe the old server had something around 128 GB of RAM and 32 CPU, I believe all 4 with cores.
I'll go out on a limb and say the new one will have at least double the amount of that.

Lots of slowness/500. =~.~=

furballs_dc said:
Lots of slowness/500. =~.~=

Same here

TheHuskyK9 said:
Same here

And here.

Azazial said:
Maybe you ought to try talking to the Danbooru people seeing as how they have over 4 times as many posts as e621 but I'm unable to find a query that bogs down Danbooru for more than 6-8 seconds. They even have numerous single tags which have a higher count than the entire number of pictures on e621. Ouroboros is a fork of the Danbooru code base anyways, IIRC.

Danbooru doesn't get as many hits as we do. Sure more content, but less usage :3.. well according to the generated stats I see..

At this very /second/ thousands of people have just used a page on e621. Next second. thousands more. It's crazy.

Butterscotch said:
And here.

yeah, the database needs faster ssds, besides that tony and I have been stripping and devouring old queries left and right.. You won't believe how unefficient the original code base was..

Aurali said:
besides that tony and I have been stripping and devouring old queries left and right.. You won't believe how unefficient the original code base was..

Thank god for housecleaning. Programming code is not my area of expertise. But I know how much of a difference it makes when someone straightens out things so that they run more smoothly underneath the hood. A very serious *high five* for anyone taking that on!

I also had no idea e621 had so much traffic. That's incredible.

You need MongoDB. It is web scale.

Error 500s upon attempting to access
http://e621.net/forum/show/78835?page=2
http://e621.net/forum/show/77987?page=4

time: just now.

Azazial said:
Error 500s upon attempting to access
http://e621.net/forum/show/78835?page=2
http://e621.net/forum/show/77987?page=4

time: just now.

I got stuck for a while trying to see my own profile page.

Edit: and when trying to post and create this edit >.<

Aurali said:
Danbooru doesn't get as many hits as we do. Sure more content, but less usage :3.. well according to the generated stats I see..

What's your source for this? I tried finding some stats, but the best I could manage with a quick google was alexa page rank which puts the sites at similar levels of traffic.

http://www.alexa.com/siteinfo/donmai.us
http://www.alexa.com/siteinfo/e621.net

I don't suppose you'd be willing to show us off some stats or graphs from e621's back end?

...The server move seems to have made things worse, I've been noticing. =~.~=

furballs_dc said:
...The server move seems to have made things worse, I've been noticing. =~.~=

Something broke, Baaad. should be fixed now.. should be. Turns out the new database's internal cleanup wasn't locking things completely and that was causing inflation of the database's physical drive size, and we were running into limits involving the disk usage. got the locks in place, so hopefully this can be marked as solved

DrNick said:
You need MongoDB. It is web scale.

You can't just drop a non relational database on top of an entirely over relational system and expect it to work! That and our search system relies very heavily on postgres's text search vector, something mongoDB isn't ready to be scalable in.

Azazial said:
What's your source for this? I tried finding some stats, but the best I could manage with a quick google was alexa page rank which puts the sites at similar levels of traffic.

http://www.alexa.com/siteinfo/donmai.us
http://www.alexa.com/siteinfo/e621.net

I don't suppose you'd be willing to show us off some stats or graphs from e621's back end?

this very second we had hits from from 2500 users... and that's with the site being just out of "Down" besides that... we've been in a slump due to performance. but once it regains we'll slowly reclaim our title.

Seems better for now. =^.^=

Edit: Taking it's sweet time loading again.

Edit 2: Worse again.

It takes still a while to load pages (mostly forum and posts for me) but it isn't happening that often now, and the only 500 I've got is when I tried to watch the forum.

Aurali said:
You can't just drop a non relational database on top of an entirely over relational system and expect it to work!

Lol, I know. It was a funny: http://www.youtube.com/watch?v=b2F-DItXtZs :)

Aurali said:
...
You can't just drop a non relational database on top of an entirely over relational system and expect it to work! That and our search system relies very heavily on postgres's text search vector, something mongoDB isn't ready to be scalable in.
...

Here's the solution to your problems:
http://en.wikipedia.org/wiki/Oracle_Exadata

Halite said:
Here's the solution to your problems:
http://en.wikipedia.org/wiki/Oracle_Exadata

Tried to read it, brain was on fire, stopped after the first sentence. I swear, it's like a different language only tech people like Aurali and Tony can understand

TheHuskyK9 said:
Tried to read it, brain was on fire, stopped after the first sentence. I swear, it's like a different language only tech people like Aurali and Tony can understand

It's a server specifically designed to run an Oracle database, particularly large and/or high traffic databases.
They can run with almost 10 times the bandwidth of a standard server configuration.
On the downside, they're stupidly expensive.

Large companies with massive amounts of data use them, like banks.

TheHuskyK9 said:
Tried to read it, brain was on fire, stopped after the first sentence. I swear, it's like a different language only tech people like Aurali and Tony can understand

Wonder if Database servers X2-8 could be converted to a gaming machine... If you had the money.

Halite said:
Here's the solution to your problems:
http://en.wikipedia.org/wiki/Oracle_Exadata

To be honest We don't need more hardware. We need the database to stop crapping on itself.. and varka's really weird recently about what things can be changed :p

Aurali said:
To be honest We don't need more hardware. We need the database to stop crapping on itself.. and varka's really weird recently about what things can be changed :p

Add green LEDs to the servers and all your problems will be gone! I wonder if you can use regular expressions on *any* search

furballs_dc said:
Wonder if Database servers X2-8 could be converted to a gaming machine... If you had the money.

Yes and no.

You run into a potential issue of operating systems.
They're not designed for Windows.

TBH, you could make a Windows OS server box that's comparable in quality for quite a bit less.
The advantages of the box that make it more expensive are things that work best specifically with databases.

Halite said:
Yes and no.

You run into a potential issue of operating systems.
They're not designed for Windows.

TBH, you could make a Windows OS server box that's comparable in quality for quite a bit less.
The advantages of the box that make it more expensive are things that work best specifically with databases.

Imagine the setup needed to run TF2 at a 2048*1536 screen resolution, RayTracing quality graphics with all effects, 5.1 Surround Audio and with no lag of any kind (connection, rendering time, processing, etc).

Also, related: Images are taking a bit to load, some load and some not, until hovered (that last part could probably be on my end, though).

Xch3l said:Also, related: Images are taking a bit to load, some load and some not, until hovered (that last part could probably be on my end, though).

You mean the thumbnails in post index? because in that case the same thing it's happening to me. Full images load fast for me.

Butterscotch said:
You mean the thumbnails in post index? because in that case the same thing it's happening to me. Full images load fast for me.

Uh-huh, and in wiki pages (like square_crossover). Several 500s when looking for that tag (forgot the name and tried looking everywhere -- wiki, tags, forum) and going page by page on each results list.

I dabbled in a thing called Solr once (like 4 years ago), by Apache. Its a search indexing program which runs on input XML. So, the search can be run out of database to keep the load down. From what I saw, it was really fast, just not super simple to figure out.

Lots of 500s right now >.< took me a while to get here.

Butterscotch said:
Lots of 500s right now >.< took me a while to get here.

I was rebooting a machine

I was unable to reach e621 for several hours today (around 2-4 hours earlier than this post or so) because of the front page giving me Error 500 over a prolonged period of time.

Azazial said:
I was unable to reach e621 for several hours today (around 2-4 hours earlier than this post or so) because of the front page giving me Error 500 over a prolonged period of time.

Yeah, this. =>.<=

Also was getting a constant error 500 for around 15 mins about 30 mins ago.

Thanks for keeping us updated in the little news widget at the top of the page. It's nice to know what's going on this time, rather then being left wondering if it's worse than it is. Or if it needs to be mentioned anywhere. Just wanted to mention how much it helps, thanks.

Is the IRC down? I'm getting
Unable to connect : java.net.ConnectException : Connection refused: connect

EDIT: Not any more, yay!

Things are working much better. I haven't gotten a 500 or connection time out today.

But I have been running into images which refused to display "the image [name of image] cannot be displayed because it contains errors."

The image is usually of a large resolution, like 2000 or so wide. I can download it fine, and view it fine on my hard drive, so I'm not so sure it's the file that has a problem. But something isn't working right. Smaller images load fine, pages have been loading faster than they used to. It's really the only problem I've been seeing today. Been running into several of them but just in the last 24 hours, so thought you might like to know about it.

the latest image which it did this on: https://e621.net/post/show/362184

[I'm using firefox. Refreshing does nothing. ctrl + F5 doesn't fix it. Cleared cache with no difference. Happens on multiple absurd res images throughout the site, but only absurdly high resolution regardless if they have that tag or not.]

EDIT: No longer having any problems loading absurdly large images. They now load perfectly. Thanks, if someone fixed it with the other site debugging going on.

Xch3l said:
How much RAM do you have?...

To be honest, I'm not sure if that could have been it or not. I only kept running into it after the site had been down for a couple hours and they started doing stuff behind the scenes to fix the 500 errors. So I thought it might be connected. However, I also restarted my computer between then and now. Currently it works fine now, even the 34.4MB one you linked to. So I'm not sure if it was my end or theirs. Thanks though.

How much RAM do you have? I have 512MB and that one loaded fine (sample and full size )

Except post #318592, which is 34.4MB (1154x7809) 137.5MB raw pixel data º~º

I constantly get 503 errors. Is the current load really that high?

The errors come in bunches. It works for about 6 to 14 posts (I don't let them load simultaneously but after each other) and suddenly I get lots of errors in a row.
... and just got a peek of 33 posts in a row, then 500-error.
... and 28 then 503.
I could save a log file, but next time. It already loaded too much to start over with.

Also: http://www.alexa.com/siteinfo/e621.net
The fact that so many people visit this page from schools creeps me out a bit.

Yeah, there's something up with Passenger, the system e621's code runs on, that is making it fall over every few minutes for a handful of requests and then return back to normal.

I will be investigating (and hopefully fixing it) this weekend!

@Varka: Nice to hear that. I could upload the log file of the batch I was running, if that helps you. You can see how many times it worked until it crashes. Also you can see the times.

As of 3 hours ago, your "Maintenance complete" announcement went out. Oddly, I'm actually having below-average performance now. Pages and vote queries are taking upwards of 20 seconds to complete. I imagine it might still be "warming up", though. I'll try back later tonight and see where it's at.

I took a look at the full announcement, e621 is still running with issues (many 500 errors). I know you guys have a lot of people to serve so it doesn't bother me so much. Keep up the good work.

On a side note, why bother implementing SSL at all? I could understand it if this site was designed to hold important personal information like credit cards... I could care less if my account got hacked by a third party monitoring my internet.

Yesterday, everything was ok, but today I obtain only wrong RSS data. Links in RSS begins with "https//..." Can Somebody fix it to correct "https://"?
Thanks.

I can tell a huge improvement. Pages loading and responding much faster for me so far. It's been very nice to use. Great job and a huge thanks to everyone who worked on it. (I'm betting it was a lot of work.)

ETA: Was loading smooth and quickly earlier today. Right now though things are having inconsistent load times. Sometimes very slow to load, then a page will suddenly load normal. Lots of unpredictable inconsistency that wasn't there this afternoon. Obviously still some kinks in the works, but steadily moving forward.

furrypickle said:
I can tell a huge improvement. Pages loading and responding much faster for me so far. It's been very nice to use. Great job and a huge thanks to everyone who worked on it. (I'm betting it was a lot of work.)

For me it loads fast sometimes, and others just goes back to be slow.

Um,i need to say something.

Recently,some pictures started to not show up.At here,and at Rule34.xxx

I did everything that e621 told,but i can't see the pictures still.Can someone help me?