Hey folks,

This is a thread for me to post updates, observations and issues with what's going on relating to performance on e621.

Update 26/8/2013:
I have finally gotten the database server stable, which has reduced the amount of times the site gets slower and slower before finally becoming unusable. We are now pegged on CPU for Ruby on Rails during peak hours, and I have ordered a third server to handle the additional load.

I've performed a few optimizations to reduce the number of 500's during peak hours; page load times during peak times will continue to go a bit high until we get another box set up, but the site should remain reliable now most of the database issues are nailed down.

As of yesterday we broke through the 1000 requests/sec barrier - a new record for us. Some fun statistics for those of you into them:

Peak pageviews/sec: 140 pageviews/sec
Peak images/sec: 888 images/sec
Peak bandwidth use: 410 mbit/sec
Bandwidth used/total: 129TB/month

Update 28/8/2013:
The third server for e621 is all set up now, and I'm working on getting it set up to handle more of the Ruby on Rails load. I know the site is pretty slow during peak times now and this upgrade will likely improve things a lot (as well as get rid of a lot of the 500 errors).

In the coming days I will be making e621 use SSL by default. This will provide additional security if you're logged in, as it means your session cookies (ie, your login to the site) will be sent encrypted.

Update 29/08/2013:
I have finished configuring the new Ruby on Rails server and fixing everything; however I will not be pushing it live just yet as I want to be sure it's all working properly and I'm very tired.

Later today I will move traffic over and e621 should improve substantially in overall speed! Whee!

Update 2/9/2013:

After a lot of butthurt and quite a few stupid bugs, e621 now has almost twice the processing power at its disposal now. The site should be a lot more responsive - as soon as I got everything stable, I noticed traffic shot up by around 30% - quite a big change!

I am still seeing some occasional 500 errors in sporadic bursts - I am keeping an eye on it and am working with the provider of one of the components of software for a solution.

Any problems? Post them in the other performance thread here! https://e621.net/forum/show/77987?page=5

Update 7/9/2013:

After a lot more butthurt than last week, e621 should now have a lot fewer bugs! RSS feeds should be fixed, DMails should work, and we have a wonderful new feature called Sets ideal for organizing and storing your favorites.

Site performance should be just as good as before now - we may be messing with some caching magic over the next week or so to reduce performance use on our end too.

We peaked at 425mbit/sec this week, and reckon we'll be hitting close to 450mbit/sec in a month or two.

Complain here! https://e621.net/forum/show/77987?page=5

Everything seems to be running very smoothly right now.

Last week, the tag history page went haywire (probably because a bot was trying to visit all of the really old pages) and it wasn't optimized, which put a ton of strain on our database server. This has since been fixed, and the site has behaved very well since.

Some more fun statistics, now everything's stable:

Peak pageviews/sec: 181 pageviews/sec
Peak scripts/css/sec: 80 scripts/sec (could use some optimization here)
Peak images/sec: 983 images/sec
Peak bandwidth use: 441 mbit/sec
Bandwidth used/total: 135TB/month
Most popular time of day: 4pm PST, closely followed by 9pm PST. (Twice as busy as 3am PST)

Complain here! https://e621.net/forum/show/77987?page=6