Hey folks,

Due to a recent heatwave in Los Angeles, the cooling at the datacenter e621 (along with a bunch of other sites I host) was unable to keep up. This resulted in 2 disks failing in another server, and necessitated a move of a large number of virtual machines to another server in the same rack (which also happened to have better cooling). This didn't affect e621.

Simultaneous with this, a hardware failure developed in the main network switch which serves e621 (most likely also caused by the heatwave), which resulted in intermittent problems with some servers working, and some not, for roughly the last 24hrs. This faulty switch has now been replaced, and services are back to normal on e621 (and most of the other sites).

It fucking sucks that this happened, as the cooling at the datacenter in question was flaky to begin with; the datacenter is having another chiller added on the roof of the building which should dramatically improve the situation (which is expected to happen tonight). If the situation is not resolved within the next 2 days, we'll be moving everything to another facility and avoiding this bullshit which should've been dealt with earlier ever again.

The site should remain up and stable until we move all our equipment, and advance warning will be posted on the forums and in other places such as my twitter account here:

http://www.twitter.com/varka

Ugh, these last 2 days have fucking sucked...

In other news, I have a new faster speed test result from the datacenter (see below).

http://www.speedtest.net/result/972078680.png

I live farther down the coast; it wasn't too bad of a heatwave for me, sad to hear it was even worse up north! Also, I always thought that e621 was hosted in Europe, surprising to hear its only a few hours north.

Oh my god. If you think it's bad there, it's worse (at least for me) in San Francisco.

It was cookin in the central coast. Its been between 108-115.

Didn't it hit like, 140, or something equally insane in LA?

Out of curiosity, this is just a colocation for you, right? Or do you own some part of the datacenter?

Hopefully there's some kind of monetary coverage for you if the system failed because the owners were inept or negligent. I hardly think "it got hot outside" qualifies as an Act of God.

Jesus. It's been nothing but cool winds and rain over here on the East coast. I hope your heat wave doesn't bounce over here to bother us.

mosh_grizzly said:
Jesus. It's been nothing but cool winds and rain over here on the East coast. I hope your heat wave doesn't bounce over here to bother us.

seconded

Haha suckas!
were expecting 30 degrees for a low by tomorrow...
:D

lol its been 60 dg in the MI lovin it but been to LA n livd Phoniex used to 120 dg at least e621 is up best wishes ^^

Haha, tropical depression type weather here in sunny Florida, you know: 30-45mph winds, blinding rain, possible tornados, power outages, and every one still has to work.
Good times :3

That's odd, I haven't noticed any bad connection to e621. I really don't see how weather effects it, anyway.

xSionx said:
That's odd, I haven't noticed any bad connection to e621. I really don't see how weather effects it, anyway.

I wasn't able to get on all day Wednesday, not even from other PC's either.

Too hot and processes arnt completed at an efficent level to keep up with standards. Or so I was told

too hot and computers melt and shit

thats why they invented heat sinks

but if its too hot heat sinks dont work

null0100 said:
too hot and computers melt and shit

thats why they invented heat sinks

but if its too hot heat sinks dont work

That's when you put HEAT SINKS on your HEAT SINKS!

tony311 said:
That's when you put HEAT SINKS on your HEAT SINKS!

This is not the BattleTech universe. It just doesn't work like that.

Yo dawg, I herd you like cooling, so I put a heatsink on your heatsink so you can cool while you cool.

Mysterious_Fox said:
Yo dawg, I herd you like cooling, so I put a heatsink on your heatsink so you can cool while you cool.

I was waiting for this XD

i hope every thing goes well...

i know """"most of the people here, hates me""""

but i wish good luck anyway...

I don't hate you. :3

i have no opinion towards you whatsoever. pretty much the same with everyone else on this site

Thank you Varka for doing all of this to keeping e621 up and running. I love you for it and i am sure all of the furry world does too.

Just wondering; What RAID configuration are you running?

We are experiencing 40-65 degree weather here in Virginia, Ill say it loud ~shouts~ I WOULD BE HAPPY TO BE IN CALI!

Travis said:
Thank you Varka for doing all of this to keeping e621 up and running. I love you for it and i am sure all of the furry world does too.

Yes, yes we do.

Surgetehfox said:
We are experiencing 40-65 degree weather here in Virginia, Ill say it loud ~shouts~ I WOULD BE HAPPY TO BE IN CALI!

Idk, I rather prefer 40-65 degree weather. Let's me wear my expansive coat collection when I go out :) Leather is not fun in the sun!

Okay, some news to follow:

We're going to be moving all of our equipment across to a new, sexier datacenter in Phoenix, AZ (yeah - in the fucking desert; somehow it's cheaper and more effective to run and cool there - go figure. It's probably the really low real estate and utility costs).

This is expected to happen in the coming month or two. It's likely that it'll either require about 12hrs of downtime, or if we do things properly and 'shuffle it across', less than an hour. More information will be available once I figure out how we're gonna move it all.

For now, the cooling in the datacenter has stabilised; one blower failure (for a single server) resulted in some internal services becoming unavailable, but so far there have been no more incidents relating to cooling.

Phoenix is a tech town.. it makes perfect sense <3

Varka said:
Okay, some news to follow:

We're going to be moving all of our equipment across to a new, sexier datacenter in Phoenix, AZ (yeah - in the fucking desert; somehow it's cheaper and more effective to run and cool there - go figure. It's probably the really low real estate and utility costs).

As somebody who's lived in that area before: A lot of the desert's heat is direct sunlight, so shade at noon can be something like 20+ degrees colder and when the sun goes down it can go down to the 60's or even 50's. A single AC could probably keep your server room cool.

Quit being such a retard!

Reshiramsmaster said:
Quit being such a retard!

I'm trying my best ;-;

Varka: just keep doin what your doing, you have people under you for a reason :)

Aurali said:
Varka: just keep doin what your doing, you have people under you for a reason :)

Varka, you are the reason e621 is still alive. You are doing a great job. e621 needs to stay alive, so keep doing your best and we are your support. Varka, you rock!

All this talk about data servers and chilling towers reminds me of the movie "Live Free Or Die Hard".

WolfieWolfie1992 said:
All this talk about data servers and chilling towers reminds me of the movie "Live Free Or Die Hard".

The new datacenter we're moving to reminds me of Oceans 13: http://www.datacenterbuilder.com/wp-content/themes/db/Images/db_secuirty_141.jpg

Varka said:
The new datacenter we're moving to reminds me of Oceans 13: http://www.datacenterbuilder.com/wp-content/themes/db/Images/db_secuirty_141.jpg

That's actually what it looks like? Purrrdy

Varka said:
The new datacenter we're moving to reminds me of Oceans 13: http://www.datacenterbuilder.com/wp-content/themes/db/Images/db_secuirty_141.jpg

oceans 11 was better. and thats really shiney for porn

big fat varka needs to be cooled constantly or he hyperventilates and the site dies

Erk said:

WTF?

Erk said:
big fat varka needs to be cooled constantly or he hyperventilates and the site dies

Hahahaha, absolutely true. <3