Data Center Fire at The Planet Disrupts Popular Blogging Service

But not completely. Disrupted any images hosted on the site. (caused it to be egads slow).

Somewhere around 27 Proboards servers are affected by this as well.

Just checked out VCDQ and it's down too.

Wow, sucks.
I also have a MPLS issue in Houston.
The internet ends in Houston?

Of all the thousands of sites down I'm surprised someone mentioned mine.. I'm honored..

-Gerard

Its back up now. Not updated since Saturday.

Digizip's servers are hosted in Texas and are also down due to power outage.

Good. They don't deserve to be online.

?

They're a bunch of scumbags. I had a DSL line with them and one day it just stopped working (it would still sync) I told them to fix it and a week later they still had no clue, so I told them to cancel. I kept on getting bills, despite countless calls to them.

Gee, at least they were communicating with you...*snicker*

IIRC it was mainly their voicemail.

How depressing.

Nice to know that, as a The Planet customer, I have to read / learn about this major outage issue from DSLR.

Thanks guys!

Well, I'll be honest about it. If I ran a company, I would be more concerned about damage control than telling you, the customer, what is going on. I mean think about it, a FIRE occured, the first thing on the owners mind was "whats down, whats up, who should I call, how many people are affected, what can we do to get back up ASAP before people sue". So before you get upset that DSLR "reported" it before the company did, you have to think, the company has to prepare a press release about how the fire got started, what they've done to make sure there is a backup in place if it happens again, etc, etc, and it all has to be worded in a long winded, lawyer friendly fashion. So I can safely say I am grateful it didn't cause the affected customers downtime resulting in days, and I happened to used quite a few of the affected sites myself within the last 24 and 48 hours reprogramming my old Verizon post paid phone to a pre-paid using QPST and the help of howardforums.com. Thanks to them, ironically, the internet on my cell phone started working properly after I seen a screen shot in the "how to" section, and found the hex code for the internet service in the actual shot that was hosted by this provider. So kudos to them for damage control and getting the site back up before I noticed it was down!

EDIT: I am actually posting this message on my PC using the same cell phone I was programming because ironically the speeds are faster than the apartments owned cable internet service. I get about 300 kpbs more download, and about 40 kbps faster upload. And although Verizon's prepaid plans suck for prices (99 cent a day to use it for that day, 10 cent a min, 10 cent per text, unlimited mobile web for 99 cent a day, thats what made me do it.). Besides, my Sprint Moto Q just died, and I have to save up for a replacement, so I am using the prepaid till I get my Mogul.

Maybe you could check in the System and Network Status forum at ThePlanet, and in particular this thread.

This is obviosly due to power grid modifications performed recently to reduce power consumption with installed air conditioners, read the article here »www.thehostingnews.com/n ··· 306.html

That article doesn't really mention anything about changing the power grid, it does talk about improvements to the cooling of the datacenters and associated energy savings.
Reducing the power consumption shouldn't cause a transformer explosion, on the contrary it should make it less likely.
The only item in the article that raised a warning sign in my mind was when they mention "Sealed power distribution units to reduce bypass airflow" since the PDUs need cooling too.

I have hosted with The Planet for about 3 years and run a hosting company.

Within two hours of the outage, we had new servers provisioned in their Dallas data center using their Automated Provisioning system.

It has been a very long night, but their provisioning system saved my company tonight.

This outage is going onto 24 hours almost. The Planet continues to not get an ETA which is very unprofessional IMO. I have 3 servers with them that are currently down due to this.

I just chatted with a buddy who was onsite during the supposedly 'electrical' fire, and said everything was business as normal.

Chat Log:
[Sunday June 01 2008 - 14:48:47] the whole thing with TP
[Sunday June 01 2008 - 14:48:51] is really fishy
[Sunday June 01 2008 - 14:49:12] i was there last night
[Sunday June 01 2008 - 14:49:17] and there was no trucks
[Sunday June 01 2008 - 14:49:21] no nothing
[Sunday June 01 2008 - 14:49:30] no electricians
[Sunday June 01 2008 - 14:49:58] so either they renamed the spring tx location tohouston 1
[Sunday June 01 2008 - 14:50:03] or nothing happened
[Sunday June 01 2008 - 14:55:07] the msg from Doug came at 11PM
[Sunday June 01 2008 - 14:55:18] i was there at 2 AM and everything was fine
[Sunday June 01 2008 - 14:55:23] and stuff is still not back.

He thinks there might have been a search warrant for servers, and they are using an electrical outage as a 'cover' story to hide the real truth of what happened.

I dont know if people realize how much of an outage this is causing. There are hundreds of thousands of sites if not more down because of this.

I have no idea about the fire being a cover. All I know is there are people losing a ton of money every minute and I imagine their will be lawsuits and lots of lost business.

Lets not forget a lot of DNS runs through theplanet.com also.

I am interested to hear more details. I cannot get to the page with the status as I assume it is getting a lot of hits.

Dean

wonder if this is why tigerdirect.com is down ?

TigerDirect routes to servers down in Miami that appear to be part of Internap. Unless they quickly switched over, I think it's probably just a coincidence.

My first thought is an electronic terrorist attack. How would a fire in any way disable a datacenter unless they had the suppression systems turned off in the whole building? I agree. Fishy.

- Andy

Perhaps the boys in blue forced everyone out, then the boys in red came with fire hoses (with water ).

Transformer exploded with enough force to take out three of the electrical room's walls. Fire department wouldn't let them turn on their backup generators. Fire suppression systems are designed to limit damage, not keep you running even though your datacentre is on fire.

I've seen a lot of backup generator installs and many have the transfer switch(s) located very near the transformer vault.
Generally this is fine but in this case I'd imagine the fact that the walls blew several feet also caused the power cables and transfer switches to move as well causing an unsafe condition.
I've seen transformer room explosions both on the primary and secondary side and they tend to be extremely powerful

edit: The CEO has a voice message explaining that the transfer switch did fail as a result of the explosion. »service-update.theplanet.com/

A website I have share-hosted at 3ix went down for over 2 hours yesterday, beginning at around 11:45 AM PDT. A traceroute shows that it's now being hosted at Hostdime in Orlando FL. I'm not sure if 3ix normally hosts from there or not. They only refer to their Texas data centers on their website. Maybe they transferred some hosting? maybe unrelated?

I probably wouldn't have caught the outage on my site, except that I happened to be running a trial of host-tracker.com monitoring... funny thing is, as of now, THEIR website is down too!

ripples....

It's not 3ix's datacenter, they resell from other hosts.

not surprised.

»www.thehostingnews.com/n ··· 306.html

lots of good stuff done here..

150 - 30 ton air conditioners!!! that is enough to chill Dodger stadium.

Glad no one was injured.

That's across their 6 data centers though. It's amazing the amount of heat that servers can put off and thanks to blades and other high density equipment, it only gets worse...

I have two servers over there - one is up and one is still down... 7:15pm - were supposed to have an update at 7:30...

from the Planet forum :
From Doug Erwin:
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.

We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.

There is no impact in any of our other five data centers.

I am sorry that this accident has occurred and I apologize for the impact.

--------------------

StatCounter Free invisible Web tracker, Hit counter and Web stats is down

Found this a few minutes ago... they don't show any pictures of the recent damage of course, but a pretty good set of pics of their Houston center ....

»blog.theplanet.com/2007/ ··· centric/

This morning my DSL went down, AT&T HSI for several hours and couldn't get online from 6:30am til after 2pm today, not sure if this has anything to do with it, but thought it would be useful to put in this thread.

I'm all the way in Miami FL and I experienced an outage of my AT&T DSL during those same hours.

we have four OC-3's at my work place here in Miami, and at&t told us last week that they were performing "major upgrades" to their network here in Miami. Found out this morning that they are migrating parts of the old bellsouth.net network onto the at&t network.

The local Fire Dept and cops and their "emergency response team" is probably looting the datacenter as we speak. Remember, if you go back into the building while its still "dangerous" you will be arreasted, and you can't stop the Fire Dept from water hosing all the server "just to be safe", even if the fire was many floors away.

The racks will also require buzz saws and the jaws of life to make sure there is no fire inside the cabinets since they have no clue how to pull out 19 inch rack systems.

So much for Halon when the Fire Dept will water it down anyways.

So what does happen when the Fire Dept demands to buzz saw and soak servers needlessly in a datacenter, or the police dept seals the building for a few days to do an investigation? Hope your insurance is paid up and you have instant offsite failover and continuous realtime DB mirroring offsite? Or this stuff doesn't happen/there are laws to protect datacenters from the demo and reclamation crew?

lujohnson's link for photos of one of the Houston centers tells the story. I was trying to understand why the explosion with three walls demolished. Battery acid fumes ignited by fire from a shorted conduit wire (some electrician's ass is in a serious sling).