Rackspace: When Bad Drivers Attack!Truck accident knocks out power transformer ( old news - 01:24PM Tuesday Nov 13 2007) tags: business · bandwidth · trouble · networkingTipped by h4x0r3d  San Antonio based Rackspace dealt with a power outage Sunday, but unlike the July outage at 365 Main, their back up contingency plans actually worked. At least they worked until a truck driver plowed his vehicle into a power transformer the next day, causing it to explode. The outage disrupted power to the company's Dallas data center and their chilling units, forcing them to take a number of customer servers offline. The outages impacted hosting providers such as Laughing Squid and startups like 37 Signals. It also impacted some ISPs such as SuddenLink -- who hosted some services at the impacted Dallas data center. After everyone was back up and running, Rackspace's CEO issued a mea culpa in an e-mail to customers, and also posted this apology to their website: We cannot promise that hardware won't break, that software won't fail or that we will always be perfect. What we can promise is that if something goes wrong we will rise to the occasion, take action, resolve the issue and accept responsibility. If you are a Rackspace customer and don't think we've lived up to this promise at anytime during the outage, please let your Account Manager know.
Related:- Comcast Sued For Traffic Shaping
- Qualcomm's UMB: DOA
- Remember How The Net Neutrality Fight Began
- Sprint, Cogent in Peering Feud
- Cogent, Sprint Kiss, Make Up
- New Docs Show FCC Glossed Over BPL Flaws
- Did AT&T's Bad Week Kill Their iPhone Exclusivity Extension?
- AT&T: No, We Didn't Misconfigure Our 3G Network
|
 |  | 
Rogue Wolf
Ate The Last Of The Pumpkin Pie
join:2003-08-12
Troy, NY
 ·RoadRunner Cable
| Re: Impacted ISPs said by exocet_cm :I still hate it when folks feel that they have to apologize for something that was beyond their control. I know it was nice, and I would have done it as well, but this was out of their hands and they still had to apologize. Yeah, but as someone who works in the retail sector, I know that some customers expect this sort of response, and even those who don't usually feel a bit appreciative of some show of contrition.
Then again, there's also those who think that their $5-a-month hosting plan should entitle them to a backup server in low Earth orbit connected through 100MBps laser-broadband rigs. 
--
I have learned to ignore such naysayers, when... quelling... them... hm?... was out of the question. | |
| 
phoneboy3
@shawcable.net | What about backup generators Don't they have backup generators like any REAL data center should? If not then they should definitely be apologizing. | |
| | 
Jahntassa
What, I can have feathers
Premium
join:2006-04-14
Conway, SC
| Re: What about backup generators Um..wow. Obviously they had backup generators, but if the generator is feeding the transformer, which then feeds power to the air handlers, then there's no real backup for that.
Backup generators can't help much if someone knocks out the device between the generator and what you're trying to power. | |
| | | jtel
join:2005-06-28
Bristol, RI
| Re: What about backup generators said by Jahntassa :Um..wow. Obviously they had backup generators, but if the generator is feeding the transformer, which then feeds power to the air handlers, then there's no real backup for that. Backup generators can't help much if someone knocks out the device between the generator and what you're trying to power. Thats not what happened though. They successfully transitioned to back up power and then switched back to utility power w/o knowing that the utility was planning to kill the power again for the safety of rescue personnel. They shut down some servers as a precautionary measure because of repeated cycling of the chillers. My take from the press release is that the chillers aren't fully on the back up system or they wouldn't be cycling repeatedly.
Whats the application for private back up generators that feed utility company transformers? Their generator would most likely connect to a switching unit that was connected to the battery room and the power co. They lose power and it switches to the generator running on battery till the generators come online. | |
| | | | 
marigolds
Gainfully employed, finally
Premium,MVM
join:2002-05-13
Saint Louis, MO
| Re: What about backup generators said by jtel :They shut down some servers as a precautionary measure because of repeated cycling of the chillers. My take from the press release is that the chillers aren't fully on the back up system or they wouldn't be cycling repeatedly. I think it went like this:
Chillers are on main power.
Main power goes off. Generators kick in immediately. Chillers reset.
Power switches from Generators to secondary. Chillers reset.
Secondary power goes off. Generators kick in again. Chillers reset.
Secondary goes back on. Chillers reset.
Main power goes back on. Chillers reset.
After four-five cycles of the chillers reseting like that, they had lost enough cooling capacity for enough time to be concerned.
--
ISCABBS - the oldest and largest BBS on the Internet
telnet://bbs.iscabbs.com
Professional Geographer
Geographic Information Science researcher | |
| | | | | | | Kearnstd
Elf Wizard
Premium
join:2002-01-22
Mullica Hill, NJ
| and that they might not have enough generator capacity to keep the chillers at full power, i mean even most supermarkets cant run the reefers on backup power even if the lights and registers are on and they stand to loose a lot more then if you shut off some servers for a few hours.
--
[65 Arcanist]Filan(High Elf) Zone: Broadband Reports | |
| | |
Moon1234
@charter.com
| Re: What about backup generators said by Kearnstd :and that they might not have enough generator capacity to keep the chillers at full power, i mean even most supermarkets cant run the reefers on backup power even if the lights and registers are on and they stand to loose a lot more then if you shut off some servers for a few hours. You would probably be surprised to know that many grocery stores have NO backup generators. I work as a night manager on the weekends in a million dollar a week grocery store in Wisconin. We have NO backup generators. When we were purchased by Roundys I asked if we would be getting them. They said no. They would never recoup the cost of the genertors. Everything in the reefers is good for around 12 hours provided the doors on the sales floor are no opened. The freezer is kept at -25 and is good for 12-24 hours provided that no doors are open. The only thing that we really have a problem with is ice cream which melts aroun 20 degrees farenheit. Some melts lower or higher depending on fat content.
Roundys has insurance on all perishable items in the store. Should we have an outage that lasts more than 4-5 hours they would be reimbursed for any loss. I have worked there for 15 years and we have had maybe 5-10 thousand dollars worth of shink due to loss of refrigeration. We have 27 compressors to run. A generator sized to keep all of them operating would be very expensive.
Almost all grocery stores work this way. We used to have a contract with a dry ice company for the main freezer in case the power went out, but Roundys eliminated that as well. They do keep about 4 pallets of bagged ice in the freezer which is intended to be sold to customers, but does act as a nice backup insurance plan as well. | |
| | | | wierdo
join:2001-02-16
Tulsa, OK
 ·Future Nine Corpor..
·Teliax VOIP
| Re: What about backup generators A few months ago I was at a nearby supermarket when the power crapped out. I was quite happy that they had a generator, since it allowed them to run half the lights and the registers.
Had it not been for the generator, I wouldn't have left with any food that evening. Not that I could do much with it when I got home, since my power was also out, but still!
Thank god for megacorp grocery stores?
Once your company gets to a certain size, the cost of insurance is greater than either absorbing the loss on your own or taking steps (like generators) to mitigate the chance of loss. | |
|
anonymous119
@stancoe.org
|  Did Charter Service go out as well?
| |
| | mecha50
join:2004-06-11
Daly City, CA
| Re: Did Charter Service go out as well? said by anonymous119 :
I have Charter Cable Internet Service... This morning i tried using the intnernet but the Internet Explorer wouldn't load my home page... I checked all connections and reset both my router and Cable Modem... nothing worked. Does this have something to do with my internet being down?
Charter is suffering internet outages in the following locations: State of Minnesota; State of Missouri; State of California; Bay City, Michigan and Bellville, Illinois. | |
| 
Oleg
Bellsouth Fastaccess
Premium
join:2003-12-08
Birmingham, AL | Say what LOL this cracks me up "a truck driver plowed his vehicle into a power transformer"  | |
| | | 
Jerm
join:2000-04-10
Richland, WA | LOL I'm lazy - I read this as "When Bad Drivers Attack" and my first reaction was "yeah it sucks when that new ethernet driver gets loaded and locks up your system!". | |
| | Radioman991
join:2001-09-24
Dayton, OH | Re: LOL Thought the same exact thing here!  | |
| 
boogi man
join:2001-11-13
Apo, AE
clubs:  | drivers yeah i was so thinking the same thing. it's cool that they are upfront but then again it is easy to be upfront when it's not really your fault | |
| NightHawke
join:2002-02-28
Rockport, TX
| Welcome to Texas Home of the crappiest infrastructure on the planet.
You have the state to thank for most of the garbage that they either put up on poles, or buried.
The PUC let things slide for so long, it would nearly require a revolution to put things back to right and tight.
Oh, and lay a little blame on TXDOT and the local city engineer for letting that transformer remain unprotected like that. Bollards and/or concrete fenders are the rule of thumb if they are close to the traffic right of way. | |
| |
qwerty1975
@sbc.com
| Re: Welcome to Texas Actually, you'd be surprised to know that the transformer isn't in the traffic right of way - it's in a parking lot. It's also not near the highway, but in a business park.
The driver even jumped a row of bushes and a curb to hit the thing.
On a side note, I drove by Rackspace's DFW datacenter this morning - they're installing concrete pillars around all three of the transformers at that location. | |
| | | |
|
|
·
·
