Tuesday's Power Outage > Eggs in one basket

Eggs in one basket

Well, considering the costs of a second location, it isn't cheap at all. When you are not making any money, it is pretty hard to justify spending another huge chunk of change for redundant servers.

That being said, I think I'm going to bite the bullet and add another site... downtime's expensive!

reply to Bobcat
I would be very angry if I were one of their customers. You shell out all this money and when the power goes out, the backup generators fail. Unbelievably stupid.

reply to Bobcat
California has the ridiculous outages because all of the "we don't want another power plant" or "you're not building it here" garbage. Way too much regulation in California is the problem.
--
Saving the world keeps me busy. However, I find Earth very primitive from my home planet of Krypton.
-Supergirl

reply to bokamba

reply to supergirl
No kidding I the Terminator would push for more power plants in California with all the support he has it can be done.
--
Team Discovery-Join the fight

reply to supergirl

reply to supergirl
NIMBY! NIMBY! Hey, how come the power's out?

reply to vanDSLuser

reply to PolarBear03
Umm, as a proud Californian, I have to chime in here...

This was not a problem of "capacity" or "generation." The California Independent System Operator (CAISO) reported plenty of capacity for this mild summer day. There were also no issues of "transmission." These are the very high voltage power lines that distribute power over long ranges. This was a problem of "distribution." Our good friends at PG&E had some issues in the complex system that leads power to our homes and businesses. There are transformers, power factor correction capacitors, power poles (that get hit by cars), and the like: things can go wrong. It is a very complicated system. To sum it up as simply as "stupid NIMBYs" really shows some... naivete.

The bankrupt PG&E may (or may not) have been paying too much attention to their distribution plant.

This leads us to what happened at the turn of this century and so called "deregulation."

I can go on for days and days why this was not good for all California consumers of electicity: from the delaying of building power plants, the disconnect of the power generators (which used to be the local utilities, now third party "Enron" type companies) from the local utilities (and their users), and the apparent greed of everybody (generators, utilities, lawmakers, and maybe some rate payers).

PS: I am moving to Washington. Don't want me there? NIMBY!!!

PPS: Just kidding!

jhrvta

reply to vanDSLuser

reply to Bobcat
a UPS switches over so fast even a computer cant notice, while on batteries the generator should be spooling up. if the diesels cant handle the load and you tout your 99.9999% uptime BS then its time to get a turbine or a 300Kw fuel cell.
--
[65 Arcanist]Filan(High Elf) Zone: Broadband Reports

reply to vanDSLuser
I thought I read somewhere that a drunk employee caused the power outage by ripping all the cords out, and that they were trying to cover that up by saying it was the power companies fault.

besides, don't these places have battery backup and diesel generators?

reply to Bobcat
WARNING: LONG RANT
SUMMARY: Their back up power plant setup, SOPs, and operations are 100% unsatisfactory and I would definitely recommend moving business elsewhere. This is from my own work experience for many years with diesel electric power plants. Anyone who directly deals with or invests with 265Main should read the very last paragraph.

After reading their log on what happened with their Gensets. N+2 setup, 2.1MW per genset, what I do not get is why their generators do not have the option to at least parallel with each other besides the back ups. I work with marine diesel electric power plants all the time, which are even harder to work with than this set up, will not get into it but I'll say a 715kw load transfer is laughable. Ok enough of my rant, now to the point.

They have 16.8MW of 100% rated capacity (8 gensets). From the looks of things their generators are not normally loaded higher than 1MW per genset, could understand to have capacity to expand and allowance for overloading. Five main gensets remained running after 3 main gensets and BOTH backup gensets failed, the power required that was put onto backup#2 was 2.5kW before it tripped off.

Now they got five remaining gensets on-line, assuming 1MW they were supplying to each of their co-lo rooms, there was still 5.5MW of safe room left on these gensets to take the 2.4MW of load that Genset 1,3,4 failed and both backups failed to take on, IF they had the option of paralleling to share the load, even if for emergency use, it would have saved the data-center from any part of it going dark. if my assumptions are correct, they had double the enough capacity amongst all their good generators to keep their data-center up. Now being they claim N+2, they ought to have a load sharing and synchronization control that can distribute amongst their power plant as needed, if it was limited to just the two backup generators then who ever came up with that set up should be fired / sued for such a bad unsafe design, and they should have their main switchboard required with the logic and power runs necessary to do this. If they did have the capability to load share and synchronize them all together as needed, then the on watch engineers should be fired immediately on the spot, as their log they posted does not indicate any attempt to load share or any problems of load sharing (except for Genset 1,3,4 passing load to the two backups and then both backups totalling failing).

That is my 0.02 on the emergency operation of the plant. Now a stab at their maintenance and SOPs. With 4 Generators failing their start up sequences right when they are needed, there is a sense that preventative maintenance and routine checks/tests were not done, and if so then they should post their logs in PDF (you laugh, I know they got them if their are what they say they are) right on-line to be scrutinized, never mind the upcoming law suit that probably will happen. There is no excuse for not doing full load testing, they could have load banks set up to work the diesels to 110% load, and all none computing loads of the building wired in to function with the load banks too. (100% normal, and 110% for 1 hour per 24 hours). At the minimum, weekly rotating of machinery with 50% load, testing all features of the genset and switchboard (I get the hint that this happens but not much happens, its started up and shutdown right away? no testing of the switchboards), transferring form the power grid to the genset on a live bus and dead bus (load banks). I could go on.

I might be a little steep with my comments, but I am talking about a company who is expected to have near 100% up-time (what 99.999%?, akin to the military and health-care), and obviously they certainly cannot meet that. I wonder if all their data-centers are like this, my biggest qualm is the apparent lack of adequate load sharing and syncing for additional redundancy. Working as an engineer on a diesel electric warship, if something like this happened cause of stupidity (loosing have the power plant instantly, risking total loss of propulsion), there would for sure be an investigation, and if it was due to human error, heads literally roll.

Anyone who has invested into this company should have some serious considerations to worry about, shake things up, and consider moving your assets elsewhere. If your investment in 365Main is large and you still have faith in them, obtain the engineering specifications of their data-center, arrange to have a private engineering firm to have a tour at it and give you their own recommendations. If 365Main wanted to keep you they would allow it, consider the fact, did they have even UPS's running to allow for genset hiccups too? I did not notice any fed/state/muni government agencies listed as clients with them quickly browsing through their site, but I can guarantee you if they do have them they will not for long. There is a comment about an ISP intentionally tripping off land power, and testing their own UPS banks and Genset power plant every week to ensure proper operations, done long enough to stress there gensets but short enough that their UPS's absorb any hiccup's, That is the BARE MINIMUM!!

Nuff said about these assholes.

reply to Bobcat
There is significant evidence that an Electromagnetic Pulse was used to shut down this datacenter as there was NO DRUNK MAN and the problem started on the roof.

Now the question is WHO and WHY (I have my hunches but will let you discuss this amongst yourselves)

roflmao...... I have not had enough beer yet.. more more more.. weeeeee

reply to EMPWasUsed
Nutcases are to post here - »/dev/null
Thank you.

reply to supergirl