donaldkPremium
join:2000-10-19
Thunder Bay, ON Reviews:
 ·Eastlink Cable
| reply to Bobcat
Re: Eggs in one basket WARNING: LONG RANT
SUMMARY: Their back up power plant setup, SOPs, and operations are 100% unsatisfactory and I would definitely recommend moving business elsewhere. This is from my own work experience for many years with diesel electric power plants. Anyone who directly deals with or invests with 265Main should read the very last paragraph.
After reading their log on what happened with their Gensets. N+2 setup, 2.1MW per genset, what I do not get is why their generators do not have the option to at least parallel with each other besides the back ups. I work with marine diesel electric power plants all the time, which are even harder to work with than this set up, will not get into it but I'll say a 715kw load transfer is laughable. Ok enough of my rant, now to the point.
They have 16.8MW of 100% rated capacity (8 gensets). From the looks of things their generators are not normally loaded higher than 1MW per genset, could understand to have capacity to expand and allowance for overloading. Five main gensets remained running after 3 main gensets and BOTH backup gensets failed, the power required that was put onto backup#2 was 2.5kW before it tripped off.
Now they got five remaining gensets on-line, assuming 1MW they were supplying to each of their co-lo rooms, there was still 5.5MW of safe room left on these gensets to take the 2.4MW of load that Genset 1,3,4 failed and both backups failed to take on, IF they had the option of paralleling to share the load, even if for emergency use, it would have saved the data-center from any part of it going dark. if my assumptions are correct, they had double the enough capacity amongst all their good generators to keep their data-center up. Now being they claim N+2, they ought to have a load sharing and synchronization control that can distribute amongst their power plant as needed, if it was limited to just the two backup generators then who ever came up with that set up should be fired / sued for such a bad unsafe design, and they should have their main switchboard required with the logic and power runs necessary to do this. If they did have the capability to load share and synchronize them all together as needed, then the on watch engineers should be fired immediately on the spot, as their log they posted does not indicate any attempt to load share or any problems of load sharing (except for Genset 1,3,4 passing load to the two backups and then both backups totalling failing).
That is my 0.02 on the emergency operation of the plant. Now a stab at their maintenance and SOPs. With 4 Generators failing their start up sequences right when they are needed, there is a sense that preventative maintenance and routine checks/tests were not done, and if so then they should post their logs in PDF (you laugh, I know they got them if their are what they say they are) right on-line to be scrutinized, never mind the upcoming law suit that probably will happen. There is no excuse for not doing full load testing, they could have load banks set up to work the diesels to 110% load, and all none computing loads of the building wired in to function with the load banks too. (100% normal, and 110% for 1 hour per 24 hours). At the minimum, weekly rotating of machinery with 50% load, testing all features of the genset and switchboard (I get the hint that this happens but not much happens, its started up and shutdown right away? no testing of the switchboards), transferring form the power grid to the genset on a live bus and dead bus (load banks). I could go on.
I might be a little steep with my comments, but I am talking about a company who is expected to have near 100% up-time (what 99.999%?, akin to the military and health-care), and obviously they certainly cannot meet that. I wonder if all their data-centers are like this, my biggest qualm is the apparent lack of adequate load sharing and syncing for additional redundancy. Working as an engineer on a diesel electric warship, if something like this happened cause of stupidity (loosing have the power plant instantly, risking total loss of propulsion), there would for sure be an investigation, and if it was due to human error, heads literally roll.
Anyone who has invested into this company should have some serious considerations to worry about, shake things up, and consider moving your assets elsewhere. If your investment in 365Main is large and you still have faith in them, obtain the engineering specifications of their data-center, arrange to have a private engineering firm to have a tour at it and give you their own recommendations. If 365Main wanted to keep you they would allow it, consider the fact, did they have even UPS's running to allow for genset hiccups too? I did not notice any fed/state/muni government agencies listed as clients with them quickly browsing through their site, but I can guarantee you if they do have them they will not for long. There is a comment about an ISP intentionally tripping off land power, and testing their own UPS banks and Genset power plant every week to ensure proper operations, done long enough to stress there gensets but short enough that their UPS's absorb any hiccup's, That is the BARE MINIMUM!!
Nuff said about these assholes. |
