51019512
join:2009-05-19
151 Front St | reply to Inssomniak
Re: TekSavvy Down (EDIT: Now back online)
said by Inssomniak  :At least it was early. What program is that? |
|
LazMan
join:2003-03-26
Angus, ON
 ·TekSavvy Solutions..
| reply to Guspaz
said by Guspaz :Consumer/business UPS are technically "sealed" lead-acid batteries that don't need to be run in any given orientation (as in they run fine upright or on their side). I'm not sure if they're gel-based or not. "Gel-Cell" is a misnomer... VRLA's (Valve regulated lead acid) batteries do use a liquid electrolyte (aka acid) but it's absorbed in fibreglass mats between the lead plates. A large VRLA will only have about 2 cups of liquid within it, unlike "flooded" cells that may have gallons.
Headspinning - not only do I remember 220 Simcoe - I was there...  Bell's union wouldn't let their people in - Nortel's had no such issue - so in we went, to try and bring equipment back online. |
|
DKS
Damn Kidney Stones
Premium,ExMod 2002
join:2001-03-22
Owen Sound, ON
clubs:
·Bell Sympatico
| reply to TSI Marc
Peer 1's official response:
From: "Ted Smith"
Date: July 7, 2009 6:04:33 PM GMT-04:00
To:
Cc: "'Dorothy Williams'"
Subject: Toronto Data Center -- 151 Front Street Post Mortem
Dear Valued Customer,
As you are aware, over the weekend the 8th floor of PEER 1’s Toronto data center located at 151 Front Street experienced a fire with a UPS cabinet that set off a chain of events that resulted in power outages in multiple suites and network outages in others. I have detailed the events below.
PEER 1 is extremely thankful that no one was injured during the fire. The fire itself was contained within a single UPS cabinet and the Toronto fire department was able to extinguish it with the use of dry chemicals. No water was used, which significantly reduced the damage and aided in our ability to restore services to all data centers as quickly as possible.
I was personally involved in PEER 1’s response to the situation, and would like to take a moment to apologize to you for the information contained within our first few email alerts regarding the outage. When we were first organizing the team, we received an early report from our on-site staff that all power was out at 151 Front Street. That was not accurate. The fire caused a power outage in PEER 1’s suite 802. The local fire department cut power to PEER 1’s suite 709 as a preventative and safety measure to contain and extinguish the fire. The network gear providing connectivity for Suite 816 are kept in a cage in Suite 802. As a result, Suite 816 suffered a network outage. Power in the rest of the building was unaffected throughout the incident. There is a fine balance to providing information during a crisis, and we do our best to make sure updates to our customers are, quick, transparent, and accurate. In no way would we try to misrepresent the situation.
PEER 1 manages all of the equipment it provides to customers with safety and performance in mind. The UPS mentioned above has been in service for approximately four years. Its last service inspection was on July 2nd and it was only at 65% load. We do not yet have the root cause, but there was nothing in the unit’s maintenance record to indicate there was potential for a problem. Our service contractor and manufacturer are both involved in the investigation.
As part of the process to restore power quickly, we had to bypass the affected UPS. This means that suite 802 is running without UPS power until we can schedule delivery and replacement. We will expedite the process and keep customers in this suite up-to-date with the latest information on the timeline.
Outage Details:
(Timelines represent impact to “most” clients. Some specific network devices failed due to power loss and restore took longer.)
· Suite 802 – Power and network outage. Start 23:32PT End 03:10PT
· Suite 816 – Network outage. Start 23:32PT End 03:10PT
· Suite 709 – Power and network outage. Start 23:32PT End 00:50PT
· Suite 402 – Limited network outage (some latency/packet loss issues). Start 23:32PT End 00:50PT
· Firewall Clients. Network outage. Start 23:32PT End 03:10PT
Ted Smith
SVP, Operations
PEER 1
1000-555 West Hastings Street
Vancouver, BC, Canada V6B 4N5
tsmith@peer1.com
T: (678)365-2896
F: (678)365-2956
www.peer1.com
--
Need-based health care not greed-based health care. |
|
Inssomniak
Premium
join:2005-04-06
Cayuga, ON
| reply to TSI Marc
At least it was early. |
|
An_Onymous
@teksavvy.com
| reply to TSI Marc
The original article reference from slashdot's headline:
»www.datacenterknowledge.com/arch···-uptime/
- power outage at its Dallas data center
- Early Thursday Equinix Inc. (EQIX) data centers in Sydney, Australia and Paris each experienced power failures.
- Google App Engine, the company’s cloud computing platform, had lengthy performance problems on Thursday, experiencing high latency and data loss.
- A fire at Fisher Plaza in Seattle late Thursday night left many of the building’s data centers without power.
- Early Sunday, July 5, a fire at 151 Front Street, the major carrier hotel in Toronto, knocked out power on several floors of the facility used by Peer 1 networks. |
|
HeadSpinning
join:2005-05-29
Windsor, ON
| reply to Davesnothere
Re: TekSavvy Down
said by Davesnothere :Interesting. I have encountered some smaller APC's where the battery had swelled to the point that it had to be Pried out. Pregnant batteries in APC UPSs is a common problem.
I had always supsected they're running the float voltage too high. A few years ago, I was at a lunch seminar and was seated at a table with one of the founders of APC, and he basically confirmed (without actually saying it...) that some units have a problem with float voltage regulation, that might cause battery failure. |
|
Davesnothere
join:2009-06-15 | reply to Thane_Bitter
Interesting.
I have encountered some smaller APC's where the battery had swelled to the point that it had to be Pried out. |
|
Thane_Bitter
join:2005-01-20
London, ON
·Bell Sympatico
| reply to EG
It could happen, here is an example of a battery issue in a APC smartUPS UPS (I think it’s the 2200 model, the pic looks identical to one I have...minus the damage ). The right most battery has broken internally (short, overcharged, or just plain broke) and had been generating gas for some time. SLA batteries do have a vent valve that is designed to release the excess gas; however nothing is 100% reliable. It failed as well and the preasure built up over time causing the cell to swell up. Assuming the rest of the UPS is unaffected you have to disconnect everything and remove the top cover do remove swollen batteries (a rackmount UPS with batteries in it is a very heavy thing to move).
»farm3.static.flickr.com/2291/226···f4_b.jpg |
|
Guspaz
Guspaz
Premium,MVM
join:2001-11-05
Montreal, QC | reply to TSI Marc
Re: TekSavvy Down (EDIT: Now back online)
Consumer/business UPS are technically "sealed" lead-acid batteries that don't need to be run in any given orientation (as in they run fine upright or on their side). I'm not sure if they're gel-based or not. |
|
jfmezei
Premium
join:2007-01-03
Beaconsfield, QC
·ELECTRONICBOX
1 edit | reply to Davesnothere
Re: Battery Types
Many moons/centuries ago, I visited the Bell CO on Belmont street in Montreal. They still have crossbar equipment feeding McGill university back then. (this was one step ahead from the first generation rotary switches, and the first to support touch tone). And they still had one rack of rotary (step by step) switches, but they had just been made offline and not yet removed).
They had a large portion of a floor dedicated to batteries. They had semi transparent plastic enclosure, and you could see bubbles flowing up between the metal plates inside. Pretty neat sight.
When I worked for a bank, I did see their turbine generators on the top floor, but I don't recall their batteries. (that would have been about 10 years after visiting the Belmont one). When I visited the Whitehorse CO in 1989 they had more modern equipment there. (and got to meet some of the operators, some of whome I spoke t later on when making calling card calls  |
|
Davesnothere
join:2009-06-15
| reply to Guspaz
It's my understanding that they are a variant of Lead-Acid called a 'Gel Cell' battery and are supposed to be completely 'maintenance free' (cannot top up the electrolyte as it is not a free flowing liquid), unlike some lead-acid car batteries which were also known as 'maintenance free', though Could be topped up if you pried off some covers believing you needed to.
Those car batteries are now are called 'Low Maintenance', due to changes in / enforcement of consumer laws, I believe. |
|
Guspaz
Guspaz
Premium,MVM
join:2001-11-05
Montreal, QC | reply to TheMG
Re: TekSavvy Down
I believe that pretty much all UPS batteries are lead acid. In fact, my consumer grade APC unit is lead acid; all of APC's stuff is. |
|
TheMG
join:2007-09-04
Edmonton, AB
·TELUS
| reply to EG
Meanwhile in Alberta not even a hiccup on my connection.
said by EG :Why assume a lack of maintenance right away ?? Maybe there was a malfunction of the re-charger and it overcharged/overheated the battery/s ?? Is that just rumors or an actual fact from an official source that the batteries were at fault?
Anyways, I do believe the batteries for these large UPS units are lead-acid batteries (just like in your car). Those do not burst into flames when overcharged, they start spitting acid and hydrogen gas. Though with the amount of ventilation, it's unlikely hydrogen could have accumulated enough to burn/explode.
More likely there was an internal fault in the actual UPS. At the power levels we're dealing with here (several hundred kVAs) we're talking serious arcs, arc flash, and vaporized metal.
Anyways, I've seen the insides of such a UPS in person. Scary stuff, even just the sheer size of the bus bars!
Another fact many people don't know: the batteries are not normally in the same cabinet as the UPS. There's a separate cabinet for the batteries alone, which can often be in a totally separate room from the UPS itself. |
|
DKS
Damn Kidney Stones
Premium,ExMod 2002
join:2001-03-22
Owen Sound, ON
clubs:
·Bell Sympatico
| reply to EG
said by EG :said by HKPolice :Battery blew?? Why wasn't it changed regularly  Why assume a lack of maintenance right away ?? Maybe there was a malfunction of the re-charger and it overcharged/overheated the battery/s ?? There is anecdotal information that there has been a smell of burning electricals at peer 1 for a long time.
--
Need-based health care not greed-based health care. |
|
EG
The wings of love
Premium
join:2006-11-18
Union, NJ
| reply to HKPolice
said by HKPolice :Battery blew?? Why wasn't it changed regularly Why assume a lack of maintenance right away ??
Maybe there was a malfunction of the re-charger and it overcharged/overheated the battery/s ?? |
|
Crowbar1
join:2009-06-23 | reply to TSI Marc
Re: TekSavvy Down (EDIT: Now back online)
Teksavvy = #1 |
|
DKS
Damn Kidney Stones
Premium,ExMod 2002
join:2001-03-22
Owen Sound, ON
clubs:
·Bell Sympatico
| reply to TSI Marc
If you want to read all about 151 Front St. and how it came to be, see here. It's got an interesting history going back to the days of the telegraph.
--
Need-based health care not greed-based health care. |
|
fourboxers
Premium,Mod
join:2003-05-04
Scarborough, ON
clubs: |  reply to TSI Marc
(topic move) troll lock
Moderator Action
The post that was here (and all 30 followups to it), has been moved to a new topic .. »troll lock |
|
AkFubar
Resistance is Futile
join:2005-02-28
Toronto Can. | reply to bdevrye
Re: TekSavvy Down (EDIT: Now back online)
Yeah
I've been checking throughout the day. Haven't had any issues either.
--
"No matter where you go, there you are." - Buckaroo Banzai |
|
