 TomS_Git-r-donePremium,MVM
join:2002-07-19
Ireland
kudos:4 | Warrnambool (Victoria, Australia) exchange fire (contd.) (Original thread too old, so heres a new one)
Report has been published:
»www.telstra.com.au/uberprod/grou···5046.pdf
Not many pretty pictured, but they have a list of 22 recommendations for improvements and action points to reduce the likelihood/impact of any similar incidents in the future.
They didn't positively identify the cause, but suspect an electrical fault within or in the roof space above a work/store room.
They suspect that some fire doors which had been choked open, and missing fire pillows in cable tray wall penetrations probably helped the fire to spread, while the air conditioning helped spread soot and other contaminants throughout the building causing damage to equipment.
Due to the fact most kit is DC powered, and DC power is usually battery backed, they are looking at ways to enable fire crews to shut down DC gear as well because while they managed to kill off all AC supplies including backup generators, the DC power systems allowed equipment to continue running and circulate air.
The lack of cable tray diversity in the building was also highlighted as a problem, since almost the entire inside cable plant was damaged.
And they've identified a need to install more smoke detectors in places like roof cavities.
Basically it has highlighted the need for improvements all round in numerous procedures, and will kick off a major program to make updates to all facilities around the country as required. |
|
|
|
| ....Question is "will they actually DO something with the list, or put it in the circular file."
quote:
indicated the following impact on service types and quantities:-
61,856 PSTN telephony services
14,409 ADSL services, consumer Broadband
56 ISDN services
40 3G-Mobile Base Stations
138 IPMAN services
13 Telstra Internet Direct services
Multi Service Edge Router failure (Impact summary 268 x Business DSL, 16,094 x ADSL, 4 x Internal use Frame Relay/ATM, 47 x Frame Relay/ATM, 40 x 3G Mobile Base Stations)
SMR services, Loss of all Radio Coverage Warrnambool & Port Campbell
Loss of communications to Digital Video Network equipment at Warrnambool & Terang node for Sky Racing
If THAT isn't an IT FML, I don't know what is...
...saved a copy for a night I have insomnia... Thanks for sharing TomS_ 
Regards |
|
| reply to TomS_
Holy bastard... FML is right. |
|
cramer
join:2007-04-10
Raleigh, NC
kudos:7 | reply to TomS_
Didn't I read something about there being no fire suppression in there? Seems like that would be the first item on the fix-this-now list. |
|
TomS_Git-r-donePremium,MVM
join:2002-07-19
Ireland
kudos:4 | Yeah. Surprisingly I don't think there was much, if any mention of fire suppression, only detection/alerting! |
|
| reply to TomS_
said by cramer:Didn't I read something about there being no fire suppression in there? Seems like that would be the first item on the fix-this-now list.
said by TomS_:Yeah. Surprisingly I don't think there was much, if any mention of fire suppression, only detection/alerting!
FML... x2. There are not enough facepalms in the world to express my feelings right now...
Regards |
|
sk1939Premium
join:2010-10-23
Washington, DC
kudos:9 Reviews:
 ·T-Mobile US
 ·Verizon FiOS
1 edit | reply to TomS_
Apparently it burned hot enough to actually melt the fibers in the fiber optic cables.
Also noted:
"The full restoration of all customer services was achieved on Wednesday 19 December 2012, following the engagement of 110 highly skilled technicians working rotating 12 hour shifts supported by experts in design, data configuration, logistics and project management."
I hope they got paid for overtime...
"Without services, the residential customers, small businesses and large commercial enterprises had to find alternative means of communicating until the services were restored. Unlike many short service disruptions the period of the outage was up to 20 days due to the scale of the damage and complexity of the restoration activity. As the Telstra Network is, in the most part highly reliable, customers found this lengthy outage inconvenient, disruptive and business and private life impacting."
In the US I can foresee a LOT of lawsuits if businesses were without connectivity for nearly a month. |
|
cramer
join:2007-04-10
Raleigh, NC
kudos:7 | said by sk1939:In the US I can foresee a LOT of lawsuits if businesses were without connectivity for nearly a month. Have you read your contract / SLA? They are not on the hook for any of your lost business. At best, you get to not pay your bill for the month. |
|
sk1939Premium
join:2010-10-23
Washington, DC
kudos:9 Reviews:
·T-Mobile US
·Verizon FiOS
3 edits | said by cramer:said by sk1939:In the US I can foresee a LOT of lawsuits if businesses were without connectivity for nearly a month. Have you read your contract / SLA? They are not on the hook for any of your lost business. At best, you get to not pay your bill for the month. For businesses that have a leased line connection with 99.999% uptime, it's a big deal. At the very least it's damaging in the form of large businesses changing providers.
"The average downtime costs vary considerably across industries, from approximately $90,000 per hour in the media sector to about $6.48 million per hour for large online brokerages."
For Standard DSL lines:
"Neither you nor we will be responsible to the other for any delay, failure in performance, loss or damage due to fire, explosion, power blackout, earthquake, volcanic action, flood, the weather elements, strike, embargo, labor disputes, civil or military authority, war, terrorist acts, acts of God, acts or omissions of carriers or suppliers, acts of regulatory or governmental agencies, or other causes beyond our reasonable control, except that you must pay for any services used."
Metro-E
"MHO’s Metro Ethernet Service is designed for 99.99% availability. If Customer experiences a network outage, MHO will restore the network connection within two
(2) hours. Customer will receive, at the Customer’s request, a Service Credit as follows:
One (1) day Service Credit of affected circuit for each additional hour after the first two (2) hours of down time
One (1) month Service Credit if Point to Point Service is down 24 hours or more after first two (2) hours of down time
Customer may obtain no more than one (1) month Service Credit for any given month.
Down time commences once customer notifies MHO and MHO issues a service ticket to the customer. The outage will conclude once MHO restores the affected
Metro Ethernet circuit."
Edit: I had forgotten about the relatively-new arbitration clause. |
|
cramer
join:2007-04-10
Raleigh, NC
kudos:7 | Which is exactly what I said:
They are not on the hook for any of your lost business.
Not paying your bill for that month may be a very small consolation vs. your lost revenues, but that's all you get. ISPs aren't stupid, and their contracts are written by good lawyers. In the US, as far as I know, every datacenter and telco pop has significant fire suppression -- I've never seen one that didn't. In the offices I've worked, it's been the second or third question from the building engineer(s) when planning the space. |
|
sk1939Premium
join:2010-10-23
Washington, DC
kudos:9 Reviews:
·T-Mobile US
·Verizon FiOS
| Agreed, I've never heard of a datacenter that didn't have some kind of fire-suppression system. I guess they did their CBA wrong during construction  . That's still a lot of angry customers they're going to have to deal with, and even without lawsuits they're going to have to launch a giant PR offensive to help quell the anger towards them. |
|
Reviews:
 ·WestNet Broadband
| Not sure on the location however there has been some fire sweeps though a lot of that area in recent years. I doubt any fire suppression helps in that weather, but hey, a failed RCD with a power short...guess they can be alerted to without fire alarms and suppression mechanics in place? |
|
| reply to sk1939
At best you will get a full month's credit. I've never seen an SLA that would allow for anything more. IF your services at that mission critical then it's up to YOU and no one else to ensure you have redundant connections that are diverse.
--
I do not, have not, and will not work for AT&T/Comcast/Verizon/Charter or similar sized company. |
|
| said by battleop:IF your services at that mission critical then it's up to YOU and no one else to ensure you have redundant connections that are diverse.
....then a provider pulls a fastie and uses a single LEC or last mile that borks at a critical time... or mother nature tosses
a fast ball. I've STILL got THAT one hanging over my head with last year's hurricane Sandy, and a couple clients out
connectivity still... fun times indeed for those of us paid to keep the bits moving.
Regards |
|
