dslreports logo
 
    All Forums Hot Topics Gallery
spc

spacer

Search Topic:
share rss forum feed

matthopp

join:2009-05-14
reply to HELLFIRE

Re: Nice job L3 :)

Yeah I work on the "other side".. just don't post over here much.

Here's the RFO.. it was a "perfect storm" and Level3 should have not done the GCR with circuits down.

Yesterday at 16:10 GMT an excavating company drove right through our pole line which took down 4 spans of fiber total fiber count 300 pairs. Poles and fiber were restored at 12:06 GMT total outage 20 hours 10 minutes.

Level 3 had a planned Global Change Request issued for over 100 Juniper Core routers in Europe and on the Eastern seaboard including Chicago and much of the Midwest. Level 3 could not postpone the Global Change Request because a maintenance window was issued to 10,000+ customers. Unfortunately when the Juniper Core was upgraded the working path was taken down and your protect path was cut due to the larger fiber issue.

---- END QUOTE ----

ARG.....

matthopp

join:2009-05-14
Also....

During a scheduled maintenance (GCR 6287157) to upgrade a Washington DC router, a configuration issue caused the two routing engines on the router not to sync. Once the Technical Service Center notified the IP NOC of the customer impact, the NOC wiped out the configuration and loaded a current configuration to restore services.

HELLFIRE
Premium
join:2009-11-25
kudos:20
....I agree, perfect storm indeed. And a total FML / date with Crown Royal moment.

As for that routing engine sync issue... I've worked with enough gear to know that that's
ALOT harder to zero in on... and if it was part of the fiber issue, it makes tracking it down
that much harder as all your other issues can mask the problem.

Moral of the story... I hope when they fixed the thing, the guys who fixed it were given
a 2week allexpenses trip to the Bahamas, minimum. Cuz I can say I wouldn't've wanted
any part of that cleanup job ... [facepalm]

Regards