Level3 Suffers Brief But Massive Outage

So it could be a TWC problem?

And TW Telecom and Global Crossing as well

That's really odd because I have t-mobile and my mobile web was still working, rather slowly but it was still working. It is usually fast though.

Test Lab? Staggered Updates? Geeesh....
Would have been the smart thing...

Lets wait till Monday morning at 8am on the
production network.

View Bulletin PSN-2011-08-327
Title MX Series MPC crash in Ktree::createFourWayNode after BGP UPDATE
Products Affected This issue can affect any MX Series router with port concentrators based on the Trio chipset -- such as the MPC or embedded into the MX80 -- with active protocol-based route prefix additions/deletions occurring.
Platforms Affected
Security
JUNOS 11.x
MX-series
JUNOS 10.x
SIRT Security Advisory
SIRT Security Notice
Revision Number 1
Issue Date 2011-08-08

PSN Issue :
MPCs (Modular Port Concentrators) installed in an MX Series router may crash upon receipt of very specific and unlikely route prefix install/delete actions, such as a BGP routing update. The set of route prefix updates is non-deterministic and exceedingly unlikely to occur. Junos versions affected include 10.0, 10.1, 10.2, 10.3, 10.4 prior to 10.4R6, and 11.1 prior to 11.1R4. The trigger for the MPC crash was determined to be a valid BGP UPDATE received from a registered network service provider, although this one UPDATE was determined to not be solely responsible for the crashes. A complex sequence of preconditions is required to trigger this crash. Both IPv4 and IPv6 routing prefix updates can trigger this MPC crash.

There is no indication that this issue was triggered maliciously. Given the complexity of conditions required to trigger this issue, the probability of exploiting this defect is extremely low.

The assertions (crash) all occurred in the code used to store routing information, called Ktree, on the MPC. Due to the order and mix of adds and deletes to the tree, certain combinations of address adds and deletes can corrupt the data structures within the MPC, which in turn can cause this line card crash. The MPC recovers and returns to service quickly, and without operator intervention.

This issue only affects MX Series routers with port concentrators based on the Trio chipset, such as the MPC or embedded into the MX80. No other product or platform is vulnerable to this issue.

Solution:
The Ktree code has been updated and enhanced to ensure that combinations and permutations of routing updates will not corrupt the state of the line card. Extensive testing has been performed to validate an exceedingly large combination and permutation of route prefix additions and deletions.

All Junos OS software releases built on or after 2011-08-03 have fixed this specific issue. Releases containing the fix specifically include: 10.0S18, 10.4R6, 11.1R4, 11.2R1, and all subsequent releases (i.e. all releases built after 11.2R1).

This issue is being tracked as PR 610864. While this PR may not be viewable by customers, it can be used as a reference when discussing the issue with JTAC.

KB16765 - "In which releases are vulnerabilities fixed?" describes which release vulnerabilities are fixed as per our End of Engineering and End of Life support policies.

Workarounds
No known workaround exists for this issue.

Understood that L3 bought up GLBX recently, but GLBX is still a unique ASN and probably will be for some time yet. I doubt support teams and network maps have been integrated yet.

It would be kind of satisfying in a Scheudenfreud kind of way if the whole cascade of router crashes was triggered because level3 was trying to integrate the GLBX routing table into it's own....

Couldn't tell what was wrong with my internets. Rebooted modem and router like a fool! Almost changed DNS servers...but couldn't Google alternatives!

Other reports claim that Jupiter Networks routers choked on a BGP update.

Test Lab? Staggered Updates? Geeesh....
Would have been the smart thing...

Lets wait till Monday morning at 8am on the
production network.

Level 3 is NOT the entire internet and Verizon has redundancies. It's probably more telling that TWC does not.