1 recommendation |
ZDNet article on this problemInternet hiccups today? You're not alone. Here's whySummary: It's not just you. Many Internet providers have been having trouble as they run into long expected (but not adequately prepared for) routing table problems.By Steven J. Vaughan-Nichols, ZDNet - August 12, 2014 » www.zdnet.com/internet-h ··· 0032566/ |
|
stvnbrs Premium Member join:2009-03-17 Cary, NC
1 recommendation |
stvnbrs
Premium Member
2014-Aug-13 2:10 pm
If only they had enough money to upgrade their aging platforms... sound like it is time for rate hikes |
|
|
y2k14
Anon
2014-Aug-13 2:15 pm
apocalypseIt has begun....... |
|
|
Y2K14Apocalypse. !!!!!! It has begun |
|
|
Actually, it's the i-Apocalypse. |
|
1 recommendation |
What a huge pain in the ass....I either adjusted the TCAM or dropped back from full tables to default+connected on all the gear I'm responsible for when this issue first showed up on NANOG. Even with all of our stuff being prepared for this I'm still hearing tons of people bitch us out because other people's shit is broken far upstream from us. |
|
|
to fiosultimate
Re: Y2K14Well that explains why my Comcast supplied IP address failed connection to dslreports earlier this week. I had to go thru a proxy to connect. DNS resolved dslreports correctly but the return path just wouldn'the work. |
|
2 recommendations |
Proactive much?So they saw this issue back in May? Why didn't they see this issue back in 2000? You can't make this stuff up. |
|
zpm join:2009-03-23 Columbus, GA
1 recommendation |
zpm
Member
2014-Aug-13 2:41 pm
TypicalGood Job Verizon....... |
|
PA23 join:2001-12-12 East Hanover, NJ
1 recommendation |
PA23
Member
2014-Aug-13 2:46 pm
Lack of maintanceThere is a work around available but the work around requires a restart. Unfortunately many places don't allow for preemptive maintenance just reactive. I remember work at place that was running code that was years out of date with known bugs that had been fixed. These bugs would occasionally cause havoc in our network, but could I perform a software upgrade??? NO! the business could never agree upon a time to allow the upgrade. |
|
|
tubbynetreminds me of the danse russe MVM join:2008-01-16 Gilbert, AZ |
said by PA23:There is a work around available but the work around requires a restart. Unfortunately many places don't allow for preemptive maintenance just reactive. the larger issue at hand is this *only* affects the providers/carriers/customers that are taking a full dfz (default-free zone) on their gear. generally -- as a carrier -- you have multiple edges/peering points where you can engineer traffic during slow periods in your network. using bgp it is entirely possible to divert traffic away from the edge that you're restarting using policy. as a hosting provider/customer -- the *only* reason you would take a dfz is if you're multihoming your internet edge -- and even then its only if you have multiple carriers at the edge (i.e. if you dual-home to the same provider using diverse paths -- they'll send you local a/s plus a 0/0). this has been approaching for a while -- and i've seen an uptick in traffic on [c-nsp] about prefix allocation in tcam, etc for pfc3xl-based boxen. the sad fact is that the reason this happened was that people were either (a) not aware of the hardware limitations of the platform or (b) thought they'd be safe by running on the razor's edge. either way -- its a self-inflicted problem -- especially because if you overflow the tcam on a sup720 -- you can watch the additional prefixes be switched in software -- and if the traffic loads are high enough -- you can see the box puke all over itself and roll around in it. q. |
|
DaveDudeNo Fear join:1999-09-01 New Jersey
1 recommendation |
to y2k14
Re: apocalypsei am sure Nostradamus predicted it. |
|
|
Cheeze_It
Anon
2014-Aug-13 3:00 pm
The platform is running on a Sup720-3BXL that had this issue...certain of itCertain aging Cisco 6500/7600 platforms with the Sup720-3BXL platforms that have been in production since the late 1990s....that could just be upgraded to a Sup 2T and none of this would have happened.
Oh wait...that would mean for them to spend money on infrastructure.
Can't have that...now can we... |
|
|
to zpm
Re: Typicalsaid by zpm:Good Job Verizon....... Really, we were so close to the 512k limit that it would have been hit eventually - Verizon just tripped it earlier. Net effect would have been the same. Sooner or later, older Cisco routers would have barfed unless they'd been adjusted to handle the larger table. |
|
BlueC join:2009-11-26 Minneapolis, MN |
to battleop
Re: What a huge pain in the ass....Right. This isn't an issue for us, but I recall seeing others discussing this months back. It was an inevitable issue and it seems a number of networks dropped the ball on being prepared. |
|
1 edit
2 recommendations |
to fiosultimate
Re: Y2K14The "Echoes of Y2K" In today's Wall Street Journal: Echoes of Y2K: Engineers Buzz That Internet Is Outgrowing Its GearRouters That Send Data Online Could Become Overloaded as Number of Internet Routes Hits '512K'By Drew FitzGerald, The Wall Street Journal - August 13, 2014 » online.wsj.com/articles/ ··· 07937617EDIT: aka "512KDay": 512KDay: How the internet BROKE (Next time, big biz, listen to your network admin)We failed the internet's management challengeBy Trevor Pott, The Register - August 13, 2014 » www.theregister.co.uk/20 ··· to_play/ |
|
tubbynetreminds me of the danse russe MVM join:2008-01-16 Gilbert, AZ |
to Cheeze_It
Re: The platform is running on a Sup720-3BXL that had this issue...certain of itsaid by Cheeze_It :Certain aging Cisco 6500/7600 platforms with the Sup720-3BXL platforms that have been in production since the late 1990s....that could just be upgraded to a Sup 2T and none of this would have happened. except of course -- to try and prove your point -- you neglected to include the vs-720 and the rsp720 -- each of which were introduced in mid-2007 and suffer from the same issue. in fact -- the s2t was released mid-2011 and has potential drawbacks that could prevent this card from being adopted in each carriers network. that being said -- if i have an edge/core box that is handling services correctly and the infrastructure needs (i.e. correct number of 1/10g interfaces) are met by the s720 -- why would i need to upgrade a box that still has life left in it? upgrading for upgrading's sake is a poor way to spend your capex dollars. q. |
|
NormanSI gave her time to steal my mind away MVM join:2001-02-14 San Jose, CA TP-Link TD-8616 Asus RT-AC66U B1 Netgear FR114P
1 recommendation |
to Cheeze_It
said by Cheeze_It :Oh wait...that would mean for them to spend money on infrastructure.
Can't have that...now can we... Of course not! It is much more important to spend the money to buy off regulation! |
|
Nanaki (banned)aka novaflare. pull punches? Na join:2002-01-24 Akron, OH |
Nanaki (banned)
Member
2014-Aug-13 3:25 pm
Was wondering what was going on.i used a web proxy to get here while it was screwed up |
|
|
to BlueC
Re: What a huge pain in the ass....I was expecting some rogue nations to get together and split their aggregated announcements into /24s to run the size of the routing table up and disrupt traffic. |
|
CosmicDebriStill looking for intelligent life join:2001-09-01 Lake City, FL |
I just waitedI just waited for the site to connect. I knew it would come back unless Justin didn't pay the hosting bill..... |
|
|
firephotoTruth and reality matters Premium Member join:2003-03-18 Brewster, WA
2 recommendations |
Profit vs infrastructureMaking money is more important than upgrading aging hardware that is so costly it's probably been paid for 100 times over it's life so far. And these are the literally the pillars of the internet, imagine how old those ISP edge routers that the shills like to cry about their costs all the time.
It's always the money. |
|
woody7 Premium Member join:2000-10-13 Torrance, CA |
to hello123454
Re: Proactive much? |
|
1 recommendation |
CDNI was about to ask Verizon to host the DSLReports CDN. |
|
KrKHeavy Artillery For The Little Guy Premium Member join:2000-01-17 Tulsa, OK |
KrK
Premium Member
2014-Aug-13 4:50 pm
Verizon crashed the InterwebzSeriously though, this was a problem that was known to be coming.
Still caught people with their pants down, however. |
|
Cobra11M join:2010-12-23 Mineral Wells, TX
1 recommendation |
to firephoto
Re: Profit vs infrastructurethey would prob save a fortune just replacing them with newer equipment in the long run.. especially on electricity! |
|
WhatNow Premium Member join:2009-05-06 Charlotte, NC
1 recommendation |
to KrK
Re: Verizon crashed the InterwebzNormal for most American business. The bigger they get the less they do because there are more people in charge to make a decision. |
|
|
ipv6ipv6 was supposed to be fully integrated into (tier-1 to isp side/edge) dns routers by now..
so much for the Skynet theory.. |
|
|
to telcodad
Re: Y2K14Dad jokes :P |
|
Rogue WolfAn Easy Draw of a Sad Few join:2003-08-12 Troy, NY
3 recommendations |
to hello123454
Re: Proactive much?These days, "proactive" means "solving the problem while it's smoking but before it explodes". |
|