dslreports logo
 
    All Forums Hot Topics Gallery
spc
view:
topics flat nest 
Comments on news posted 2014-08-13 14:53:43: If you've been having problems accessing DSLReports.com and a flood of other websites this week, you're not alone. The problem, as it turns out, was experienced by tier-one and last mile ISPs alike across much of North America. ..

page: 1 · 2 · next

telcodad
MVM
join:2011-09-16
Lincroft, NJ

1 recommendation

telcodad

MVM

ZDNet article on this problem

Internet hiccups today? You're not alone. Here's why
Summary: It's not just you. Many Internet providers have been having trouble as they run into long expected (but not adequately prepared for) routing table problems.
By Steven J. Vaughan-Nichols, ZDNet - August 12, 2014
»www.zdnet.com/internet-h ··· 0032566/

stvnbrs
Premium Member
join:2009-03-17
Cary, NC

1 recommendation

stvnbrs

Premium Member

If only they had enough money to upgrade their aging platforms... sound like it is time for rate hikes

y2k14
@24.243.21.x

y2k14

Anon

apocalypse

It has begun.......

fiosultimate
join:2014-06-09
San Antonio, TX

fiosultimate

Member

Y2K14

Apocalypse. !!!!!! It has begun

telcodad
MVM
join:2011-09-16
Lincroft, NJ

telcodad

MVM

said by fiosultimate:

Apocalypse. !!!!!!

Actually, it's the i-Apocalypse.

battleop
join:2005-09-28
00000

1 recommendation

battleop

Member

What a huge pain in the ass....

I either adjusted the TCAM or dropped back from full tables to default+connected on all the gear I'm responsible for when this issue first showed up on NANOG. Even with all of our stuff being prepared for this I'm still hearing tons of people bitch us out because other people's shit is broken far upstream from us.

OhHell
@66.249.83.x

OhHell to fiosultimate

Anon

to fiosultimate

Re: Y2K14

Well that explains why my Comcast supplied IP address failed connection to dslreports earlier this week. I had to go thru a proxy to connect. DNS resolved dslreports correctly but the return path just wouldn'the work.

hello123454
Premium Member
join:2002-02-02
21845

2 recommendations

hello123454

Premium Member

Proactive much?

So they saw this issue back in May? Why didn't they see this issue back in 2000? You can't make this stuff up.

zpm
join:2009-03-23
Columbus, GA

1 recommendation

zpm

Member

Typical

Good Job Verizon.......

PA23
join:2001-12-12
East Hanover, NJ

1 recommendation

PA23

Member

Lack of maintance

There is a work around available but the work around requires a restart. Unfortunately many places don't allow for preemptive maintenance just reactive. I remember work at place that was running code that was years out of date with known bugs that had been fixed. These bugs would occasionally cause havoc in our network, but could I perform a software upgrade??? NO! the business could never agree upon a time to allow the upgrade.

tubbynet
reminds me of the danse russe
MVM
join:2008-01-16
Gilbert, AZ

tubbynet

MVM

said by PA23:

There is a work around available but the work around requires a restart. Unfortunately many places don't allow for preemptive maintenance just reactive.

the larger issue at hand is this *only* affects the providers/carriers/customers that are taking a full dfz (default-free zone) on their gear. generally -- as a carrier -- you have multiple edges/peering points where you can engineer traffic during slow periods in your network. using bgp it is entirely possible to divert traffic away from the edge that you're restarting using policy.

as a hosting provider/customer -- the *only* reason you would take a dfz is if you're multihoming your internet edge -- and even then its only if you have multiple carriers at the edge (i.e. if you dual-home to the same provider using diverse paths -- they'll send you local a/s plus a 0/0).

this has been approaching for a while -- and i've seen an uptick in traffic on [c-nsp] about prefix allocation in tcam, etc for pfc3xl-based boxen. the sad fact is that the reason this happened was that people were either (a) not aware of the hardware limitations of the platform or (b) thought they'd be safe by running on the razor's edge. either way -- its a self-inflicted problem -- especially because if you overflow the tcam on a sup720 -- you can watch the additional prefixes be switched in software -- and if the traffic loads are high enough -- you can see the box puke all over itself and roll around in it.

q.

DaveDude
No Fear
join:1999-09-01
New Jersey

1 recommendation

DaveDude to y2k14

Member

to y2k14

Re: apocalypse

i am sure Nostradamus predicted it.

Cheeze_It
@73.181.84.x

Cheeze_It

Anon

The platform is running on a Sup720-3BXL that had this issue...certain of it

Certain aging Cisco 6500/7600 platforms with the Sup720-3BXL platforms that have been in production since the late 1990s....that could just be upgraded to a Sup 2T and none of this would have happened.

Oh wait...that would mean for them to spend money on infrastructure.

Can't have that...now can we...
HeadSpinning
MNSi Internet
join:2005-05-29
Windsor, ON

HeadSpinning to zpm

Member

to zpm

Re: Typical

said by zpm:

Good Job Verizon.......

Really, we were so close to the 512k limit that it would have been hit eventually - Verizon just tripped it earlier. Net effect would have been the same. Sooner or later, older Cisco routers would have barfed unless they'd been adjusted to handle the larger table.
BlueC
join:2009-11-26
Minneapolis, MN

BlueC to battleop

Member

to battleop

Re: What a huge pain in the ass....

Right. This isn't an issue for us, but I recall seeing others discussing this months back. It was an inevitable issue and it seems a number of networks dropped the ball on being prepared.

telcodad
MVM
join:2011-09-16
Lincroft, NJ

1 edit

2 recommendations

telcodad to fiosultimate

MVM

to fiosultimate

Re: Y2K14

The "Echoes of Y2K"

In today's Wall Street Journal:

Echoes of Y2K: Engineers Buzz That Internet Is Outgrowing Its Gear
Routers That Send Data Online Could Become Overloaded as Number of Internet Routes Hits '512K'
By Drew FitzGerald, The Wall Street Journal - August 13, 2014
»online.wsj.com/articles/ ··· 07937617

EDIT: aka "512KDay":

512KDay: How the internet BROKE (Next time, big biz, listen to your network admin)
We failed the internet's management challenge
By Trevor Pott, The Register - August 13, 2014
»www.theregister.co.uk/20 ··· to_play/

tubbynet
reminds me of the danse russe
MVM
join:2008-01-16
Gilbert, AZ

tubbynet to Cheeze_It

MVM

to Cheeze_It

Re: The platform is running on a Sup720-3BXL that had this issue...certain of it

said by Cheeze_It :

Certain aging Cisco 6500/7600 platforms with the Sup720-3BXL platforms that have been in production since the late 1990s....that could just be upgraded to a Sup 2T and none of this would have happened.

except of course -- to try and prove your point -- you neglected to include the vs-720 and the rsp720 -- each of which were introduced in mid-2007 and suffer from the same issue. in fact -- the s2t was released mid-2011 and has potential drawbacks that could prevent this card from being adopted in each carriers network.

that being said -- if i have an edge/core box that is handling services correctly and the infrastructure needs (i.e. correct number of 1/10g interfaces) are met by the s720 -- why would i need to upgrade a box that still has life left in it? upgrading for upgrading's sake is a poor way to spend your capex dollars.

q.

NormanS
I gave her time to steal my mind away
MVM
join:2001-02-14
San Jose, CA
TP-Link TD-8616
Asus RT-AC66U B1
Netgear FR114P

1 recommendation

NormanS to Cheeze_It

MVM

to Cheeze_It
said by Cheeze_It :

Oh wait...that would mean for them to spend money on infrastructure.

Can't have that...now can we...

Of course not! It is much more important to spend the money to buy off regulation!
Nanaki (banned)
aka novaflare. pull punches? Na
join:2002-01-24
Akron, OH

Nanaki (banned)

Member

Was wondering what was going on.

i used a web proxy to get here while it was screwed up

battleop
join:2005-09-28
00000

battleop to BlueC

Member

to BlueC

Re: What a huge pain in the ass....

I was expecting some rogue nations to get together and split their aggregated announcements into /24s to run the size of the routing table up and disrupt traffic.

CosmicDebri
Still looking for intelligent life
join:2001-09-01
Lake City, FL

CosmicDebri

Member

I just waited

I just waited for the site to connect. I knew it would come back unless Justin didn't pay the hosting bill.....

firephoto
Truth and reality matters
Premium Member
join:2003-03-18
Brewster, WA

2 recommendations

firephoto

Premium Member

Profit vs infrastructure

Making money is more important than upgrading aging hardware that is so costly it's probably been paid for 100 times over it's life so far. And these are the literally the pillars of the internet, imagine how old those ISP edge routers that the shills like to cry about their costs all the time.

It's always the money.

woody7
Premium Member
join:2000-10-13
Torrance, CA

woody7 to hello123454

Premium Member

to hello123454

Re: Proactive much?

biochemistry
Premium Member
join:2003-05-09
92361

1 recommendation

biochemistry

Premium Member

CDN

I was about to ask Verizon to host the DSLReports CDN.

KrK
Heavy Artillery For The Little Guy
Premium Member
join:2000-01-17
Tulsa, OK

KrK

Premium Member

Verizon crashed the Interwebz

Seriously though, this was a problem that was known to be coming.

Still caught people with their pants down, however.
Cobra11M
join:2010-12-23
Mineral Wells, TX

1 recommendation

Cobra11M to firephoto

Member

to firephoto

Re: Profit vs infrastructure

they would prob save a fortune just replacing them with newer equipment in the long run.. especially on electricity!
WhatNow
Premium Member
join:2009-05-06
Charlotte, NC

1 recommendation

WhatNow to KrK

Premium Member

to KrK

Re: Verizon crashed the Interwebz

Normal for most American business. The bigger they get the less they do because there are more people in charge to make a decision.
tmc8080
join:2004-04-24
Brooklyn, NY

tmc8080

Member

ipv6

ipv6 was supposed to be fully integrated into (tier-1 to isp side/edge) dns routers by now..

so much for the Skynet theory..

Coelispex
join:2013-06-03
Howell, MI

Coelispex to telcodad

Member

to telcodad

Re: Y2K14

Dad jokes :P

Rogue Wolf
An Easy Draw of a Sad Few
join:2003-08-12
Troy, NY

3 recommendations

Rogue Wolf to hello123454

Member

to hello123454

Re: Proactive much?

These days, "proactive" means "solving the problem while it's smoking but before it explodes".
page: 1 · 2 · next