dslreports logo
site
spacer

spacer
 
   
spc
story category
Time Warner Cable Suffers Nationwide Morning Outage
by Karl Bode 08:32AM Wednesday Aug 27 2014
On the heels of a nationwide Charter DNS outage over the weekend, Time Warner Cable is the latest ISP to suffer a nationwide outage. According to user comments in our forums, the ISP early this morning suffered a routing problem, knocking all of the company's customers offline. "At 430am ET this morning during our routine network maintenance, an issue with our Internet backbone created disruption with our Internet and On Demand services," Time Warner Cable said in a statement. "As of 6am ET services were largely restored as updates continue to bring all customers back online."

view:
topics flat nest 

Frank
is chilling
Premium
join:2000-11-03
somewhere

bright house affected as well.

From the looks of the bright house forum it seems that brighthouse internet was affected as well by the twc outage.
greatshow09

join:2013-07-07
Minneapolis, MN

Re: bright house affected as well.

TWC does own a bit of BHN. That's what makes the merger even more complex. They'd have to sort out how they'd deal with the parts of BHN that TWC owns.

Surfer

@71.47.213.x

Re: bright house affected as well.

No they don't. They doing bidding, and buy modems in boxes in bulk for bright house, but don't own any of them. It is a private company with no stocks to own.

buzz_4_20

join:2003-09-20
Limestone, ME

Odd Outage.

I could pull IPs on my cable modem. And got DNS responses from tw servers and from some others.

Glad I got to work and saw this I was scratching my head pretty bad.
gene32

join:2004-05-03
Reynoldsburg, OH

It's for the children

Eh, they were just cutting over to the NSA routers so your data can be protected from terrorists.
elefante72

join:2010-12-03
East Amherst, NY

1 recommendation

Re: It's for the children

The must be furiously updating routers because of the IPv4 route memory exhaustion going on. And when you rush, things break

telcodad

join:2011-09-16
Lincroft, NJ
kudos:7

Re: It's for the children

said by elefante72:

The must be furiously updating routers because of the IPv4 route memory exhaustion going on. And when you rush, things break

That was my thought also: »Why You're Having Problems Reaching DSLReports (and Other Sites)
smcallah

join:2004-08-05
Home
TWC's backbone routers are well advanced beyond the routers that can't handle the IPv4 route table.
ke4pym
Premium
join:2004-07-24
Charlotte, NC

Someone's change control got hosed

Bet there's some serious phone calls going on right now.
Joe12345678

join:2003-07-22
Des Plaines, IL

jumped the gun on the comcast merger?

Did jumped the gun on the comcast merger?

Mr Clean

@24.211.191.x

Lies

My TWC internet was out @ 2AM.
smcallah

join:2004-08-05
Home

Re: Lies

TWC is doing maintenance all the time overnights. Your Internet being out at 2am may or may not have been related to this other maintenance they had that went wrong.

Simba7
I Void Warranties

join:2003-03-24
Billings, MT

Router issue?

I remember Bresnan had a router that didn't want to play nice with others, causing a massive network outage to their entire footprint.

They resolved it a few days later. They probably had to get into the routers and force an updated config or update the tables.

telcodad

join:2011-09-16
Lincroft, NJ
kudos:7

Re: Router issue?

It does look like that:

Time Warner Cable hit by service outage for almost two hours
Reuters - August 27, 2014
»www.reuters.com/article/2014/08/···20140827
quote:
The outage, which the company said was caused by routine network maintenance, affected 2 percent of the routed networks in the United States, according to Renesys, a U.S. firm that monitors global Internet activity.
Also:
quote:
The outage came one day after the company agreed to pay $1.1 million to U.S. regulators who found the company did not did not file all the proper paperwork to report multiple network outages last year.
I guess they got some major paperwork to do now!

telcodad

join:2011-09-16
Lincroft, NJ
kudos:7

Re: Router issue?

Massive Internet Outage Points to Flaws in Policy and Technology
Time Warner Cable’s 11.4-million-customer broadband blackout was one of the largest to affect U.S. customers.
By David Talbot, MIT Technology Review - August 28, 2014
»www.technologyreview.com/news/53···hnology/

koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

2 recommendations

Technical details/explanation

...albeit briefly were discussed on outages.org and NANOG:

»puck.nether.net/pipermail/outage···943.html
»puck.nether.net/pipermail/outage···944.html
»mailman.nanog.org/pipermail/nano···464.html

In English: either TWC's or L3's routers announced to the rest of the Internet "don't bother sending us traffic for 69.134.0.0/17 or 76.178.0.0/17 any more" (this isn't the same thing as a route expiring; this is administrative, as in someone made a routing table config change hence the withdraw). Then ~1.5 hours later, requested traffic to those two /17s be sent back to them.

Whether or not this was truly a Level3-induced mistake is unknown at this time, but it's certainly possible depending upon the relationship between TWC and L3 (who controls and manages what equipment, etc.).

RIPE's BGPlay replays (you may want to choose "Options" and set animation speed to 4 or 5 to speed things up, as there's a lot of nonsense going on):

»stat.ripe.net/widget/bgplay#w.re···4.0.0/17
»stat.ripe.net/widget/bgplay#w.re···8.0.0/17

You'll see some route changes (at 08:43 UTC, 09:19 UTC, and 09:28 UTC), followed immediately by a complete mess of route path changes and some widthdrawls (from 09:29 UTC onward), until 09:31 UTC where over the next 4 minutes the rest of the Internet said "withdraw? Okay you got it" and they disappeared from the 'net. 1 hour and 15 minutes later (at 10:50 UTC) routes were reannounced.

outages.org thread: »puck.nether.net/pipermail/outage···935.html

NANOG thread: »mailman.nanog.org/pipermail/nano···452.html
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.

kpfx

join:2005-10-28
San Antonio, TX

Re: Technical details/explanation

Great info. You sir, have earned 1 Internet.
spdickey

join:2002-11-17
Pacific Palisades, CA

1 recommendation

Re: Technical details/explanation

From TWC Untangled

This Morning's Outage

During an overnight network maintenance activity in which we were managing IP addresses, an erroneous configuration was propagated throughout our national backbone, resulting in a network outage.

We immediately identified and corrected the root cause of the issue and restored service by 7:30 am ET. We apologize for any inconvenience this caused our customers.

A failure of this size is very serious and we are taking the necessary steps to improve our processes with the objective of making sure this doesn’t happen again.

»www.twcableuntangled.com/2014/08···-outage/

telcodad

join:2011-09-16
Lincroft, NJ
kudos:7

DirecTV also quietly experiences its own technical issues

»www.fiercecable.com/story/twc-ex···14-08-28

DIRECTV Suffers Local Channel Outage
By Phillip Swann, TVPredictions.com - August 28, 2014
»www.tvpredictions.com/directv082814.htm

PaulJinLA

@142.129.102.x

Re:Time Warner Cable Suffers Nationwide Morning Outage

Im a TWC customer in Los Angeles and was on the internet when the outage occurred. I was on a site in Miami area when all the sudden I was unable to get through to site. I ran several traceroutes to different east and west coast sites. Traces started out only getting as far as Dallas,Tx then was unable to trace anything beyond the L.A. area with routes being bounced around between Aggxx routers. Finally started seeing the unable to resolve host name error message when running traces. I use OpenDNS nameservers as well as one of TWC's.

I have to wonder if the stated percentage of network out of commission was just the routes/networks directly affected by the erroneous update or does that also cover the networks with business and private users who were unable to get out on the Internet due to no path to nameservers?

Also if I'm not mistaken I experienced the same sort of outage early this morning (8/29) a little after midnight. I was on the same site in Miami when all the sudden the webpage froze and I lost link. I ran a tracert and saw the unable to resolve host name error. I logged into my router as well as my cable modem and didn't see any error messages. About 6 minutes later the link came back up.

Now I sure wish TWC would stop routing me from Van Nuys to LAX to Tustin (way down south in Orange County), then all the way back to LAX, before route disappears into a cloud on way to final destination. Sheeesh. -Paul