Level 3 Issues Cause Outage for Comcast, Other ISPs

I’m On Comcast - but outage only affected my Facebook & Twitter access. Access to other sites was ok. Problems started about 11 a.m. ET and lasted until 3 p.m.

Yep, we had troubles in SW Washington. Ooma phone service was next to useless (very choppy), audio streaming basically didn't work, and many websites weren't accessible at all.

CenturyLink has just taken over. This is just the beginning of network connectivity issues. A bud once described dealing with CTL as a bunch of "assholes".

FYI - The latest report on ComcastCares »twitter.com/comcastcares at 3:38 PM ET:

quote:

---

Internet issues should now be resolved for almost all customers. We apologize for the inconvenience & thank you for your patience.

---

... which I why I continue to advocate for people to subscribe to POTS service.

I was wondering why Facebook and several other sites were slow to load or otherwise not working this morning. Now I know why. At least the issue seems to be resolved.

Comes just as they announced Century Link closing the acquisition, most likely caused by some sort of post merger process to bind the two to a more uniform service.

From: »news.centurylink.com/201 ··· mmission

quote:

---

"We anticipate closing the transaction effective November 1, 2017."

---

Yeah, wow, that sure didn't take long!

 

First log entry was at 12:51:06, high latency but still able to get the data through.

At 12:53:35 data stopped going through.

The issue continued off and on until just after 14:21:20, when it appeared to be resolved.

I haven't seen any issues since then.

(edit: times are EST)

There were outages? I guess I was just lucky...no slowness or outages for me...and I have Spectrum (TWC previously). oh yeah, my landline (CLink) was working also.

From:
Comcast, Others Dealing With Internet Outage
Level 3 says ‘configuration error’ caused disruption and service has been restored
By Jeff Baumgartner, Multichannel News - November 6, 2017
»www.multichannel.com/new ··· e/416363

quote:

---

... Update: Level 3 issued this statement at about 3:30 pm ET: "On Monday, Nov. 6, our network experienced a service disruption affecting some customers with IP-based services. The disruption was caused by a configuration error. We know how important these services are to our customers. Our technicians were able to restore service within approximately 90 minutes."

---

Issues like this can be prevented using products that test upstream connectivity and reroute to alternate carriers if there's a problem. No excuses for not employing this other than cheaping out.

my latency is better than usual now
level 3 should screw up more often

reminds me after a charter local fiber cut,
my coax isp enjoyed the best connection ever!
sadly, it was right before i cancelled it

Or at least a cellular backup.

pretty sure they did, seemed like the alternate paths were just completely swamped

Screwing up one network at a time. Almost as bad as Charter.

Charter "Hey, we screwed up the network *AND* we're charging more for our services and really gouging our customers!"

CenturyLink "Hold my beer"

Speculum was flawless all day during my telecommute.

That's what I have here. Ooma is my primary home phone and the cell phone is my home phone backup. Since Ooma has had a way to forward calls to a cell phone in the event of an internet outage for some time. So, even when someone does call my Ooma line and the network is down, the call comes through my cell phone.

This is just the start of issues we’re going to have with Level3 being acquired by CenturyLink.

FTFY

I’ve dealt with CenturyLink and to call them incompetent would be an understatement. It does not at all surprise me that they would misconfigure something.

I've been tracking problems across the US starting is SF to DC to ATL to DAL to SEA over the weekend, as I transmit high volumes of data to cloud services located in the Tysons Corner area of Virginia, so across the country from Seattle. Just because L3 has the probs it affects all the other major transmission companies and their throughput as well. I've been jumping from one route to another all day today trying to get my data stream back approching 'normal' speeds almost to no avail.

I think there are more problems out there in transmission/router plants than simply L3/Centurylink, after spending close to 40 years constructing and operating transcontinental and transoceanic fiber systems, now retired, I see most operators simply not paying attention to their systems, outsourcing maintenance and engineering to service companies (and I worked for the myself) and mostly take their paycheck and run. Same as what we see with programming 'services' that allow not just the hack of the month, but hack of the day and even hour.

I'm sure they're good professional folks out there but I think it's getting few and far between from what I see.

Wired has posted an analysis of the outage: »www.wired.com/story/how- ··· -the-us/ which concludes that this was a route leak (happened before, likely will happen again).

That's what we use (cell backup). Basic Century Link POTS here is $22 plus another $7 or so in taxes/fees. And that doesn't include any calling outside the area, which is a problem given that Portland is just 12 miles away and is interstate long distance. So the basic POTS service wouldn't be a replacement for Ooma. It would just be an emergency phone. POTS service that includes long distance would be over $70 a month all in.

Go to our basements for some POTS. But really, for a utility* ISPs are doing poorly. One point of failure for everything including VoIP.

*Harking back to all those "broadband is necessary and should be a utility" discussions.

"we're charging more for our services and really gouging our customers!"

This is standard operating procedure for Level(3). They are the most expensive quote I've ever seen for on network bandwidth (56 Marietta) of any company I've ever been quoted. They are generally twice or more expensive than AT&T DIA that's delivered from a CO into 56 Marietta that's not in the same building.

I keep telling the Level(3) guy that's always calling on me to stop wasting my time with their insanely expensive quotes.

but alt-east their crap works

Centurylink has been decent on enterprise type services but they flat out suck on SMB and Residential service. I'm afraid even that's going to go away now that the L3/CTL merger is done. Once a company gets to a certain size it's pretty much impossible to keep things under control.

...I was online around 12:30, and everything was fine... Until just before 1:00pm, when everything ground to a halt. I'm on Verizon FIOS, and have noted multiple connectivity issues for the past several days... Blinking white network light on the router. Figured it was Verizon, but the router checked okay. I tried a ping test on the DNS address I use: 4.2.2.1, which I think is Level3. Pings all failed. Added an alternate DNS address, and everything took off and worked fine...

Still wondering about the Verizon router issue over the past several days. Since installation 11 months ago, I've never noticed the white "network" light blinking...

Level3 was good for at least 3 nationwide outages a year as it was before Centurylink. Now they'll be overachievers.