dslreports logo
 story category
AT&T Responds To Outage
Under investigation -- equipment 'impairment' to blame...

Last night we reported that AT&T was suffering a significant outage in BellSouth territory. Our resident users reported that DNS functionality was derailed, and AT&T support mechanisms all but failed -- with users unable to reach support representatives. The Associated Press ultimately picked up on your complaints. We asked AT&T's Brad Mays what caused the outage and received this official company statement:

quote:
On Monday evening, AT&T experienced a disruption in its Internet service in the nine-state Southeast region which impacted customers' ability to surf the Web. The root cause of the disruption is still being investigated but appears to be an equipment impairment. Network technicians were able to restore service slightly before 11 p.m. We continue to monitor the situation, but it this point it appears that service has been successfully restored. We apologize for any inconvenience this is causing customers.
Just as when Comcast DNS servers ran into trouble in 2005 and last June, customers have found that alternative DNS operators such as OpenDNS are frequently helpful during such outages. We're sure OpenDNS doesn't mind the added business, either. OpenDNS's John Roberts even stopped by our forums to lend users a hand.
quote:
Ironically, we even took our Washington DC location offline for maintenance during this period (not knowing about these ISP DNS issues), and -- as designed -- there was zero interruption of service. Traffic was re-routed to our other locations seamlessly. Check our system page (here or here) in case your DNS is ever down.
Of course your ISP would probably prefer you switch back to their DNS servers once the smoke clears, given the growing use of DNS redirection to create an additional revenue stream (though to our knowledge neither AT&T or Comcast use DNS redirection yet).

Notable here is that the support systems for the nation's largest ISP absolutely collapsed during this outage, with several users e-mailing us to complain that they simply could not reach the company.

"From the earliest part of the problem throughout the evening, callers trying to contact the help desk received either an immediate fast busy or a recording which simply said, 'we're too busy, call back later,'" notes one long-time user. "That's just not an acceptable response for thousands of customers who suddenly find themselves without internet access, through no fault of their own."

Update:We've spoken further with Brad and have found that AT&T's investigation concluded that the outage was due to an "an isolated router failure."
view:
topics flat nest 
page: 1 · 2 · next

DotMac4
Shill H8r
Premium Member
join:2007-10-26
Huntington Beach, CA

DotMac4

Premium Member

...Yet

Give the greedy bastards some time, they'll get around to it.

tao
Frazzlebats
Premium Member
join:2000-12-03
Lansing, MI

tao

Premium Member

OpenDNS

Is the perfect DNS usage to subvert the DNS redirection methods some ISPs use for profit. DNS redirection is a perversion of the Internet that users themselves can render useless to employ.

In fact, maybe someone should start a list of ISP redirection. We should even start a campaign to have Google, and other search engines, delist the sites designed to take advantage of user misspellings. Google already offers you respelled word options when you search, maybe an additional step is necessary to protect the integrity of the DNS servers.

gatorkram
Need for Speed
Premium Member
join:2002-07-22
Winterville, NC

1 recommendation

gatorkram

Premium Member

Re: OpenDNS

You do realize OpenDNS does the same thing?

Although, you can turn it off..

I admit, I have been a happy OpenDNS user for a long time now. All it took for me to switch, was to have a dns outage at my current isp, and I have never looked back.

jgkolt
Premium Member
join:2004-02-21
Avon, OH

1 recommendation

jgkolt

Premium Member

Re: OpenDNS

you are correct. One nice thing is all the little changes you cna do to opendns without actually running your own dns server.
Jerkface
join:2005-06-05
Hackettstown, NJ

Jerkface

Member

what i think

i think this is something more than "a server crash". Wouldn't it be funny if they just took down those servers to implement DNS redirect software? Hrmmmm.... Yup.

Just because its At&t.

en102
Canadian, eh?
join:2001-01-26
Valencia, CA

en102

Member

Re: what i think

I agree... I highly doubt that AT&T has 1 server for DNS for its '9 state south east' region.
Here in California, there's DNS for LA/Irvine, San Diego, San Jose, etc. (or at least there was a while back).
I suspect they had corruption, and not a hardware outage.

ifarrell
join:2000-08-10
Willow Spring, NC

ifarrell

Member

Re: what i think

said by en102:

I agree... I highly doubt that AT&T has 1 server for DNS for its '9 state south east' region.
Here in California, there's DNS for LA/Irvine, San Diego, San Jose, etc. (or at least there was a while back).
I suspect they had corruption, and not a hardware outage.
Corruption and AT&T in the same sentence, that makes sense. Especially since it's probably all that NSA equipment hanging off their network that probably caused the problem.

mrsynical
@comcast.net

mrsynical

Anon

how does openDNS make money???

They have to pay the bills! How do they do it? If they give it away for too long, it may become closedDNS.

djrobx
Premium Member
join:2000-05-31
Reno, NV

djrobx

Premium Member

Re: how does openDNS make money???

They have the DNS redirect feature (on by default) to generate ad revenue. I've chosen NOT to disable this feature in support of their service.

ponon
@ameritech.net

ponon

Anon

Chicago also

from 1-4 am

NormanS
I gave her time to steal my mind away
MVM
join:2001-02-14
San Jose, CA

NormanS

MVM

Re: Chicago also

Unrelated to the AT&T Southeast problem.
b10010011
Whats a Posting tag?
join:2004-09-07
united state

b10010011

Member

Roll your own DNS server

TreeWalk DNS is a Domain Name Server program which fetches and converts Web Site human-readable names into the numbered addresses your computer needs so that your browser, email, instant messenger, and FTP programs can surf the Web. For Microsoft Windows® XP, 2000, .NET and 2003 workstations or servers (Vista version pending).

TreeWalk is FREE for personal use!

»ntcanuck.com/

I run TreeWalk as primary and Open-DNS as secondary.
Stumbles
join:2002-12-17
Port Saint Lucie, FL

Stumbles

Member

Re: Roll your own DNS server

Well good for you. But no need to tell everyone to install yet another piece of software when ALL you really need to do is point your DSL modem to a different DNS server. It's not rocket science.
b10010011
Whats a Posting tag?
join:2004-09-07
united state

1 recommendation

b10010011

Member

Re: Roll your own DNS server

said by Stumbles:

Well good for you. But no need to tell everyone to install yet another piece of software when ALL you really need to do is point your DSL modem to a different DNS server. It's not rocket science.
I guess that is fine if you like having to relay on somebody else to do their job.

Then you can whine about it and blame someone else when something goes wrong.

Personally I like to be self reliant whenever possible.

Besides running your own DNS server is not rocket science either.
Stumbles
join:2002-12-17
Port Saint Lucie, FL

Stumbles

Member

Re: Roll your own DNS server

Besides running your own DNS server is not rocket science either.

Ya for sure..... I'm just to lazy to do it.
Stumbles

Stumbles

Member

Oh really.

Hmm, an "isolated router" caused this. Appears said router was not so "isolated" after all..... and they don't have a backup? Sounds par for the course.

djrobx
Premium Member
join:2000-05-31
Reno, NV

djrobx

Premium Member

Re: Oh really.

One isolated router should not cause this. That's the whole point behind configuring two redundant DNS servers. Competent network design would have these DNS servers in different physical locations so they are not affected by a single outage.

Matt3
All noise, no signal.
Premium Member
join:2003-07-20
Jamestown, NC

Matt3

Premium Member

Re: Oh really.

said by djrobx:

One isolated router should not cause this. That's the whole point behind configuring two redundant DNS servers. Competent network design would have these DNS servers in different physical locations so they are not affected by a single outage.
Agreed. We're a tiny shop and we have 2 DNS servers in our main data center and we pay for 1U of rack space all the way across the country for our tertiary DNS server.

en102
Canadian, eh?
join:2001-01-26
Valencia, CA

1 recommendation

en102 to djrobx

Member

to djrobx
I tend to agree.
Even IF they decided to have their 2 DNS servers in the same facility, they should AT LEAST:
1. Have them in separate racks (avoid fires, heat, rack damage, etc.)
2. Connected to different electrical circuits (duh - don't want a tripped breaker or UPS outage to pull them down)
3. Connected to the Internet through separate routers / backbone connections (less of an issue, but a failed switch/cable/router or maintenance could bring it all down)
4. Each DNS server should also have a failover switch configured as well.

Personally, I could live with 2 DNS servers in the same facility, assuming that I had 3 DNS servers configured on my PC/Linux box, and the 3rd one being at some other facility.
Eg. in my case, 66.51.205.100 (SoCal), 66.41.206.100 (SoCal) and 66.218.44.5 (NorCal).

dellsweig
Extreme Aerobatics
MVM
join:2003-12-10
Campbell Hall, NY

1 edit

dellsweig

MVM

single router failure??

A network the size of ATT should never have experienced an outage of this scale on a "SINGLE" isolated router failure....

Even then - the advanced NMS systems in use at ATT should have picked up the root cause of this failure and automatically dispatched a tech (VERY big grin)

FFH5
Premium Member
join:2002-03-03
Tavistock NJ

FFH5

Premium Member

Re: single router failure??

said by dellsweig:

A network the size of ATT should never have experienced an outage of this scale on a "SINGLE" isolated router failure....
A router "FAILURE" is usually handled easily. The problem is when the router is still up and running but malfunctioning. Problems like this are much harder to detect and repair quickly.

ONiall
Yum, Citizen
Premium Member
join:2002-11-18
Portland, OR

1 recommendation

ONiall

Premium Member

Re: single router failure??

said by FFH5:

A router "FAILURE" is usually handled easily. The problem is when the router is still up and running but malfunctioning. Problems like this are much harder to detect and repair quickly.
I agree. Sometimes there is a loss of routing protocols, or some other issue that doesn't show up in active monitoring.

Those issues take time to identify, and the support engineers need solid information from the service desk to figure out where to start looking.

Unfortunately, that can be hard to obtain. Non-technical service desk personnel and customers often don't know how to generate the troubleshooting data required for a quick problem determination.

Raphion
join:2000-10-14
Samsara

Raphion to FFH5

Member

to FFH5
said by FFH5:

said by dellsweig:

A network the size of ATT should never have experienced an outage of this scale on a "SINGLE" isolated router failure....
A router "FAILURE" is usually handled easily. The problem is when the router is still up and running but malfunctioning. Problems like this are much harder to detect and repair quickly.
Indeed. Just the other day, I was unable to access DSLReports, pinged out everytime. I could get to any other site though, and everything else was working perfectly. But finally I rebooted my own router, and *poof*, suddenly I could get to DSLReports again. Somehow, my router was locking me out of just this one site, and doing everything else just fine.

NormanS
I gave her time to steal my mind away
MVM
join:2001-02-14
San Jose, CA
TP-Link TD-8616
Asus RT-AC66U B1
Netgear FR114P

NormanS to dellsweig

MVM

to dellsweig
said by dellsweig:

A network the size of ATT should never have experienced an outage of this scale on a "SINGLE" isolated router failure....

Even then - the advanced NMS systems in use at ATT should have picked up the root cause of this failure and automatically dispatched a tech (VERY big grin)
Interesting. As an AT&T DSL customer, this DNS outage had zero impact on me.

IPingUPing
N4BFR
Premium Member
join:2002-08-30
Atlanta, GA

IPingUPing

Premium Member

Re: single router failure??

From your profile, it doesn't appear you are in the 9 state impacted region.

NormanS
I gave her time to steal my mind away
MVM
join:2001-02-14
San Jose, CA

NormanS

MVM

Re: single router failure??

Well, the AT&T network is larger than even the 13 state former Bellsouth region.

geek
Mad Scientist at Work
Premium Member
join:2002-01-07
Southbury, CT

geek

Premium Member

Anycast?

I thought AT&T was using Anycast for DNS. I guess a router failure at a key point would still prove to be detrimental.

David
Premium Member
join:2002-05-30
Granite City, IL

David

Premium Member

Re: Anycast?

Bellsouth I believe can use the anycast IP's if they choose.

NormanS
I gave her time to steal my mind away
MVM
join:2001-02-14
San Jose, CA
TP-Link TD-8616
Asus RT-AC66U B1
Netgear FR114P

NormanS to geek

MVM

to geek
said by geek:

I thought AT&T was using Anycast for DNS. I guess a router failure at a key point would still prove to be detrimental.
Unless the AT&T/BS customers are using 68.94.156.1/68.94.157.1, they are not using the Anycast DNS servers that the rest of us ('at&t Yahoo! HSI') are using.

David
Premium Member
join:2002-05-30
Granite City, IL

David

Premium Member

Confirmed

Bellsouth can use the anycast 68.94.156.1/68.94.157.1 if they choose!

dot_null
Premium Member
join:2004-06-28
Kennesaw, GA

1 edit

dot_null to NormanS

Premium Member

to NormanS

Re: Anycast?

The BellSouth/at&t Southeast DNS servers I am using are: 205.152.37.23/205.152.144.23

NormanS
I gave her time to steal my mind away
MVM
join:2001-02-14
San Jose, CA
TP-Link TD-8616
Asus RT-AC66U B1
Netgear FR114P

NormanS

MVM

Re: Anycast?

Anycast allows an IP address to point to the closest servers to the source of the DNS request. AT&T Midwest customers, AT&T Southwest customers, AT&T Northeast customers, and AT&T West customers all get the same pair of DNS server IP addresses. However, my DNS requests don't go to the same DNS serves as a customer in Los Angeles, California, Houston, Texas, Chicago, Illinois, or Hartford, Connecticut.

»en.wikipedia.org/wiki/Anycast

If the DNS servers you are using are Anycast servers, your route would be different than another AT&T Southeast customer's route, if you and the other customer are widely separated in the Southeast region.

However, if that other customer has a different IP address pair than you have, the servers are likely not Anycast servers.

koitsu
MVM
join:2002-07-16
Mountain View, CA

koitsu

MVM

Redundancy?

One router, 9 states -- makes no sense.

KeysCapt

join:2001-07-11
Carson City, NV

1 edit

KeysCapt

Far more than "equipment impairment"

The other complete failure during this apparently (to them) insignificant event was the total meltdown of their help desk. From the earliest part of the problem throughout the evening, callers trying to contact the help desk received either an immediate fast busy or a recording which simply said, "we're too busy, call back later". That's just not an acceptable response for thousands of customers who suddenly find themselves without internet access, through no fault of their own.

Those who found their way to the forum were the fortunate ones, quickly learning of alternative DNS numbers which solved the problem for them. But as one poster put it, really too bad for all the others who weren't so lucky and spent their evening trying to figure out, without AT&T's help, what had happened to their internet.
whiteyonenh
join:2004-08-09
Keene, NH

whiteyonenh

Member

Re: Far more than "equipment impairment"

said by KeysCapt:

From the earliest part of the problem throughout the evening, callers trying to contact the help desk received either an immediate fast busy or a recording which simply said, "we're too busy, call back later". That's just not an acceptable response for thousands of customers who suddenly find themselves without internet access, through no fault of their own.
Well... it's a good thing that they haven't started bundling the CallVantage service with dry-line yet then. I know of many people that would switch in an instant to save money. Imagine if they switched everyone over to VOIP, what a DNS failure could do to an ATA. I would like to see their response if people across 9 states lost phone service.

I would think that internet service is approaching the necessity of phone service in many areas. This is also a reason that I keep my cell phone and a VOIP line, at least one of them will work at any given time.

Bama_Lady12
@bellsouth.net

Bama_Lady12

Anon

Looking at the alternatives

AT&T has never been one of my favorite utility companies, but since they started streamlining aka outsourcing associates and reorganizing Ma Bell, my displeasure has only grown. I am not a tech junkie, but I like to be able to depend on my Internet. We had this same thing happen to us for about 30 minutes Friday evening and then were without our service for about 4 hours or more last night.
I don't care for TimeWarner, but after last night and with the deals that they are having with digital cable and their high speed Internet services I am more than a little tempted to switch. Unlimited calling for me and my family out of state are the only things that keep me connected to this stinking company!
boggaf05
CCNA, SCVU
Premium Member
join:2006-06-21
SC

boggaf05

Premium Member

Use OpenDNS

Just use OpenDNS, I found that it fixed many connection issues our business line was having not loading many webpages.

djatsw
@nc.us

djatsw

Anon

cable outage

my service with charter Communications was also out for hours last evening. They tried to blame it on the Vonage phone service I have. But the phone was working just fine.

mrsynical
@comcast.net

mrsynical

Anon

how does open DNS make their money??

how do they pay their bills? If they keep giving away free DNS, then at some point they could be closedDNS.

bsnihls
join:2001-02-07
Jacksonville Beach, FL

bsnihls

Member

"an isolated router failure."

"an isolated router failure." is bull, once again AT&T treats us like idiots.

Maybe I should have "an isolated router failure." when i go to pay the bill this month.

How about this "fail over"?

Hey BellSux now AT&Sux, If a router fails and no one answers the support line, does it really matter?

HAHAHA the phone company is to busy. That is your job! Answer the damn phones or place a message that says, "You are not alone, we are not working either"

MediaDavid
@bellsouth.net

1 recommendation

MediaDavid

Anon

Re: "an isolated router failure."

ha....what ATT/bellsouth claims was their Fix was in reality not a fix for any of their customers.
the reason is unless you know the arcane and complicated system needed to log yourself out of your server and reboot your modem, the "fix" did nothing to actually restore service. It only permitted you to have the ability to try to contact someone to walk you through the process.
It took me over 16 hours to actually get a call through and after 45 mins on hold, someone walked me through the fix.
how on earth could they A) try to lie their way through this when they knew their customers still did not have service?
b) allow their customer service operations to fail so totally and completely?

total bullcrap

Anon

i Caught their BS.

I used to work for Bell. a bit before ATT came into the Game.

Bell used Multiple DNS servers for each area. usually by city. Georgia alone had i believe 4 DNS servers for Atlanta alone.

so if they had a 'simple router failure' then it happened simultaneously between all there DNSs.
SwanSong89
join:2007-12-04
Virginia Beach, VA

SwanSong89

Member

Bellsouth/AT&T so-called Outage a hoax

My DSL provider is Bellsouth.net/AT&T and my service was shut down for three days for no apparant reason and without warning during this so-called "OUTAGE".when I called the tech lines were busy, i got through to a recording that stated that an "OUTAGE" in my area was being fixed but when they finally restored it today anytime i try to watch a divix video on any divx based site i get the error message that sates: "Divx webplayer plugin has performed an illegal operation and that it is strongly advised that I restart firefox."(this never happened before the so-called outage) It automatically shuts down firefox. It doesn't seem to matter what divx file i try to watch on Stage 6 or elsewhere it blocks them all. It is plain obvious that they (AT&T) used this so-called outage downtime to make adjustments to their customers services. Without informing them that changes were being made that would allow them to block certain types of internet surfing,downlading,viewing etc. Please pass this message along to anyone u can find. I am going to Drop them ASAP!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
I googled AT&T and found that they are now allowing the NSA and MPAA,RIAA to have access to its clients emails,web browising etc.
I also received a warning from Firefox while i was trying to login that stated: "It is possible though unlikely that someone may be trying to intercept your communication with this website."

Heterman
Premium Member
join:2004-02-28
Fayetteville, AR

Heterman

Premium Member

Re: Bellsouth/AT&T so-called Outage a hoax

Is there anything you "tried" on your machine while the "Internet" was down?

a.) I can play Stage 6 DivX fine; I am on at&t. Are you sure there is nothing wrong with your DivX player?
b.) They have been tangled in the NSA stuff for a while now, that is nothing new. Wait 'till you read about the datacenter they built to route the Internet through.
c.) The FireFox warning about "Someone intercepting" is a message received when you go from a secure site to a non-secure site. It usually shows up when you first install the browser and will continue to show until you check the box to not bother you anymore. It is a standard default message, which leads me to believe you un-installed one or more things to try and "fix" your Internet problem. So, that takes us back to my very first sentence.

NormanS
I gave her time to steal my mind away
MVM
join:2001-02-14
San Jose, CA
TP-Link TD-8616
Asus RT-AC66U B1
Netgear FR114P

NormanS to SwanSong89

MVM

to SwanSong89
said by SwanSong89:

I googled AT&T and found that they are now allowing the NSA and MPAA,RIAA to have access to its clients emails,web browising etc.
AT&T was giving access to the NSA since long before SBC bought them. To the best of my knowledge, SBC never gave up customer information to the **AA. They, and Verizon, were even taken to court by the **AA over that. Verizon, for sure (I don't know about SBC) appealed, the **AA win, and got the lower court decision overturned.

BTW, SBC took over AT&T, so the company you are now dealing with is, for all intents and purposes, SBC.

And, while I am at it, I had no DNS problems over the past period when the AT&T Southeast DNS servers went FUBAR. You could try 68.94.156.1 and 68.94.157.1 for primary and secondary DNS servers, if you like. They have been working fine for me for a couple of years, now.
ryouga12
join:2001-06-12
Cleveland, TN

ryouga12

Member

Same problems mentioned above

I'll throw in my two cents. I had this same problem. My only regret was not saving the information about the opendns servers and could not access any site at all. I do find it strange that we had these problems after the merger, although it could just be a strange coincidence. I never had any problems before ATT and BS merged.

NormanS
I gave her time to steal my mind away
MVM
join:2001-02-14
San Jose, CA
TP-Link TD-8616
Asus RT-AC66U B1
Netgear FR114P

NormanS

MVM

Re: Same problems mentioned above

said by ryouga12:

I never had any problems before ATT and BS merged.
I detect the post hoc ergo propter hoc logical fallacy, here...
LeeWL
join:2002-11-10
Morrisville, NC

LeeWL

Member

I use OpenDNS so I did not even know there was an outage.

I have been using OpenDNS for at least a year, maybe since the very beginning and found it was much better and faster than the Bellsouth DNS.

Yes, they do redirect, but since their business model depends on advertising on the redirect pages, they need users to stay with them (and therefore hit the redirect page every once in a while) to make money. Since users must actively enter the DNS addresses to use them, it is of primary importance that they (OpenDNS) provide a good service so people will use them, so you get somethign in return for the redirect page. When Verizon or someone else does it by default you get nothing in return.

•••
page: 1 · 2 · next