dslreports logo

story category
AT&T Responds To Outage
Under investigation -- equipment 'impairment' to blame...
by Karl Bode 11:56AM Tuesday Dec 04 2007
Last night we reported that AT&T was suffering a significant outage in BellSouth territory. Our resident users reported that DNS functionality was derailed, and AT&T support mechanisms all but failed -- with users unable to reach support representatives. The Associated Press ultimately picked up on your complaints. We asked AT&T's Brad Mays what caused the outage and received this official company statement:
On Monday evening, AT&T experienced a disruption in its Internet service in the nine-state Southeast region which impacted customers' ability to surf the Web. The root cause of the disruption is still being investigated but appears to be an equipment impairment. Network technicians were able to restore service slightly before 11 p.m. We continue to monitor the situation, but it this point it appears that service has been successfully restored. We apologize for any inconvenience this is causing customers.
Just as when Comcast DNS servers ran into trouble in 2005 and last June, customers have found that alternative DNS operators such as OpenDNS are frequently helpful during such outages. We're sure OpenDNS doesn't mind the added business, either. OpenDNS's John Roberts even stopped by our forums to lend users a hand.
Ironically, we even took our Washington DC location offline for maintenance during this period (not knowing about these ISP DNS issues), and -- as designed -- there was zero interruption of service. Traffic was re-routed to our other locations seamlessly. Check our system page (here or here) in case your DNS is ever down.
Of course your ISP would probably prefer you switch back to their DNS servers once the smoke clears, given the growing use of DNS redirection to create an additional revenue stream (though to our knowledge neither AT&T or Comcast use DNS redirection yet).

Notable here is that the support systems for the nation's largest ISP absolutely collapsed during this outage, with several users e-mailing us to complain that they simply could not reach the company.

"From the earliest part of the problem throughout the evening, callers trying to contact the help desk received either an immediate fast busy or a recording which simply said, 'we're too busy, call back later,'" notes one long-time user. "That's just not an acceptable response for thousands of customers who suddenly find themselves without internet access, through no fault of their own."

Update:We've spoken further with Brad and have found that AT&T's investigation concluded that the outage was due to an "an isolated router failure."

70 comments .. click to read