nwrickert
sand groper
Premium,MVM
join:2004-09-04
Geneva, IL
 ·AT&T U-Verse
 ·AT&T Midwest
| reply to scooby
Re: No service on the West Loop at midnight 8/1/08
Around 11:57pm, I lost total connectivity but everything was up. PPPOE connection was up and connected and I had an IP address but routing was completely dead. Yes, that's about what I saw.
After about 15 minutes I rebooted my router and it could not authenticate. That, I didn't try. It was pretty obvious what would happen if I did. It was time for sleeping anyway, so I waited till the morning.
--
AT&T dsl; Westell 327w modem/router; openSuSE 11.0; firefox 3.0.1 |
|
scooby
Premium
join:2001-05-01
Schaumburg, IL
| reply to sparky672
said by sparky672  :That's not quite what I experienced. I was connected just fine according to the SS-4100 GUI and all modem lights were normal. (I also use OpenDNS exclusively.) I could not access the Internet at all when the problem occurred. After I rebooted the modem then, of course, I could not reconnect and got the auth errors. Around 11:57pm, I lost total connectivity but everything was up. PPPOE connection was up and connected and I had an IP address but routing was completely dead. Not just authentication and DNS. After about 15 minutes I rebooted my router and it could not authenticate.
Working second shift it really sucks that they do these maintenance windows so early. |
|
sparky672
join:2006-04-03
Manhattan, IL
| reply to lev
said by lev :Wolverine said only authentication and DNS were affected. So if you were already connected, there would not have been any problem, unless you tried to use DNS during that time. That's not quite what I experienced.
I was connected just fine according to the SS-4100 GUI and all modem lights were normal. (I also use OpenDNS exclusively.)
I could not access the Internet at all when the problem occurred.
After I rebooted the modem then, of course, I could not reconnect and got the auth errors. |
|
ssj4android
Redefining Reality
join:2002-04-14
Wyoming, MI
1 edit | reply to Tw0lazy
said by Tw0lazy :
After reading about the black hats vs the DNS flaws, perhaps this was a regional DNS upgrade? Would explain alot...
That's what I thought. If not last night, I believe the DNS servers were patched withing the last week (DoxPara reported them as vulnerable for quite some time after the patch).
But that doesn't explain established connections being dropped. |
|
kherr
Premium
join:2000-09-04
Collinsville, IL
clubs:
| reply to DubYou
said by DubYou :
I called support and a recording said something about "network enhancements". They also mentioned everything going back to normal by 6 AM.
It's crazy how wide spread this is. I'm in River Grove (Kiddie Land / Triton College). Also, friend in Schiller Park was out.
In other words an improved connection to the NSA .... |
|
lev
Napoleon is always right
Premium,Ex-mod 2002-08
join:2001-05-30
Chicago, IL
clubs: | reply to Late_Tech
I misunderstood Wolverine.
Heck, stuff worked for me.  |
|
Late_Tech
join:2000-08-06
Northbrook, IL
clubs:
1 edit | reply to lev
Well.....any downloads that were running at that time of service interruption ended with time-outs, no remote connections to Terminal Servers could be completed, VPN tunnels were dropped, pings & traceroutes were denied as hosts were unavailable.
It wasn't the worst of times, but it was a bit of an inconvenience. I had to make up a lot of work this morning that was planned for the overnight hours. |
|
Tw0lazy
@ameritech.net | reply to jafo1972
After reading about the black hats vs the DNS flaws, perhaps this was a regional DNS upgrade? Would explain alot... |
|
StillLearn
Premium
join:2002-03-21
Streamwood, IL
·AT&T Midwest
| reply to wolverine_99
said by wolverine_99 :Not that it matters all that much to you at this point, but the maintenance window info on the FAQ is a bit out of date. The windows are M-F. Thanks.
»dedicated.sbcis.sbc.com/NDWS/sbc···ance.jsp talks of Tuesday and Thursday morning maintenance windows.
»helpme.att.net does not seem to have maintenance window info, but maybe I missed it. |
|
Tw0lazy
@ameritech.net
| reply to jafo1972
Out here near Gurnee, IL (Think Six flags and Gurnee Mills), the dsl went out exactly at 11:57, only reason why i know is I was downloading a backup from my site, and happened to look at my task bar when the net went out. Funny part is, I checked my logs, the whole dsl system was up. The auth server is what was down. Hope this can shed some light... |
|
gdm
Premium,MVM
join:2001-06-15
Mchenry, IL
clubs: | reply to lev
Yeah that's what I get for trying to type and watch a live meeting on another screen. |
|
wolverine_99
Premium
join:2004-12-07
Mckinney, TX
| reply to lev
Lev,
I should have been more clear. All traffic was isolated to the POP. You couldn't get beyond the POP for anything even if you had a session. Sorry for the confusion I may have created about what was impacted and not. It's been a long night. I'm just glad my team is not the one explaining today. However, I feel for that team's Director since I've been there and it's not fun. |
|
wolverine_99
Premium
join:2004-12-07
Mckinney, TX
| reply to jafo1972
StillLearn,
As I stated, that FAQ is outdated. The maintenance windows are M-F. This would qualify as mtce window work.
nwricket,
I think we both know how it happened...I just wasn't going to mention it.
GDM,
Yes it impacted both DSL & U-verse services in the impacted POPs. The Chicago POP was not impacted, only Elmurst in the Chicago area. If you were authenticated and had a session up it wouldn't have dropped. However, you wouldn't have been able to get TV or internet services during the impact time. |
|
lev
Napoleon is always right
Premium,Ex-mod 2002-08
join:2001-05-30
Chicago, IL
clubs: | reply to gdm
Wolverine said only authentication and DNS were affected.
So if you were already connected, there would not have been any problem, unless you tried to use DNS during that time. |
|
gdm
Premium,MVM
join:2001-06-15
Mchenry, IL
clubs: | reply to Mr Anon
I checked my RG and it telling me I have had a connection for 17 days. I am not sure the adsl users are related to uverse but wolverine could say for sure. |
|
StillLearn
Premium
join:2002-03-21
Streamwood, IL | reply to nwrickert
Midnight 8/1/08 did not start a maintenance window; that window would have been 24 hours earlier. »Ameritech - SBC FAQ »Is there an official DSL network maintenance window? |
|
nwrickert
sand groper
Premium,MVM
join:2004-09-04
Geneva, IL
·AT&T U-Verse
·AT&T Midwest
| reply to wolverine_99
Why was it started before midnight? That answer didn't come out this morning, but it will be addressed later this morning. LOL
They probably did some "initial preparatory work that couldn't possibly hurt anything" - and took the whole system down by mistake.
Thanks for the information.
And yes, I recognize that the outage would prevent authentication and DNS from working. It's just that those were secondary problems, not the primary cause of the outage.
--
AT&T dsl; Westell 327w modem/router; openSuSE 11.0; firefox 3.0.1 |
|
SlamDance
@sbcglobal.net | reply to jafo1972
Check authentication with PPP Server was down in Burbank, IL around 12:40. went to bed, back up in the morning. |
|
Mr Anon
@il.us | reply to jafo1972
It was not just the Auth Server, my U-Verse connection lost interent around midnight but I did not loose TV, it was back when I got up around 6am this morning no telling what time it actually restarted. |
|
wolverine_99
Premium
join:2004-12-07
Mckinney, TX
| reply to nwrickert
I can't go into the details of the outage, but it was maintenance work that did not go as planned. Why was it started before midnight? That answer didn't come out this morning, but it will be addressed later this morning. As stated, it was a large outage impacting more than the Elmhurst POP. DNS and authentication servers would have been impacted as a result.
nwricket,
Since it appears you did not reboot your modem, you kept your PPPOE session to rback10. However, anyone that rebooted their modem would not have been able to authenticate until routing was resolved. |
|
