dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
1748
share rss forum feed

mystica

join:2001-10-30
Denver, CO

[Rant] Comcast Maintenance at 2am ends in null-routed subnet

Click for full size
Modem diag screen showing maintenance at 2:16am when Hector went offline
I just want to smack the network engineer at some headend or hubsite who screwed this up.

It seems that during last night's maintenance, some idiot null routed my friend's subnet.
Addresses 71.196.204.1-71.196.205.254 inclusive (71.196.204.0/23) are ALL unreachable. Traceroutes below.

Small backstory: I'm the go-to network guy for some friends of mine in the Denver area. I've got custom OpenWRT running on their routers, with SSH open for remote access and each router has a dyndns address in case I need to fix something remotely.

I get a call from my friend Hector. He tells me that since about 2am last night (~02:00 December 14 2011) his internet has been dead.

I explain to him that Comcast often will perform network upgrades and maintenance at this time of night, as when I subscribed to their service, staying awake all night long, I'd lose connectivity around that time for a few hours every few months.

So I asked if it came back on later, and no, it was still dead as of 7am December 14 before he left for work.

So I traceroute from my house, which from CentQwesturyLink goes thru texas instead of _directly in town_ (i'll NEVER understand why the 2 main ISPs in denver screw their customers over by routing to DALLAS first. HLRN means Highlands Ranch, a suburb of Denver where, apparently, the DSL services are hubbed)

traceroute to hectors.dyndns.host (71.196.204.25), 30 hops max, 60 byte packets
 1  Heimdall.mystica.lan (192.168.x.x)  0.163 ms  0.185 ms  0.220 ms
 2  192.168.0.1 (192.168.0.1)  0.870 ms  0.884 ms  0.904 ms
 3  hlrn-dsl-gw91-156.hlrn.qwest.net (207.225.112.156)  21.901 ms  22.038 ms  22.315 ms
 4  hlrn-agw1.inet.qwest.net (71.217.190.209)  22.325 ms  22.900 ms  22.911 ms
 5  dap-brdr-04.inet.qwest.net (67.14.2.162)  46.566 ms  47.289 ms  46.719 ms
 6  ix-0-1-0-0.tcore2.DT8-Dallas.as6453.net (206.82.142.1)  48.026 ms  46.924 ms  45.620 ms
 7  206.82.142.22 (206.82.142.22)  47.866 ms  47.925 ms 206.82.142.18 (206.82.142.18)  47.874 ms
 8  pos-2-2-0-0-cr01.dallas.tx.ibone.comcast.net (68.86.86.149)  46.524 ms  46.374 ms  45.685 ms
 9  pos-2-10-0-0-cr01.denver.co.ibone.comcast.net (68.86.87.185)  60.631 ms  60.842 ms  60.801 ms
10  * * *
11  *^C
 

This traceroute is very unique, because of where it dies. It does not die on the 'ar' hop. Not on the 'ur' hop and not on the 'cdn' hop. It dies on the 'cr' hop, presumably meaning 'core router'. The IBONE HOP. Baffled at this, I tried from my cellphone.

traceroute to 71.196.204.25 (71.196.204.25), 30 hops max, 38 byte packets
1  10.170.205.48 (10.170.205.48)  701.783 ms  116.394 ms  118.317 ms
 
[10.x addresses in tmobile's internal network were removed for space ]
 
9  10.176.188.190 (10.176.188.190)  118.408 ms  96.497 ms  119.812 ms
10  te-9-3.car2.Denver1.Level3.net (4.53.14.73)  119.782 ms  96.710 ms  119.690 ms
11  ae-2-52.edge3.Denver1.Level3.net (4.69.147.104)  119.812 ms  135.681 ms  110.535 ms
12  COMCAST-IP.edge3.Denver1.Level3.net (4.79.82.54)  119.721 ms  COMCAST-IP.edge3.Denver1.Level3.net (4.28.20.6)  100.037 ms  COMCAST-IP.edge3.Denver1.Level3.net (4.79.82.54)  96.680 ms
13  *  *  *
14  *^C
/ #
 

No Ibone, but, again, the hop just before the Denver Comcast core router, this time on Level3's side.

Stumped at traces being blocked I had someone presently on Comcast try:

traceroute to 71.196.204.25 (71.196.204.25), 30 hops max, 60 byte packets
 1  10.120.150.240 (10.120.150.240)  0.663 ms  1.440 ms  1.821 ms
 2  96.152.122.1 (96.152.122.1)  24.303 ms  25.202 ms  27.116 ms
 3  te-9-2-ur01.aurora.co.denver.comcast.net (68.86.105.81)  16.506 ms  16.629 ms  16.737 ms
 4  * * *
 5  * * *
 

So this seems to be a largish routing failure. It never reaches the CDN hop that a proper trace shows:

 9  pos-2-15-0-0-cr01.denver.co.ibone.comcast.net (68.86.85.173)  60.671 ms  58.924 ms  60.328 ms
10  pos-0-13-0-0-ar02.aurora.co.denver.comcast.net (68.86.95.246)  64.588 ms  60.150 ms  64.050 ms
11  te-8-3-ur01.aurora.co.denver.comcast.net (68.86.103.38)  60.068 ms  59.640 ms  59.886 ms
12  te-17-10-cdn12.aurora.co.denver.comcast.net (68.85.221.190)  72.856 ms  61.788 ms  79.876 ms
13  c-71-196-255-xxx.hsd1.co.comcast.net (71.196.255.xxx)  68.618 ms  70.031 ms  71.013 ms
 

So I meet him at home after he gets out of work, and I find his router is PROPERLY DHCPING THE IP address. IT HAS PROPER DNS SERVERS. IT HAS THE PROPER GATEWAY. IT SIMPLY CANNOT ROUTE.

I try the laptop directly to the modem? works perfectly fine. Of course, its a different IP, as the IPs are based on MAC address and assigned by DHCP.

I CHANGE THE ROUTERS MAC ADDRESS to DHCP a new IP; IT ROUTES FINE!

How did you MESS THIS UP SO BAD COMCAST?!

Without me actively understanding IP networks, my friend would STILL be without internet.

I called in a trouble ticket and explained to the first level tech that the packets were not routing at all, gave the IP inside the subnet that was affected... I just hope that said trouble ticket gets to the right techs, else a few more people are likely without internet right now and DON'T KNOW WHY.

--
:wq!
Wrote dslreports_forum_post
mike@thor:~>_


Mike Wolf

join:2009-05-24
Beachwood, NJ
kudos:4
exact same thing happened to me this morning while i was in a skype group audio call. my logs match up exactly to yours line by line. Mine happend around midnight but it came back after about five minutes. maybe they broke something important in his area? maybe his modem got fried with a bad firmware update?

mystica

join:2001-10-30
Denver, CO
If your line goes out, but comes back within 5 minutes, thats a normal maintenance window.

If your line goes out, and stays out the entire day, and the entire subnet you are on cannot route *anywhere* FROM *anywhere* this is a wholly different problem.

Someone typoed a routing table update. The modem never flashed any new firmware; The internet works fine once I changed the MAC address on the router to force a new DHCP IP.

--
:wq!

Wrote dslreports_forum_post

mike@thor:~>_


Mike Wolf

join:2009-05-24
Beachwood, NJ
kudos:4
possibly but its unsual for that to happen and no one to notice in the company. broken hardware on comcast's end comes to mind, or something physical thats wrong.

mystica

join:2001-10-30
Denver, CO
Definitely unusual, but, it did happen, and its definitely on Comcast's end. If a packet to a subnet dies at the core router, yeah, the routing table is messed up.
--
:wq!
Wrote dslreports_forum_post
mike@thor:~>_


Mike Wolf

join:2009-05-24
Beachwood, NJ
kudos:4
well lets just hope it gets fixed soon before something else happens


netcool
Premium,VIP
join:2008-11-05
Englewood, CO
kudos:108
reply to mystica
said by mystica:

I called in a trouble ticket and explained to the first level tech that the packets were not routing at all, gave the IP inside the subnet that was affected... I just hope that said trouble ticket gets to the right techs, else a few more people are likely without internet right now and DON'T KNOW WHY.

Should be fixed now.

mystica

join:2001-10-30
Denver, CO
said by netcool:

Should be fixed now.

Thanks for this. I'll keep my custom MAC in the router until I can get over there again and reboot the modem to clear its MAC cache.
--
:wq!

Wrote dslreports_forum_post

mike@thor:~>_


Reggie

@myvzw.com
reply to mystica
Well I am also experiancing something similar that occurred at the same time. I have a Windows Server 2003 that I am using as my router/gateway with comast. When I have just that machine connected to the internet it works great. As soon as I plug in the other NIC and add my LAN to the network I have internet across the LAN for about 5-10 min then I lose it. At that point I can only ping Comcast's gateway but no further. The tracert also dies at the gateway. After restarting the server box with the LAN disconnected it works again. Reconnect LAN and it dies. Is there possibly something in this upgrade that would be stopping me. I dont understand why, because I am just using a compter as a router in place of a typical one. Any suggestions would be great.


Mike Wolf

join:2009-05-24
Beachwood, NJ
kudos:4
That sort of sounds more like a business setup rather then a residental setup with a router, so maybe something is detecting the server OS and raising a red flag? I know that Comcast has been working hard at eliminating businesses using residential service.


beachintech
There's sand in my tool bag
Premium
join:2008-01-06
kudos:5
reply to Reggie
The problem lies with your router, and not with comcast. If your connection works with just the main router box connected, it's good to go. You adding a nic with your LAN connected won't impact the first connection unless it breaks the setup on the OS.
--
Ex-Tech at the Beach.
I speak for myself, not my former employer.


Al_S

@comcastbusiness.net
Did anybody experience any problems over this past weekend in Chicago? I lost internet on Saturday and per Comcast it's my modem. I replaced it and it still didn't want to work. Had to call them a few times and now it's working but my router doesn't work. Replaced it with an old one I had and it kind of works. My PC in the bedroom doesn't want to connect. I thought I got a power spike or something but another coworker had a similar problem.