This was a major, MAJOR outage. And based on what the phone rep said, I'm betting it was maintenance. Almost 3 hours of downtime. Here's my timestamp log:
0123 -- Lost all IP access, but cable modem still has sync
0124 -- Modem interface (via web) no longer works... odd
0126 -- Power-cycled modem. Got sync, but no DHCP/TFTP. Appears to be 100% identical to yesterday's situation
0128 -- Now able to access web interface on modem. DHCP discover sent, no offer received... yep, looks like a repeat of yesterday
0132 -- Called Comcast; got message saying they were experiencing longer than average hold times. Hung up and will call back later
0133 -- Modem rebooted itself. Again, got sync but no DHCP/TFTP
0139 -- Modem rebooted itself again; behaviour same as before
0142 -- Called Comcast, spoke to rep. Rep confirmed "outage in my area". I asked if it was maintenance, and was put on hold. After a few minutes, rep came back on and said he wasn't able to confirm that it's maintenance, but that someone did put an end/completion time of the outage into their system. End time set approximately 4 hours from now (0545 PST)
0357 -- Still down. Sometime between 0145 and now, behaviour changed. Now modem "Receive" LED is lit, but "Send" blinks constantly. Modem status shows "In Progress" for "Upstream Ranging". Modem log says "Init RANGING Critical Ranging Request (17) Retried exhausted" and "T3 time-out (US 7)", and sometimes US 8 or US 9
0400 -- Called Comcast; got automated message "we are experiencing an outage in your area"
0403 -- Modem suddenly synced up, able to access Internet! Periodic traceroute confirms.
0404 -- Reset modem to factory defaults and rebooted modem. Signal levels didn't look quite right (upstream power level was ~4-5dB higher than previous)
0407 -- Modem finally synced up. There are errors in the log about "No Ranging Response Received, T3 time-out (US 7 and US 9)", but it did sync up. Upstream power level is 42.9dB, while prior to all of this insanity, it was 39.7dB. I suppose that's "better" in the sense that it's more balanced (between +35dB and +52dB).
My signal levels before:
Thu Jan 29 23:00:00 PST 2009
Downstream SNR: 38.0 dB
Downstream Power: 10.4 dBmV
Upstream Power: 39.7 dBmV
And after this whole fiasco:
Fri Jan 30 04:15:31 PST 2009
Downstream SNR: 38.1 dB
Downstream Power: 10.4 dBmV
Upstream Power: 42.9 dBmV
Also, something very interesting: something has indeed changed in the IP network routing as well. I'm on a completely different netblock, and traceroutes look very, very different (including increased hop count). Previously I was on the 69.181.141.0/24 netblock (more or less), and now I'm on 67.169.182.0/24.
=== Thu Jan 29 23:58:00 PST 2009 (1233302280)
HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst
1. 192.168.1.1 0.0% 30 30 0.6 0.4 0.3 0.6
2. ??? 100.0 30 0 0.0 0.0 0.0 0.0
3. 68.86.142.113 0.0% 30 30 9.0 11.6 8.2 20.6
4. 68.87.226.185 0.0% 30 30 10.3 12.9 9.5 24.3
5. 68.86.90.153 0.0% 30 30 11.8 15.6 11.2 31.1
6. 4.71.118.5 0.0% 30 30 13.7 18.6 10.5 80.6
7. 4.68.18.3 0.0% 30 30 13.3 39.7 10.3 210.3
8. 4.79.219.106 0.0% 30 30 13.3 15.1 11.1 29.9
9. 209.128.116.111 0.0% 30 30 16.0 13.5 10.6 16.4
10. 72.20.106.125 0.0% 30 30 13.3 15.2 11.7 24.7
=== END
After:
=== Fri Jan 30 04:19:00 PST 2009 (1233317940)
HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst
1. 192.168.1.1 0.0% 30 30 0.4 0.4 0.3 0.6
2. ??? 100.0 30 0 0.0 0.0 0.0 0.0
3. 68.85.191.37 0.0% 30 30 7.0 8.2 5.9 10.7
4. 68.87.226.86 0.0% 30 30 9.3 8.5 5.9 21.1
5. 68.87.226.221 0.0% 30 30 8.8 10.3 7.7 22.0
6. 68.86.90.141 0.0% 30 30 10.8 11.8 9.8 14.3
7. 68.86.85.78 0.0% 30 30 16.3 14.9 12.7 18.0
8. 4.71.118.29 0.0% 30 30 13.2 21.5 12.2 122.5
9. 4.68.18.3 0.0% 30 30 13.9 36.2 12.7 151.6
10. 4.79.219.106 0.0% 30 30 13.7 14.6 12.9 17.4
11. 209.128.116.111 0.0% 30 30 13.9 15.2 12.9 18.4
12. 72.20.106.125 0.0% 30 30 14.1 15.0 13.1 17.9
=== END
And here is a normal traceroute, with DNS resolution:
$ traceroute -P icmp -q 1 -f 3 anubis.sc1.parodius.com
traceroute to anubis.sc1.parodius.com (72.20.106.125), 64 hops max, 72 byte packets
3 68.85.191.37 (68.85.191.37) 6.713 ms
4 ml-1-sr01.coliseumway.ca.sfba.comcast.net (68.87.226.86) 7.167 ms
5 te-0-3-0-0-ar01.oakland.ca.sfba.comcast.net (68.87.226.221) 8.781 ms
6 pos-0-2-0-0-cr01.sacramento.ca.ibone.comcast.net (68.86.90.141) 11.406 ms
7 pos-0-7-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.85.78) 14.533 ms
8 xe-10-0-0.edge1.SanJose1.Level3.net (4.71.118.29) 13.784 ms
9 ae-11-69.car1.SanJose1.Level3.net (4.68.18.3) 14.520 ms
10 BAY-AREA-IN.car1.SanJose1.Level3.net (4.79.219.106) 14.291 ms
11 sr2sc1-ge0.bayarea.net (209.128.116.111) 14.536 ms
12 anubis.sc1.parodius.com (72.20.106.125) 14.285 ms
What the heck is "coliseumway"? And why isn't colosseum spelled correctly? EDIT: As it turns out, Coliseum Way is some street in Oakland. I wonder why it's spelled that way? Bizarre.
Why are my packets going to Oakland, then supposedly to some device in Sacramento, then back down to San Jose? (The destination IP is located in Sunnyvale/Santa Clara)
Very fishy.