how-to block ads
|
koitsu
Premium
join:2002-07-16
Mountain View, CA
4 edits | [Connectivity] Bay Area outage (Mountain View), 01/30
This was a major, MAJOR outage. And based on what the phone rep said, I'm betting it was maintenance. Almost 3 hours of downtime. Here's my timestamp log:
0123 -- Lost all IP access, but cable modem still has sync
0124 -- Modem interface (via web) no longer works... odd
0126 -- Power-cycled modem. Got sync, but no DHCP/TFTP. Appears to be 100% identical to yesterday's situation
0128 -- Now able to access web interface on modem. DHCP discover sent, no offer received... yep, looks like a repeat of yesterday
0132 -- Called Comcast; got message saying they were experiencing longer than average hold times. Hung up and will call back later
0133 -- Modem rebooted itself. Again, got sync but no DHCP/TFTP
0139 -- Modem rebooted itself again; behaviour same as before
0142 -- Called Comcast, spoke to rep. Rep confirmed "outage in my area". I asked if it was maintenance, and was put on hold. After a few minutes, rep came back on and said he wasn't able to confirm that it's maintenance, but that someone did put an end/completion time of the outage into their system. End time set approximately 4 hours from now (0545 PST)
0357 -- Still down. Sometime between 0145 and now, behaviour changed. Now modem "Receive" LED is lit, but "Send" blinks constantly. Modem status shows "In Progress" for "Upstream Ranging". Modem log says "Init RANGING Critical Ranging Request (17) Retried exhausted" and "T3 time-out (US 7)", and sometimes US 8 or US 9
0400 -- Called Comcast; got automated message "we are experiencing an outage in your area"
0403 -- Modem suddenly synced up, able to access Internet! Periodic traceroute confirms.
0404 -- Reset modem to factory defaults and rebooted modem. Signal levels didn't look quite right (upstream power level was ~4-5dB higher than previous)
0407 -- Modem finally synced up. There are errors in the log about "No Ranging Response Received, T3 time-out (US 7 and US 9)", but it did sync up. Upstream power level is 42.9dB, while prior to all of this insanity, it was 39.7dB. I suppose that's "better" in the sense that it's more balanced (between +35dB and +52dB).
My signal levels before:
And after this whole fiasco:
Also, something very interesting: something has indeed changed in the IP network routing as well. I'm on a completely different netblock, and traceroutes look very, very different (including increased hop count). Previously I was on the 69.181.141.0/24 netblock (more or less), and now I'm on 67.169.182.0/24.
After:
And here is a normal traceroute, with DNS resolution:
What the heck is "coliseumway"? And why isn't colosseum spelled correctly? EDIT: As it turns out, Coliseum Way is some street in Oakland. I wonder why it's spelled that way? Bizarre.
Why are my packets going to Oakland, then supposedly to some device in Sacramento, then back down to San Jose? (The destination IP is located in Sunnyvale/Santa Clara)
Very fishy.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer. | |
devnuller
join:2006-06-10
Hollis, NH
1 edit | said by koitsu  :This was a major, MAJOR outage. And based on what the phone rep said, I'm betting it was maintenance. Almost 3 hours of downtime. Here's my timestamp log: Must have been major work. Based on your timestamps it happened in a large portion of time windows used for occational planned maintenance. Changes will occasionally happen.
said by koitsu :Why are my packets going to Oakland, then supposedly to some device in Sacramento, then back down to San Jose? (The destination IP is located in Sunnyvale/Santa Clara) Very fishy. Internet routes do not always follow driving paths. The way routing tables work are very different than mapquest. From a network speed perspective, this change in path doesn't look like it added any material time.
Not very fishy.... very normal | |
koitsu
Premium
join:2002-07-16
Mountain View, CA
| said by devnuller :said by koitsu :Why are my packets going to Oakland, then supposedly to some device in Sacramento, then back down to San Jose? (The destination IP is located in Sunnyvale/Santa Clara) Very fishy. Internet routes do not always follow driving paths. The way routing tables work are very different than mapquest. From a network speed perspective, this change in path doesn't look like it added any material time. Not very fishy.... very normal Oh believe me, I'm quite aware of that.  But this change supposedly indicates traffic is being sent almost a hundred miles away, only to have to come right back. I guarantee BGP route preferencing can address this -- I do this all the time at my day job.
But my question isn't "why are my IP packets supposedly going here?" My concern is why on a physical transport level, someone would engineer things that way. There's a much greater chance of interruption in service (think: fibre cut) based on increased cabling distance between Oakland and Sacramento than there is between Oakland and Santa Clara. There's a ~2ms increase in latency as a result (between hops #5 and #6 in my mtr, not the single-query traceroute), but I'm not crying over that.
There's also the chance it's a "DNS thing" as a result of tonight's work; possibly some netblocks were reassigned, and no one's taken the time to update in-addr.arpa DNS yet.
Either way, I'm hoping Comcast can shed some light on what all went on tonight. If this is how things are going to be routed, that's fine -- it's just a very large change compared to how things were previously. | |
devnuller
join:2006-06-10
Hollis, NH
2 edits | Yes it could be a delay in geo-INADDR post change.
There could be many reasons for this, but if the 0.2ms increase isn't causing a problem, the level of detailed tweaking you are talking about is very different for a small BGP connection compared to Internet backbone routing. Changes in backbone routing tables are not small quick fixes and can cause other major issues. | |
bayareanon
@pacbell.net
| reply to koitsu
I also had another outage last night. If they are doing maint. then they need to inform their service people because none of the ones I've spoken to knew anything about it.
Also make sure you call and complain EVERY time it happens. If they start getting lots of complaints then maybe they will do something about it, instead of blow it off as a local wiring issue (like they did with me - had my whole house re-wired but the outages continue). | |
-
|
