dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
559
share rss forum feed


xymox1
Premium
join:2008-05-20
Phoenix, AZ

Rebooted modems all over phx.

I monitor systems at residential homes where I install gear. I monitor systems with automated pings from Alertra.com. Been doing this for a very long time. It allows me to know if a client has a issue right away.

So over the last 3hrs. 4pm-7pm I saw every system I monitor in phoenix on Cox cable go down and then back up in a minute. I have clients all over the valley and slowly they all rebooted modems, or something in the plant, over the last 3 hrs. None of my DSL clients or clients with entriprize level connections. ONLY Cox cable clients.

I did some remote checks and ran speedtest on some systems and speeds did not increase. Modem firmware looks the same. Just some strange things happening insdie Cox tonight.

Maybe they patched Heartbleed on thier systems.



liht
Acryllicht

join:2000-07-11
Paradise Valley, AZ

That's weird. I got notice my SIP trunks were down, which pretty much never happens.
Wonder what's going on.
--
werd



Hard Harry

join:2010-10-19
Narragansett, RI
kudos:2
Reviews:
·Cox HSI
reply to xymox1

said by xymox1:

Maybe they patched Heartbleed on thier systems.

Don't think so. Most of Cox system never used OpenSSL to start with and any security patch wouldn't effect general connectivity. What exactly are you monitoring? Packets from clients to you? Client to Alertra? Maybe something happened at the CMTS level? Rebooting a rack or something, though im sure there are engineers who could take a much more educated guess.


xymox1
Premium
join:2008-05-20
Phoenix, AZ
reply to xymox1

Heartbleed is OpenSSL 1.0.1 and its used in many many ways. Cisco has advisories on it and what systems it effects. I have not looked up if it effects any CMTS systems, but with all the switches and other network, support and control gear involved at Cox from the plant all the way up to the control center in atlanta im sure they have vulnerable systems.

It would be a OS upgrade and that might actually cause some reboot of even a CMTS ?

Something happened. Different clients were effected at different times all over the valley. Only once per client area. It was like max 2 minutes.

Monitoring. I have Alertra send a TCP connect to a specific port which is port forwarded thru the router to a device. It does this once a minute. If the TCP connect fails Alertra alerts me. This then monitors the clients modem, connection, router usally a switch or 2 and the device. If there is any issue then I get a alert.

I also have the router run a script and check for outbound connectivity every 5 mins by pinging a few very solid IP's on the net and if there is a issue to email me a alert.

I also have the router ping every critical device on the clients internal network once every 5 minutes and report any issues.

I always know when there is a issue with a clients system before they do. Or at least at the same time they do.

None of my clients noticed this because it was so quick.



Hard Harry

join:2010-10-19
Narragansett, RI
kudos:2

Does Alerta record the reason for failure? Like unable to route? Or unable to resolve?

Also, did you notice if any of the public IP's change after the hiccup?



CoxDowntime

@cox.net
reply to xymox1

Yeah noticed it here in Deer Valley. DNS was failing to COX and 4.2.2.2 according to tcpdump on my OpenWRT router. Power-cycled my 3010 modem once and it failed to reconnect on first try. Power-cycled again and it resync'd first time. Connectivity was back up then. Downtime about 5 minutes for me. Public IP did not change. Speedtest.net right after to San Diego was showing full 195/40 on Ultimate. Not sure why they didn't do it at night during regular downtime. Probably wanted the A/day team on board...


Guzzler

join:2002-09-24
Tempe, AZ
kudos:1
reply to xymox1

Add cable box to the list of rebooted devices.



xymox1
Premium
join:2008-05-20
Phoenix, AZ
reply to xymox1

Back when they changed to 8x4 and turned on 200/40 I saw this sort of thing. They were doing stuff in prep for the upgrade I guess.

So, maybe after Time-Warner turned on 300 mb/s service in LA, maybe they felt the need to keep up and we might see yet another speed boost.

I cant wait for DOCSIS 3.1 and 1Gb-8Gb/s



xymox1
Premium
join:2008-05-20
Phoenix, AZ
reply to xymox1

Alertra,, im not sure if I can see what the failure was. They do have logs of the checks..

Im currently on modem firmware d3000-v302r125572-130314a-COX With a DPC3010 which has a date of Mar 14 13:17:39 2013

I dont remember what firmware I had before. Maybe they updated firmware in the modems ?



odog
Cable Centric Vendor Biased
Premium,VIP
join:2001-08-05
Atlanta, GA
kudos:11
Reviews:
·Comcast

1 recommendation

reply to xymox1

We don't tend to do things during the day, unless it is an emergency.

What do the modem logs say? They should give an indication if this was a modem to cmts disruption or something else.

At first thought it sounds like a route ripple, and alerta is now taking a different route into our network. As the networks swung between routers in the routing table you had what looked like a disconnect, but likely only affected specific routes/networks.



xymox1
Premium
join:2008-05-20
Phoenix, AZ
reply to xymox1

Hi Odog.. Ahhh the Cox Master !

I cant view the DPC3010 log. Wish I could. Motorola has a log a normal user could look at but alas not in the DPC..

It was a strange event. Ive been using Alertra for like 10 years and I have only seen this like one other time.

It could be so many things. Well, you would know if it was in your system, so, it sounds like it wasnt. And I was hoping it was a prelude to a speed boost



Anonguy

@cox.net

Cox had to do an Emergency CMTS patch and reboot to fix a No Dial Tone issue.