dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
993
ChadDa3mon
join:2015-04-16
Warren, MI

ChadDa3mon

Member

Weekly packet loss issue - Detroit, MI

So for months I have had a weekly issue with packet loss. Every Thursday, starting at around 8:30 AM, until 9:30 AM, I have around 50% packet loss. The issue resolves itself after that, and the rest of the week everything is fine.

At first I thought maybe it was someone on my end, saturating my measly 5M upload cap. Some scheduled task I wasn't thinking of. But I've shut down everything, and just used various laptops to test, and the problem is still there.

At this point I am suspecting some issue with WoW outside of my cable modem. I have data charting this issue back for months now and I'm getting fed up with it. I work from home, as a level 3 network engineer, and it's a joke among my peers that I'm suffering this after so long.

So, I finally tried to call WoW today and see if I could speak to someone more advanced than their first line techs, the ones that tell me to just reboot my cable modem. No luck.

So next I called their cancellation department and said I would be canceling my service for Comcast if they can't get me someone competent to talk to - no luck.

Both of them said the only option was to send a service tech on site to my house, which I said would be useless, mainly because there's no way they can promise me a tech will show up between 8:30 and 9:00 AM, when the problem is most prevalent. Plus, as the speeds return to normal shortly after 9 AM, I highly doubt there's a hardware issue in my house.

I don't want to switch to Comcast, I despise Comcast, but they're my only other option in this area. I tell all of my friends here to sign up for WoW, but I'm left now wondering if I can even suggest them after this problem.

Is there any trick you guys may have to speak to someone a bit more knowledgable who can at least take a look at the data I have and investigate things? I'd be fine if they came back without finding anything, just so long as I had a feeling they understood the problem and at least tried to make sure there wasn't a problem on their end.

Thanks in advance,
A very frustrated WoW customer

bdnhsv
join:2012-01-20
Huntsville, AL

bdnhsv

Member

Are you seeing any T3/4 timeouts in the modem's log during the timeframes you're experiencing the packet loss? You might share some of your graphs with the WOW employees here on the site, and ask them to confirm if they are seeing your modem flapping in their CMTS.
ChadDa3mon
join:2015-04-16
Warren, MI

ChadDa3mon

Member

Not sure how to get to the modem logs for this, I got my own surfboard instead of the equipment WoW provided, which was much more unstable.

With my own surfboard modem, speeds and service are generally great, besides this Thursday morning issue :/

aes128
join:2003-12-19
Warren, MI

aes128

Member

said by ChadDa3mon:

Not sure how to get to the modem logs for this, I got my own surfboard instead of the equipment WoW provided, which was much more unstable.

With my own surfboard modem, speeds and service are generally great, besides this Thursday morning issue :/

Generally you would go here:

»192.168.100.1/cmLogs.htm
ChadDa3mon
join:2015-04-16
Warren, MI

ChadDa3mon

Member

Thanks AES, that link did work, I would have never guessed that.

I'll look at the logs once the issue repeats and post any findings.

I don't see any mention of "graphs" which was mentioned earlier, or the option to find such a thing. Just Status | Signal | Addresses | Configuration | Logs | Open Source | Help
ChadDa3mon

ChadDa3mon

Member

Bringing this back up. The issue strangely enough quit when I first posted this. The past few weeks it has started happening again. I'm getting around 40% packet loss. I looked at my modem logs and put the results here

»pastebin.com/FFUgZ8VW

Lots of errors and ugly looking information, I just have no idea what any of it means.
ChadDa3mon

ChadDa3mon

Member

Signal levels during high packet loss

»postimg.org/image/ikuvig9wh/
wkcole
Premium Member
join:2007-11-08
Eastpointe, MI

wkcole

Premium Member

Top guesses:
1. Loose coax connection somewhere between your CM and the drop to your home, possibly allowing moisture in.
2. Squirrels or other vermin have gnawed on a cable.
3. Some other device on your cable (TV, splitter,amplifier, etc.) is misbehaving and injecting noise

The bottom 3 channels (13-15) that are showing a lot of errors are relatively close in frequency and the signal & power values for 14 & 15 are borderline in that snapshot. You should see steady upstream 36-40dB SNR numbers and power levels very close to zero dBmV. As I understand it, the 9dBmV power level on channel 14 is a sign that something is boosting power on that frequency, possibly to compensate for detected noise. The error rates on those 3 channels are absolutely a sign of a problem in the physical layer. The way cable Internet links work makes that entirely the cable company's realm, which is why cable modems don't provide users many knobs and buttons.

If you can't see/feel an obvious problem in your accessible wiring, such as a coax connector that can be tightened with your fingers or a chewed line, this is probably something you need to have WOW diagnose & fix. The signal problems on those 3 channels could be cause by something very close to you or anywhere between your cable modem and the WOW end of the coax, and there is no way (by design) for you to see any physical layer details past the cable modem.
wowMike
join:2013-12-02
Allen Park, MI

wowMike to ChadDa3mon

Member

to ChadDa3mon
So I've looked into your modem and I see what causing your problem. There is about 10 dB of power randomly getting injected. Seems to happen throughout the night but around 9am it suddenly drops off.
I'll forward this on to our maintenance team to check everything in your area, perhaps there is a cracked cable that's allowing some outside signal in.

Wkcole was pretty spot on with their guesses.
ChadDa3mon
join:2015-04-16
Warren, MI

ChadDa3mon to wkcole

Member

to wkcole
said by wkcole:

Top guesses:
1. Loose coax connection somewhere between your CM and the drop to your home, possibly allowing moisture in.
2. Squirrels or other vermin have gnawed on a cable.
3. Some other device on your cable (TV, splitter,amplifier, etc.) is misbehaving and injecting noise

The bottom 3 channels (13-15) that are showing a lot of errors are relatively close in frequency and the signal & power values for 14 & 15 are borderline in that snapshot. You should see steady upstream 36-40dB SNR numbers and power levels very close to zero dBmV. As I understand it, the 9dBmV power level on channel 14 is a sign that something is boosting power on that frequency, possibly to compensate for detected noise. The error rates on those 3 channels are absolutely a sign of a problem in the physical layer. The way cable Internet links work makes that entirely the cable company's realm, which is why cable modems don't provide users many knobs and buttons.

If you can't see/feel an obvious problem in your accessible wiring, such as a coax connector that can be tightened with your fingers or a chewed line, this is probably something you need to have WOW diagnose & fix. The signal problems on those 3 channels could be cause by something very close to you or anywhere between your cable modem and the WOW end of the coax, and there is no way (by design) for you to see any physical layer details past the cable modem.

Thanks for such a detailed write up. My question is this: If this was a physical connection issue, why does it only seem to occur Thursday mornings? I don't do anything before it starts, and I don't do anything to resolve it, it clears up on its own.

I guess my fear is if I get a wow tech out here and the issue isn't occuring at the time they visit (which is likely) they won't find any problem.
ChadDa3mon

ChadDa3mon to wowMike

Member

to wowMike
Thank you wowMike.
ChadDa3mon

ChadDa3mon

Member

Below is a section of the log file from the *10 minute ping test I run now to track this. This is why I'm afraid it will be so hard to get a tech to find the problem, it only happens during a 2 or 3 hour window, then seems to resolve itself.

Output below is Time, packet loss%, IP I test with. Every 10 minutes I do a small ping flood on 3 IPs, a backbone DNS server, my internal router, and the WOW gateway IP.

Thu Jul 23 08:20:01 EDT 2015,0%,4.2.2.2
Thu Jul 23 08:20:07 EDT 2015,0%,192.168.168.1
Thu Jul 23 08:20:08 EDT 2015,0%,24.192.208.1
Thu Jul 23 08:30:01 EDT 2015,0%,4.2.2.2
Thu Jul 23 08:30:08 EDT 2015,0%,192.168.168.1
Thu Jul 23 08:30:08 EDT 2015,0%,24.192.208.1
Thu Jul 23 08:40:01 EDT 2015,39%,4.2.2.2
Thu Jul 23 08:40:08 EDT 2015,0%,192.168.168.1
Thu Jul 23 08:40:08 EDT 2015,38%,24.192.208.1
Thu Jul 23 08:50:01 EDT 2015,2%,4.2.2.2
Thu Jul 23 08:50:08 EDT 2015,0%,192.168.168.1
Thu Jul 23 08:50:08 EDT 2015,7%,24.192.208.1
Thu Jul 23 09:00:01 EDT 2015,39%,4.2.2.2
Thu Jul 23 09:00:08 EDT 2015,0%,192.168.168.1
Thu Jul 23 09:00:09 EDT 2015,37%,24.192.208.1
Thu Jul 23 09:10:01 EDT 2015,0%,4.2.2.2
Thu Jul 23 09:10:08 EDT 2015,0%,192.168.168.1
Thu Jul 23 09:10:08 EDT 2015,0%,24.192.208.1
Thu Jul 23 09:20:01 EDT 2015,0%,4.2.2.2
Thu Jul 23 09:20:09 EDT 2015,0%,192.168.168.1
Thu Jul 23 09:20:09 EDT 2015,0%,24.192.208.1
Thu Jul 23 09:30:02 EDT 2015,36%,4.2.2.2
Thu Jul 23 09:30:09 EDT 2015,0%,192.168.168.1
Thu Jul 23 09:30:09 EDT 2015,46%,24.192.208.1
Thu Jul 23 09:40:02 EDT 2015,0%,4.2.2.2
Thu Jul 23 09:40:09 EDT 2015,0%,192.168.168.1
Thu Jul 23 09:40:09 EDT 2015,0%,24.192.208.1
Thu Jul 23 09:50:02 EDT 2015,0%,4.2.2.2
Thu Jul 23 09:50:10 EDT 2015,0%,192.168.168.1
Thu Jul 23 09:50:10 EDT 2015,0%,24.192.208.1
Thu Jul 23 10:00:03 EDT 2015,0%,4.2.2.2
Thu Jul 23 10:00:10 EDT 2015,0%,192.168.168.1
Thu Jul 23 10:00:10 EDT 2015,0%,24.192.208.1
Thu Jul 23 10:10:01 EDT 2015,0%,4.2.2.2
Thu Jul 23 10:10:08 EDT 2015,0%,192.168.168.1
Thu Jul 23 10:10:09 EDT 2015,0%,24.192.208.1
wowMike
join:2013-12-02
Allen Park, MI

wowMike

Member

So a part of the plant was replaced in your area on Friday and so far I'm not seeing the massive swings in power. Your receive level is a little low but I'm not sure if your modem is off the first splitter or not as that would make the difference. Otherwise your levels are consistent with a +/- 2 variance which is pretty normal.
When issues are seen at a given time of day it's generally an environment issue that's causing some sort of RFI/EMI noise. If the plant is damaged slightly such as the sheath to a cable is cracked or a squirrel has chewed part of the insulation off, then that noise can leak into the plant causing a problem. Similarly the ground in some housing might be lose from vibration that would normally block some noise, or the isolation partially failing letting a neighbors bad TV bleed noise in locally.
You only saw the issue on Thursday while I was seeing a nightly problem, we'll have to wait and see if they were resolved together or if the problem I saw was masking something else. We'll keep an eye on it and if you have an issue again on Thursday please let me know, likewise if I see something pop up in the next week I'll let you know here.
ChadDa3mon
join:2015-04-16
Warren, MI

ChadDa3mon

Member

said by wowMike:

So a part of the plant was replaced in your area on Friday and so far I'm not seeing the massive swings in power. Your receive level is a little low but I'm not sure if your modem is off the first splitter or not as that would make the difference. Otherwise your levels are consistent with a +/- 2 variance which is pretty normal.
When issues are seen at a given time of day it's generally an environment issue that's causing some sort of RFI/EMI noise. If the plant is damaged slightly such as the sheath to a cable is cracked or a squirrel has chewed part of the insulation off, then that noise can leak into the plant causing a problem. Similarly the ground in some housing might be lose from vibration that would normally block some noise, or the isolation partially failing letting a neighbors bad TV bleed noise in locally.
You only saw the issue on Thursday while I was seeing a nightly problem, we'll have to wait and see if they were resolved together or if the problem I saw was masking something else. We'll keep an eye on it and if you have an issue again on Thursday please let me know, likewise if I see something pop up in the next week I'll let you know here.

Thank you thank you thank you. I can't tell you how happy I am to finally have someone looking into this instead of asking me if I tried to reboot my modem. I will keep an eye on my internet and let you know if I experience any further issues.

Seriously, thank you.
ChadDa3mon

ChadDa3mon

Member

Been having some latency issues for a while now it seems. Only 5% packet loss, but horrible ping times.

$ sudo ping -f -c 500 24.192.208.1
PING 24.192.208.1 (24.192.208.1) 56(84) bytes of data.
.......
--- 24.192.208.1 ping statistics ---
500 packets transmitted, 493 received, 1% packet loss, time 6756ms
rtt min/avg/max/mdev = 343.913/468.059/616.399/57.416 ms, pipe 48, ipg/ewma 13.539/444.016 ms

Looked at the logs on my cable modem, nothing jumps out, this is the only entry from today.

DHCP RENEW WARNING - Field invalid in response v4 option;CM-MAC=f8:0b:be:de:56:2e;CMTS-MAC=00:01:5c:63:5c:66;CM-QOS=1.1;CM-VER=3.0;

Current signal values can be seen here

»postimg.org/image/r455zu2rl/

Hard to say when things started, looking at my ping results, I can see the heaviest packet loss here

Tue Jul 28 20:30:01 EDT 2015,11%,4.2.2.2
Tue Jul 28 20:30:09 EDT 2015,0%,192.168.168.1
Tue Jul 28 20:30:09 EDT 2015,6%,24.192.208.1
Tue Jul 28 20:40:01 EDT 2015,3%,4.2.2.2
Tue Jul 28 20:40:09 EDT 2015,0%,192.168.168.1
Tue Jul 28 20:40:10 EDT 2015,5%,24.192.208.1
Tue Jul 28 20:50:01 EDT 2015,7%,4.2.2.2
Tue Jul 28 20:50:10 EDT 2015,0%,192.168.168.1
Tue Jul 28 20:50:10 EDT 2015,13%,24.192.208.1
ChadDa3mon

ChadDa3mon

Member

Latency is spiking again.

# ping -f -c 500 24.192.208.1
PING 24.192.208.1 (24.192.208.1) 56(84) bytes of data.
...........................
--- 24.192.208.1 ping statistics ---
500 packets transmitted, 473 received, 5% packet loss, time 6578ms
rtt min/avg/max/mdev = 216.629/551.325/854.296/108.762 ms, pipe 58, ipg/ewma 13.183/811.051 ms
ChadDa3mon

ChadDa3mon

Member

Packet loss has begun, things started right around 8:30 with a modem reboot, but came back fine. Now I am noticing the packet loss.

--- 24.192.208.1 ping statistics ---
500 packets transmitted, 461 received, 7% packet loss, time 6418ms
rtt min/avg/max/mdev = 7.383/18.370/60.062/8.042 ms, pipe 5, ipg/ewma 12.862/16.223 ms

Refreshing the logs lost the first few lines where my modem rebooted, but here is what I have now

»pastebin.com/0xz1zvpn

Signal Values here

»postimg.org/image/9y47c772t/
ChadDa3mon

ChadDa3mon

Member

23% packet loss

--- 24.192.208.1 ping statistics ---
500 packets transmitted, 383 received, 23% packet loss, time 6645ms
rtt min/avg/max/mdev = 9.534/21.768/52.677/7.996 ms, pipe 4, ipg/ewma 13.317/18.257 ms

»postimg.org/image/b2lef3n9x/

No logs in the past 14 minutes to show.
wkcole
Premium Member
join:2007-11-08
Eastpointe, MI

wkcole to ChadDa3mon

Premium Member

to ChadDa3mon
2 side notes:
said by ChadDa3mon:

$ sudo ping -f -c 500 24.192.208.1
PING 24.192.208.1 (24.192.208.1) 56(84) bytes of data.
.......
--- 24.192.208.1 ping statistics ---
500 packets transmitted, 493 received, 1% packet loss, time 6756ms
rtt min/avg/max/mdev = 343.913/468.059/616.399/57.416 ms, pipe 48, ipg/ewma 13.539/444.016 ms

Yes, those are atrocious times, but your choice to use ping in "flood" mode may be influencing your results. Once upon a time that was a common DoS attack tool, which is why you need to run ping as root to use it: back in the Dark Ages it was reasonable to assume that 'root' was always a highly skilled IT professional exercising careful judgment. That is also why some devices detect and suppress ping floods, in some cases rather subtly (e.g. using QoS approaches to slow them rather than just dropping packets.)
said by ChadDa3mon:

Looked at the logs on my cable modem, nothing jumps out, this is the only entry from today.

DHCP RENEW WARNING - Field invalid in response v4 option;CM-MAC=f8:0b:be:de:56:2e;CMTS-MAC=00:01:5c:63:5c:66;CM-QOS=1.1;CM-VER=3.0;

That is entirely normal for WOW and apparently harmless. I am unclear on what exactly triggers them but I see identical messages (aside from the 2 MAC addresses, of course) at irregular multi-day intervals that don't correlate to any discernible problems. Since I have a block of static addresses, I don't really care about the constant low level of DHCP chatter that comes down the wire and I don't know why my CM seems to think this is a problem.
ChadDa3mon
join:2015-04-16
Warren, MI

ChadDa3mon

Member

said by wkcole:

Yes, those are atrocious times, but your choice to use ping in "flood" mode may be influencing your results. Once upon a time that was a common DoS attack tool, which is why you need to run ping as root to use it: back in the Dark Ages it was reasonable to assume that 'root' was always a highly skilled IT professional exercising careful judgment. That is also why some devices detect and suppress ping floods, in some cases rather subtly (e.g. using QoS approaches to slow them rather than just dropping packets.)

I appreciate the note, but ping floods haven't been a threat in well over 10 years. I use them constantly today to measure this issue, and when things are working fine, there's never an issue. It's just a much faster way of gathering meaningful test data then trying to wait for a regular ping to catch up.

Plus - the IP in question is the default gateway of my router, so it's not even really touching their core network.

Sadly, even with a normal ping, latency is horrible

$ ping 24.192.208.1
PING 24.192.208.1 (24.192.208.1) 56(84) bytes of data.
64 bytes from 24.192.208.1: icmp_seq=1 ttl=61 time=162 ms
64 bytes from 24.192.208.1: icmp_seq=2 ttl=61 time=195 ms
64 bytes from 24.192.208.1: icmp_seq=3 ttl=61 time=206 ms
64 bytes from 24.192.208.1: icmp_seq=4 ttl=61 time=178 ms

This is literally to the very next hop outside my network, totally unacceptable