dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
958
darkpixel
join:2014-02-03
Winlock, WA

darkpixel

Member

[Business] Packet loss on new install, techs say there are no issues

Click for full size
I am a 3rd party contractor. I recently recommended a customer switch from a local provider to Comcast because they were paying about $3,000/mo for 25/25 internet and a PRI, where Comcast quoted 100/25 and the PRI at $400/mo.

We are switching in phases. Internet was installed 02/27/2014.

I immediately ran into problems with packet loss and latency.

After doing a bunch of testing, here's what I've found and *verified* using multiple machines (Linux, Windows, and BSD):

If I ping from any device on 'my side' of the Comcast modem (i.e. something plugged in to the ethernet jacks and assigned an IP from our Comcast-assigned block), to any device on my side of the Comcast modem, I have no packet loss.

I can also ping from a device on my side *to* the modem's public IP without packet loss.

If I ping anything past the modem, I get (over a period of ~6 hours) 30% packet loss and an increase of ~25 msec RTT.

I tested this with a few IPs like Google's DNS servers, my office (which is on Comcast), and the next upstream hop past my router (as determined by a traceroute).

From the opposite side of the modem (from my home and from my office), I also get zero packet loss to the Comcast modem itself.

But I get ~30% packet loss to any device I plug in to the modem and an increase of ~25 msec RTT.

...so I figured it was something bizarre with the modem and called support again.
They were zero help.
They can ping the modem IP just fine with no packet loss. (And so can I from either inside the building or remotely from my house)

The moment they ping something on 'my' side of the modem, they see packet loss which they attribute to my network being screwed up. (Actually, they said it's because I'm not using their DNS servers which us utter crap)

So I called the sales guy and asked if he could get someone to replace the modem.
He did. The tech came out, replaced the modem, re-provisioned it with our IP block, plugged himself in, and runs a few tests over 10 minutes. He sees no packet loss, no unreasonable delays, and leaves.

A few minutes later everything starts timing out with the occasional packet coming through, and then returns to intermittent periods of packet loss and 'normal' service.

Everyone says it's something on my end. I don't believe it.
I've tested with multiple pieces of hardware, multiple operating systems, different cables every time, various combinations of devices plugged into the modem--including one device at a time with no other hardware plugged into the modem.

I understand the position of the Comcast techs--they can ping their modem just fine, so my device must be bad. And my position is that my devices are working fine, but once traffic crosses the modem to the Comcast network there are problems. ..but the modem has been replaced, equipment has been swapped, signal levels have been checked.

Attached is a smokeping graph from a Linux box plugged directly into the modem pinging the 'next hop' past the modem with nothing else plugged into the modem.

Any pointers?

Kasoah
join:2013-08-20
Merced, CA

Kasoah

Member

I don't think this is the case, but your modem ip is be assigned from the CMTS and they could be pinging that.

You didn't post your signals btw.
darkpixel
join:2014-02-03
Winlock, WA

darkpixel

Member


upstream
 
Click for full size
downstream
said by Kasoah:

I don't think this is the case, but your modem ip is be assigned from the CMTS and they could be pinging that.

You didn't post your signals btw.

Modem signal levels?

Anon1114
@comcast.net

Anon1114

Anon

Try disabling ipv6.
quesix
join:2005-12-19
Cary, IL
ARRIS SB6141
Cisco 2851
Asus RT-AC66

quesix to darkpixel

Member

to darkpixel

some things to try:
1) new ethernet cable from modem to device
2) check speed/duplex settings of device
3) look at data following over connection from firewall, look for large amount of outbound traffic, even if small packets such as port scanning (coax connection doesn't like large volume of upload packets) this could indicate your linux box or one of the PCs has been/are rooted/part of botnet
darkpixel
join:2014-02-03
Winlock, WA

darkpixel to Anon1114

Member

to Anon1114
That was one of the first things the techs suggested.
darkpixel

darkpixel to quesix

Member

to quesix
said by quesix:

some things to try:
1) new ethernet cable from modem to device
2) check speed/duplex settings of device
3) look at data following over connection from firewall, look for large amount of outbound traffic, even if small packets such as port scanning (coax connection doesn't like large volume of upload packets) this could indicate your linux box or one of the PCs has been/are rooted/part of botnet

1. We've tried multiple devices, each with their own factory-made cable.
2. Speed and duplex settings were all set to auto/full. I tried 1000/full and 100/full since all the devices had gig-e adapters, no change.
3. We've tried this with a freshly installed Windows and Linux box. I haven't fired up Wireshark on my linux laptop yet, but I'm considering it...
darkpixel

darkpixel

Member

Click for full size
...so I noticed something a bit odd after letting smokeping run for a day. It's not really obvious on the '3 hour graph', but it is on the '30 hour graph'.

My tests showed no problems pinging the modem, only when you 'crossed' the modem.

Attached in a graph of pinging the modem.
There is only one box plugged into the modem--it's a Linux box running a fresh install of Ubuntu, Apache, and SmokePing.

The ping times are gradually increasing...
darkpixel

darkpixel

Member

said by darkpixel:

The ping times are gradually increasing...

I should also mention, Comcast came out and replaced the modem thinking that could be the issue. This is the new modem.

gar187er
I DID this for a living
join:2006-06-24
Seattle, WA

gar187er to darkpixel

Member

to darkpixel
a 64qam centered at 15.7 mhz....YIKES.....

i thought having one centered at 21mhz was bad....

telcodad
MVM
join:2011-09-16
Lincroft, NJ

telcodad

MVM

said by gar187er:

a 64qam centered at 15.7 mhz....YIKES.....

i thought having one centered at 21mhz was bad....

Wow, yeah, they usually only have a 1/2-rate (2.56Ksymb/sec) 16QAM (DOCSIS 1.x) or 64QAM carrier in that noisy sub-20MHz region: »[Signals] My signal is good? It appears so..

And I haven't seen them below 18 MHz either. Must be a really "clean" system, at least on the reverse path (upstream) side!
darkpixel
join:2014-02-03
Winlock, WA

darkpixel

Member

Another tech came out and looked at everything.

He tested from his laptop. He ran a few pings and the speed test page, said everything was good and was getting ready to leave. I asked him to keep a few pings open for 5 minutes. He did. He saw packet loss, and his response was "We don't control anything outside the Comcast network, so we can't guarantee it". It's an understandable statement, but when it's *every* site and IP I can think of, I think it's a Comcast issue.

So after a lot of debate, he went back to his truck to check things again.

After a few minutes he called me over, and showed me a test he ran. I am completely lost when it comes to cable frequencies, noise, etc... The test basically showed two 'grids'. On the left was a grid with small black dots all over the place. He said "That's your signal. It's noisy. I've never seen one like that before". He pointed at the other one which was the same grid, but all the small black dots were clustered in to groups making up larger dots which were in the middle of the grid boxes. He said "That's your signal after the modem cleans it up. It's perfect.".

Meh.

Plugging my laptop directly into the modem (with othe devices plugged in as well) wasn't as helpful as I initially thought. The modem is a switch (duh), so I couldn't see any of the traffic from the servers or firewall.

I unplugged everything from the modem except my laptop, rebooted the modem, and assigned myself an IP address from our block.

I captured while I idly surfed around for 5-10 minutes.

Out of ~100,000 packets captures, there were quite a few "ETHERNET FRAME CHECK SEQUENCE INCORRECT" errors along with duplicate ACK's ("TCP DUP ACK"), and a lot of re-transmissions.

tshirt
Premium Member
join:2004-07-11
Snohomish, WA

tshirt

Premium Member

What do your signals look like now that it has been connected for a day or 2?*** The SnR's shown in the first screenshots are unrealistically high/most likely inaccurate, and a noise problem should NOW show as much lower SnR's and lots of errors
*** If you have just/recently rebooted the modem wait until you have a day or so of undisrupted surfing/other basic activity.
darkpixel
join:2014-02-03
Winlock, WA

darkpixel

Member

said by tshirt:

What do your signals look like now that it has been connected for a day or 2?*** The SnR's shown in the first screenshots are unrealistically high/most likely inaccurate, and a noise problem should NOW show as much lower SnR's and lots of errors
*** If you have just/recently rebooted the modem wait until you have a day or so of undisrupted surfing/other basic activity.

Yeah--just rebooted a few minutes ago while testing. I'll let it build up over the weekend.
darkpixel

darkpixel

Member

Better news. After 6 calls in to tech support, the latest technician *finally* saw 6% packet loss on the line. She scheduled a technician to come out tomorrow along with a supervisor to double-check line conditions.
darkpixel

darkpixel

Member

said by darkpixel:

Better news. After 6 calls in to tech support, the latest technician *finally* saw 6% packet loss on the line. She scheduled a technician to come out tomorrow along with a supervisor to double-check line conditions.

Thanks everyone for the help. It appears the issue has been resolved.

After trying two netgear modems, a particular tech that I've worked with before did a 90-minute round trip to a warehouse and managed to pry an SMC modem out of their hands. The SMC modem came up and has no issues.

I've left pings running for ~48 hours to my router, other devices on my assigned block, the next hop past the router, and Google's DNS servers. Zero packet loss across the board.

I remember years ago when they first introduced the SMC modems there were all sorts of weird issues that were slowly corrected by firmware updates. I guess the new Netgear modems need a few tweaks still...

Thanks again for the advice everyone.

-A