dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
3160
share rss forum feed

TimHaag

join:2012-10-26
Sebastian, FL

Latency and Packet Loss Problems - Over 2 months

Good evening everyone.

I have a post over at the main comcast forums. I've been having latency and packet loss problems for over 2 months. I've had 3 techs out and 2 maintenance tech support requests put in. I believe the problem has been determined to not be local to my home or the main box at my condo unit. I have also ruled out my computers in my home. I haven't heard from anyone at Comcast in nearly 2 weeks despite making a couple more calls since then.

I live in Sebastian, FL (South East Florida). The problem typically begins sometime after 6 PM EST in the evenings and lasts for several hours. I have also seen the problem happen in the mornings and last as late as noon.

The thread on the main comcast forums is a bit overcrowded and unruly so here are the important highlights.

My modem has been replaced once. My modem is not likely the problem.
Pings typically look like this to any website when the problem exists
Berkeley Netalyzr doesn't show much of consequence. Sometimes it will show the bursts of packetloss
3D TraceRoute shows problems on the first few hops. This is to the google.com website.
Smoke ping is useful for seeing the very high latency when it occurs. The huge spikes (3000 pings) in latency died down last Thursday, but still receiving high pings (1000+) and packetloss.
• A second 3D TraceRoute showing the packetloss and latency. Here is the 3D TraceRoute graphed for comedic effect.
• I have been running Ping Plotter since 10pm Thursday night. Here is a screenshot of some of the problems since then.

I'm hoping by posting here I may get some additional assistance in getting this problem resolved. Please avoid posting "me too" type replies in this thread. My problem is most likely centered on the CMTS here in my town. Since it is such a small town in south east Florida, it is unlikely to cause problems for anyone not local to this area. If you are having a similar problem, please start a new thread.

Thanks so much for reading my post.



flwpwr

@comcast.net

Most of what you will get here is help determining issues not official comcast support.

Recommendation #1 run two cmd windows in window one
ping www.comcast.net -t
windows 2 look up gateway ip using ipconfig then ping that ip using -t

This sets up 2 ping commands running one your internet network to the gateway the other the external network to comcast.net

The gateway will wither be your modem or your router depending on which equiptment you have.

You can also try the line quality test at this website during problematic times, but suffices to say the other data we need is your modem levels and error logs if available form 192.168.100.1.


TimHaag

join:2012-10-26
Sebastian, FL

The 3D TraceRoutes and Ping Plotter do this exactly. The data is already provided in the screenshots above. The problem begins at the first hop after my gateway (hop 2 in the graphs).

For modem logs, they show 2 things from time to time
• MTA PROV: Failed (almost daily)
• No Ranging Response received - T3 time-out (once every 3-4 days at most)

I have only had the T3 time-out coincide once with when I'm having the problems. The MTA PROV has to do with my modem being VOIP enabled. I do not have VOIP through Comcast so this is of no consequence.



koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23
reply to TimHaag

Up front and honest: I have been working with TimHaag See Profile on this problem. We've been discussing things at length privately, and I've stepped him through how to properly set up PingPlotter so that he can determine where the problem actually begins.

Obviously I have no visibility into the Comcast CMTS network, nor visibility into the network devices or interfaces (router-wise or CMTS-wise), but to me at this point, especially since the issue seems to happen for long periods of time (10+ hours), and is often happening in the early evenings (but not always), I'm inclined to believe the CMTS is overloaded or there is a network link being saturated somewhere (not Tim's).

Now that's a bold statement for me to make without ruling some things out first, yeah? So let me cover what Tim and I have discussed already privately:

- He has ruled out any/all possibility of devices on his network saturating his network connection to Comcast (my line of thinking was maybe there was a compromised computer or device being involved in DDoS). Specifically this involved shutting off any computers or wireless devices, and keeping only one device (the machine running PingPlotter) powered on. AFAIK, Tim has done this, and has also done things like shut off the "PingPlotter machine" and instead run it from another device and saw the same results.

- Probably the most important data point: his downstream power level, downstream SNR, and upstream power level look great while this issue is happening and when it's not. He does not see increased corrected or uncorrected frames (FEC) during this issue. Statistics:

His downstream power is between 1 and 4.7dB, and his downstream SNR is between 36-37dB. This is seen across all 8 bonded channels/frequencies. Upstream power is 48dB, and has occasionally seen worse (going up to 50dB in one case), but has also seen better (46dB). It never exceeds 50dB. He does not ever experience loss of sync.

- The problem is recurring in nature. Based on the PingPlotter results I have analysed myself, these are the times and their durations of major events/hits. Timezone is Eastern:

11/09 @ 17:00 (roughly) -- not a major incident though
11/11 @ 06:00 (roughly -- similar to 11/09
11/12 @ 00:00 (roughly) -- major issue. This lasted almost 10 hours long
11/13 @ 03:00 (roughly) -- lasted almost 4 hours
11/13 @ 15:00 (roughly) -- ongoing

(Note that the 11/13 @ 15:00 incident I don't have an end time for)

As such, I am left to believe the issue is not with his individual connection, but more so something upstream.

He has not tried replacing his modem (some Zoom model, not sure which) at this time, but that is something that can probably be done as a troubleshooting point. I don't know if he owns his modem or rents it. (Note for Tim: if you own the modem and go this route, make sure you call Comcast to do the provisioning. Do not do it yourself! I can explain why in more detail if need be, but for now just trust me.)

This customer has had an issue for over 8 weeks, and has had service techs out who do absolutely nothing -- and in some cases, service techs who haven't even shown up for the job. Yes, that's a separate problem, but it doesn't diminish the amount of annoyance/stress the customer has. Tim has gotten absolutely no where trying to get someone at Comcast of more seniority to look at this (i.e. a head-end tech).

SteveTeow See Profile apparently did try one thing, but no technical details were given other than a "modem reprovision" and a "back-end change".

I would really like jlivingood See Profile and/or SteveTeow See Profile to be engaged on this matter.

My opinion as to what needs to happen: a senior network tech (either head-end or IP network or both) needs to be "on-call" (for lack of better term) so that when the issue begins happening, Tim can call that tech to begin investigating immediately. This is because the issue tends to not happen during the normal working day, only during evenings (when Comcast employees have gone home for the day) or absurd hours of the night.

I think Tim's been patient and reasonable, much like myself during my 11+ week incident, but like me, has reached a point where he has exhausted all options and has no idea where to turn.

This is a serious problem with Comcast -- the customer is never given any idea or path to escalate something, so what we're stuck with is repeatedly calling 800-COMCAST where the SOP for the CSRs is to continue sending out generic service techs for issues they (service techs) cannot solve. There does not seem to be an efficient/proper way to get the proper teams looking at things of this nature, and that's bad.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


TimHaag

join:2012-10-26
Sebastian, FL

Thanks alot Koitsu for summarizing. I have actually had the modem replaced once. The second time I asked for my modem to be replaced, it wasn't. The tech whom arrived did not bring one with him. The problem on 11/13 ended roughly around 20:30. I haven't checked the logs since last night, so I don't have any updates on if the problem has manifested again since then.

The "modem reprovision" that ComcastSteve performed was on the replaced modem. It is worth noting that the replaced modem was the identical model as the last one. My old modem worked for 8 months before I had a problem. Considering the circumstances, I do think it is unlikely that both modems are faulty.

I agree with the problems contacting people. I've made a couple phone calls to executive support and haven't received any return calls. I've been told someone would call me back by noon the following day both times. The original person who helped me when techs didn't show up, also hasn't called me back. He said the problem should be fixed by the end of day Saturday, November 3rd. He never gave me return contact information, instead citing he would follow up with me. I spoke with him on November 2nd, the return phone number dials directly into 1-800-COMCAST.

As a shot in the dark last week, I emailed koitsu asking for advice because he's been so successful in at least diagnosing his problem, if not downright assisting in the fix. I personally built my home computer (I've built over 20 systems). I am a software developer by trade. I understand the typical troubleshooting steps to determine the root cause of a problem. I've narrowed down to a single device and run diagnostics during the problem. I've then switched to a different device to see if it still existed. Both devices, as the sole device connected to the gateway via ethernet cable, have been able to document the problem. I believe I have done all the necessary steps to rule out my devices. I have updated all drivers and run multiple AV software scans. I've used the various windows tools to diagnose anything. The techs that have been at my house, have ruled out the lines local to my condo unit. However, as koitsu said, they have never been at my house while the problem was ongoing. I have the cell phone number of one of the techs who came out to my house and called him late Saturday on the 3rd when the problem was occurring. He said he would tell his manager, but there wasn't much else he could do.

I've emailed the we_can_help@cable.comcast.com email requesting help. I haven't received a return email or a phone call. After 8 weeks, I still feel like I haven't even gotten to the person I need to speak with to have the problem fixed. I understand why Comcast shields their network engineers. The vast majority of problems, probably 999 out of 1000, are local to the user. It is very frustrating that every tech I've worked with have ruled out the problem being local to me, yet can't assist me in getting the problem fixed.


kelfa

join:2012-10-27
Fort Myers, FL

I apologize for joining the thread like this, but I must say that these problems are exactly the same as mine. I live in Fort Myers, FL a couple hours away.

Problem has been ongoing 1-2 months now since upgrading to Blast!
1) 2 techs have been out to replace all lines/splitters.
2) My signals are perfect all the time, during the problems as well.
3) Line techs have been called and Im always the line techs are working.
4) Been monitoring ping plotter/tracers like yourself and am getting EXACTLY everything you have described.
5) Comcaststeve also replied to me and "reprovisioned" which seemed to help, but the problem returned quickly
6)Tried a rental, sb6121, and zoom. All three the same problem

Some days are better than others, some days online gaming, skype are just plain unusable.

Seems impossible to get someone on site when the problems are happening the most



koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

kelfa See Profile, please do not "append" to the thread. I'm not picking on ya (honest), but it's very important people remember:

Every single issue/ordeal needs to be handled individually/separately, at least initially.

When a thread/issue of a single customer begins to result in "me too" posts, it becomes difficult (bordering impossible) for an ISP or any technician/engineer to get concise details. Literally one engineer could spend 2-3 hours of his time every day just trying to sort out who said what and what means what. The official Comcast forums are *filled* with this sort of thing, which is almost certainly why it seems like nothing there ever gets responded to + fixed.

I would strongly suggest starting a separate thread here on the forum for your issue, and if you want it's cool to say "It's very similar to {this thread here}" but please keep it separate. You will need to post signal levels and do the exact same sort of analysis as what was done here. Don't post "summaries", post actual hard data/evidence.

In TimHaag See Profile's case, at this point it looks like something very close to the CMTS, and I would be surprised if people multiple hours apart were using the same CMTS. If this was a "Comcast backbone" (IP network) problem of some kind, it would be showing up starting at hops further away (i.e. hops #3 or #4), but it happens very close to home. My first inclination is an oversubscribed CMTS, but that is purely speculative and it is impossible for me to get evidence proving that at this time. We need to wait for jlivingood See Profile or other actual engineers to get involved -- and if they do not get involved, there are other options available, but as I told Tim privately, I don't' want to jump the gun.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


Fievel

join:2005-11-10

Any update on this situation ?

I am very curious to know the resolution ( yes, I am and have had for over 6 months this same issue -yes..the same exact issue...we could swap usernames and the details would be 100 percent the same...except im in pt charlotte, fl)

and no, that wasnt a "me too" ...just a heads up that comcast has this same problem in another area of florida



dmk08809

join:2001-09-08
Clinton, NJ
Reviews:
·CenturyLink
·Vonage
·Comcast

It seems almost systemic and more than just any given geographic service territory -- chronic and/or intermittent troubles are very difficult if not impossible to have addressed or followed up upon. Why? I think its because there's no real "ownership" of such problems. Just keep closing out the tickets & sweep 'em under the rug hoping it'll go away or maybe the customer will bail and go bother some other broadband provider (assuming they even have that as an option).

My own experiences -- nearly identical to what you guys enumerated -- have been posted here on DSLR. ~3 months later the problem still exists and we're on the 11th or 12th dispatch now.

Just know you're not alone and in good company guys... Be persistent and don't give up.


TimHaag

join:2012-10-26
Sebastian, FL

Sorry I didn't update sooner. The problem still exists. I had about a 2 week reprieve during the holidays, but am back to daily packet loss for 4 to 8 hours each day.


harald

join:2010-10-22
Columbus, OH
kudos:2
reply to TimHaag

Looking at your pings, I do not see evidence of packet loss. I do see lots of indication of intermittent connectivity.

Looking at the weather history for your city, it appears that this problem started when the evening temperatures started to cool off.

First guess: A "suckout" on some connection int the cable network. A trunk or feeder cable with an intermittently degraded/loose/thermally challenged connection.

Packet loss, especially from an overloaded upstream or downstream channel, doesn't normally look like your plots.


TimHaag

join:2012-10-26
Sebastian, FL

The only thing that makes me not think that is the very cold weather over the holiday. I did not have packet loss with the temperature in the 40s. I think a more likely correlation is people traveling away for Christmas, thus reducing load. The packet loss continued once the holiday was over.


TimHaag

join:2012-10-26
Sebastian, FL

Quick update. I have a network engineer working with me now. He has been absolutely amazing. He has found problems in the local network with line noise feeding in. He has made configuration and hardware upgrades at my primary node and another node that feeds into my primary node. He also found a local hotel/resort with degraded cabling that was feeding in a ton of line noise into the network from time to time. This place is only a couple miles down the road from me. He and his team finished replacing that wiring yesterday. We are cautiously optimistic that he has found the root cause of my packet loss problems. His impression is that the line noise was pushing my modem too hard which caused it to stop transmitting for a second or two every 10 to 20 seconds when the noise was too high.