Up front and honest: I have been working with TimHaag
on this problem. We've been discussing things at length privately, and I've stepped him through how to properly set up PingPlotter so that he can determine where the problem actually begins.
Obviously I have no visibility into the Comcast CMTS network, nor visibility into the network devices or interfaces (router-wise or CMTS-wise), but
to me at this point, especially since the issue seems to happen for long periods of time (10+ hours), and is often happening in the early evenings (but not always), I'm inclined to believe the CMTS is overloaded
or there is a network link being saturated somewhere (not Tim's).
Now that's a bold statement for me to make without ruling some things out first, yeah? So let me cover what Tim and I have discussed already privately:
- He has ruled out any/all possibility of devices on his network saturating his network connection to Comcast (my line of thinking was maybe there was a compromised computer or device being involved in DDoS). Specifically this involved shutting off any computers or wireless devices, and keeping only one device (the machine running PingPlotter) powered on. AFAIK, Tim has done this, and has also done things like shut off the "PingPlotter machine" and instead run it from another device and saw the same results.
- Probably the most important data point:
his downstream power level, downstream SNR, and upstream power level look great while this issue is happening and when it's not. He does not see increased corrected or uncorrected frames (FEC) during this issue. Statistics:
His downstream power is between 1 and 4.7dB, and his downstream SNR is between 36-37dB. This is seen across all 8 bonded channels/frequencies. Upstream power is 48dB, and has occasionally seen worse (going up to 50dB in one case), but has also seen better (46dB). It never exceeds 50dB.
He does not ever experience loss of sync.-
The problem is recurring in nature. Based on the PingPlotter results I have analysed myself, these are the times and their durations of major events/hits. Timezone is Eastern:
11/09 @ 17:00 (roughly) -- not a major incident though
11/11 @ 06:00 (roughly -- similar to 11/09
11/12 @ 00:00 (roughly) -- major issue. This lasted almost 10 hours long
11/13 @ 03:00 (roughly) -- lasted almost 4 hours
11/13 @ 15:00 (roughly) -- ongoing
(Note that the 11/13 @ 15:00 incident I don't have an end time for)
As such, I am left to believe the issue is not with his individual connection, but more so something upstream.
He has not tried replacing his modem (some Zoom model, not sure which) at this time, but that is something that can probably be done as a troubleshooting point. I don't know if he owns his modem or rents it. (Note for Tim: if you own the modem and go this route, make sure you
call Comcast to do the provisioning. Do not do it yourself! I can explain why in more detail if need be, but for now just trust me.)
This customer has had an issue for over 8 weeks, and has had service techs out who do absolutely nothing -- and in some cases, service techs who haven't even shown up for the job. Yes, that's a separate problem, but it doesn't diminish the amount of annoyance/stress the customer has. Tim has gotten absolutely no where trying to get someone at Comcast of more seniority to look at this (i.e. a head-end tech).
SteveTeow
apparently did try one thing, but no technical details were given other than a "modem reprovision" and a "back-end change".
I would really like jlivingood
and/or SteveTeow
to be engaged on this matter.
My opinion as to what needs to happen: a senior network tech (either head-end or IP network or both) needs to be "on-call" (for lack of better term) so that when the issue begins happening, Tim can call that tech to begin investigating immediately. This is because the issue tends to not happen during the normal working day, only during evenings (when Comcast employees have gone home for the day) or absurd hours of the night.
I think Tim's been patient and reasonable, much like myself during
my 11+ week incident, but like me, has reached a point where he has exhausted all options and has no idea where to turn.
This is a serious problem with Comcast -- the customer is never given any idea or path to escalate something, so what we're stuck with is repeatedly calling 800-COMCAST where the SOP for the CSRs is to continue sending out generic service techs for issues they (service techs) cannot solve. There does not seem to be an efficient/proper way to get the proper teams looking at things of this nature, and that's bad.