Comcast XFINITY → What to do about 11 weeks of node/network problems?

As I'm sure some of you know or remember, I've been dealing with a fairly annoying issue since August 14th or thereabouts:

»forums.comcast.com/t5/Co ··· /1379219

It's a long thread but I'll sum it up for folks at this point, based on what a fantastic Comcast line tech determined over 5 weeks ago:

The issue is with frequencies 747MHz and higher, which are used for downstream channels (Internet only) here in my area, having crappy SNR. The root cause of the crappy SNR is interference caused by cellular towers (either AT&T or Verizon -- tech is not sure yet). This same situation was seen in San Carlos (about 15 miles from here) some months ago, which Comcast had to track down and fix by replacing a long run of cable which had degraded shielding or something to that effect.

I confirmed that Verizon (and possibly AT&T) uses these frequencies for their 4G/LTE service.

I was assured by the line tech that this can/will get fixed, especially since they were able to confirm its existence, but it involves physically walking around with some kind of equipment to determine where leakage occurs. The line tech promised that he'd keep me in the loop with status updates as they progress, and just "FYI we haven't forgotten you!" calls.

So here's the situation...

The last time I heard from anyone at Comcast -- managerial or technical -- was 5 weeks ago. And this problem still exists -- it hasn't been fixed.

I have no contact numbers for anyone except an "escalations manager" of some sort (who has not returned my calls). Please note this individual is not "the manager of a service tech", but an actual manager who handles situations where service techs, or service tech managers, do not do their job (he was not able to give me an official title so I don't know what to call him).

I have posted about this issue in the past on the Comcast Direct forum and it was read a couple times but never solicited a reply from SteveTeow or anyone else.

I have kept a very precise/accurate, factual, and fair Word document outlining dates/times, everything that has transpired from day 1 until present, including all calls/in-person talks and so on I've had and what I've been told. I do have all related CRs, names of people, etc. -- I did this because I had one of those "engineer gut feelings" that told me this was going to turn out to be a serious PITA and thus keeping all my facts straight.

I'm at my wit's end here. I've been a reasonable customer, literally been as respectful and cool with the techs as possible (especially the line techs -- you guys are awesome), ditto with any of the managers I've spoken to. I know our city (Mountain View) was, a few months ago, handed off to the Menlo Park line techs to handle (historically its been handled by Santa Clara), so there's a transitional learning period. I've tried to be respectful of that. And as an engineer I understand things take time, but I feel 11 weeks is unreasonable, especially when it's affecting an entire node (at least from here all the way to the head end, since that's where the line tech went and was able to reproduce the problem reliably there).

In the back of my mind I've considered writing a letter to the Comcast CEO + including my Word doc, begging for some help, but deep inside I feel like that's borderline cowardly and that it could cause lots of drama/problems for the techs involved -- I don't want anyone fired or reprimanded! I just want this damn thing fixed!

I feel like I'm out of options because I have no contact number for the line tech who is most familiar with the issue (and I'd need to know his extension or last name to be able to look him up in the PBX directory), and the only "manager" contact I have isn't returning my calls.

I've also thought about reaching out to the California PUC, since this is an interference-related problem, but then I realised the PUC probably wouldn't care: the frequencies affected are for Internet-only service and do not affect cable TV, but could affect VoIP reliability (i.e. lots of jitter/packet loss); but generally speaking the Internet isn't considered a utility, it's a "nicety". But again, I don't want to go to that extreme because of what it could do to the great techs I've spoken to and met with in person.

I'm looking for recommendations from folks on what to do. This is a really crummy situation. :-( Help?

Well, I see I am NOT alone when it comes to Comcast and their rediculously poor customer support. Maybe we should form our own club?
I too have been losing information packets for the last 2 weeks, and all the phone techs can do is reboot my (brand-new) Modem. So, I went further, and here is what I did.
I ran a 'tracert' and a 'pathping' to the server I am trying to connect to (as the phone tech didn't know what to do), and saved them as .txt so I can e-mail/send. It is evident that on my 2nd and 4th hop I am losing 100% sent/recieved packets, and my HSI is dropping and/or super slow with extremely high latency. I brought this up to the phone tech, and they scheduled me a 'tech' visit for Friday the 2nd 3pm. He came and went like there's no tomorrow! The only thing I got out of him is that from the telephone pole to my house connection, my upstream @ the tap is 48.75, which is, obviously, 'extremely high'. He couldn't do anything else, said he would tell 'maintenance', and left.
I called this morning to see where I stood with Comcast, and guess what? They had 0 knowledge about the tech, and that he notified maintenance, and she wanted to 'reset' my Modem (again). I stopped her dead in her tracks, asked for a supervisor, as it's obvious she didn't know crap, and she 'refused' to get me a supervisor. (???) I told her to give my # to any supervisor so they can call me, and as of date, 0 phone call.
I currently attend college, and I need my internet for reference, contact with school, etc.
Someone tell me...where do I go from here???
Verizon???

I would now try a different approach...
You have tried providing detailed technical information/troubleshooting to them but can not get them to focus on the issue.

If you you can, get 25 of your neighbors to call and complain that they are having "problems with youtube and facebook" it will become a larger blip on their radar. Sure they will all get to reboot their modems etc but it now is a "bigger problem". An area problem gets more attention - fact.

Hand out a flyer and initiate a more grass roots effort.
It can't hurt and might be more effective.

From my observations they always focus on putting out the larger fires first. A single customer complaint always falls low on the list and rarely move to the top - it always falls off the radar for support resource allocation.

Good luck!

I want to be very clear here with my words:

I do not want this thread to become a "me too" or "f*** Comcast!" thread. I see too many of those and they do absolutely no good.

Every person's situation is different. I sympathise that you've been dealing with "high pings" for 2 weeks (although the evidence you've presented is only 50% applicable; your "pathping" shows high latency when talking to the CMTS, but your "tracert" doesn't), but that's a very different situation and handled in a very different way (and at a different layer) than what my issue is. Mine relates to actual cellular tower interference making it into a cable node, affecting a large number of people (I don't have actual statistics but I can confirm my neighbour who has the same setup I do sees the same problem), while yours relates to what could be your own connection being saturated or a CMTS capacity issue. Comcast can help diagnose your situation much easier than mine, and there are employees on this forum who can help. In my case, the only people who can resolve the issue are network/line techs, and I have no escalation path for those.

I recommend rather than "hijacking" (I use the term lightly) this thread for your problem, you start your own. What I requested in this thread is for feedback/advice from people on where to go from here with regards to my situation. Again: each situation must be handled individually.

Have you tried posting in the Comcast Direct Forum to see if ComcastSteve can kick it up and over for you?

You can try writing the FCC and reporting the issue. Also, at the bottom of your bill states who your local franchise authority is. Contact them regularly about the problem and they'll pressure comcast to work on a fix. Also, if you know of a business customer on that node that can complain, it will become a higher priority.

When you say the tech was able to reproduce the problem at the headend, do you know what methods he used to do that or what exactly was reproduced?

The OP wrote, "I have posted about this issue in the past on the Comcast Direct forum and it was read a couple times but never solicited a reply from ComcastSteve See Profile or anyone else."

To be more specific I've included a screenshot. Like I said, no response, but 2 people have looked at it. In that ComcastDirect post thread I did provide the official Comcast Forums post/thread.

My bills are digital/paperless; let's see what the billing statement PDFs have in them...

quote:

---

Your Local Franchise Authority Is:
City Of Mountain View
500 Castro St P.o. Box 7540
Mountain View, CA 94039
Ph:(650) 903-6301, FCC Unit #CA0906
(THIS IS NOT A PAYMENT OR SERVICE CENTER)

---

How convenient -- that's about 3 blocks from where I live.

I do not know of anyone with business-class service around here. I only know my immediate neighbours in my 6-apartment unit; most of the folks have DOCSIS 1.1 and 2.0 modems so they don't have channel bonding, and their modems aren't actively using frequencies at or above 747MHz (I've looked -- really! A little odd to ask neighbours if I can use their PC for a moment, but true story). Two of the neighbours don't use Comcast at all, so my demographic is limited.

Regarding the local franchise authority: what exactly do I tell them? I've never done something like this before.I don't know what methods were used, only that "in the field" he claims he could not reproduce the problem. The problem is abysmally bad SNR (29-32dB) starting at 747MHz or thereabouts, and it gets worse as the frequency increases (so 753MHz is really bad).

Taken directly from my notes with names and numbers XXX'd out for obvious reasons. XXX in this case is the line tech.

said by koitsu:

---

09/18 – Afternoon (3:50pm) – Call from XXX. Stated that he took readings at the head-end and confirmed the fluctuations in SNR/noise for 747/753MHz, indicating that the issue is “at the node”. XXX said he’ll be working tomorrow with a head-end engineer to try and track things down further, but that they didn’t have time today because there was only one head-end engineer available due to a major issue/problem which happened last night + kept them awake for long hours. XXX said he’ll call me tomorrow with details of what transpires/ask for verification of improvement/etc. I told XXX in the meantime I’d be rebooting my cable modem to try and dismiss use of 753MHz for the time being; he said that’s fine. Also took down XXX’s number per Caller ID: 650-XXX-XXXX

...

09/21 – Morning – Call from XXX. After discussing it with many of his colleagues, apparently a couple of them had seen this exact situation happen before up in San Carlos (specifically 747/753MHz). The root cause turned out to be interference injected into a node by a nearby AT&T cellular tower. As it turned out there was a faulty cable they replaced (visual inspection of the cable showed no problems but the cable was 20+ years old) to resolve the issue.

So at present they’re fairly certain this is what’s causing the problem here in Mountain View, but need to put in the hours/time to try and figure out where the noise is being injected and inspect and/or replace cables. XXX also noted that he had an on-site injury 2 days ago (damaged his knee and hip) but he has another tech working with him directly to get this issue fixed. The downside is that it may take some time (few weeks, maybe longer). I told XXX that is completely 100% acceptable, and that I considered all of this good news not bad.

So for now, my plan is to stick with using frequencies other than 747/753 (may take repeated reboots of my modem), since I have a working connection in that situation. I asked XXX give me periodic updates (weekly or bi-weekly if need be) to let me know how things progress, and that if they need anything from me to give me a call. He assured me he’d do just that.

09/21 – Morning – Wanted to called XXX back to let him know that based on some Internet resources, the issue may be with Verizon 4G/LTE and not AT&T:

»www.wpsantennas.com/cell ··· ion.aspx

However, 650-XXX-XXXX required either knowing the individual’s last name or their extension, of which I know neither. I’ll tell him of this the next time he calls me.

---

You can see the SNR for yourself here (Firefox, Chrome, IE9, or IE10 required -- nothing I can do about that, sorry):

»jdc.koitsu.org/snrgraph/

I can make the data for all of August available if need be; just ask.

You can see that there's no issue at present only because I've rebooted my modem a bazillion times to get it to pick frequencies other than 747 and 753MHz. But you can see at least on October 9th the issue was still there.

Regarding contacting your franchise authority, you're best bet is to call them and explain the situation. From there they'll let you know what steps they will take to help resolve the problem.

So, I'm not sure if I'm reading your notes correctly but it sounds like the tech called in to the headend about those last two QAMs. I'm guessing that the headend threw the analyzer on there and saw no problems on their end which is why the tech stated the issue was at the node. If he's seeing the problem at the testpoint of the node, then there is an issue with the RF at the physical node. If he doesn't see the problem there, but at your end, then he's got to go down the line until he finds the problem. It would be unusual if the problem wasn't found after all this time. There's a lot of posibilites so its really hard to say. Worst case, call them again and tell them to roll a truck out to you. The guys in the truck will know more than anyone you speak to on the phone. Hell, ask the guy that shows up for his shop manager's email address and contact info, then you'll be able to have documented correspondence with someone that has the authority to finally fix the issue.

Some clarification about my notes:

1. By "line tech" I am referring to an actual network line tech (the guys who drive around with bucket trucks and fix problems), not a service technician (the guys who come to your house and check everything between the cable modem all the way up to where the cable comes in off the pole).

2. Line tech was not able to see the issue in the field (was unable to see bad SNR for 747MHz or higher). I wasn't give any details other than that; just that "in the field" he couldn't see the problem. He stated that he was going to go to the head-end later that day and see if he could see it from there.

3. The follow-up call I got consisted of him saying the following, which didn't make any sense to me (this is why I put "at the node" in quotes): "I went to the head-end and yeah I can see the problem there at the node". I said "the node? Did you mean the head-end?" and he said "no, at the node".

As I understand it, "the node" refers to the cable node/segment (which correlates with a small or large geographic area, i.e. a "segment" of the cable network from which customers get their drops) while "head-end" means where (or near where) the CMTS is. So this is why I put "at the node" in quotes because I'm not sure what exactly he meant by what he said, given the terminology I've learned. I often get the impression all these techs (service and line/network) "re-use" terms to describe multiple things.

4. Line tech stated that after chatting with fellow techs about this problem, one reminded him of this issue happening in San Carlos (about 15 miles from here) but he himself had forgotten about it. He then told me the story about having to replace a long run of cable up there which looked fine but obviously was taking noise. I asked how they managed to figure out it was that cable. He said there's a device they have which allows them to look for leakage, but it's a long and drawn out task because it requires 2 people and a lot of walking.

Now for a question:

Doesn't a "truck roll" refer to sending out a line/network tech? If so, from whom do I ask this? Comcast? I have talked to Comcast CSRs (800-COMCAST) more times than I can count, and have requested line/network techs twice already -- what they send is a standard service tech every single time, who goes "yeah you have your ducks in a row, sounds like a network problem, not much I can do, you should talk to that network/line tech you've met". (Yeah that's nice, I would if I had his surname/extension...)

Or is this something I'm supposed to ask my LFA?

The node is a glorified media converter essentially. It takes the optical feed from the hubsite and converts it to the coax running around neighborhoods. The CMTS generally exists at the hubsite level not the headend level. Some might reuse headend to mean hubsite but with everyone having their own ways of saying things it's really anyone's guess.

Think of the headend as the main aggregation point for a geographic area. It feeds many different hubsites that themselves feed many different nodes. There may be more than one headend for an area depending on size, Atlanta has two that I know of: Stone Mountain and Vinings but I believe there are two others that probably came from other cable system acquisitions and general expansion.That is a cable leak detector and there can be vehicle and hand-held versions. Looks like they have really changed since the last one I had my hands on. The one I had really resembled a radar gun and had two telescoping antennas coming out each side. It could require two people but sometimes will be done with one depending on what they are looking for.

If they are looking for signal egress (signal leaking out) that can be done with one person driving around until he/she gets to the general area and then going on foot. If they are looking for signal ingress (signal making it's way in), that really does require two people. One to sit at your house and/or designated test points with a sensor and the other to drive around broadcasting a specific signal that sensor will be looking for.

Suffice it to say, depending on the area this issue could be fixed either from you pushing them or the FCC/FAA pushing them. If you live near a major airport or are in a major flight path, the FAA will do flyovers and identify hot spots that need attention. As such, your area may get flagged and fixed eventually even if you don't push them (not that I'm saying you shouldn't try, just that it may one day "fix itself" so to speak).

He could have had the headend techs check the transmitter for your node. They could then see whether or not those last two QAMs had MER/BER issues or if the signal from the NSG was degraded on those last two QAMs. If he truly saw the problem at the headend for your node, then that means those last two QAMs are degraded at the NSG because of a faulty card. A simple card swap at the headend and it's fixed. However, I doubt that's the problem or else it would have been fixed already. So, honestly I'm not sure what he meant there.

What you call "line tech" is what Comcast calls "Maintenance Techs". Maintenance techs do have the ability to walk the line with sniffer guns to locate where interference is entering the plant or where signal is leaking. I don't know how CSRs dispatch calls these days, but next time you call in, specifically request that you want a maintenance tech or a CommTech 5 tech to come diagnose an issue with the plant.

Also still contact the LFA and report the problem. Municipalities love fining cable companies for not being in compliance with maintaining their plant. Fines tend to jump start maintenance efforts.

Isn't it basically their S.O.P. that the phone reps can not do this and that only the premise facing techs can escalate an issue to the maintenance / network dept.

This is correct. The CSRs will open up a new CR (ticket/case) and will only schedule a standard service technician to come out. That technician and his supervisor are the ones who make the decision to "escalate the problem to network" (as in make a case and hand it off to network/line techs (maintenance techs) to deal with), and that's quite literally the last the customer ever hears of things (i.e. the assumption is made by all involved that network/line will resolve the issue). I speak from experience on this one. :-(

In my case, I was able to meet the network/line tech (maintenance tech) solely because of a Comcast employee on the official Comcast Forums (who shall remain anonymous per their request) "putting in a good word" for me behind-the-scenes + reaching out to someone at my local repair office (in Menlo Park) to look at the case. It was total chance that I actually *met* the guy -- I happened to be looking out my window the next day and saw a Comcast truck (with bucket) outside, where the driver got out and wandered around near our property looking up at utility poles. I went out and talked to him, asked him if he was handling my CR #, and he was. That's the guy, by the way, who I mentioned above (re: went to head-end, reproduced issue with 747MHz and higher, determined issue was same as what was seen in San Carlos).

If I call 800-COMCAST, all the CSRs are going to do is open up a new CR/case, link it/tie it to the old ones, and schedule to have a service technician come out. I've been down this road 3 times already, and all it does is waste service technicians' time since they can't solve the problem.

Thanks, HeadendJoe, and JoelC707 (hi dude ).

As surprising as it might sound, I actually understand most of the terminology you've used, sans two terms: MER and NSG. I know what QAMs are, and I know what BER is (bit error rate; from my experience with DS1/DS3/OC3/OC12 and SONET), but the other two are new to me.

I put in a call to the "general manager" who I'd spoke to in the past (about service technicians). I think I'm going to let things sit until Wednesday.

If I don't get a call back from him by then, I'm going to try an alternate approach -- I do have a contact at Comcast who handles the PR and IP networking side of things, and I have a good relationship with that person. Possibly they can get me in touch with someone who can drive this.

Otherwise if that goes no where, I'll give my LFA a call and approach it from that angle.

I guess on the positive side, there's lots of hard data/evidence that I can provide, and there is a part of me (the engineer part ) that's happy that Comcast was able to confirm existence of the issue. Sometimes tracking down a problem is the hardest part. In this case though it seems to be getting it fixed that's a pain.

Will let folks know what transpires.

This is a NSG »harmonicinc.com/product/ ··· -9000-6g

Essentially its an edgeQam that receives optical info and generates the info into RF QAM channels. So how Comcast works is that their CMTS routers are uBR10012 Cisco routers. The Downstream info (signal from the headend device to the node) leaves the router via fiber to the Harmonic NSG. The NSG outputs the info into multiple RF QAMs. Those QAMs eventually reach a transmitter where its converted back to optical, sent to the node, and then converted back to RF. Comcast also uses NSGs to convert VOD optical streams to RF QAMs.

Just a few definitions for you. MER (Modulation Error Rate) is the digital equivalent to SNR, which is the baseband measurement equivalent to Carrier to Noise. MER measures the ratio of error power to average power in an ideal QAM signal. MER measures the difference in quality between the transmitted modulation of a digital signal and the received modulation. Carrier to noise ratio is the measurement of the distance between a modulated RF carrier and the inherent noise floor. You can find a little more detail about these on Wikipedia.

Keep us updated for sure, hopefully it will get fixed. If you end up calling the FCC or your LFA, make sure you ask about Comcast's POP (proof of performance) testing. I believe by law the public has the right to review the outcome of those tests. They do the tests twice a year. May be worth glancing over if you're interested in where they found leaks in the plant during the test.

Today I received a call from a Comcast "technical supervisor" of some seniority, from the San Jose office, stating that he was calling multiple customers in my area -- specifically Hope Street (the street I live on) -- to find out what people's experiences were with their service. More specifically, they had started to receive many reports of problems pertaining to "interference" isolated to this region and wanted to chat with people about their experience/issues.

The timing of this is uncanny. I asked him if he was calling me specifically or if he was asked to call me on behalf of someone else -- he said no, he was just calling customers in my immediate vicinity. Hmmm... Very intriguing.

We chatted for almost 20 minutes, and I gave him the history, re: over 11 weeks of problems, been tracking this one issue pertaining to 747/753MHz and bad SNR, etc.. and asked if this was the issue they were trying to track down. He said that yes, that sounded very much like the problem they were chasing. He didn't want to state confidently that this is necessarily a 4G/LTE interference issue because "they just don't have the hard data right now that confirms it 100%", but that "it sounds very likely". Also noted that if it turns out to be that kind of problem, the downside is that it does take quite some time to narrow down.

He sounded very eager to talk to the maintenance tech who I had last spoken to over 5 weeks ago who knew a lot about this problem, so I was able to give him the name/main branch number of the tech. He sounded very eager to talk to this tech, as "it sounds like he has the most knowledge about this problem and it'd be good to get his insights" (and I fully agree). He said he should be able to reach out to that tech, since he plans on calling the maint. tech supervisor out of the Menlo Park office who is probably that tech's boss.

He also sounded a little surprised -- I don't think he was expecting a plethora of highly detailed information, at least from a generic end customer. Surprise! Us engineers are a wily bunch.

He also gave me his name (obviously), Email address, and phone number and urged me to shoot him an Email (in progress), and absolutely assured me that I'd be hearing back from him via one of those means.

So, at least someone is looking into this (though it sounds like there were duplicated efforts), and it def. looks like there are other customers in my area who have the same issue. Maybe one of them already went the LFA route and lit a fire under someone's butt about this, who knows.

I know, I think I always throw techs for a loop when I start rattling off my line stats, networking stuff, etc. Networking isn't my business, but engineering is and much of my job revolves around solving complex problems and pinpointing a root cause. So naturally, I get a little OCD when it comes to issues I see with my connection and trying to solve them.

Glad you have a contact now and it sounds like the issue is getting some traction! Good luck!

Traction indeed. Today has been an absolutely amazing (and insane) day.

A few hours after my above post, my connection went down. Within about 5 minutes I saw a Comcast maintenance tech (network/line; saw the bucket truck) walking around outside. He had an analyser wired to what appeared to be a cable with an RF modulator (or something) on the end of it. I asked if he was looking for interference pertaining to 747/753MHz, and he said he was looking for general interference issues. So I began telling him my situation, as well as the phone call I got earlier that morning. Once I gave him details he wanted to see the "hub" behind our building so I showed him that.

While he was looking at that, suddenly a 2nd tech appeared -- the same guy out of Menlo Park who had been handling my issue (who I hadn't heard from in 5 weeks). Turns out the damage to his hip was a little more severe than originally thought so it took him longer to recover. I'm glad he's doing okay now.

They both ended up climbing poles and doing *tons* of analysis, both at our line amp as well as at the "hub" behind our building. I'd say 70% of what was being said I didn't understand (I particularly laughed at a term they used: "Dairy Queening"). What I could make out is that they were looking for interference and taking level readings up at the pole / hub, as well as at the line amp, then comparing them between locations (behind our building vs. the line amp about 300 feet away). I watched both techs work the majority of the time, I guess because I was partially concerned for their safety (see bottom of post) and partially interested in the tech bits.

I think they were here for 2 of 3 hours. All I was able to discern from the many hours of work was that they had found some loosened ends near some hubs/splitters (not sure which) near the line amp, and also saw some sub-par signal level within the line amp itself (they opened it up) -- to address the former they re-tighten some brackets/clamps, and to address the latter they replaced what they referred to as "pads". I asked to see one of the latter, and they just looked like basic circuits; little copper coil on a small PCB. I'm guessing they're just line attenuators of some kind. One of the techs explained them to me but it was way over my head (lots of EE).

However, despite all their work, they still were not able to reproduce the crummy SNR that I've been reporting. They both believe me (especially given all the data I have), but they need to see it when it's happening.

More interestingly: at about the 70% mark, a third Comcast employee showed up. I didn't even see him until I went to check on the existing two techs. The fellow turned out to be the "Network Engineering Supervisor" for my region (I have his card so that's his official title) and was super friendly. He had gotten a call from the previous guy I spoke to that morning, and was in the area at the time + decided to come see things himself. He oversaw what was going on and took quite a few mental notes as far as what the conditions of everything was, mainly pertaining to the numbers (signal levels I believe) that the techs were reading off.

After all the adjustments/etc., my cable modem chose to sync up using 747MHz (we all were hoping for 753MHz, but even after 7-8 reboots it wouldn't pick it; just luck of the draw). 747MHz showed an SNR of 36/37dB -- that's about 4-5dB better than before. It's been like that for about an hour. That's better than how it's historically been, yes, but the supervisor and I both agree that we should let things sit because it's just too soon to tell if it's fixed.

I have numbers for the manager I spoke with this morning, the aforementioned supervisor, and (finally) the network tech who has been handling this issue. So I definitely have escalation points going forward, and will be keeping in direct communication with the NE supervisor for my area. Just gotta keep an eye on things and see if it's looking better. If it's not, we have some other choices/paths available to us, but we don't want to jump the gun.

Also, a footnote comment for you maintenance techs: you guys are truly incredible. I've never seen someone actually climb a 30-foot ladder, at a 16-17 degree angle, plus have to lean another 4-5 degrees, just to get to a line amp. And in drizzle to boot. Plus dropping their tools (multiple times, hehe ). Just absolutely amazing. You guys have my utmost respect for doing what you do -- I hope you get at least a little bit of happiness knowing there's at least 1 customer who really does appreciate the pains and efforts you go through during the course of your jobs.

Awesome! Hope it holds up for you. Sounds like you have a good bunch of folks working in your area and it's great to have contacts for the future.

My business escalations person I wound up working with on a line issue a couple of years ago was terrific. She unfortunately moved on to a different position, but she really helped to drive resolutions to my issues. And in my particular case, the line tech also had to sit outside with a meter near the ped for 2-3 hours in the rain It was pure dedication!

Fingers crossed for you!

No matter what the field is, intermittent issues are a PITA to diagnose and fix. At this point I would definitely let things sit and see how things progress. Since you weren't able to get it to sync on the culprit frequency, you don't yet know if it's fixed and any further "tinkering" could basically "re-break" things if it's truly fixed right now. Only time will tell but I might suggest occasionally rebooting the modem and see if you can catch 753 to check that one too.

Yeah, that's the plan, Joel. I plan on sticking with 747 for the time being (will let the NE supervisor and the maint tech know how things look after 24 hours), then after some time (maybe a week?), reboot it and see if I can get 753 and keep an eye on that for a while. It's a good thing I wrote my polling/logging script, saves me a bundle of trouble.

This does not indicate the issue is solved, but I thought I'd provide a brief status update (I also called the NE supervisor and let him know this -- he's in agreement, still too soon to tell).

Modem uptime is almost 6 hours and I haven't seen the SNR for 747MHz drop below 36dB once. Number of uncorrectable frames is also stable (compared to when the issue happens, where uncorrectable FECs shoots through the roof).

Graph of the past 6 hours is attached (or folks can just refer to my website), specifically showing only 747MHz.

Data:

Nov 08 20:58:00    Device Model: Motorola SB6121 (hardware version 5.0)
Nov 08 20:58:00        Firmware: SB_KOMODO-1.0.6.8-SCM01-NOSH (Aug 30 2012 15:35:46)
Nov 08 20:58:00       Boot Code: PSPU-Boot(25CLK) 1.0.12.18m3
Nov 08 20:58:00    Modem Uptime: 0 days 5h:57m:51s
Nov 08 20:58:00    Modem Status: Operational
Nov 08 20:58:00 Down Channel  4: 729000000 Hz,   1 dBmV power, 36 dB SNR, mod: QAM256, corr: 3108, uncorr: 595
Nov 08 20:58:00 Down Channel  5: 735000000 Hz,   1 dBmV power, 36 dB SNR, mod: QAM256, corr: 3106, uncorr: 502
Nov 08 20:58:00 Down Channel  6: 741000000 Hz,   1 dBmV power, 37 dB SNR, mod: QAM256, corr: 2814, uncorr: 472
Nov 08 20:58:00 Down Channel  7: 747000000 Hz,   1 dBmV power, 36 dB SNR, mod: QAM256, corr: 2536, uncorr: 532
Nov 08 20:58:00   Up Channel  9:  23700000 Hz,  46 dBmV power, 5.120 Msym/sec, status: Success
Nov 08 20:58:00   Up Channel  7:  35400000 Hz,  47 dBmV power, 2.560 Msym/sec, status: Success
Nov 08 20:58:00   Up Channel  8:  30600000 Hz,  46 dBmV power, 5.120 Msym/sec, status: Success
 

It's been a while since I've posted in this thread, but I wanted to provide a follow-up.

For about 3 weeks the 747/753MHz issue was solved. Well, at least I can say that it was for 747MHz, I couldn't get my modem to pick 753MHz (just total chance on that, no technical reason for it) so I couldn't test it. Everything was peachy keen, great SNRs all times of the day, etc.. The same for my neighbour.

Then on 11/30 we had a fairly large storm (very heavy winds, and what I would classify as "medium to heavy rain" but what Californians would classify as "absolutely crazy heavy rain" ), starting at around 04:30. At about 05:30 the wind really kicked in hard and my SNR on all frequencies started crapping out, along with T3 timeouts, etc... After a few hours, things settled down and sort of stabilised -- or so I assumed.

It wasn't until ~16 hours later when I looked at my SNR graphs that I realised the SNR issue had returned. Meaning: as a result of what happened on 11/30, the SNR issue is back. It's 100% identical -- issue tends to start around 7am and last until midnight, with SNRs dropping down to 31dB at times. Correlates directly with hours of the day when cellular traffic increases. My neighbour's seeing the same thing with his SNR (for 747MHz) as well.

I reached out to the Network Engineering Supervisor in my area to inform him of this situation, to get those network/maint techs back out here to figure out what is going on. I was told the following, verbatim:

"I will have {tech} look into this issue as soon as he has a chance too. We have a lot of clean up to do after the storm. There was
some damage cause to an amplifier on the node that feeds your home so this could be the issue out there."

I have no idea if this is the same line amp they worked on 4 weeks ago or not.

Anyway, I'm inclined to think whatever's causing this issue is likely some loose wire or bad cable run (between poles, not to my apt building) or a line amp that is just downright horked/busted in some way (allowing interference), probably as a result of wind. The interesting (amusing?) part of this story is that a busted line amp was proposed by a Comcast Forum tech at something like the 6-7 week mark.

I don't know what strings I'm going to have to pull to try and get credits either, but I'll tell ya, for the amount of time I'm having to put in on this (especially feeling like I have to micromanage people), I really do feel like I'm entitled to credits. Plus, as admitted above, this affects an entire node, not just me -- so I'm not the only customer who's affected.

Been a couple weeks since my last update, though I'd provide insights to where things are now.

After another set of mails on Monday (17th), the Network Engineering Supervisor took a look at everything remotely (my apartment's connection and my neighbour's) and confirmed that things were hitting 31-32dB on 747MHz range (I also included screenshots in my mails to him). This was around 13:00 or thereabouts. Within 15 minutes he had the previous network/maint tech from weeks ago out here taking readings, etc..

Tech took readings at the cable box, looking at 747 and 753MHz on his analyser, and confirmed the problem. Even more interesting was that he took a reading from an open input (literally holding up the input jack in the air) and looked at what the gain was at. Normally in our area, for 73x-75xMHz, the reading is in the -40dB region. These were his readings from behind our building (I saw them myself):

735MHz = between -44dB and -46dB
741MHz = between -43dB and -46dB
747MHz = between -31dB and -37dB
753MHz = between -27dB and -33dB

Note the large jump in average gain starting at 747MHz.

The tech told me that a few weeks prior, he took readings in the same manner for every cable/line segment down Church Street (which is where our "main connection" runs to the node), and all the numbers on all the frequencies were in the -40dB range. However, when you approach this area of my block, the gain for 747MHz and above starts going off the charts. What this means is that there is absolutely something in the air (i.e. wireless signal of some kind) that is pounding on 747MHz and above. That's perfectly fine too, but the issue (as the tech explained) is that that signal is seeping in to the node somewhere along this block -- specifically in the general line of sight of that cellular tower that was found via visible inspection weeks ago. (It would really help if I could draw a diagram... Maybe I'll work on that later)

The tech stated that to diagnose the issue, that tomorrow (12/18) he would come out early in the morning (6:30am) and install a tap/splitter on the cable that runs *to* the line amp, and hook me -- *just me, not my neighbours* -- up to that tap (this is apparently called a "ground line"), then monitor both my connection as well as my neighbour's. So in effect, I would have a direct line to the hub that connects to the node (i.e. no line amp for me), while my neighbours setup wouldn't change. This would be a temporary measure solely to try and narrow down which set of cable or piece of equipment is most likely the issue.

The theory is that if both myself and my neighbour continue to see the SNR issue, then it means the bad cable or bad piece of equipment is either at/around the tap I'm on, or further back from that (i.e. closer to the node). However, if only my neighbours see the SNR issue, then that means the bad cable/equipment is either the line amp or the stretch of cable that's between the line amp and where their connection comes in. (Sorry, this is really hard to describe without a visual diagram) We all agreed it'd be the best troubleshooting method for this, and scheduled to have it done between 0630 and 0830 the following morning.

So here comes the worst part of it all..

At 1930 that same night the tech came out and took readings/proposed the above, just out of no where, the SNR issue ceased. I changed nothing. My neighbour changed nothing. It just suddenly stopped for no reason. This has been one of the only times where things suddenly working actually made me furious!

The following morning the tech came out as promised and began the work proposed. I told him of the SNR issue ceasing the previous night and he was bummed out by that too, but regardless (understandably) went ahead with the proposed changes to the cabling to try and narrow this down. He padded the tap/splitter (so no need to install additional splitters at our cable box) and voilà, all done.

I also had him do another open input check with his analyser (I was wondering if the cellular tower had been shut off or simply stopped broadcasting) -- nope, still broadcasting nice and strong, i.e. 747/753MHz still have really high gain.

Anyway, my signal stats before:

Dec 17 14:56:00    Device Model: Motorola SB6121 (hardware version 5.0)
Dec 17 14:56:00        Firmware: SB_KOMODO-1.0.6.8-SCM01-NOSH (Aug 30 2012 15:35:46)
Dec 17 14:56:00       Boot Code: PSPU-Boot(25CLK) 1.0.12.18m3
Dec 17 14:56:00    Modem Uptime: 4 days 13h:10m:34s
Dec 17 14:56:00    Modem Status: Operational
Dec 17 14:56:00 Down Channel  1: 705000000 Hz,   0 dBmV power, 37 dB SNR, mod: QAM256, corr: 161369, uncorr: 733
Dec 17 14:56:00 Down Channel  5: 735000000 Hz,   0 dBmV power, 37 dB SNR, mod: QAM256, corr: 187313, uncorr: 590
Dec 17 14:56:00 Down Channel  6: 741000000 Hz,  -1 dBmV power, 37 dB SNR, mod: QAM256, corr: 179164, uncorr: 479
Dec 17 14:56:00 Down Channel  7: 747000000 Hz,  -1 dBmV power, 32 dB SNR, mod: QAM256, corr: 208803, uncorr: 560
Dec 17 14:56:00   Up Channel  8:  30600000 Hz,  46 dBmV power, 5.120 Msym/sec, status: Success
Dec 17 14:56:00   Up Channel  7:  35400000 Hz,  47 dBmV power, 2.560 Msym/sec, status: Success
Dec 17 14:56:00   Up Channel  9:  23700000 Hz,  46 dBmV power, 5.120 Msym/sec, status: Success
 

Signal stats after being given a "ground line":

Dec 19 19:02:00    Device Model: Motorola SB6121 (hardware version 5.0)
Dec 19 19:02:00        Firmware: SB_KOMODO-1.0.6.8-SCM01-NOSH (Aug 30 2012 15:35:46)
Dec 19 19:02:00       Boot Code: PSPU-Boot(25CLK) 1.0.12.18m3
Dec 19 19:02:00    Modem Uptime: 1 days 10h:25m:38s
Dec 19 19:02:00    Modem Status: Operational
Dec 19 19:02:00 Down Channel  2: 717000000 Hz,  -1 dBmV power, 36 dB SNR, mod: QAM256, corr: 59185, uncorr: 635
Dec 19 19:02:00 Down Channel  3: 723000000 Hz,  -1 dBmV power, 36 dB SNR, mod: QAM256, corr: 61702, uncorr: 635
Dec 19 19:02:00 Down Channel  6: 741000000 Hz,  -1 dBmV power, 37 dB SNR, mod: QAM256, corr: 60489, uncorr: 722
Dec 19 19:02:00 Down Channel  8: 753000000 Hz,  -2 dBmV power, 36 dB SNR, mod: QAM256, corr: 54379, uncorr: 634
Dec 19 19:02:00   Up Channel  8:  30600000 Hz,  38 dBmV power, 5.120 Msym/sec, status: Success
Dec 19 19:02:00   Up Channel  7:  35400000 Hz,  39 dBmV power, 2.560 Msym/sec, status: Success
Dec 19 19:02:00   Up Channel  9:  23700000 Hz,  37 dBmV power, 5.120 Msym/sec, status: Success
 

So now we simply wait for the SNR issue to happen again (to either myself and my neighbour, or just my neighbour).

I'll see if I can draw up a diagram of how all the cabling and stuff is run, just because it makes things a lot easier to understand.

As far as a root cause goes? The supervisor, tech, and myself are all in agreement -- the most likely candidate is something environmental, ex. a hair-line crack in some cable that gets stretched open when it's windy, a line amp that's allowing interference, or possibly a tap/splitter that's doing the same. We don't know which, but we're trying to narrow down the geographic location of which lines/cable/equipment it is.

Attached are diagrams. I'm not the best artist, sorry.

All the pink/magenta lines are my neighbours, which are coming off the tap/splitter that's off of the "red cable" coming from the line amp. After the work done on 12/18, I'm coming off a tap/splitter that's along the "blue cable". Hopefully this makes things a little more clear.

Obligatory Google Maps link for those who want to scout around my area to get an idea.

And you can actually see the cellular tower using Google Maps' Street View (centre of picture), but it's hard to spot since it's painted the same colour as the building. But it's there. *shakes fist*

Koitsu, Thanks for your detailed postings and chronology of events! A story like this gives me hope that maybe my issue will someday be fixed. I think a key difference is that you actually have a real inside contact that was able to push the issue. From what I've seen here in NJ using normal escalations processes for mere mortals is slow, inconsistent, and quite possibly a waste of time. Furthermore, providing details (like SNR) to most techs seem to be a waste of breath -- they're only interested in what they see on their meters at that exact moment. *sigh* Good luck and keep accounting and sharing the story!

You're welcome. As far as getting an "inside contact", that happened through strange/bizarre means in itself. Details are here (first part):

»Re: What to do about 11 weeks of node/network problems?

What I can't figure out is if this Comcast manager really was calling people along my street or if it was the result of a VM I left someone else at Comcast stating that I was basically at the end of my rope + I felt like the only choice I had left was to send a letter (along with a diary which I've been keeping of all the events from day one) to the CEO.

If you're struggling though, HeadendJoe recommended contacting your LFA (local franchise authority) -- particularly if there's a widespread issue affecting citizens in an area, or "plant issues". The address/phone of your LFA is printed on your bill (on paper or digitally/PDF available via the Comcast site).

The bottom line is that customers should not have to go through this kind of rigmarole to get something fixed. The only outlet that people know of is 800-COMCAST, and all the CSRs can send out are standard service technicians (who are the ones who have to make the call on whether or not to send a CR (case/ticket) to network, if the issue is a cable network issue). And even if it makes it to network, there's absolutely zero communication with the customer from that point forward -- so if the issue persists, the customer has no idea what to do, calls 800-COMCAST again, schedules another service tech visit, rinse lather repeat. This system is just downright broken, and it's been like this for at least 10 years. It doesn't work.

Welcome to my fun...

I have an issue that 2-3 times a night my modem reboots, Upstream SNR dips below 29. Downstream SNR great, 42, power levels are good with the use of forward path attenuator.

I have maint sup direct cell phone and he has me on monitoring and sees the drop every night. His direct reports I have they cell phone also and they have seen it and it's the entire node but it is so quick and random on the time (beyond it being at night) they can't figure it out.

Sucks. Something is causing noise but only for a moment every night at a different time.

I feel bad, I am a pain in the butt customer but I am paying for service I want. They are trying but is a crappy situation. When issue is random and happens for a minute tops no way to catch it to track it because it's a different time every day.