dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
27
share rss forum feed


koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23
reply to koitsu

Re: What to do about 11 weeks of node/network problems?

It's been a while since I've posted in this thread, but I wanted to provide a follow-up.

For about 3 weeks the 747/753MHz issue was solved. Well, at least I can say that it was for 747MHz, I couldn't get my modem to pick 753MHz (just total chance on that, no technical reason for it) so I couldn't test it. Everything was peachy keen, great SNRs all times of the day, etc.. The same for my neighbour.

Then on 11/30 we had a fairly large storm (very heavy winds, and what I would classify as "medium to heavy rain" but what Californians would classify as "absolutely crazy heavy rain" ), starting at around 04:30. At about 05:30 the wind really kicked in hard and my SNR on all frequencies started crapping out, along with T3 timeouts, etc... After a few hours, things settled down and sort of stabilised -- or so I assumed.

It wasn't until ~16 hours later when I looked at my SNR graphs that I realised the SNR issue had returned. Meaning: as a result of what happened on 11/30, the SNR issue is back. It's 100% identical -- issue tends to start around 7am and last until midnight, with SNRs dropping down to 31dB at times. Correlates directly with hours of the day when cellular traffic increases. My neighbour's seeing the same thing with his SNR (for 747MHz) as well.

I reached out to the Network Engineering Supervisor in my area to inform him of this situation, to get those network/maint techs back out here to figure out what is going on. I was told the following, verbatim:

"I will have {tech} look into this issue as soon as he has a chance too. We have a lot of clean up to do after the storm. There was
some damage cause to an amplifier on the node that feeds your home so this could be the issue out there."

I have no idea if this is the same line amp they worked on 4 weeks ago or not.

Anyway, I'm inclined to think whatever's causing this issue is likely some loose wire or bad cable run (between poles, not to my apt building) or a line amp that is just downright horked/busted in some way (allowing interference), probably as a result of wind. The interesting (amusing?) part of this story is that a busted line amp was proposed by a Comcast Forum tech at something like the 6-7 week mark.

I don't know what strings I'm going to have to pull to try and get credits either, but I'll tell ya, for the amount of time I'm having to put in on this (especially feeling like I have to micromanage people), I really do feel like I'm entitled to credits. Plus, as admitted above, this affects an entire node, not just me -- so I'm not the only customer who's affected.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23
Been a couple weeks since my last update, though I'd provide insights to where things are now.

After another set of mails on Monday (17th), the Network Engineering Supervisor took a look at everything remotely (my apartment's connection and my neighbour's) and confirmed that things were hitting 31-32dB on 747MHz range (I also included screenshots in my mails to him). This was around 13:00 or thereabouts. Within 15 minutes he had the previous network/maint tech from weeks ago out here taking readings, etc..

Tech took readings at the cable box, looking at 747 and 753MHz on his analyser, and confirmed the problem. Even more interesting was that he took a reading from an open input (literally holding up the input jack in the air) and looked at what the gain was at. Normally in our area, for 73x-75xMHz, the reading is in the -40dB region. These were his readings from behind our building (I saw them myself):

735MHz = between -44dB and -46dB
741MHz = between -43dB and -46dB
747MHz = between -31dB and -37dB
753MHz = between -27dB and -33dB

Note the large jump in average gain starting at 747MHz.

The tech told me that a few weeks prior, he took readings in the same manner for every cable/line segment down Church Street (which is where our "main connection" runs to the node), and all the numbers on all the frequencies were in the -40dB range. However, when you approach this area of my block, the gain for 747MHz and above starts going off the charts. What this means is that there is absolutely something in the air (i.e. wireless signal of some kind) that is pounding on 747MHz and above. That's perfectly fine too, but the issue (as the tech explained) is that that signal is seeping in to the node somewhere along this block -- specifically in the general line of sight of that cellular tower that was found via visible inspection weeks ago. (It would really help if I could draw a diagram... Maybe I'll work on that later)

The tech stated that to diagnose the issue, that tomorrow (12/18) he would come out early in the morning (6:30am) and install a tap/splitter on the cable that runs *to* the line amp, and hook me -- *just me, not my neighbours* -- up to that tap (this is apparently called a "ground line"), then monitor both my connection as well as my neighbour's. So in effect, I would have a direct line to the hub that connects to the node (i.e. no line amp for me), while my neighbours setup wouldn't change. This would be a temporary measure solely to try and narrow down which set of cable or piece of equipment is most likely the issue.

The theory is that if both myself and my neighbour continue to see the SNR issue, then it means the bad cable or bad piece of equipment is either at/around the tap I'm on, or further back from that (i.e. closer to the node). However, if only my neighbours see the SNR issue, then that means the bad cable/equipment is either the line amp or the stretch of cable that's between the line amp and where their connection comes in. (Sorry, this is really hard to describe without a visual diagram) We all agreed it'd be the best troubleshooting method for this, and scheduled to have it done between 0630 and 0830 the following morning.

So here comes the worst part of it all..

At 1930 that same night the tech came out and took readings/proposed the above, just out of no where, the SNR issue ceased. I changed nothing. My neighbour changed nothing. It just suddenly stopped for no reason. This has been one of the only times where things suddenly working actually made me furious!

The following morning the tech came out as promised and began the work proposed. I told him of the SNR issue ceasing the previous night and he was bummed out by that too, but regardless (understandably) went ahead with the proposed changes to the cabling to try and narrow this down. He padded the tap/splitter (so no need to install additional splitters at our cable box) and voilà, all done.

I also had him do another open input check with his analyser (I was wondering if the cellular tower had been shut off or simply stopped broadcasting) -- nope, still broadcasting nice and strong, i.e. 747/753MHz still have really high gain.

Anyway, my signal stats before:

Dec 17 14:56:00    Device Model: Motorola SB6121 (hardware version 5.0)
Dec 17 14:56:00        Firmware: SB_KOMODO-1.0.6.8-SCM01-NOSH (Aug 30 2012 15:35:46)
Dec 17 14:56:00       Boot Code: PSPU-Boot(25CLK) 1.0.12.18m3
Dec 17 14:56:00    Modem Uptime: 4 days 13h:10m:34s
Dec 17 14:56:00    Modem Status: Operational
Dec 17 14:56:00 Down Channel  1: 705000000 Hz,   0 dBmV power, 37 dB SNR, mod: QAM256, corr: 161369, uncorr: 733
Dec 17 14:56:00 Down Channel  5: 735000000 Hz,   0 dBmV power, 37 dB SNR, mod: QAM256, corr: 187313, uncorr: 590
Dec 17 14:56:00 Down Channel  6: 741000000 Hz,  -1 dBmV power, 37 dB SNR, mod: QAM256, corr: 179164, uncorr: 479
Dec 17 14:56:00 Down Channel  7: 747000000 Hz,  -1 dBmV power, 32 dB SNR, mod: QAM256, corr: 208803, uncorr: 560
Dec 17 14:56:00   Up Channel  8:  30600000 Hz,  46 dBmV power, 5.120 Msym/sec, status: Success
Dec 17 14:56:00   Up Channel  7:  35400000 Hz,  47 dBmV power, 2.560 Msym/sec, status: Success
Dec 17 14:56:00   Up Channel  9:  23700000 Hz,  46 dBmV power, 5.120 Msym/sec, status: Success
 

Signal stats after being given a "ground line":

Dec 19 19:02:00    Device Model: Motorola SB6121 (hardware version 5.0)
Dec 19 19:02:00        Firmware: SB_KOMODO-1.0.6.8-SCM01-NOSH (Aug 30 2012 15:35:46)
Dec 19 19:02:00       Boot Code: PSPU-Boot(25CLK) 1.0.12.18m3
Dec 19 19:02:00    Modem Uptime: 1 days 10h:25m:38s
Dec 19 19:02:00    Modem Status: Operational
Dec 19 19:02:00 Down Channel  2: 717000000 Hz,  -1 dBmV power, 36 dB SNR, mod: QAM256, corr: 59185, uncorr: 635
Dec 19 19:02:00 Down Channel  3: 723000000 Hz,  -1 dBmV power, 36 dB SNR, mod: QAM256, corr: 61702, uncorr: 635
Dec 19 19:02:00 Down Channel  6: 741000000 Hz,  -1 dBmV power, 37 dB SNR, mod: QAM256, corr: 60489, uncorr: 722
Dec 19 19:02:00 Down Channel  8: 753000000 Hz,  -2 dBmV power, 36 dB SNR, mod: QAM256, corr: 54379, uncorr: 634
Dec 19 19:02:00   Up Channel  8:  30600000 Hz,  38 dBmV power, 5.120 Msym/sec, status: Success
Dec 19 19:02:00   Up Channel  7:  35400000 Hz,  39 dBmV power, 2.560 Msym/sec, status: Success
Dec 19 19:02:00   Up Channel  9:  23700000 Hz,  37 dBmV power, 5.120 Msym/sec, status: Success
 

So now we simply wait for the SNR issue to happen again (to either myself and my neighbour, or just my neighbour).

I'll see if I can draw up a diagram of how all the cabling and stuff is run, just because it makes things a lot easier to understand.

As far as a root cause goes? The supervisor, tech, and myself are all in agreement -- the most likely candidate is something environmental, ex. a hair-line crack in some cable that gets stretched open when it's windy, a line amp that's allowing interference, or possibly a tap/splitter that's doing the same. We don't know which, but we're trying to narrow down the geographic location of which lines/cable/equipment it is.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23
Click for full size
Network cabling diagram
Click for full size
Cellular tower location vs. home
Attached are diagrams. I'm not the best artist, sorry.

All the pink/magenta lines are my neighbours, which are coming off the tap/splitter that's off of the "red cable" coming from the line amp. After the work done on 12/18, I'm coming off a tap/splitter that's along the "blue cable". Hopefully this makes things a little more clear.

Obligatory Google Maps link for those who want to scout around my area to get an idea.

And you can actually see the cellular tower using Google Maps' Street View (centre of picture), but it's hard to spot since it's painted the same colour as the building. But it's there. *shakes fist*
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


dmk08809

join:2001-09-08
Clinton, NJ
Reviews:
·CenturyLink
·Vonage
·Comcast
Koitsu, Thanks for your detailed postings and chronology of events! A story like this gives me hope that maybe my issue will someday be fixed. I think a key difference is that you actually have a real inside contact that was able to push the issue. From what I've seen here in NJ using normal escalations processes for mere mortals is slow, inconsistent, and quite possibly a waste of time. Furthermore, providing details (like SNR) to most techs seem to be a waste of breath -- they're only interested in what they see on their meters at that exact moment. *sigh* Good luck and keep accounting and sharing the story!


koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23
You're welcome. As far as getting an "inside contact", that happened through strange/bizarre means in itself. Details are here (first part):

»Re: What to do about 11 weeks of node/network problems?

What I can't figure out is if this Comcast manager really was calling people along my street or if it was the result of a VM I left someone else at Comcast stating that I was basically at the end of my rope + I felt like the only choice I had left was to send a letter (along with a diary which I've been keeping of all the events from day one) to the CEO.

If you're struggling though, HeadendJoe recommended contacting your LFA (local franchise authority) -- particularly if there's a widespread issue affecting citizens in an area, or "plant issues". The address/phone of your LFA is printed on your bill (on paper or digitally/PDF available via the Comcast site).

The bottom line is that customers should not have to go through this kind of rigmarole to get something fixed. The only outlet that people know of is 800-COMCAST, and all the CSRs can send out are standard service technicians (who are the ones who have to make the call on whether or not to send a CR (case/ticket) to network, if the issue is a cable network issue). And even if it makes it to network, there's absolutely zero communication with the customer from that point forward -- so if the issue persists, the customer has no idea what to do, calls 800-COMCAST again, schedules another service tech visit, rinse lather repeat. This system is just downright broken, and it's been like this for at least 10 years. It doesn't work.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


AnonMan

@comcast.net
Welcome to my fun...

I have an issue that 2-3 times a night my modem reboots, Upstream SNR dips below 29. Downstream SNR great, 42, power levels are good with the use of forward path attenuator.

I have maint sup direct cell phone and he has me on monitoring and sees the drop every night. His direct reports I have they cell phone also and they have seen it and it's the entire node but it is so quick and random on the time (beyond it being at night) they can't figure it out.

Sucks. Something is causing noise but only for a moment every night at a different time.

I feel bad, I am a pain in the butt customer but I am paying for service I want. They are trying but is a crappy situation. When issue is random and happens for a minute tops no way to catch it to track it because it's a different time every day.


koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23
AnonMan, that situation is indeed crummy. Due to the highly intermittent nature tracking it down (to a specific cable/area of the node) sounds very difficult and time consuming. On the other hand, if it's happening on the entire node, that means more than just you is impacted -- bringing this up with your LFA may be the next step for you. Node-level issues I think are given more attention than individual customer issues.

Also, I've been told by both the Network Engineering Supervisor as well as the network/maint tech who was out here, that Comcast does have a "history view" for SNR levels on a per-channel basis (including support for bonding), so they should be able to see it there. Meaning, they have something similar to what I provide here (from my own modem) but probably with less granularity (I poll every 2 minutes): »jdc.koitsu.org/snrgraph/ -- I know Comcast having that information doesn't help narrow down where on the cable network/segment the problem lies, but it would act as evidence (for your LFA) that Comcast acknowledges there's a problem affecting multiple people on your node.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.

JoeHemi

join:2011-05-06
Bellevue, WA

1 recommendation

Yes, the system is called scout monitor and shows the history and real time data for each STB & modem for every customer. The other system, which I like better, is called Watch Tower. Much more detailed analysis of the relationship between the customer's equipment & the CMTS/Narrowcast. Other nice thing about watch tower is that techs can submit notes when they troubleshoot that node and those notes are visible to anyone troubleshooting your node in the future. Other field tool the techs can use to monitor the signal from your node is Pathtrack which shows a virtual representation of the return signal so they can see a quick visual of noise, carrier problems, etc and take snapshots of any problems at any interval. They can also call in to the headend and have them turn up the return sweep. The techs sweep that node segment and can determine issues. Many, many tools out there, but, all the tools can do is tell the tech there is a problem, not how, where, or why.