dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
9086

pflog
Bueller? Bueller?
MVM
join:2001-09-01
El Dorado Hills, CA

pflog to koitsu

MVM

to koitsu

Re: What to do about 11 weeks of node/network problems?

Awesome! Hope it holds up for you. Sounds like you have a good bunch of folks working in your area and it's great to have contacts for the future.

My business escalations person I wound up working with on a line issue a couple of years ago was terrific. She unfortunately moved on to a different position, but she really helped to drive resolutions to my issues. And in my particular case, the line tech also had to sit outside with a meter near the ped for 2-3 hours in the rain It was pure dedication!

Fingers crossed for you!
JoelC707
Premium Member
join:2002-07-09
Lanett, AL

JoelC707 to koitsu

Premium Member

to koitsu
said by koitsu:

That's better than how it's historically been, yes, but the supervisor and I both agree that we should let things sit because it's just too soon to tell if it's fixed.

No matter what the field is, intermittent issues are a PITA to diagnose and fix. At this point I would definitely let things sit and see how things progress. Since you weren't able to get it to sync on the culprit frequency, you don't yet know if it's fixed and any further "tinkering" could basically "re-break" things if it's truly fixed right now. Only time will tell but I might suggest occasionally rebooting the modem and see if you can catch 753 to check that one too.

koitsu
MVM
join:2002-07-16
Mountain View, CA
Humax BGW320-500

koitsu

MVM

Yeah, that's the plan, Joel. I plan on sticking with 747 for the time being (will let the NE supervisor and the maint tech know how things look after 24 hours), then after some time (maybe a week?), reboot it and see if I can get 753 and keep an eye on that for a while. It's a good thing I wrote my polling/logging script, saves me a bundle of trouble.
koitsu

koitsu

MVM

Click for full size
747MHz SNR graph 11/08
This does not indicate the issue is solved, but I thought I'd provide a brief status update (I also called the NE supervisor and let him know this -- he's in agreement, still too soon to tell).

Modem uptime is almost 6 hours and I haven't seen the SNR for 747MHz drop below 36dB once. Number of uncorrectable frames is also stable (compared to when the issue happens, where uncorrectable FECs shoots through the roof).

Graph of the past 6 hours is attached (or folks can just refer to my website), specifically showing only 747MHz.

Data:

Nov 08 20:58:00    Device Model: Motorola SB6121 (hardware version 5.0)
Nov 08 20:58:00        Firmware: SB_KOMODO-1.0.6.8-SCM01-NOSH (Aug 30 2012 15:35:46)
Nov 08 20:58:00       Boot Code: PSPU-Boot(25CLK) 1.0.12.18m3
Nov 08 20:58:00    Modem Uptime: 0 days 5h:57m:51s
Nov 08 20:58:00    Modem Status: Operational
Nov 08 20:58:00 Down Channel  4: 729000000 Hz,   1 dBmV power, 36 dB SNR, mod: QAM256, corr: 3108, uncorr: 595
Nov 08 20:58:00 Down Channel  5: 735000000 Hz,   1 dBmV power, 36 dB SNR, mod: QAM256, corr: 3106, uncorr: 502
Nov 08 20:58:00 Down Channel  6: 741000000 Hz,   1 dBmV power, 37 dB SNR, mod: QAM256, corr: 2814, uncorr: 472
Nov 08 20:58:00 Down Channel  7: 747000000 Hz,   1 dBmV power, 36 dB SNR, mod: QAM256, corr: 2536, uncorr: 532
Nov 08 20:58:00   Up Channel  9:  23700000 Hz,  46 dBmV power, 5.120 Msym/sec, status: Success
Nov 08 20:58:00   Up Channel  7:  35400000 Hz,  47 dBmV power, 2.560 Msym/sec, status: Success
Nov 08 20:58:00   Up Channel  8:  30600000 Hz,  46 dBmV power, 5.120 Msym/sec, status: Success
 
koitsu

koitsu

MVM

It's been a while since I've posted in this thread, but I wanted to provide a follow-up.

For about 3 weeks the 747/753MHz issue was solved. Well, at least I can say that it was for 747MHz, I couldn't get my modem to pick 753MHz (just total chance on that, no technical reason for it) so I couldn't test it. Everything was peachy keen, great SNRs all times of the day, etc.. The same for my neighbour.

Then on 11/30 we had a fairly large storm (very heavy winds, and what I would classify as "medium to heavy rain" but what Californians would classify as "absolutely crazy heavy rain" ), starting at around 04:30. At about 05:30 the wind really kicked in hard and my SNR on all frequencies started crapping out, along with T3 timeouts, etc... After a few hours, things settled down and sort of stabilised -- or so I assumed.

It wasn't until ~16 hours later when I looked at my SNR graphs that I realised the SNR issue had returned. Meaning: as a result of what happened on 11/30, the SNR issue is back. It's 100% identical -- issue tends to start around 7am and last until midnight, with SNRs dropping down to 31dB at times. Correlates directly with hours of the day when cellular traffic increases. My neighbour's seeing the same thing with his SNR (for 747MHz) as well.

I reached out to the Network Engineering Supervisor in my area to inform him of this situation, to get those network/maint techs back out here to figure out what is going on. I was told the following, verbatim:

"I will have {tech} look into this issue as soon as he has a chance too. We have a lot of clean up to do after the storm. There was
some damage cause to an amplifier on the node that feeds your home so this could be the issue out there."

I have no idea if this is the same line amp they worked on 4 weeks ago or not.

Anyway, I'm inclined to think whatever's causing this issue is likely some loose wire or bad cable run (between poles, not to my apt building) or a line amp that is just downright horked/busted in some way (allowing interference), probably as a result of wind. The interesting (amusing?) part of this story is that a busted line amp was proposed by a Comcast Forum tech at something like the 6-7 week mark.

I don't know what strings I'm going to have to pull to try and get credits either, but I'll tell ya, for the amount of time I'm having to put in on this (especially feeling like I have to micromanage people), I really do feel like I'm entitled to credits. Plus, as admitted above, this affects an entire node, not just me -- so I'm not the only customer who's affected.
koitsu

koitsu

MVM

Been a couple weeks since my last update, though I'd provide insights to where things are now.

After another set of mails on Monday (17th), the Network Engineering Supervisor took a look at everything remotely (my apartment's connection and my neighbour's) and confirmed that things were hitting 31-32dB on 747MHz range (I also included screenshots in my mails to him). This was around 13:00 or thereabouts. Within 15 minutes he had the previous network/maint tech from weeks ago out here taking readings, etc..

Tech took readings at the cable box, looking at 747 and 753MHz on his analyser, and confirmed the problem. Even more interesting was that he took a reading from an open input (literally holding up the input jack in the air) and looked at what the gain was at. Normally in our area, for 73x-75xMHz, the reading is in the -40dB region. These were his readings from behind our building (I saw them myself):

735MHz = between -44dB and -46dB
741MHz = between -43dB and -46dB
747MHz = between -31dB and -37dB
753MHz = between -27dB and -33dB

Note the large jump in average gain starting at 747MHz.

The tech told me that a few weeks prior, he took readings in the same manner for every cable/line segment down Church Street (which is where our "main connection" runs to the node), and all the numbers on all the frequencies were in the -40dB range. However, when you approach this area of my block, the gain for 747MHz and above starts going off the charts. What this means is that there is absolutely something in the air (i.e. wireless signal of some kind) that is pounding on 747MHz and above. That's perfectly fine too, but the issue (as the tech explained) is that that signal is seeping in to the node somewhere along this block -- specifically in the general line of sight of that cellular tower that was found via visible inspection weeks ago. (It would really help if I could draw a diagram... Maybe I'll work on that later)

The tech stated that to diagnose the issue, that tomorrow (12/18) he would come out early in the morning (6:30am) and install a tap/splitter on the cable that runs *to* the line amp, and hook me -- *just me, not my neighbours* -- up to that tap (this is apparently called a "ground line"), then monitor both my connection as well as my neighbour's. So in effect, I would have a direct line to the hub that connects to the node (i.e. no line amp for me), while my neighbours setup wouldn't change. This would be a temporary measure solely to try and narrow down which set of cable or piece of equipment is most likely the issue.

The theory is that if both myself and my neighbour continue to see the SNR issue, then it means the bad cable or bad piece of equipment is either at/around the tap I'm on, or further back from that (i.e. closer to the node). However, if only my neighbours see the SNR issue, then that means the bad cable/equipment is either the line amp or the stretch of cable that's between the line amp and where their connection comes in. (Sorry, this is really hard to describe without a visual diagram) We all agreed it'd be the best troubleshooting method for this, and scheduled to have it done between 0630 and 0830 the following morning.

So here comes the worst part of it all..

At 1930 that same night the tech came out and took readings/proposed the above, just out of no where, the SNR issue ceased. I changed nothing. My neighbour changed nothing. It just suddenly stopped for no reason. This has been one of the only times where things suddenly working actually made me furious!

The following morning the tech came out as promised and began the work proposed. I told him of the SNR issue ceasing the previous night and he was bummed out by that too, but regardless (understandably) went ahead with the proposed changes to the cabling to try and narrow this down. He padded the tap/splitter (so no need to install additional splitters at our cable box) and voilĂ , all done.

I also had him do another open input check with his analyser (I was wondering if the cellular tower had been shut off or simply stopped broadcasting) -- nope, still broadcasting nice and strong, i.e. 747/753MHz still have really high gain.

Anyway, my signal stats before:

Dec 17 14:56:00    Device Model: Motorola SB6121 (hardware version 5.0)
Dec 17 14:56:00        Firmware: SB_KOMODO-1.0.6.8-SCM01-NOSH (Aug 30 2012 15:35:46)
Dec 17 14:56:00       Boot Code: PSPU-Boot(25CLK) 1.0.12.18m3
Dec 17 14:56:00    Modem Uptime: 4 days 13h:10m:34s
Dec 17 14:56:00    Modem Status: Operational
Dec 17 14:56:00 Down Channel  1: 705000000 Hz,   0 dBmV power, 37 dB SNR, mod: QAM256, corr: 161369, uncorr: 733
Dec 17 14:56:00 Down Channel  5: 735000000 Hz,   0 dBmV power, 37 dB SNR, mod: QAM256, corr: 187313, uncorr: 590
Dec 17 14:56:00 Down Channel  6: 741000000 Hz,  -1 dBmV power, 37 dB SNR, mod: QAM256, corr: 179164, uncorr: 479
Dec 17 14:56:00 Down Channel  7: 747000000 Hz,  -1 dBmV power, 32 dB SNR, mod: QAM256, corr: 208803, uncorr: 560
Dec 17 14:56:00   Up Channel  8:  30600000 Hz,  46 dBmV power, 5.120 Msym/sec, status: Success
Dec 17 14:56:00   Up Channel  7:  35400000 Hz,  47 dBmV power, 2.560 Msym/sec, status: Success
Dec 17 14:56:00   Up Channel  9:  23700000 Hz,  46 dBmV power, 5.120 Msym/sec, status: Success
 

Signal stats after being given a "ground line":

Dec 19 19:02:00    Device Model: Motorola SB6121 (hardware version 5.0)
Dec 19 19:02:00        Firmware: SB_KOMODO-1.0.6.8-SCM01-NOSH (Aug 30 2012 15:35:46)
Dec 19 19:02:00       Boot Code: PSPU-Boot(25CLK) 1.0.12.18m3
Dec 19 19:02:00    Modem Uptime: 1 days 10h:25m:38s
Dec 19 19:02:00    Modem Status: Operational
Dec 19 19:02:00 Down Channel  2: 717000000 Hz,  -1 dBmV power, 36 dB SNR, mod: QAM256, corr: 59185, uncorr: 635
Dec 19 19:02:00 Down Channel  3: 723000000 Hz,  -1 dBmV power, 36 dB SNR, mod: QAM256, corr: 61702, uncorr: 635
Dec 19 19:02:00 Down Channel  6: 741000000 Hz,  -1 dBmV power, 37 dB SNR, mod: QAM256, corr: 60489, uncorr: 722
Dec 19 19:02:00 Down Channel  8: 753000000 Hz,  -2 dBmV power, 36 dB SNR, mod: QAM256, corr: 54379, uncorr: 634
Dec 19 19:02:00   Up Channel  8:  30600000 Hz,  38 dBmV power, 5.120 Msym/sec, status: Success
Dec 19 19:02:00   Up Channel  7:  35400000 Hz,  39 dBmV power, 2.560 Msym/sec, status: Success
Dec 19 19:02:00   Up Channel  9:  23700000 Hz,  37 dBmV power, 5.120 Msym/sec, status: Success
 

So now we simply wait for the SNR issue to happen again (to either myself and my neighbour, or just my neighbour).

I'll see if I can draw up a diagram of how all the cabling and stuff is run, just because it makes things a lot easier to understand.

As far as a root cause goes? The supervisor, tech, and myself are all in agreement -- the most likely candidate is something environmental, ex. a hair-line crack in some cable that gets stretched open when it's windy, a line amp that's allowing interference, or possibly a tap/splitter that's doing the same. We don't know which, but we're trying to narrow down the geographic location of which lines/cable/equipment it is.
koitsu

koitsu

MVM

Click for full size
Network cabling diagram
Click for full size
Cellular tower location vs. home
Attached are diagrams. I'm not the best artist, sorry.

All the pink/magenta lines are my neighbours, which are coming off the tap/splitter that's off of the "red cable" coming from the line amp. After the work done on 12/18, I'm coming off a tap/splitter that's along the "blue cable". Hopefully this makes things a little more clear.

Obligatory Google Maps link for those who want to scout around my area to get an idea.

And you can actually see the cellular tower using Google Maps' Street View (centre of picture), but it's hard to spot since it's painted the same colour as the building. But it's there. *shakes fist*

dmk08809
join:2001-09-08
Clinton, NJ

dmk08809

Member

Koitsu, Thanks for your detailed postings and chronology of events! A story like this gives me hope that maybe my issue will someday be fixed. I think a key difference is that you actually have a real inside contact that was able to push the issue. From what I've seen here in NJ using normal escalations processes for mere mortals is slow, inconsistent, and quite possibly a waste of time. Furthermore, providing details (like SNR) to most techs seem to be a waste of breath -- they're only interested in what they see on their meters at that exact moment. *sigh* Good luck and keep accounting and sharing the story!

koitsu
MVM
join:2002-07-16
Mountain View, CA
Humax BGW320-500

koitsu

MVM

You're welcome. As far as getting an "inside contact", that happened through strange/bizarre means in itself. Details are here (first part):

»Re: What to do about 11 weeks of node/network problems?

What I can't figure out is if this Comcast manager really was calling people along my street or if it was the result of a VM I left someone else at Comcast stating that I was basically at the end of my rope + I felt like the only choice I had left was to send a letter (along with a diary which I've been keeping of all the events from day one) to the CEO.

If you're struggling though, HeadendJoe recommended contacting your LFA (local franchise authority) -- particularly if there's a widespread issue affecting citizens in an area, or "plant issues". The address/phone of your LFA is printed on your bill (on paper or digitally/PDF available via the Comcast site).

The bottom line is that customers should not have to go through this kind of rigmarole to get something fixed. The only outlet that people know of is 800-COMCAST, and all the CSRs can send out are standard service technicians (who are the ones who have to make the call on whether or not to send a CR (case/ticket) to network, if the issue is a cable network issue). And even if it makes it to network, there's absolutely zero communication with the customer from that point forward -- so if the issue persists, the customer has no idea what to do, calls 800-COMCAST again, schedules another service tech visit, rinse lather repeat. This system is just downright broken, and it's been like this for at least 10 years. It doesn't work.

AnonMan
@comcast.net

AnonMan

Anon

Welcome to my fun...

I have an issue that 2-3 times a night my modem reboots, Upstream SNR dips below 29. Downstream SNR great, 42, power levels are good with the use of forward path attenuator.

I have maint sup direct cell phone and he has me on monitoring and sees the drop every night. His direct reports I have they cell phone also and they have seen it and it's the entire node but it is so quick and random on the time (beyond it being at night) they can't figure it out.

Sucks. Something is causing noise but only for a moment every night at a different time.

I feel bad, I am a pain in the butt customer but I am paying for service I want. They are trying but is a crappy situation. When issue is random and happens for a minute tops no way to catch it to track it because it's a different time every day.

koitsu
MVM
join:2002-07-16
Mountain View, CA
Humax BGW320-500

koitsu

MVM

AnonMan, that situation is indeed crummy. Due to the highly intermittent nature tracking it down (to a specific cable/area of the node) sounds very difficult and time consuming. On the other hand, if it's happening on the entire node, that means more than just you is impacted -- bringing this up with your LFA may be the next step for you. Node-level issues I think are given more attention than individual customer issues.

Also, I've been told by both the Network Engineering Supervisor as well as the network/maint tech who was out here, that Comcast does have a "history view" for SNR levels on a per-channel basis (including support for bonding), so they should be able to see it there. Meaning, they have something similar to what I provide here (from my own modem) but probably with less granularity (I poll every 2 minutes): »jdc.koitsu.org/snrgraph/ -- I know Comcast having that information doesn't help narrow down where on the cable network/segment the problem lies, but it would act as evidence (for your LFA) that Comcast acknowledges there's a problem affecting multiple people on your node.
JoeHemi
join:2011-05-06
united state

1 recommendation

JoeHemi

Member

Yes, the system is called scout monitor and shows the history and real time data for each STB & modem for every customer. The other system, which I like better, is called Watch Tower. Much more detailed analysis of the relationship between the customer's equipment & the CMTS/Narrowcast. Other nice thing about watch tower is that techs can submit notes when they troubleshoot that node and those notes are visible to anyone troubleshooting your node in the future. Other field tool the techs can use to monitor the signal from your node is Pathtrack which shows a virtual representation of the return signal so they can see a quick visual of noise, carrier problems, etc and take snapshots of any problems at any interval. They can also call in to the headend and have them turn up the return sweep. The techs sweep that node segment and can determine issues. Many, many tools out there, but, all the tools can do is tell the tech there is a problem, not how, where, or why.

koitsu
MVM
join:2002-07-16
Mountain View, CA
Humax BGW320-500

koitsu

MVM

Been a few weeks since my previous update -- keeping it short this time. Things are still continuing to be investigated (maint/line techs have been out twice in the past 48 hours, for example) and still working on narrowing down what section of cable or devices are susceptible to interference in the 747MHz and above freq. range.

One key point I thought I'd mention: putting me on a "direct line" or "ground line" did not alleviate the issue, meaning the interference is presumably coming in somewhere along Church Street (two spots are being focused on at this time).

I've kept in very close contact with the Network Engineering Supervisor during all of this -- in fact, he's who I mainly communicate with when giving updates/information.
koitsu

1 recommendation

koitsu

MVM

Click for full size
Click for full size
Strange things are afoot at the Circle K! Today a couple trucks from CableCom were out working along Church Street (particularly at the Hope/Church intersection) with a very large spool of cable (not pictured).

I'll have to ask my Comcast contact what was done, but my initial guess is a new set of cable has been run, meaning the existing cable may have degraded/been damaged in some way (identical to that in San Carlos some time ago (reference)). Comcast maint/line techs were out here yesterday for 10-12 hours straight, spending their time along the section of Church Street (map).

I imagine there will be some lengthy downtime in the morning sometime within the next week or so as Comcast switches over to the new cable run, but that's perfectly fine by me assuming it solves the issue.
koitsu

1 edit

koitsu

MVM

Yesterday (01/04) morning I lost sync and used the opportunity to run an errand. Both Comcast line/maint techs were out tinkering with the cable which CableCom had installed the day prior, so I had an opportunity to chat with them.

The techs spent a lot of time at very early hours (5-6am) on 01/02 narrowing down exactly where the interference was seeping in. Doing this required placement of terminators on the main cable run each step of the way (starting at behind our building, then down to the spot on Church Street where the cable heads south-east, and so on), then examining the SNR with an analyser. Time consuming but it's what it took.

They found that the interference was being introduced in the stretch of cable between the spot behind our building and Church Street (see map -- referring to the blue line that goes from behind my building heading north-east, where it intersects with Church Street, before going south-east). The cable had no visual anomalies (e.g. no deep gnaw marks from our aggro squirrels ), but was definitely bad in some way. They've taken the cable and will be using it for "educational purposes" + doing further analysis on it using a TDR. The cable was also encompassed by all sorts of tree branches (the trees are on private property; the guy who owns the house next to our building really needs to have someone out to prune them. They're a good 5 or 6 years overdue).

The tech doing most of the analyser work said that with his analyser hooked up and with the terminators in place, on that cable segment, he could see SNR fluctuate badly for the 747 and 753MHz range, jumping around constantly between roughly -28dB and -39dB. "It was all over the place; it shouldn't be like that".

During the troubleshooting process, the techs also found that the cable between that point at Church Street, and the Hope/Church intersection, was also damaged (this time visibly so). This cable was not the source of the interference seepage, but the techs told me that they preemptively had CableCom run a new cable for that as well, because "in 1-2 years the cable would have degraded to the point where the interference issue would have recurred". The root cause of damage to that cable seems to be the (city-owned) trees along the street; they didn't go into details.

On the morning of 01/04 they switched everything over to use the new cables.

Lo and behold, my SNR on all frequencies is now 37dB and steady. With their analyser it shows something like -35 with little to no variance. On my end, there's no variance at all -- it's absolutely rock solid and that's quite apparent in my SNR graphs. In fact, the occasional +/-1dB I'd been seeing on some of the other frequencies too are now also gone.

Rephrased: since the cable replacement, my connection has been rock solid; Youtube videos load quickly, I'm not seeing massively incrementing uncorrected error counts, and no longer do I see packet loss.

The techs and I both agree there is a huge improvement since the cable replacements, but also agree that we should revisit things in some weeks (maybe 4-6 weeks). They'll also use that opportunity to get rid of the "ground line" and put me back on the same splitter/hub as my neighbours (i.e. reverting back to the original cable configuration prior to the work done on 12/19). I also get to discuss with appropriate managers the possibility of service credits.

It's evening time here, but tomorrow I'll go out with my camera and record a short video showing the general vicinity/area of where the cables were replaced.

So assuming the cable replacement fixes the issue: from start to finish, the entire ordeal lasted from roughly August 10th 2012 until January 4th 2013. That's almost 5 months.