dslreports logo
 
    All Forums Hot Topics Gallery
spc
uniqs
55
cramer
Premium Member
join:2007-04-10
Raleigh, NC
Westell 6100
Cisco PIX 501

cramer to bluepoint

Premium Member

to bluepoint

Re: How to lease SB6141?

The logical extension of this... they tested a specific firmware version. So, if you show up with any other version, they don't let your modem on the network. They're still NOT going to upgrade customer owned modems. They've been down this bumpy road and have no desire to try it again. It sucks, but I agree with their reasons.

Ideally, they should have a portal to allow *you* to request an update. But such a system would have a boatload of security implications.

bluepoint
join:2001-03-24

bluepoint

Member

said by cramer:

The logical extension of this... they tested a specific firmware version. So, if you show up with any other version, they don't let your modem on the network.

Why not, they recommended it, they provisioned it? Why not let it in the network?

They're still NOT going to upgrade customer owned modems. They've been down this bumpy road and have no desire to try it again. It sucks, but I agree with their reasons.

Do you have a link or evidence that what your saying is TWC's position?
Implications of letting subscriber's buy their own modem is huge but just to think they did not think of consequences is bad on their part. My position is, TWC told us to use certain brand and model which is understandably reasonable decision on their part so they can manage them. If their position is, "they don't care", they would have just let us get any DOCSIS 3 model. No matter how you think there will still be support generated if there is a problem. Would you imagine the phone support calls it will generate if the recommended modems are not compatible with their system? If firmware updates are needed, I think they're better off pushing it to their subscribers instead of truck rolls and truck rolls to determine the problem.

Ideally, they should have a portal to allow *you* to request an update. But such a system would have a boatload of security implications.

If they don't want to do anything with subs owned modem, allowing subs install the firmware on their own would not be a bad idea provided they provide a secure server that people can get the file and that they have tested the firmware. Either way they will be involved.
cramer
Premium Member
join:2007-04-10
Raleigh, NC
Westell 6100
Cisco PIX 501

cramer

Premium Member

Do you have a link or evidence that what your saying is TWC's position?

No, I do not have any TWC internal memos. I can only go on the word of many people across the internet who have dealt with TWC w.r.t. their own owned modem.

To the letter of the law, any DOCSIS 3 certified modem *should* work. However, we have ample evidence showing that testing an certification is almost useless... in the real world, any random modem on any random network could (and usually does) run into some sort of compatibility issues. ISPs test hundreds of modems -- different brands, models, and yes firmwares -- before putting them in customer hands. This is why you see many more modems in the "leased" box vs. the "allowed to be owned". Leased modems are much more closely controlled -- no matter how they come from the factory, they'll get the tested and approved for the local system firmware. They have no control over what version is on the retail modem at Frys.

Ordinarily, it wouldn't be an issue to reflash the customer owned modems. HOWEVER, there was a huge mess some years ago when they upgrade *all* the (buggy as hell to begin with) SB6120's. They bricked a lot of them -- their own, too. And generally just made the bugginess worse. Customers started *watching* the version on their modem and bitched when the ISP "upgraded" them. So now, we have TWC's apparent hands off policy. (there are plenty of posts right here of people getting the same "no" to upgrades. I don't understand why it's so hard for you to accept.)

The issue of allowing customer firmware upgrades was carved in stone MANY years ago. (in the original DOCSIS 1.0 specs?) Firmware updates MUST come from the HFC interface (as per the DOCSIS config file.) You are obviously too young to remember the era of uncapped cable modems... there were two ways of getting there... feed it a config file from the LAN side, or hack the f'ing firmware. And thus the rule was born.

(I'm not saying that rule is still relevant. Any modem that disobeys it's QoS rules should be quickly detectable by the headend.)

bluepoint
join:2001-03-24

bluepoint

Member

said by cramer:

Do you have a link or evidence that what your saying is TWC's position?

No, I do not have any TWC internal memos. I can only go on the word of many people across the internet who have dealt with TWC w.r.t. their own owned modem.

Your conviction seems to be strong on the basis of what you see in the internet.

Leased modems are much more closely controlled -- no matter how they come from the factory, they'll get the tested and approved for the local system firmware. They have no control over what version is on the retail modem at Frys.

They are all the same hardware from the same manufacturer, no difference in my opinion.

Ordinarily, it wouldn't be an issue to reflash the customer owned modems. HOWEVER, there was a huge mess some years ago when they upgrade *all* the (buggy as hell to begin with) SB6120's. They bricked a lot of them -- their own, too.

SB6120 isn't an old model that I will not see people complain TWC bricked their modem. You are making it look like TWC bricked many of this modem but I have not seen them here. Do you have a link? Maybe I missed them. I've seen people complain about being locked out of the modem but nothing else?
cramer
Premium Member
join:2007-04-10
Raleigh, NC

cramer

Premium Member

No, I didn't bookmark them. (3-4years ago?) You seem to be unable to accept what anyone else says anyway, so what's it matter.

swintec
Premium Member
join:2003-12-19
Alfred, ME

swintec

Premium Member

said by cramer:

No, I didn't bookmark them. (3-4years ago?) You seem to be unable to accept what anyone else says anyway, so what's it matter.

I seem to remember this being a charter problem, not TW...actually made DSL Reports front page.
cramer
Premium Member
join:2007-04-10
Raleigh, NC

cramer

Premium Member

I found a single reference about a SB6120 and some Dlink with a few seconds of googling. Not that he'd believe anything off the internet.

bluepoint
join:2001-03-24

bluepoint to cramer

Member

to cramer
said by cramer:

No, I didn't bookmark them. (3-4years ago?) You seem to be unable to accept what anyone else says anyway, so what's it matter.

What matters is truthfulness. What you read without basis to show for will always not be accepted as fact.
Mele20
Premium Member
join:2001-06-05
Hilo, HI

Mele20

Premium Member

I don't recall TWC bricking any 6120 modems but we have not used that one in Hawaii so I suppose it is possible I missed this happening...unlikely though as I read this forum regularly and have since before I joined this site back in 2001.

You guys don't seem to realize that it is RARE for TWC to upgrade the firmware on any of their modems. In the almost 12 years I have had Road Runner, I have seen TWC upgrade the firmware on Surfboards TWO times only. The first time, I was responsible for forcing them to either upgrade the firmware on the SB 4100 due to a security problem or face lawsuits. Since every other cable ISP of reasonable size world wide had, a year or more earlier, upgraded the firmware to fix the security hole EXCEPT TWC which was still on the original factory issued firmware version, TWC didn't have much choice when I challenged them on it. They agreed to push a firmware upgrade across all divisions (not just Hawaii where my complaint started out) and agreed to start keeping timely firmware updates as Comcast and others were already doing.

They pushed the firmware update shortly afterwards. Their excuse for ignoring the security update for so long when other cable companies pushed it as soon as Motorola released it was that they had bricked some modems once with a firmware update...but that was many years ago...*long before the 6120) and a rather feeble excuse since other cable companies had updated the SB 4100 firmware multiple times with no ill effects. The other firmware update pushed by TWC between 2001-2013 was to make my SB 5100 DOCSIS 2 capable.

I got the SB6141 from Oceanic TWC when I upgraded to 20/2 recently. It was a brand new one still shrink wrapped and had notes with it in its factory box. It has the firmware from 2011 that you guys are griping about TWC not updating your personal bought ones that have that firmware. Why would TWC update yours when they have not updated their own? They won't push a firmware update unless absolutely necessary. TWC is notorious for this bad practice and they have not kept up that part of the agreement we reached many years ago.

DocDrew
How can I help?
Premium Member
join:2009-01-28
SoCal
Ubee E31U2V1
Technicolor TC4400
Linksys EA6900

4 edits

1 recommendation

DocDrew to cramer

Premium Member

to cramer
Click for full size
It wasn't a modem firmware update that caused the problem. It was a CMTS software update to meet updated DOCSIS 3 specs that wasn't compatible with older 6120 firmware, specifically version 1.0.1.8.

The bigger problem was Motorola was still distributing version 1.0.1.8 in it's retail modems several months after the problem came to light (1.0.1.8 was over a year old at that point, the new firmware release on retail after Moto acknowledged the problem was 1.0.3.3). The 1.0.1.8 modems wouldn't sync on CMTSs with the newer software, but the modems weren't bricked. TWC or the other cable companies couldn't update them normally since they wouldn't sync. TWC had updated the 6120 modems on their network before the CMTS update. I'd guess this is why TWC doesn't officially support new customer owned 6120s on their network, since there are still ones with 1.0.1.8 new in the box wandering around. 6121s had newer firmware from the start.

This hit all the MSOs: Charter, Comcast, Cox, BrightHouse, Optimum, and TWC all had threads about the issue here on DSLR:
»[Connectivity] Motorola SB6120 doesn't work with CC-Resolved
»[AZ] SB6120 Provisioning
Many mistakenly thought it was a ISP modem firmware update failure since Motorola never said anything about it publicly. ISPs just had to deal with the fallout and old stock of 6120s ever since. Even Karl misreported it: »Firmware Update Cripples Some Motorola SB6120 DOC 3 Modems [49] comments

If anything, TWC learned to update customer owned modem firmware more often. As far as I know TWC does update customer modem firmware, when they update other modems of the same make and model. It's can't be requested from customer support, they can't do it, most don't even know how it's done so you get a bunch of odd and wrong answers about it. Modem firmware updates are pushed by an automated process initiated/scheduled by TWCs Engineers.

BTW, the majority of 6120 modems I've seen in my area on TWC are running firmware 1.0.6.3. Retail modems never shipped with that. It was released Aug 2011 by Moto and mid last year by TWC. Motorola has averaged a new firmware for the 6120 every 2-3 months for almost 4 years now, most ISPs are 3-6 months behind at best and none I've seen push every release. The latest I've seen is 1.0.6.10 released Dec 2012.

bluepoint
join:2001-03-24

1 edit

bluepoint

Member

Good info DocDrew See Profile, thanks.

If anything, TWC learned to update customer owned modem firmware more often. As far as I know TWC does update customer modem firmware, when they update other modems of the same make and model.

cramer
Premium Member
join:2007-04-10
Raleigh, NC
Westell 6100
Cisco PIX 501

cramer to DocDrew

Premium Member

to DocDrew
Am I the only engineer who read that tech note and thought, "well that's easy enough to fix." (not having used an Arris CMTS, I don't know how hard it is to remove that attr from it's response.) After all, why send a D3 attr to a D2 modem? (TWC's database does know what your modem is. myservices/myaccount shows the correct modem for my accounts.)

DocDrew
How can I help?
Premium Member
join:2009-01-28
SoCal
Ubee E31U2V1
Technicolor TC4400
Linksys EA6900

3 edits

DocDrew

Premium Member

said by cramer:

Am I the only engineer who read that tech note and thought, "well that's easy enough to fix." (not having used an Arris CMTS, I don't know how hard it is to remove that attr from it's response.) After all, why send a D3 attr to a D2 modem? (TWC's database does know what your modem is. myservices/myaccount shows the correct modem for my accounts.)

Huh? The 6120 is a D3 modem.

I also don't think that TLV option is particular to D3, if I remember correctly it's a parameter used in "PowerBoost" configs. The problem was 1.0.1.8 firmware didn't handle that option properly and it was being sent whether or not it was configured on newer CMTS software due to changes in the D3 spec making it default in some way. Firmware before or after didn't have that problem. The vast majority of other modems didn't have an issue with it.

Motorola and their distributors made it worse by selling modems with old firmware for months, even after the problem was found.
cramer
Premium Member
join:2007-04-10
Raleigh, NC
Westell 6100
Cisco PIX 501

cramer

Premium Member

Right. No sleep and no caffeine.

Either way, how hard is it to not send the TLV? On a Cisco, it's usually as easy as "no" in the right place to turn off a default. (maybe not to a specific subset of modems, 'tho)

As for "old stock"... I know first hand how much of a pain in the ass it is to get a new firmware image into the manufacturing processes. (granted, our products are trivial to fix in the field.)

DocDrew
How can I help?
Premium Member
join:2009-01-28
SoCal
Ubee E31U2V1
Technicolor TC4400
Linksys EA6900

1 edit

DocDrew

Premium Member

said by cramer:

Either way, how hard is it to not send the TLV? On a Cisco, it's usually as easy as "no" in the right place to turn off a default. (maybe not to a specific subset of modems, 'tho)

It's possible, but I think it was made more complicated by the use of PowerBoost.

Some operators using Walled Garden setups changed their default config files for non-provisioned modems in specific ways to allow the modems to sync. That coupled with an automated firmware update system for those specific modems allowed them online. Obviously that meant some extra hoops to jump through for modems already authorized.

So between the failure, time to troubleshoot down to the modem firmware/CMTS software incompatibility, solution testing, and large scale solution deployment, it was several months. All for 1 model modem with 1 specific firmware version which frequently users didn't know had a problem until it was plugged in.

This sort of firmware incompatibility situation happens somewhat frequently with modems, usually not to the point of modems not syncing, but it was made magnitudes worse by the retail distribution of the 6120s and old firmware.

j1349705
Premium Member
join:2006-04-15
Holly Springs, NC

j1349705 to cramer

Premium Member

to cramer
said by cramer:

The logical extension of this... they tested a specific firmware version. So, if you show up with any other version, they don't let your modem on the network. They're still NOT going to upgrade customer owned modems. They've been down this bumpy road and have no desire to try it again. It sucks, but I agree with their reasons.

Ideally, they should have a portal to allow *you* to request an update. But such a system would have a boatload of security implications.

This is kind of off topic, but OBIHAI VoIP ATAs (commonly discussed in the VoIP Tech Chat forum here) have a nice optional web management portal that lets you centralize administration of all of your OBi devices. When a firmware update is available, they do not automatically push the update - instead, an alert icon pops up next to any devices with an update. If you want to install it, just click the icon and within about 1 minute, your ATA will reboot with the latest firmware. Obviously they have it a lot easier since this is a single vendor solution, but it is a nice implementation.

For residential customers, the My Services portal already has a feature that will show whether or not your equipment is online or not. If it is easy enough to query the firmware version of the modem (possibly over SNMP?), it shouldn't be too hard to add an update function that would be reasonably secure, possibly with a customer interface similar to what I described above. They can force the customer to accept the terms and conditions before pushing the update. This (should) make both sides happy. Customers can get bug fixes when they want, and customers who are happy with their modems can skip the update if they are concerned about bugs. If they can securely offer remote management of DVR recordings and Digital Phone features, it seems like a secure way to update modems should also be possible. Obviously, the update function would have to be set up to only allow official firmware from the cable modem vendor. If TWC wants to do more testing before approving the update, I'm all for that as well. I'm not suggesting that they should allow customers to update firmware images... that would not be a good idea, and most of the time customers can't get them anyways.

There is definitely a lot of conflicting information on how they have handled or will handle updates for customer owned equipment in the future. And both sides have valid points. However, I think the policy of not updating will be far more likely to cause problems for both TWC and customers in the long run. Not many cable modem firmware release notes get released to the public, but some of them I have seen definitely show some important bug fixes.
cramer
Premium Member
join:2007-04-10
Raleigh, NC
Westell 6100
Cisco PIX 501

1 edit

cramer

Premium Member

Not to keep kicking the dead horse, but... Either way they go, there will be issues. No updates mean there will be buggy modems out there. Doing the updates means some of them are going to fail -- yes, with a sample size as large as an ISP, there. will. be. failures. (I've been there, signed the PO for the t-shirts.) If you screw up a modem, you'll have to replace it. This makes for a very pissed customer -- esp. if they owned the modem. Angry customers are very vocal customers.

DOCSIS 1.1 introduced a number of security processes for firmware updates. The process is specifically engineered to only come from the network, so no manufacturer even has a channel to get a new firmware to end users.

Given that, it follows that the firmware on any given modem will either be a "GA" release loaded from the factory, or an ISP specific firmware they loaded on it. The operator has to authenticate every modem on the network. So they (should, can, etc.) know the manufacturer, model, serial number, and firmware version of every device out there. Proper testing means having one of everything in your testlab. How the 6120 bug didn't happen in their lab leaves me bald. This wasn't some random Chinese modem; it was a very popular, and approved, modem they themselves used, and they were coming from the factory with the old/buggy firmware. It makes no sense that they wouldn't have *one* 6120 with that firmware on it in the lab. Or that they wouldn't've retrieved one from the field as soon as this started to put it in the lab to see *why* it's failing. (I'm also assuming motorola didn't have it striped down so far it wouldn't log why it was rebooting. "invalid field" doesn't help much if it doesn't say which field; that doesn't mean you cannot find it, just that the engineers will have to earn their massive paycheck to find it.)

I've been there and worked through many a panic. The test lab cannot simulate an entire universe. Bugs are going to be missed; this is a fact. The 6120 issue should've been seen in the lab, but it wasn't. (we'll come back to that when we're looking for someone to fire.) This is why it's suicide to do any rapid, large scale roll out. When an issue surfaces -- and it will -- you now face a network-wide problem instead of localized problem. Rolling back an entire network is, well, more of mess than the upgrade. "Staged Upgrades", say it with me class. Given the size of TWC (Comcast, etc.) network, they should never be finished with an upgrade -- they should never have the same version on every system... there should be several versions rolling across the network like ripples in pond. </rant>

[Edit: I've worked for and with several ISPs over the decades, but TWC isn't one of them. I have no idea what they test, or how -- and it's never come up in conversation with the people I know there. There are pictures of someone's test lab over in the bbphotos forum.]

rchandra
Stargate Universe fan
Premium Member
join:2000-11-09
14225-2105
ARRIS ONT1000GJ4
EnGenius EAP1250

rchandra

Premium Member

What really annoyed me is that, according to the status Web page from the modem, it was coming online enough that, if they so chose, I think they could specify firmware to download (TFTP of parameters, IIRC), but it was outside their SOPs. So I was forced to negotiate with the vendor (Best Buy in this case) to swap it.

They also ran into an issue where the modem would "upgrade" itself every night, and therefore reboot...for weeks. I basically didn't care because very close to 100% of the time it happened while I was asleep, so I didn't notice it at all until I looked at the log page for a reason why my DSLR monitor showed unreachability .