xymox1 Premium Member join:2008-05-20 Phoenix, AZ ARRIS SB8200 MikroTik CCR1036-8G-2S+
3 edits |
xymox1
Premium Member
2021-Oct-23 3:45 pm
Re: SB6190 Puma6 TCP/UDP Network Latency Issue DiscussionSide note on my doomsday scenario.. I forgot one tricky aspect. If a MSO maintenance network got taken over by 16 million devices it might be hard to even find enough bandwidth to upload a firmware fix, assuming you could. A bot net might also cause a whole network crippling overload. Thank the cableLords ( cablelabs ) for priorities at the CMTS level for packets and maintenance packets are the highest level. Wait.. Does that mean the CMTS might treat packets from compromised CPE at that highest level ? Wow if so, EEEgads.. Good luck trying to get hold of that. Maybe there is a GOD level priority only known to Arris / Cisco in the CMTS.
I guess the CMTS will be the gatekeeper under network duress. Thank god for that.
So there should be a way to take control of that. Maybe on the CMTS just lock out everybody, then let in small groups and firmware update. Go block by block.
If they turned them into RF killers tho, which I would think would be easy, and then go ransom ware,, well #$@#$@%..They are all spraying out of sync crap everywhere on all channels. . God,, imagine if they had control of the botnet and could cripple the ISP at will on a RF level.. Can you say ransomware ??.. Jeeze how do you bring that back on-line just to fix it ?
I gotta stop posting. This is a insane issue with more doomsday scenarios then i can imagine. AND IT APPEARS TO BE SECURED BY, uhhhhhhhhhhhh.... Hmmm...
I should write a short story. "The day the earth stood still - for MSOs".. Hmm I will watch that again later.. The original B&W of course..
BUT I have more fun things to do as hobbies today then this. I am restoring a CRT video projector, which is way fun...
I will ponder and craft a form letter I will send to all the parties involved later this evening. I will also post it here.
Does a big wide open network hole like this qualify to do CVEs ? Would the target be each MSO ? Can a CVE apply to a industry ? Is this just a BP only ? I need to start planning how to escalate this to resolution. I need a plan. Maybe start to inform press. Maybe LightReading might be a good place for a story too. |
|
kevinds Premium Member join:2003-05-01 Calgary, AB |
kevinds
Premium Member
2021-Oct-23 5:48 pm
I would suspect that the management IP networks would be isolated from each other, but modems on the same IP network wouldn't be.
It would be difficult to, at scale, make rules preventing modems from talking to each other. Easy to make firewall rules on the CMTS side preventing 10.100.100.0/23 and 10.100.102.0/23 from communicating.
At best, such an attack, if possible, would be used to make a bot-net..
Recovery.. As long as the bootloader is intact, cold-boot the modem and the bootloader will download new firmware from the DOCSIS network. |
|
xymox1 Premium Member join:2008-05-20 Phoenix, AZ ARRIS SB8200 MikroTik CCR1036-8G-2S+
|
xymox1
Premium Member
2021-Oct-23 6:30 pm
said by kevinds:Recovery.. As long as the bootloader is intact, cold-boot the modem and the bootloader will download new firmware from the DOCSIS network. 16 million attempting that the same day would be interesting. If they just pick up the same firmware they will be reinfected. Causing loops and it stays persistent. And most likely loop further complicating things. Further making the network unstable. But I donno, I am not a IT guy. And a DOCSIS ISP is WAY beyond me. NOW... If you had a rapid firmware deployment, it could grab a updated firmware and in short order come under control. But weeks to get a patch is not really a option. So you will have to stand the network back up using know susceptible firmware. The only way to do that is to isolate each one and pray. It might be possible to load a whole environment into a CMTS that has one goal. Isolate and update. Then flip it back to normal once recovered. Maybe some module could be loaded that could do that. For the clients, the fastest way back online tho is to switch to DSL/Fiber/5G. However there most likely will be a overload of human power to hook all those people up and maybe even bandwidth issues if a huge number switch over. YOU CAN BET those people that do make it, never come back. Hmmm... I think I will look into what the least expensive DSL is I can have for backup. Its a good option anyway. |
|
kevinds Premium Member join:2003-05-01 Calgary, AB |
kevinds
Premium Member
2021-Oct-23 6:57 pm
said by xymox1:16 million attempting that the same day would be interesting. If they just pick up the same firmware they will be reinfected. Causing loops and it stays persistent. As the situation described above... Modem is 'broken'.. Customer calls into support, support walks through the cold-boot procedure, modem then downloads 'good' firmware and gets back online, or even rapidily patched version, getting the customer back online.. Not 16 million devices all at once. Not that a modem bug would affect every modem deployed on an ISP anyways. Even so CMTS ACL rules would severly reduce the spead in the mean time. |
|
DocDrewRF Medic Premium Member join:2009-01-28 dv streaming Ubee E31U2V1 Technicolor TC4400 ARRIS TG1672
3 edits |
to xymox1
said by xymox1: If they just pick up the same firmware they will be reinfected. Pick up the same firmware from where? Another infected modem? Is that modem also providing DHCP, config files, and TFTP? It'll also need signed firmware for that make/model of modem with the proper CVC for the target modem and firmware to accept it. Among other things. Hacking a modem in your possession from the LAN side or using serial is a totally different situation than hacking it across the network. » www.haxorware.com/forums ··· hp?fid=7The modems won't accept any old firmware sent it's direction, even if it's for the right make and model. said by xymox1:But I donno, I am not a IT guy. And a DOCSIS ISP is WAY beyond me. Please take your own advice and stop. |
|
cramer Premium Member join:2007-04-10 Raleigh, NC Westell 6100 Cisco PIX 501
|
to kevinds
said by kevinds:I would suspect that the management IP networks would be isolated from each other, but modems on the same IP network wouldn't be. This is called "AP Isolation" in wifi, and "port isolation" on switches. Preventing nodes on a network from communicating directly is a fairly common security practice. As a cablemodem only talks to the CMTS, and only listens to the CMTS, any modem-to-modem traffic always goes through the CMTS. [BPI/BPI+ means modems can only understand the CMTS, even if they could directly talk to each other.] So, it's a fairly simply process (in theory) to prevent HFC interfaces from interacting -- they have no known reason to need to. It should be stressed this does not apply to CPE traffic -- my router must be able to talk to your router, even if we're on the same node. (TWC had that broken long long ago.) Recovery.. As long as the bootloader is intact, cold-boot the modem and the bootloader will download new firmware from the DOCSIS network. For starters, the bootloader does not have the complex code to run the RF interface; any activity that requires DOCSIS will need the full firmware running. Second, even decades old hacker firmware IGNORES maintenance requests for firmware updates -- it'd remove the hacks. The ONLY way to wipe the evil off would be physical disassembly, and full flash replacement. A botnet of cablemodems would, indeed, be a mighty force for evil. Unlike a compromised Ring doorbell, where you can see the traffic from it, isolate it from the network, just f'ing turn it off, and possibly replace the firmware... End users can't see the traffic their modem generates. They can't filter that traffic. And it's not like they can unplug it to stop it. And no one outside the factory can change the firmware that's on it. I suspect the only reason this "Pox Eclipse" hasn't happened is simply because IoT trash is such trivial lower hanging fruit. Just look at the 6190... how many versions of firmware have there been for that trash? Dozens? Times how many provider tweaks / rebuilds? You'd have to have an RCE for every version to get a toe in the door to get your special evil version loaded. Multiply that by how many different hardware versions? Is this a v4 or v6 network? Is the CPE side being tucked into a CGN? Web forced through a proxy? Is this even a provisioned modem with internet access? Can I passively listen to find targets? Do you want to get fancy and hack the router side firmware as well? (MiTM for triple the evil.) |
|
mackey Premium Member join:2007-08-20 |
to DocDrew
said by DocDrew:said by xymox1: If they just pick up the same firmware they will be reinfected. Pick up the same firmware from where? Another infected modem? Is that modem also providing DHCP, config files, and TFTP? It'll also need signed firmware for that make/model of modem with the proper CVC for the target modem and firmware to accept it. Among other things. Hacking a modem in your possession from the LAN side or using serial is a totally different situation than hacking it across the network. The modems won't accept any old firmware sent it's direction, even if it's for the right make and model. There is no need to muck with DHCP or TFTP, the CMTS can handle that like usual (since if it wanted to refuse it would be easier to just deny network access completely). If this hypothetical worm exploits a 0-day then an operator re-flashing the same version would just get reinfected. Who's to say a WAN-accessible RCE bug does not exist? There was already a LAN-side one found in the spectrum analyzer of Broadcom-based modems. If you have access to the MGMT interface then there are a number of services exposed that could be explored for similar vulnerabilities. Once a bad guy is running his own code then the CVC signing is moot as he can just write to the flash chip directly. For the record, I rooted my SB6190 a number of years ago and said root survived 3 firmware upgrades before I retired it, so it's definitely possible to hijack these things without messing with the base image. |
|
xymox1 Premium Member join:2008-05-20 Phoenix, AZ ARRIS SB8200 MikroTik CCR1036-8G-2S+
|
to kevinds
A worm could spread thru all 16 million, in a very hypothetical ideal situation in what 30 mins ? Code red got the whole planet of IIS servers in a day. Fast enough you would end up with a crippling wave of phone calls from residential and buis customers. Aint no way your gonna get a customer service agent.
16 mil is a low estimate. there are 30 million on Comcast alone. Looks like about 80 million total for the US. I would guess a worm would be OS specific, no idea how that breaks out tho.
But for the US alone 30 million across all the ISPs seems reasonable. In the world tho, 50 million total ? 75 million ?
With each box being quite powerful that would be, I think, one of the most powerful bot nets ever created.
Of course it would be battle from the first hour, and I am sure it would be brought under control, BUT might be a crazy week. 5G will be the winner.
Hey, isn't one of the big things with CableLabs 10G the ability to isolate each device on the client side to stop virus spread ? But they don't do it for themselves. Wow.. The whole SDN thing with 10G. |
|
|
| xymox1 |
xymox1
Premium Member
2021-Oct-23 10:38 pm
I am going to blast this everywhere I can.. I will also send this as email with adjustment to CableLabs security. Hey look at this ! hahaha... Lets see if someone is manning that desk tonight. hehehe.. Give them something to ponder  » my.xfinity.com/vulnerabi ··· tyreportThere appears to be a very serious gap in your security best practices and policies that could result in a very widespread serious incident.
VERY serious issues are being discussed publicly. »SB6190 Puma6 TCP/UDP Network Latency Issue Discussion
Your maintenance network, which controls all the devices on your network, is susceptible to attack. In fact its nearly criminally negligent in its lack of security and appears to be based on 1990's security protocols.. A subscriber on the LAN side can determine his address on the maintenance network and can ping ANY CPE on the network as long as they are on the same ISP. The CPE are not walled off from each other in any way. This could result in a VAST compromise of your entire network nationwide from a worm that self spreads via the wide open maintenance network connecting all devices. . . ALL susceptible devices on your network, 10's of millions, could be taken over in hours with a nearly impossible task of clean up and maybe a week of complete ISP downtime. This would also result in the largest loss of subscribers in history for cable as people flee to DSL and 5G that day trying to get internet. You would need new firmware for every device that addresses the issue, and getting new firmware will take weeks. All the modems might be bricked with no hope of recovery. The news coverage would be devastating. Each modem/router could attack the subscriber side and scrape data and files. On the ISP side it would lock out all maintenance access and recovery of the devices, and the whole network, nearly impossible. It would setup a serious botnet - possibly the largest ever created when combined with the other top world wide ISPs. It might even result in a Ransom ware attack on a massive scale with all the CPE locked out from the ISP. A silent malware could spread stealthy and then sit on CPE and attack the suscribers by doing fake DNS and even MiM attacks. A botnet of CPE would be incredibly powerful
This wide open gap appears to exist in most ISPs. So it is a CableLabs lack of proper security vision and best practices for the maintenance network. I have contacted CableLabs.
You need to do a security audit of your maintenance network and secure it. The kinda emergency level, possible fairly easy temp fix is simple. Isolate each piece of CPE. Right now all CPE can see each other and spread worms. Simply doing a config change could wall off each device with NO downside. This might be able to be implemented maybe in a day. This alone would reduce the issue to nearly zero. BLocking access to the maintenance network from the subscriber is also key and most likely easy. You REALLY need to do this and because these discussions are going on now, badguys could be reading, so RIGHT NOW is the time to secure your network BEFORE a incident occurs.
There may be simple quick solutions to avoid this doomsday scenario.. See the thread discussion for full details of this issue and possible solutions. »SB6190 Puma6 TCP/UDP Network Latency Issue Discussion
I will be following up to be sure you got this message.
You can contact me for any further details or respond to this email.
Chris Stephens xxx-xxx-xxxx |
|
|
xymox1 1 edit |
xymox1
Premium Member
2021-Oct-23 11:06 pm
OK top 10 ISP informed to their security departments. Now on to the main event.. I want to blast Cablelabs and hit the right people. Email address at CableLabs is easy. f.last@cablelabs.com... So I will go scrape the site and linked-in for the right people. Once I am done with that.. I will find some people at LightReading and pass the info along to some IT press friends as well.. Webbug  So I know when they come read this. Assuming they are on a corp IP and I can figure it out. |
|
cramer Premium Member join:2007-04-10 Raleigh, NC Westell 6100 Cisco PIX 501
|
to DocDrew
said by DocDrew:Pick up the same firmware from where? Another infected modem? Is that modem also providing DHCP, config files, and TFTP? It'll also need signed firmware for that make/model of modem with the proper CVC for the target modem and firmware to accept it. Among other things. And this is why, personally, I cannot accept the word of any cable industry person. They're either drinking the coolaid, unable to see past their own lack of imagination, or they're actively trying to keep things swept under the rug. Hacking a modem in your possession from the LAN side or using serial is a totally different situation than hacking it across the network. Correct. Flashing firmware directly through physical access is actually quite trivial. (plenty of youtube videos) Getting that code on any random modem over the internet is basically impossible. Modems simply bridge internet traffic; they don't look at it or interact with it, so you can't talk to the modem directly. HOWEVER, if some enterprising Russian hackers publish a hacked firmware that some idiot (and let's be honest, anyone using hacker firmware is an idiot) loads on their modem and then puts online, then that gang potentially has a point to access that system's management (HFC side) network. That network, as we've been trying to paint for days here, is woefully unprepared for intrusion. The only security in sight is the BELIEF no one has access to it. That's a bad way to secure anything. The cpe side increasingly has little access -- even the "read-only" informational UI is often disabled. (I can't even ping mine. It answers ARP and that's it.) But they have little, if any, protections on the HFC side. Modems can talk to other modems. My modem has the same management snmp settings as every other modem. My modem has the same management user and password as every other modem. (and it's shockingly common for that password to be cleartext!) Thus, once access is found to one modem, access can quickly be had to other modems. The hacked firmware on the first modem can be pushed to the target modem over the management network -- modem to modem -- so long as they're the same type. (a single 6190 can infest every 6190 it finds, for example.) In a matter of minutes, it will worm it's way to an internet provisioned modem where it can MiTM it's way to an internet CnC system to get firmware for other types of modems. ( this is precisely how many internet worms have functioned for decades.) |
|
xymox1 Premium Member join:2008-05-20 Phoenix, AZ ARRIS SB8200 MikroTik CCR1036-8G-2S+
|
xymox1
Premium Member
2021-Oct-24 12:22 am
To Cablelabs: A list of 6 key people. All on the same email, I sent it from Badmodems.com admin.. There were no reflections. All the emails landed. Hi all..
You guys hate me First it was Puma and the badmodems.com list and now this..
I am sorry to directly email you. No need to respond. Its OK, I understand its a legal thing. I wont email again. Sorry for this hassle. Sorry for a long read.
There appears to be a very serious gap in your security best practices and policies that could result in a very widespread serious incident that could effect all DOCSIS systems worldwide and result in a worldwide incident.
This appears to be from MSOs deploying horrendously bad security on the maintenance network.
The issues are being discussed publicly. This thread begins with discussion of firmware and then turns to the maintenance network which appears to have little if any security implemented possibly because there is no modern published best practices for the maintenance network beyond something from the 1990's. »SB6190 Puma6 TCP/UDP Network Latency Issue Discussion
The maintenance network, which controls all the devices on a DOCSIS network, is susceptible to attack. In fact its nearly criminally negligent in its lack of security and appears to be based on 1990's security protocols of mostly security thru obscurity. .. A subscriber on the LAN side can determine his address on the maintenance network and can ping ANY CPE on the network as long as they are on the same ISP. The CPE are not walled off from each other in any way. This could result in a VAST compromise of the entire MSO network nationwide from a 0-day worm that self spreads via the wide open maintenance network connecting all devices. . . ALL susceptible devices on your network, 10's of millions, could be taken over in hours with a self spreading worm with a nearly impossible task of clean up and maybe a week of complete ISP downtime. This would also result in the largest loss of subscribers in history for cable as people flee to DSL and 5G that day trying to get internet. You would need new firmware for every device that addresses the issue, and getting new firmware will take weeks. All the susceptible CPE might be bricked with no hope of recovery once taken over. The current security practices are inadequate.. The news coverage would be devastating. Each modem/router could attack the subscriber side and scrape data and files. On the ISP side it would lock out all maintenance access and recovery of the devices, and the whole network, nearly impossible. It would setup a serious botnet - possibly the largest ever created when combined with the other top world wide ISPs. It might even result in a Ransom ware attack on a massive scale with all the CPE locked out from the ISP. A silent malware could spread stealthy and then sit on CPE and attack the subscribers quietly by doing fake DNS and even MiM attacks. This could already be the case. A botnet of CPE would be incredibly powerful
This wide open gap appears to exist in most ISPs. So it is CableLabs' lack of proper security vision to keep up with modern threats by doing best practices for the maintenance network that seems to be the main issue. 10G offers micronets and SDN containment of LAN devices,,, yet the ISP has nothing like it to protect its own network and its subscribers and CPE.
Each ISP will need to do a 3rd party security audit and pentest of all the MSO's maintenance networks and secure them. The kinda emergency level, possible fairly easy temp fix is simple. Isolate each piece of CPE. Right now all CPE can see each other and spread worms. Simply doing a config change could wall off each device with NO downside. This might be able to be implemented maybe in a day. This alone would reduce the issue to nearly zero. BLocking access to the maintenance network from the subscriber is also key and most likely easy. MSOs REALLY need to do this and because these discussions are going on now, badguys could be reading, so RIGHT NOW is the time to secure MSO networks BEFORE a incident occurs.
There may be simple quick solutions to avoid this doomsday scenario for now..
Make sure you read up to the current postings. »SB6190 Puma6 TCP/UDP Network Latency Issue Discussion
I will be following up.
You can contact me for any further details or respond to this email.
I am the guy who found the Puma issue. So you guys know I can be persistent and noisy. I would really like to hear that CableLabs is going to pursue a whole new approach to device security on the maintenance network including RAPID firmware deployment. EVERYBODY wins..
Sorry for blasting the email. Sorry to start your monday kinda ruff. Think of it as a cool new feature.
I have contacted all the top 10 MSOs and sent reports to the security teams. They are the guys who made this mess, but, they need a good best practice to follow and that does not seem to be there.
Gone are the days of junk boxes with poor CPUs. MSOs are dropping POWERFUL devices with lots of RAM and Flash. They run Microsoft or Linux. They are connected to a massive bandwidth pipe. It looks possible to take over whole ISPs. These are prime targets no one has noticed yet apparently. Gone are the days of old.. These are high value targets and a bot net of incredible scale... Its time for a top down new approach to firmware and device security..
Of course none of my doomsday scenarios most likely will ever happen.. And most likely everything is fine.. BUT MSO's can't just keep these maintenance networks so 1990s sloppy. Change really does need to occur.
IMHO..
Chris Stephens xxx-xxx-xxxx |
|
| xymox1 |
xymox1
Premium Member
2021-Oct-24 12:33 am
OK done for the evening.. Lots of emails. Now I wait to see what happens and watch web stats for that web bug.. |
|
kevinds Premium Member join:2003-05-01 Calgary, AB 1 edit |
to cramer
said by cramer:Modems simply bridge internet traffic; they don't look at it or interact with it, so you can't talk to the modem directly. One would think, but this has been proven not to be the case.. One example, there is a modem-only-modem out there that by default performs SIP-ALG on the bridged traffic... As expected, it only causes issues.. TC4400, from memory. Previous modems I messed with, a 30-30-30 reset wouldn't load the OS, it would instead re-download the firmware. Once I wiped the bootloader partition, then it was a brick.. Some embedded devices, the bootloaders have network abilities. |
|
cramer Premium Member join:2007-04-10 Raleigh, NC Westell 6100 Cisco PIX 501
|
cramer
Premium Member
2021-Oct-24 4:03 am
Network, as in ethernet (intel, marvel, realtek, etc.), sure, there's very little involved with talking to them. DOCSIS is a huge pile of ****; bootloader partitions are rarely large enough for all that crap. This sounds like the OS seeing that sequence and initiating a download, 'tho I am unaware of any DOCSIS provisions for a modem to ask the network for firmware. (it's the other way around. if their were, customer owned modems could get updated.)
(which is why many cisco routers need a "boothelper" to network boot. ROMMON doesn't have drivers for DSL, T1, etc. cards, but can drive the builtin ethernet -- 'tho cisco defaults to the boothelper for any non-flash boot.)
WTH is it "ALG'ing"? It doesn't have any information on what to turn into what. Sounds like someone included something they shouldn't when they built the firmware. |
|
DocDrewRF Medic Premium Member join:2009-01-28 dv streaming Ubee E31U2V1 Technicolor TC4400 ARRIS TG1672
4 edits |
to xymox1
Unless you actually have a real world example case and procedure you can demonstrate, emailing a bunch of people about this hypothetical attack won't get you much.
It certainly won't get you faster turn around time between modem firmware releases and provider installation. It also won't get end users the ability to install firmware on their own. If anything they'll lock down the installation process even tighter after longer testing periods.
Just look at the Puma6 issue as an example of industry reactions. Some providers still issue them and they can still be found on retailer shelves. That's with real world examples and a easy repeatable test proceedure.
Unauthorized communication between modems is a CMTS issue. |
|
xymox1 Premium Member join:2008-05-20 Phoenix, AZ ARRIS SB8200 MikroTik CCR1036-8G-2S+
|
xymox1
Premium Member
2021-Oct-24 12:26 pm
said by DocDrew:Some providers still issue them Comcast gave one to a resi client of mine this week in Boulder CO. They don't give them to buis clients. Go figure. I still find it amazing that thing, and all Puma 6's, are still in use. I wonder what the stock of Puma 6's was ? I don't think Intel makes them anymore ? Maybe they made enough to take over the world and , oops.. |
|
xymox1 2 edits |
xymox1
Premium Member
2021-Oct-24 1:08 pm
I think what might help overall security is there is some background server that goes and scans each bit of CPE on the whole network. It could generate a checksum for the firmware running and even get other metrics and verify the CPE's integrity. This would run low priority continuously and be super low impact. There would need to be a protocol and BP. This could catch all sorts of stuff before it gets out of hand. Like a virus scanner but for hardware CPE. That would be a cool tool to bake into the DOCSIS std. That first scan would be interesting. What is already out there ? Webbug on new page  |
|
kevinds Premium Member join:2003-05-01 Calgary, AB |
to cramer
said by cramer:WTH is it "ALG'ing"? It doesn't have any information on what to turn into what. Sounds like someone included something they shouldn't when they built the firmware. Exactly. But it does prove that in 'bridge' mode modems will still pay attention to, and will react to traffic that passes through them. PITA to troubleshoot it, when we were not expecting the modem to fk with traffic passing through it.. said by cramer:'tho I am unaware of any DOCSIS provisions for a modem to ask the network for firmware. (it's the other way around. if their were, customer owned modems could get updated.) Some networks update customer owned modems.. Shaw specifically will update modems regardless of provisioned access. I can get one of their accepted models, plug it in, it will firmware update regardless if it is provisioned or not. |
|
mackey Premium Member join:2007-08-20 |
to xymox1
said by xymox1:It could generate a checksum for the firmware running and even get other metrics and verify the CPE's integrity. Except if you hack the firmware you could just tell your modem to report the desired checksum instead of the actual checksum. They already do scan and look at the reported firmware version. |
|
xymox1 Premium Member join:2008-05-20 Phoenix, AZ ARRIS SB8200 MikroTik CCR1036-8G-2S+
|
xymox1
Premium Member
2021-Oct-24 4:40 pm
said by mackey:Except if you hack the firmware you could just tell your modem to report the desired checksum instead of the actual checksum. They already do scan and look at the reported firmware version. Well I tried. Hahaha.. Maybe there is a more secure version then just a checksum. I donno. I am just thinking out loud. Maybe people are listening and this might be a good time to suggest new things. said by kevinds:Shaw specifically will update modems regardless of provisioned access. There are ISP that update devices... And some that never will.. Some are quick and proactive when new firmware comes, some NEVER bother looking. This VAST variation in this hyper critical aspect of security is again a failure of policy and or lack of BP from CableLabs. Again 1990's mentality. If your the only one responsible for firmware updates, and the subscriber is denied the ability, then you must have a consistent policy that is SECURITY based. Being lazy, or not hiring enough people to handle this cuz your MSO is cheap, OR actually setting a policy that punishes people who OWN their CPE by denying updates MUST END. Its clear that some MSOs cannot be allowed these options because they do not have the resources to do it, or they abuse the process. Firmware updates for DOCSIS products need a policy that all members must follow exactly. ------------------------ I have sent a email to soc@us-cert.gov to ask how to proceed. Included my email to CableLabs with links here. |
|
xymox1 2 edits |
xymox1
Premium Member
2021-Oct-24 5:22 pm
For example. My MSO, Cox, does a AWESOME job. If its on their network, its updated to the very most current firmware. New updates go out really fast. They obviously have people dedicated to this.
Cox is privately owned. i think they were the only ones to take a stand on the Puma thing, even tho it cost them money and delayed thier gigabit rollout. They did not push Puma's.
I also understand they created their own testing lab to spot things like the issues with Puma.
They are just clearly proactive and doing the responsible thing.
So there are MSOs out there who are doing the right thing and seem to have ethics and common sense. I still think that there must be MSOs who have a tightly controlled maintenance network. Maybe its just a few, very large, bad apples.
So maybe what is needed to bring these bad apples into line is some form of financial penalty.
Maybe its time to bring the FCC in to look at this. Maybe we need some rules. I will start looking into that.
I would bet the MSOs that are still handing out Puma 6's are the same ones with out of control maintenance networks. Rouge MSOs thinking its the wild west and they set their own rules.
I think some MSOs will need to be dragged kicking and screaming all the way. I do not think they will make these changes and develop modern security practices unless forced. I think the FCC is the only one who could force that and it would require rule making and you can bet there would be a pitched battle as these MSO loose control.
There is a infrastructure package coming. Maybe I also need to look into that and see what stuff they have planned for cable. Then contact the lawmakers involved and point them to this thread. Maybe a provision that MSOs get lots of money for infrastructure upgrade, but, they must do regular 3rd party security audits and, provide timely firmware. That way we skip the FCC process. Who knows... Nothing ventured, nothing gained. I have been involved with the political process before.
I suppose I could file complaints locally with cities. They do hold the contract with the MSO.
I have NO illusion. This is a LONG impossible looking battle. But I like those.
Intel sold off its connected home division. Little guys can win these big battles. |
|
xymox1 1 edit |
xymox1
Premium Member
2021-Oct-24 6:36 pm
Looks like MSOs are after that infrastructre money big time.. So I think this might be a angle I will pursue. If we are gonna give them big bucks, lets get something for our money.. Like, uhhh,,, secure MSOs that have proper security updates rolled out in firmware.. » www.opensecrets.org/news ··· -hurdle/ |
|
cramer Premium Member join:2007-04-10 Raleigh, NC Westell 6100 Cisco PIX 501
|
to kevinds
Not "will", "can". The example given is a clear case of buggy software. The only thing the bridge does is filter what it's told to block. (i.e. keep the management network away from the CPE. It's also where "server" blocks are placed -- port 25 blocking is commonly done on the modem.)
TWC used to update customer hardware, but they moved away from that after bricking a small percentage of devices. Charter absolutely will not. In any case, there's no mechanism for the modem to initiate the process; during registration, the network decides if it's running the correct version and will send a configuration file specifying what to load, where to get, how to verify it, etc. |
|
kevinds Premium Member join:2003-05-01 Calgary, AB |
kevinds
Premium Member
2021-Oct-24 7:21 pm
said by cramer:port 25 blocking is commonly done on the modem No it is not. This is at the CMTS. It doesn't work 'at scale' to apply at the modem. Static IPs are allowed outgoing TCP port 25, the same modem, a DHCP IP is not allowed outgoing TCP port 25. |
|
xymox1 Premium Member join:2008-05-20 Phoenix, AZ ARRIS SB8200 MikroTik CCR1036-8G-2S+
|
to cramer
said by cramer:TWC used to update customer hardware, but they moved away from that after bricking a small percentage of devices. Again... Its all about standards and BP. The fact they bricked devices shows a lack of proper standards and best pratices. That they could not apply right firmware and caused bricks was also a failure of stds and BP. ISP need to know things wont break. CableLabs needs to work this out and still provide rapid firmware updates to EVERY device on the network. >> EVERY DEVICE |
|
cramer Premium Member join:2007-04-10 Raleigh, NC Westell 6100 Cisco PIX 501
|
to xymox1
said by xymox1:I think what might help overall security is there is some background server that goes and scans each bit of CPE on the whole network. There's nothing in current specifications, firmware, or custom tweaks, that allows the network to pull contents of the modem (ram or flash.) Even if there were, you're asking an untrustable system to provide trusted information... it'll just lie. Hacker firmware always has. [side note: that lie is easy to detect.] Plus, with the memory of modern systems -- and the dual boot safety -- a modem could have the network provided "blessed" image in the alternate partition(s) for just such integrity checks. (if you can find it, read up on the history of "blessed clients" for the ages old X11 game xtrek/nettrek.) Physical security is extremely hard to do. Sony, Microsoft, and Nintendo still haven't managed to get there, and they have some of the brightest minds (with unlimited budgets) thinking about it. There is no way you'll ever get a cable modem secured for as long as they last -- 3-5 years is considered a success for a game console; modems last decades. It would be prohibitively expensive to resort to the methods used on PIN-pads and FIPS hardware. (eevblog on YT has a few tear downs of those things.) [or the old military serial line encryptor modules. even Hughes (maker) didn't like them. the ones I remember had a tiny thermite charge to destroy the SRAM. rumor was, they couldn't be x-ray'd without setting it off.] (Search for the stories around Directv's masterful stroke of frying hacked "football" cards days before the superbowl many years back. A friend of mine has a few of those framed. It helped that those cards ran pretty hot to begin with; the processing loop DTV put them in melted them.) |
|
DocDrewRF Medic Premium Member join:2009-01-28 dv streaming Ubee E31U2V1 Technicolor TC4400 ARRIS TG1672
2 edits |
to cramer
said by cramer:TWC used to update customer hardware, but they moved away from that after bricking a small percentage of devices. Charter absolutely will not. TWC did and updating the firmware wasn't responsible for the "bricking". It was an incompatibility between old firmware on the SB6120 modems and new CMTS software that added new DOCSIS features. Retail versions of the modem has very old software. All the MSOs who updated their CMTSs were hit with that issue. » Firmware Update Cripples Some Motorola SB6120 DOC 3 Modems [49] comments» Charter Exploring What Killed Many User Modems [48] comments» [Connectivity] SB6120 - old firmware and will not go OnlineCharter does update customer owned modems. There are some recent reports on Netgear modems being updated. » [HSI] Netgear CM1200 new firmwareMy personal SB6183 was also updated on Charter more than once. |
|
cramer Premium Member join:2007-04-10 Raleigh, NC Westell 6100 Cisco PIX 501
|
to DocDrew
said by DocDrew:Unless you actually have a real world example case and procedure you can demonstrate... ... "we're going to continue to stick our heads in the sand." "It's not a problem until it's a problem." Keep assuming that nuke is safe until it explodes. Being the internet, here's your car analogy: Cable Customer: My car only has one lug on the wheel. [out of 4 or 5] MSO: Has the wheel fallen off the car? CC: No?!? MSO: (thinking: why the f*** are you bothering me) It's fine. Keep driving it. Call us back if it falls off. ("if" not "when") (Or are you goading us into committing multiple felonies?) If anything they'll lock down the installation process even tighter after longer testing periods. The modems that are already out there CANNOT be fixed. There is no way to even attempt to secure the software they run. (Ask Tivo, Inc. how well having the bios self-check itself worked. It didn't; it took minutes to find the single byte to turn that crap off. It took years to get those levels of non-security addressed so cablelabs would certify the Series3 -- their first cable(card) box.) (puma6)... That's with real world examples and a easy repeatable test[s] It's also a "rare" problem for which they really don't give a rats ass. Again, more head-in-the-sand BS. The only reason anything at all was done is because a lot of people bitched about it. (even involved lawyers.) The insecurity of the network isn't something people are going to even understand, much less crusade for. The industry isn't going to care until the bridge falls down and people die (as one of my engineering professors loved to say.) Yes, currently, the odds are low... Russian hackers are making bank with ransomeware, North Korea is playing with nukes, and China is busy hacking our cellular network. (if china wanted to 0wnz the US cable system, they've had that ability since day-one... every one of our modems is built there.) Unauthorized communication between modems is a CMTS issue. Yes it is. But nobody has applied even that simple bandaid in two decades. (some, apparently, don't even hide the HFC network from the CPE network.) Blocking inter-modem traffic would greatly reduce the attack surface, but it's still not zero. How hardended is the CMTS, and other infrastructure, from the modems? The CMTS is certainly protected from the internet, but how about its services exposed to the HFC side? How about the windows/linux/solaris/??? servers running DHCP/TFTP/SDV/PPV/etc.? |
|
| cramer |
to kevinds
said by kevinds:It doesn't work 'at scale' to apply at the modem It's a single line in the config file. As this is 25/tcp outbound, the best place to put it is at the modem... before it ever crosses the network. Granted, I've not looked at a config in many years, but when I asked to have that block removed from my office's account -- which was NOT static addresses (4 DHCP) -- the modem was rebooted to do it. That strongly suggests it was in the config. ('tho the filter can be changed live via SNMP) (all the addresses I have recorded resolve to *.res.rr.com today. Currently, Spectrum doesn't appear to be blocking port 25 at all.) |
|