Hello Folks, I thought it was probably about time for an update. Were still working on the various issues discussed in the forum, and this remains a top priority for us.
There is no full solution for all issues yet then ? In fact this whole statement seems to imply that there is no full solution for the Puma 5/6 or 7.
We have started to provide the production releases for Puma 6 to OEMs,
So patches have been provided to some modem makers. So far im not aware of any vendor qualifying one of these patches and providing this fix to a MSO and then the MSO qualifying it for field deployment ? Because of previous patches not providing solutions we will want to test this solution.
I want to be clear on this. The 90 day CVE publication clock begins when you provide the last vendor, of every vendor on earth to get a Puma 5/6/7, a patch ?
Thats a fairly unacceptable time frame considering the 0-day exploit code is published and the CVE info would help the public make decisions to protect their systems from attack.
This time frame could be extended into years. I will take this up with MITRE.
and in parallel we have been working on fixes for Puma 5 and Puma 7.
The Puma 7 does not have a fix released by intel yet to a mfgr. So any Puma 7 fix is a long way from being deployed to a end user, assuming a full fix can be found and that is not certain at this point it sounds like.
Having said that, as many of you know, getting a fix from a vendor like Intel to an end user has a few steps to go through: Intel must develop & test the new code and deliver the code to the OEM. Then the OEM has to integrate and test the code in their specific end product, and then deliver the code to the cable operator. And then the cable operator tests it in their environment before sending it to the end user.
Its then weeks to months once you release a fix. Also some MSOs do not update firmware at all. So the fix wont be applied to some users ever.
It is a chain of events, and everyone has their role to play and wants to test to make sure the new code plays nicely with all the other components. Thats why it takes a while for the new code to appear.
This 1980's process needs new standards so a emergency patch can be applied in days vs months.
Also, regarding the CVE, heres what Ive been told regarding getting an update out: It is standard practice for vendors to hold the actual CVE submission with details until mitigations are available to all customers.
This makes sense when the exploit code is not out there. You hold off so no one uses the published CVE details to exploit the vulnerability. In this case the code is out there and the details known.
The responsible thing to do is issue a alert making end users aware of the issue.
As end users are at least months to years away from a patch right now,, without a fix even established for the 5 and 7, issuing a advisory is the correct action and providing some temporary mitigation remedies the end user can apply in that advisory while a full fix is worked out and delivered to clients seems a correct course of action. Even if that mitigation is to replace the device.
Intels practice, which is known to both MITRE and CERT, is to publish the details to the CVE within 90 days of mitigations being delivered to all customers, and we fully intend to adhere to this practice here.
I bet...
_______________________________________________
So as far as I can tell, nothing has changed from 4 months ago ?
What has been clarified is that the Puma 7 has no fix in sight yet.
Also we have no assurance the fixes in works address all the issues.
I will take up your CVE publication plan here with MITRE as this exposes millions of end users to a DoS, memory corruption reboot attack and UDP DoS for maybe years before a CVE or advisory is released. This is a irresponsible security practice that exposes millions to attack and undermines trust in the CERT / MITRE process.
You note production releases for XB3/Puma6 deploying? Is there a specific firmware to be on the lookout for that has proven to truly fix the issue? The last Comcast post for the Arris XB3/1682g didn't help the latency issues at all.
They said they gave the code to modem makers. They did not say modem makers have given it to ISPs or that ISPs have tested it and have given it to any end users. Its possible this is months to years away.
You picked the bones clean. Its all talk and no substance until we see other wise. Maybe just MAYBE. Intel will be hit with a lawsuit as well for this and not just modem vendors like Arris atm. Its all smoke and mirrors right now from Intel even with that new response. Nothing is coming any time soon if EVER for some customers as well.
When you ping inside your network you just going thru the switch chip. When you ping outside your network your go thru NAT and a list of "router" things. Performance is impacted by std consumer grade routers.
It has nothing to do with switch chip. Puma 6 produces regular lag spikes (~60ms) even if you ping it from local network. These spikes add up to spikes produced by DOCSIS engine.
That's why it strongly looks to me like fault of ARM core unable to produce realtime response, sometimes unable to respond in time due to some other task hogging it (I beleive ARM core handles IP routing, e.g. packet passing from LAN to WAN - it happens even in bridge mode).
Mackey has in-depth research into this. He was the guy who discovered the DoS and memory corruption. He also had root inside the modem.
Its been so long since I aimed Pingplotter at the modem, rather then thru the modem, ive forgotten what it does with ICMP Ping.
I might put my Puma 6 on my network tonight and do some playing around.
YES. You are correct that some process runs every 1.96 seconds causing a very large spike. The spike is actually misordered packets as I remember and this shows up as latency.
You picked the bones clean. Its all talk and no substance until we see other wise. Maybe just MAYBE. Intel will be hit with a lawsuit as well for this and not just modem vendors like Arris atm. Its all smoke and mirrors right now from Intel even with that new response. Nothing is coming any time soon if EVER for some customers as well.
It does seem that way...
I want to test this fix being given to vendors. It *might* still be possible they have fixed everything. I do hold out hope.
But as there is no determined fix for the 7 yet, I donno.. And then all this delay on the CVE... This all looks very suspicious to me.
Its clearly irresponsible security practice for a 0-day rated CVSS 9.1 or more. Thats fairly WTF..
You picked the bones clean. Its all talk and no substance until we see other wise. Maybe just MAYBE. Intel will be hit with a lawsuit as well for this and not just modem vendors like Arris atm. Its all smoke and mirrors right now from Intel even with that new response. Nothing is coming any time soon if EVER for some customers as well.
It does seem that way...
I want to test this fix being given to vendors. It *might* still be possible they have fixed everything. I do hold out hope.
But as there is no determined fix for the 7 yet, I donno.. And then all this delay on the CVE... This all looks very suspicious to me.
Its clearly irresponsible security practice for a 0-day rated CVSS 9.1 or more. Thats fairly WTF..
As we all know. Intel has some very VERY deep pockets here. One reason why I think they are getting away with this crap right now and overwhelming strict NDA's.
Im outraged. This is just BS. This debacle is never ending.
If it cant be fixed, say so, get over it and move on. If it can be fixed GET IT DONE NOW.
Exposing millions of end users to this crap for years now with a fix months to years away yet, even if it has a fix, is abysmal behavior and really is just voluntary disregard. Its also abuse of established security protocols meant to protect infrastructure and end users so Intel can avoid financial loss.
No to mention im still concerned we got here because maybe Intel had some backroom dealings with vendors and MSOs to become exclusive in the market.
This whole thing is a huge multi-national ****** ****
Indeed. And Shaw Cable in BC and Alberta has no non-Puma6 modems available for plans above 15/1.5. Some people have gotten the older DPC3825 provisioned on the 75/7.5 plan though.
Be thankful you Americans mostly have other options.
Indeed. And Shaw Cable in BC and Alberta has no non-Puma6 modems available for plans above 15/1.5. Some people have gotten the older DPC3825 provisioned on the 75/7.5 plan though.
Be thankful you Americans mostly have other options.
That was my point to the press in email just now. A LOT of people cant get out from under this device. Its forced on them.. They cant get away from the DoS, reboot or performance issues.
On Shaw cable alone they have 1.9 million internet subscribers and I would imagine a large percentage of them are on a plan above 15/1.5. »www.shaw.ca/corporate/ab ··· mpanies/
The total number of Pumas in service this second might be staggering world wide. AND GROWING at a really impressive rate becoming exclusive in many markets.
On Shaw cable alone they have 1.9 million internet subscribers and I would imagine a large percentage of them are on a plan above 15/1.5. »www.shaw.ca/corporate/ab ··· mpanies/
Yeah. And their users don't have any other option - you can't buy your own modem up there
I think long ago, the conversation between Shaw and the CRTC went like this, with regards to consumer protection laws and similar.
CRTC: People must be able to buy their own modem. Shaw: No [Insert technical/business reason here] C: Can you offer the modem rental for free then? S: Yes C: End of discussion
It's actually quite amazing. I always find it queer when other countries are more regressive and corporate-oriented than America is. They usually have far less of an excuse.
CRTC: People must be able to buy their own modem. Shaw: No [Insert technical/business reason here] C: Can you offer the modem rental for free then? S: Yes C: End of discussion
Yeah, at least modem rentals are free - though I'm sure this helps contribute to higher costs baked into the sticker price. But allowing this level of control still strikes me as pretty dubious. And at least Shaw's modem/router combo (currently a Hitron) allows for the use of another router (via a secondary IP, interestingly enough, not a bridge) - for now...
The Hitron does have a bridge mode if you call in for it to be enabled.
In addition to the secondary IP method? Interesting. I'm not sure what advantage there is to using it as a bridge, but still cool to know that it's possible.
I started out wanting to help Arris and Intel fix this. Offered help a lot while others were screaming at them. I really want a solution, not more headache.
I think this last statement has me over a edge now. Maybe they intend to just keep pushing this crap onto consumers, everyone be damned. Maybe they think "No one notices these issues but a little group of people on some minor forum".. "Just look at the sales".... Maybe MSOs also dont care, maybe they think consumers are so stupid they can just push this stuff down thier throat. World wide..
Intel looks ready to degrade consumer internet performance worldwide and not care at all.
Sales are still going at full production speeds, MSOs are rolling out features that make it so Intel is the only company that can supply a chip. They have exclusive control over HUGE markets with monopoly powers.
This whole thing is a horror whow. Its a possible glimpse of the future where performance does not matter at all. If all the world knows is intel Puma performance adn there is nothing to compare to, there is no issue right ? Its alll equally shit.
Intel is now officially dead to me. I give up. I tried to help, tried to get things worked out. Its not working. Even under impressive pressure like press reports and a web site. I kept hoping the pressure would cause a fix to pop out. Nope.
Im going to swap back to my 6190 tonight.. Im gonna do more playing around. I want some tests, just in case a firmware comes along. I wantto be able to AB it very clearly.
One unanswered question I would really like to know. What is that 1.95 second spike ? Whats going on that causes it ? The processing seems to stop dead and then come back. Like some super high priority process runs every 1.95 seconds. What is this ? Why does it halt the whole processor ?
Sadly the reviews are overwhelmingly positive at big places like Amazon etc. People are very happy with their broken modem and see no issue. Those are the clueless sheeple that cant be helped or convinced other wise. Intel and all the modem vendors are simply laughing to the bank quite literally. This has had ZERO effect on their sales obviously. The scary thing is I don't know what is worse. The fact Intel and vendors can continue selling broken modems. Or the people that still buy them by the masses knowing it has issues. As a matter of fact here is a recent review and Arris response 9hrs ago. So Arris is still spewing off about magical firmware that addresses this issue even though it does not exist. I wonder if this could be used in court against them with the class action going on.
STAY AWAY FROM THIS MODEM ByVictor J. Velasco IIIon August 3, 2017 Capacity: Download Speed: 1.4 Gbps|Model: Cable Modem|Color: White|Verified Purchase DO NOT BUY THIS MODEM AS ITS CHIPSET HAS MAJOR FLAWS. DO A GOOGLE ON THE MODEM ITSELF AND THE INTEL PUMA 6 CHIPSET ITS USES.
Dear Victor J. Velasco III, ARRIS is committed to providing the best broadband experience for all users of all our devices and we’ve been working actively with Intel to address the latency issue with some SURFboard® SB6190. We've recently completed the notification process for the Service Providers who will begin testing and applying the SB6190 firmware update.
If I ever get my server running again I'm going to make a website of my own where you can run torture tests against your modem. I have some ideas on how to improve the (benign) DSLR P6 test, plus I'm going to have more intrusive tests such as the DoS, reboot, and brick (older SB61xx firmware, not P6) tests.
So I hooked the damned cable modem of doom back up tonight.. I had forgotten the boot time is literally 2.5 TIMES longer then my SB8200.. Its FOREVER... Of course it does have 2 CPUs to boot and tie together in a Frankenstein manner..
After it got booted up.... I ran lots of tests.. Its been a while and i wanted some fresh charts... Wanted to play with a few things.
Basic setup for those of you who are new to me testing. I have a high performance PC for these tests. Rampage iV MB, quad Samsung SSD Raid 0, high performance ram. Intel X990 CPU overclocked to 4.8Ghz. I use a boot partition that has A virgin Win 7 Pro 64 and Pingplotter. THATS ALL. Nothing else. I use Deepfreeze on it so its the same on every boot. I use the embedded MB Ethernet controller but also have a high performance 10Gbs Intel network card. Its got a expensive EVGA water cooled graphics card. Whole system is water cooled using 1/2 lines, a big pump and a big radiator.
Wowee.. I forgot... So here I do a high temporal resolution look at it.. 100 pins a second.. Really wanted to see the spikes. They are VERY narrow. Whatever the 2 sec process is, its brief.
These are all 60 second charts.
I targeted my ISP DNS server as its on my same network and low latency. So it has the least noise in doing these charts. This is TCP port 53.
I also did some slower plots to google port 80
Im going to run a overnite plot so show how PingPlotter when it does averaging masks the issue unless you look closely or at the MAX stats spec. I need this plot over at the VM forums.
I did confirm that the spike is not visible if you do ICMP Ping pointed at the modem. Like not AT ALL visible.. Not the 5ms full scale in this plot.. This is scary deceptive..
Also doing ICMP ping to google hides the issue as well..
·
·
