| |
to NetDog
SB6190 Puma6 TCP/UDP Network Latency Issue DiscussionHello Folks, I thought it was probably about time for an update. We’re still working on the various issues discussed in the forum, and this remains a top priority for us. We have started to provide the production releases for Puma 6 to OEMs, and in parallel we have been working on fixes for Puma 5 and Puma 7.
As I shared earlier, we are delivering these fixes as firmware patches and we are working to get the fixes out ASAP. Having said that, as many of you know, getting a fix from a vendor like Intel to an end user has a few steps to go through: Intel must develop & test the new code and deliver the code to the OEM. Then the OEM has to integrate and test the code in their specific end product, and then deliver the code to the cable operator. And then the cable operator tests it in their environment before sending it to the end user. It is a chain of events, and everyone has their role to play and wants to test to make sure the new code plays nicely with all the other components. That’s why it takes a while for the new code to appear.
Also, regarding the CVE, here’s what I’ve been told regarding getting an update out: “It is standard practice for vendors to hold the actual CVE submission with details until mitigations are available to all customers. In this way, all customers are ready by the time the details are made publicly available via the CVE publication.” Intel’s practice – which is known to both MITRE and CERT, is to publish the details to the CVE within 90 days of mitigations being delivered to all customers, and we fully intend to adhere to this practice here. |
|
 jmorlanHmm... That's funny.
MVM
join:2001-02-05
Pacifica, CA
 ·AT&T Wireless Br..
 ·Comcast XFINITY
ARRIS SB8200
Asus RT-AC66U B1
Obihai OBi200
|
said by ChipHeadx86:“It is standard practice for vendors to hold the actual CVE submission with details until mitigations are available to all customers Does this mean that a CVE for which no mitigation is possible is never submitted? |
|
| |
to ChipHeadx86
You note production releases for XB3/Puma6 deploying? Is there a specific firmware to be on the lookout for that has proven to truly fix the issue? The last Comcast post for the Arris XB3/1682g didn't help the latency issues at all. |
|
| |
Anon0ee77 to xymox1
Anon
2017-Aug-4 3:04 pm
to xymox1
said by xymox1:When you ping inside your network you just going thru the switch chip. When you ping outside your network your go thru NAT and a list of "router" things. Performance is impacted by std consumer grade routers. It has nothing to do with switch chip. Puma 6 produces regular lag spikes (~60ms) even if you ping it from local network. These spikes add up to spikes produced by DOCSIS engine. That's why it strongly looks to me like fault of ARM core unable to produce realtime response, sometimes unable to respond in time due to some other task hogging it (I beleive ARM core handles IP routing, e.g. packet passing from LAN to WAN - it happens even in bridge mode). Top - TP-Link Archer C2600, Bottom - Virgin Media SuperHub3 (Modem/Bridge Mode): » i.imgur.com/WB4fIyh.png |
|
 train_wreckslow this bird down
join:2013-10-04
Antioch, TN
Cisco ASA 5506
|
to ChipHeadx86
said by ChipHeadx86:It is a chain of events, and everyone has their role to play and wants to test to make sure the new code plays nicely with all the other components. never gets to your device, so you have to buy a new one How those Puma 7 sales coming along? |
|
 NetDog
Premium Member
join:2002-03-04
Parker, CO |
to Anond644b
said by Anond644b :You note production releases for XB3/Puma6 deploying? Is there a specific firmware to be on the lookout for that has proven to truly fix the issue? The last Comcast post for the Arris XB3/1682g didn't help the latency issues at all. I will do some digging and see what I can find out where the RDK builds are at with the latency issue.. |
|
|
 xymox1
Premium Member
join:2008-05-20
Phoenix, AZ
ARRIS SB8200
MikroTik CCR1036-8G-2S+
4 edits |
to ChipHeadx86
said by ChipHeadx86:Hello Folks, I thought it was probably about time for an update. Were still working on the various issues discussed in the forum, and this remains a top priority for us. There is no full solution for all issues yet then ? In fact this whole statement seems to imply that there is no full solution for the Puma 5/6 or 7. said by ChipHeadx86:We have started to provide the production releases for Puma 6 to OEMs, So patches have been provided to some modem makers. So far im not aware of any vendor qualifying one of these patches and providing this fix to a MSO and then the MSO qualifying it for field deployment ? Because of previous patches not providing solutions we will want to test this solution. I want to be clear on this. The 90 day CVE publication clock begins when you provide the last vendor, of every vendor on earth to get a Puma 5/6/7, a patch ?
Thats a fairly unacceptable time frame considering the 0-day exploit code is published and the CVE info would help the public make decisions to protect their systems from attack.
This time frame could be extended into years. I will take this up with MITRE.
said by ChipHeadx86:and in parallel we have been working on fixes for Puma 5 and Puma 7. The Puma 7 does not have a fix released by intel yet to a mfgr. So any Puma 7 fix is a long way from being deployed to a end user, assuming a full fix can be found and that is not certain at this point it sounds like. said by ChipHeadx86:As I shared earlier, we are delivering these fixes as firmware patches and we are working to get the fixes out ASAP. Thats what was said 4 months ago. What exactly is the time frame ? said by ChipHeadx86: Having said that, as many of you know, getting a fix from a vendor like Intel to an end user has a few steps to go through: Intel must develop & test the new code and deliver the code to the OEM. Then the OEM has to integrate and test the code in their specific end product, and then deliver the code to the cable operator. And then the cable operator tests it in their environment before sending it to the end user. Its then weeks to months once you release a fix. Also some MSOs do not update firmware at all. So the fix wont be applied to some users ever. said by ChipHeadx86:It is a chain of events, and everyone has their role to play and wants to test to make sure the new code plays nicely with all the other components. Thats why it takes a while for the new code to appear. This 1980's process needs new standards so a emergency patch can be applied in days vs months. said by ChipHeadx86:Also, regarding the CVE, heres what Ive been told regarding getting an update out: It is standard practice for vendors to hold the actual CVE submission with details until mitigations are available to all customers. This makes sense when the exploit code is not out there. You hold off so no one uses the published CVE details to exploit the vulnerability. In this case the code is out there and the details known. The responsible thing to do is issue a alert making end users aware of the issue. As end users are at least months to years away from a patch right now,, without a fix even established for the 5 and 7, issuing a advisory is the correct action and providing some temporary mitigation remedies the end user can apply in that advisory while a full fix is worked out and delivered to clients seems a correct course of action. Even if that mitigation is to replace the device.said by ChipHeadx86: In this way, all customers are ready by the time the details are made publicly available via the CVE publication. which at this point is months to years away with no firm solution found yet for the 5 and 7. said by ChipHeadx86:Intels practice, which is known to both MITRE and CERT, is to publish the details to the CVE within 90 days of mitigations being delivered to all customers, and we fully intend to adhere to this practice here. I bet... _______________________________________________ So as far as I can tell, nothing has changed from 4 months ago ? What has been clarified is that the Puma 7 has no fix in sight yet. Also we have no assurance the fixes in works address all the issues. I will take up your CVE publication plan here with MITRE as this exposes millions of end users to a DoS, memory corruption reboot attack and UDP DoS for maybe years before a CVE or advisory is released. This is a irresponsible security practice that exposes millions to attack and undermines trust in the CERT / MITRE process. |
|
| xymox1 |
to Anond644b
said by Anond644b :You note production releases for XB3/Puma6 deploying? Is there a specific firmware to be on the lookout for that has proven to truly fix the issue? The last Comcast post for the Arris XB3/1682g didn't help the latency issues at all. They said they gave the code to modem makers. They did not say modem makers have given it to ISPs or that ISPs have tested it and have given it to any end users. Its possible this is months to years away. |
|
KoRnGtL15
Premium Member
join:2007-01-04
Grants Pass, OR |
to xymox1
You picked the bones clean. Its all talk and no substance until we see other wise. Maybe just MAYBE. Intel will be hit with a lawsuit as well for this and not just modem vendors like Arris atm. Its all smoke and mirrors right now from Intel even with that new response. Nothing is coming any time soon if EVER for some customers as well. |
|
xymox1
Premium Member
join:2008-05-20
Phoenix, AZ ·Cox HSI
ARRIS SB8200
MikroTik CCR1036-8G-2S+
1 edit |
to Anon0ee77
said by Anon0ee77 :said by xymox1:When you ping inside your network you just going thru the switch chip. When you ping outside your network your go thru NAT and a list of "router" things. Performance is impacted by std consumer grade routers. It has nothing to do with switch chip. Puma 6 produces regular lag spikes (~60ms) even if you ping it from local network. These spikes add up to spikes produced by DOCSIS engine. That's why it strongly looks to me like fault of ARM core unable to produce realtime response, sometimes unable to respond in time due to some other task hogging it (I beleive ARM core handles IP routing, e.g. packet passing from LAN to WAN - it happens even in bridge mode). Mackey has in-depth research into this. He was the guy who discovered the DoS and memory corruption. He also had root inside the modem. Its been so long since I aimed Pingplotter at the modem, rather then thru the modem, ive forgotten what it does with ICMP Ping. I might put my Puma 6 on my network tonight and do some playing around. YES. You are correct that some process runs every 1.96 seconds causing a very large spike. The spike is actually misordered packets as I remember and this shows up as latency. |
|
| xymox1 |
to KoRnGtL15
said by KoRnGtL15:You picked the bones clean. Its all talk and no substance until we see other wise. Maybe just MAYBE. Intel will be hit with a lawsuit as well for this and not just modem vendors like Arris atm. Its all smoke and mirrors right now from Intel even with that new response. Nothing is coming any time soon if EVER for some customers as well. It does seem that way... I want to test this fix being given to vendors. It *might* still be possible they have fixed everything. I do hold out hope. But as there is no determined fix for the 7 yet, I donno.. And then all this delay on the CVE... This all looks very suspicious to me. Its clearly irresponsible security practice for a 0-day rated CVSS 9.1 or more. Thats fairly WTF.. |
|
KoRnGtL15
Premium Member
join:2007-01-04
Grants Pass, OR |
said by xymox1:said by KoRnGtL15:You picked the bones clean. Its all talk and no substance until we see other wise. Maybe just MAYBE. Intel will be hit with a lawsuit as well for this and not just modem vendors like Arris atm. Its all smoke and mirrors right now from Intel even with that new response. Nothing is coming any time soon if EVER for some customers as well. It does seem that way... I want to test this fix being given to vendors. It *might* still be possible they have fixed everything. I do hold out hope. But as there is no determined fix for the 7 yet, I donno.. And then all this delay on the CVE... This all looks very suspicious to me. Its clearly irresponsible security practice for a 0-day rated CVSS 9.1 or more. Thats fairly WTF.. As we all know. Intel has some very VERY deep pockets here. One reason why I think they are getting away with this crap right now and overwhelming strict NDA's. |
|
xymox1
Premium Member
join:2008-05-20
Phoenix, AZ ·Cox HSI
ARRIS SB8200
MikroTik CCR1036-8G-2S+
|
xymox1
Premium Member
2017-Aug-4 9:30 pm
Im outraged. This is just BS. This debacle is never ending. If it cant be fixed, say so, get over it and move on. If it can be fixed GET IT DONE NOW. Exposing millions of end users to this crap for years now with a fix months to years away yet, even if it has a fix, is abysmal behavior and really is just voluntary disregard. Its also abuse of established security protocols meant to protect infrastructure and end users so Intel can avoid financial loss. No to mention im still concerned we got here because maybe Intel had some backroom dealings with vendors and MSOs to become exclusive in the market. This whole thing is a huge multi-national ****** **** |
|
 jtl999
join:2012-11-24
Vancouver BC |
jtl999
Member
2017-Aug-4 9:32 pm
Indeed. And Shaw Cable in BC and Alberta has no non-Puma6 modems available for plans above 15/1.5. Some people have gotten the older DPC3825 provisioned on the 75/7.5 plan though.
Be thankful you Americans mostly have other options. |
|
xymox1
Premium Member
join:2008-05-20
Phoenix, AZ ·Cox HSI
ARRIS SB8200
MikroTik CCR1036-8G-2S+
|
xymox1
Premium Member
2017-Aug-4 9:37 pm
said by jtl999:Indeed. And Shaw Cable in BC and Alberta has no non-Puma6 modems available for plans above 15/1.5. Some people have gotten the older DPC3825 provisioned on the 75/7.5 plan though.
Be thankful you Americans mostly have other options. That was my point to the press in email just now. A LOT of people cant get out from under this device. Its forced on them.. They cant get away from the DoS, reboot or performance issues. Its a incredibly mishandled situation. |
|
| xymox1 |
xymox1
Premium Member
2017-Aug-4 9:43 pm
On Shaw cable alone they have 1.9 million internet subscribers and I would imagine a large percentage of them are on a plan above 15/1.5. » www.shaw.ca/corporate/ab ··· mpanies/The total number of Pumas in service this second might be staggering world wide. AND GROWING at a really impressive rate becoming exclusive in many markets. |
|
Asus RT-AC68
Asus RT-N66
Linksys E3000
|
Yeah. And their users don't have any other option - you can't buy your own modem up there  |
|
| |
said by bilditup1:you can't buy your own modem up there That right there needs to be fixed! |
|
jtl999
join:2012-11-24
Vancouver BC 176.0 159.6
 ·TELUS
(Software) pfSense
MikroTik CRS125-24G-1S-RM
Ubiquiti UniFi AP-LR
|
jtl999
Member
2017-Aug-4 11:52 pm
I think long ago, the conversation between Shaw and the CRTC went like this, with regards to consumer protection laws and similar.
CRTC: People must be able to buy their own modem.
Shaw: No [Insert technical/business reason here]
C: Can you offer the modem rental for free then?
S: Yes
C: End of discussion |
|
Asus RT-AC68
Asus RT-N66
Linksys E3000
|
to chpalmer
said by chpalmer:That right there needs to be fixed! It's actually quite amazing. I always find it queer when other countries are more regressive and corporate-oriented than America is. They usually have far less of an excuse. said by jtl999:CRTC: People must be able to buy their own modem.
Shaw: No [Insert technical/business reason here]
C: Can you offer the modem rental for free then?
S: Yes
C: End of discussion Yeah, at least modem rentals are free - though I'm sure this helps contribute to higher costs baked into the sticker price. But allowing this level of control still strikes me as pretty dubious. And at least Shaw's modem/router combo (currently a Hitron) allows for the use of another router (via a secondary IP, interestingly enough, not a bridge) - for now... |
|
jtl999
join:2012-11-24
Vancouver BC |
jtl999
Member
2017-Aug-4 11:57 pm
The Hitron does have a bridge mode if you call in for it to be enabled.  |
|
Asus RT-AC68
Asus RT-N66
Linksys E3000
|
said by jtl999:The Hitron does have a bridge mode if you call in for it to be enabled. In addition to the secondary IP method? Interesting. I'm not sure what advantage there is to using it as a bridge, but still cool to know that it's possible. |
|
jtl999
join:2012-11-24
Vancouver BC 176.0 159.6
·TELUS
(Software) pfSense
MikroTik CRS125-24G-1S-RM
Ubiquiti UniFi AP-LR
|
jtl999
Member
2017-Aug-5 12:01 am
Probably reduces the chance of the modem firmware messing up (Puma6 not withstanding) and leaves one public IP free. » community.shaw.ca/thread ··· t-135709 |
|
xymox1
Premium Member
join:2008-05-20
Phoenix, AZ ·Cox HSI
ARRIS SB8200
MikroTik CCR1036-8G-2S+
|
to NetDog
|
|
 Devious
Premium Member
join:2002-08-22
Seattle, WA |
to NetDog
said by NetDog:I will do some digging and see what I can find out where the RDK builds are at with the latency issue Perhaps check on this as well. said by ChipHeadx86:We have started to provide the production releases for Puma 6 to OEMs |
|
xymox1
Premium Member
join:2008-05-20
Phoenix, AZ ·Cox HSI
ARRIS SB8200
MikroTik CCR1036-8G-2S+
|
xymox1
Premium Member
2017-Aug-5 12:44 am
I started out wanting to help Arris and Intel fix this. Offered help a lot while others were screaming at them. I really want a solution, not more headache.
I think this last statement has me over a edge now. Maybe they intend to just keep pushing this crap onto consumers, everyone be damned. Maybe they think "No one notices these issues but a little group of people on some minor forum".. "Just look at the sales".... Maybe MSOs also dont care, maybe they think consumers are so stupid they can just push this stuff down thier throat. World wide..
Intel looks ready to degrade consumer internet performance worldwide and not care at all.
Sales are still going at full production speeds, MSOs are rolling out features that make it so Intel is the only company that can supply a chip. They have exclusive control over HUGE markets with monopoly powers.
This whole thing is a horror whow. Its a possible glimpse of the future where performance does not matter at all. If all the world knows is intel Puma performance adn there is nothing to compare to, there is no issue right ? Its alll equally shit.
Intel is now officially dead to me. I give up. I tried to help, tried to get things worked out. Its not working. Even under impressive pressure like press reports and a web site. I kept hoping the pressure would cause a fix to pop out. Nope.
Im going to swap back to my 6190 tonight.. Im gonna do more playing around. I want some tests, just in case a firmware comes along. I wantto be able to AB it very clearly.
One unanswered question I would really like to know. What is that 1.95 second spike ? Whats going on that causes it ? The processing seems to stop dead and then come back. Like some super high priority process runs every 1.95 seconds. What is this ? Why does it halt the whole processor ? |
|
| |
to xymox1
and the first test was PC directly wired to modem... |
|
KoRnGtL15
Premium Member
join:2007-01-04
Grants Pass, OR |
to xymox1
Sadly the reviews are overwhelmingly positive at big places like Amazon etc. People are very happy with their broken modem and see no issue. Those are the clueless sheeple that cant be helped or convinced other wise. Intel and all the modem vendors are simply laughing to the bank quite literally. This has had ZERO effect on their sales obviously. The scary thing is I don't know what is worse. The fact Intel and vendors can continue selling broken modems. Or the people that still buy them by the masses knowing it has issues. As a matter of fact here is a recent review and Arris response 9hrs ago. So Arris is still spewing off about magical firmware that addresses this issue even though it does not exist. I wonder if this could be used in court against them with the class action going on.
STAY AWAY FROM THIS MODEM
ByVictor J. Velasco IIIon August 3, 2017
Capacity: Download Speed: 1.4 Gbps|Model: Cable Modem|Color: White|Verified Purchase
DO NOT BUY THIS MODEM AS ITS CHIPSET HAS MAJOR FLAWS. DO A GOOGLE ON THE MODEM ITSELF AND THE INTEL PUMA 6 CHIPSET ITS USES.
Dear Victor J. Velasco III,
ARRIS is committed to providing the best broadband experience for all users of all our devices and we’ve been working actively with Intel to address the latency issue with some SURFboard® SB6190. We've recently completed the notification process for the Service Providers who will begin testing and applying the SB6190 firmware update. |
|
 mackey
Premium Member
join:2007-08-20 |
to xymox1
If I ever get my server running again I'm going to make a website of my own where you can run torture tests against your modem. I have some ideas on how to improve the (benign) DSLR P6 test, plus I'm going to have more intrusive tests such as the DoS, reboot, and brick (older SB61xx firmware, not P6) tests. |
|
jtl999
join:2012-11-24
Vancouver BC 176.0 159.6
·TELUS
(Software) pfSense
MikroTik CRS125-24G-1S-RM
Ubiquiti UniFi AP-LR
|
jtl999
Member
2017-Aug-5 1:42 am
said by mackey:plus I'm going to have more intrusive tests such as the DoS, reboot, and brick Have a few warnings there, and CSRF protection too, so it can't be accidentally triggered by a malicious attacker. Maybe have something such as "type \"I agree to these conditions\" to begin the tests and press enter" to cover yourself. |
|
·
·
·
