train_wreckslow this bird down join:2013-10-04 Antioch, TN Cisco ASA 5506 Cisco DPC3939
3 recommendations |
to mackey
Re: SB6190 Puma6 TCP/UDP Network Latency Issue Discussionsaid by mackey:After disabling the packet processor with # echo "disable" > /proc/net/ti_pp iptraf-ng shot up to 5100 pps +/- 100 to match the server and web browsing went back to normal. Just for curiosity, when you disable the processor how much of a total performance drop do you see? |
|
mackey Premium Member join:2007-08-20
4 recommendations |
mackey
Premium Member
2017-Apr-26 3:16 am
Speed test maxes out at ~60 mbps instead of my normal ~350 mbps. The 6190's CPU goes to 100% as well. |
|
train_wreckslow this bird down join:2013-10-04 Antioch, TN
7 recommendations |
Yeah figured as much. Weak CPU, with hardware assisted ASICs which come with many of their own "limitations". |
|
6 recommendations |
I think this pretty much seals the deal with the Puma 6 devices. The CPU can't handle the traffic on its own, and the packet processor can't handle more than a tiny amount of traffic before it becomes overloaded. And together performance is still well below any reasonable expectations.
And by the nomenclature in the command line, this is a Texas Instruments packet processor, yes?
What this set of tests confirms is that the Puma 6 can be perfectly fine at ~60 Mbit/sec or below regardless of traffic, or mediocre due to packet loss at any amount of traffic if the packet processor is left on. That's one hell of a design flaw and indicates these need to be recalled now rather than later.
Now the next phase of these tests: ≤24 bonded channels vs. >24 bonded channels using the same test methods. The reason I bring this up is because baseline latency doubles on puma 6 devices once you surpass 24 bonded channels on the downstream side, indicating a rather significant bottleneck occuring, presumably in either the ARM CPU or the packet processor, or both.
I'm very interested to see whether or not number of bonded channels affects either of these processors when they're isolated. |
|
|
I still think the odds of a recall are infinitesimal.
A wild swag at what will probably happen is a software patch that will be more aggressive about just dropping packets destined for new destination ports if CPU usage is above 80% or so. |
|
7 recommendations |
said by kucharsk:I still think the odds of a recall are infinitesimal.
A wild swag at what will probably happen is a software patch that will be more aggressive about just dropping packets if CPU usage is above 80% or so. If they do that, then the device no longer falls within any respectable standard and would be deemed ultimately defective and would end up recalled. I'm sure you can imagine the support nightmare that would ensue when customers and businesses show constant dropped packets, especially for gaming, VOIP, and VPNs. No, unless there's some magic bullet that can strike the components of puma 6 and fuse them together into a lean mean networking machine, it's done for and a recall is very likely. This is all going to end up as mounting evidence against Intel's puma 6 architecture, that much you can bank on. The reason the architecture and thus the devices would be considered defective is that if you have to drop packets before even reaching max load, these devices are defective for the purposes of their intended markets, which are high speed/low latency applications. It certainly makes them unfit for anything higher than ~75 Mbit/sec. And if you still believe a recall can't happen, take a quick look back at the Canadian MSOs which have already done recalls of puma 6 devices (prioritizing their gigabit tier customers first). The dominos keep falling, and they aren't falling in Intel's favor. |
|
1 recommendation |
You can jump up and down and demand a recall all you want, but if the DoS vulnerability is fixed it still remains most people are thrilled with their 6190s. You keep saying "considered defective" - by whom? Which of the specs on the box is it not meeting? Intellectually, and to people knowledgeable about networking, sure. To the vast majority of customers, not really. ("So, if a bad guy is trying to attack my modem to shut off my Internet service, I might see slower than usual responses. This affects me how, precisely?")
Intel has already moved on and so likely doesn't care if Puma6 sales fall to zero tomorrow.
Cable companies can recall devices all they want - doesn't mean they'll see a penny back from the manufacturer, and it doesn't mean the manufacturer will issue one. In fact it would likely be a matter of Intel cutting them a deal on Puma7s or 8s or some other under-the-table payment with zero admission of "fault."
In all seriousness, except for the need to issue firmware that deals with the DoS, the Puma6 devices are no worse off today than they were yesterday and will continue to sell briskly until the DOCSIS 3.1 rollout is complete.
Then again since I can still get a SB6183, SB6190s may continue to be sold for years. |
|
kucharsk
1 recommendation |
to train_wreck
said by train_wreck:Yeah figured as much. Weak CPU, with hardware assisted ASICs which come with many of their own "limitations". While true, that makes for good system design; there's no sense in selling an engineered solution with more computing horsepower than it "needs" according to the design. Yes, this falls apart in cases like this where a hardware processor doesn't seem to be performing as it should, but in most cases it saves a huge amount of money. |
|
5 recommendations |
nallar
Member
2017-Apr-26 5:43 am
Exactly. Hardware acceleration is fine if engineered correctly and we shouldn't be asking manufacturers to never use it. The problem here is that it doesn't work properly. The tiny lookup table issue explains why I've seemingly randomly had more severe connection issues... I have many connections going through the modem and the state table in my firewall always has more than 1024 entries. It has much more relaxed timeouts than the 1 second the puma 6 uses. Whenever more than 1000 of those flows are sending packets during the same second I'm effectively running the same DoS attack against my modem - except entirely through normal usage. |
|
mackey Premium Member join:2007-08-20
3 recommendations |
to Squishy Tia
said by Squishy Tia:What this set of tests confirms is that the Puma 6 can be perfectly fine at ~60 Mbit/sec or below regardless of traffic Not sure how you got to that conclusion, even with the packet processor disabled it still has the "Puma 6" syndrome with delayed or lost packets (i.e. the DSLR Puma 6 test looks the same). In fact it's worse as with the PP disabled even established connections see the packet loss. |
|
|
mackey
3 recommendations |
mackey
Premium Member
2017-Apr-26 6:21 am
First run, PP Enabled:
2nd run, PP Disabled:
3rd run, PP still Disabled:
4th run, PP re-Enabled:
|
|
Datalink Premium Member join:2014-08-11 Ottawa ON
5 recommendations |
Datalink
Premium Member
2017-Apr-26 6:41 am
Taking into consideration the discussion over the last three pages of posts, have a look at a couple of previous posts by college IT admins. What has been said over the last three pages reminded me of what the admins were seeing. I haven't reread thru all of the posts below, just the first half page or so. » [Equip] DPC3941B Modem Capacity» [Connectivity] Business 100/20 High Latency/High Loss when 400+ users onComments, given the discussion in this thread and the fact that both threads use a Puma 6 modem? |
|
mackey Premium Member join:2007-08-20
3 recommendations |
mackey
Premium Member
2017-Apr-26 6:47 am
Yeah, it really looks like they hit this Puma 6 SPS (sessions per second) issue. |
|
1 recommendation |
to Squishy Tia
Problem with this is that 5000+ UDP pps is not typical for a 90% of packets would realistically reside. The standards speak to throughput Mbps throughput, not PPS; so, I am interested in what testing of real traffic mix shows, 768 to 1500 byte packets both UDP and TCP, vs academia type testing of unrealistic traffic under the guise of "normal." Is this shows to be fine, let's call it what it is, DoS vulnerable, not flat out broken. |
|
mackey Premium Member join:2007-08-20
6 recommendations |
mackey
Premium Member
2017-Apr-26 8:48 am
said by Anon07dd2 :Problem with this is that 5000+ UDP pps is not typical for a 90% of packets would realistically reside. At max length (1500 bytes), 5000 PPS is only 7.5 mbps. I have a few IP security cameras which spit out more then that. said by Anon07dd2 :The standards speak to throughput Mbps throughput, not PPS Every router I've ever seen which is worth anything has both mbps *and* PPS throughput listed in the specs. Heck, even Ubnt's ERL has the PPS plastered all over their website. |
|
1 recommendation |
said by mackey:Every router I've ever seen which is worth anything has both mbps *and* PPS throughput listed in the specs. Heck, even Ubnt's ERL has the PPS plastered all over their website. You have to talk about what class of router you are talking about. Certainly I haven't seen an everyday home usage router, say your everyday WiFi router list such specs. |
|
|
Tchaika join:2017-03-20 New Orleans, LA
5 recommendations |
to nallar
said by nallar:The tiny lookup table issue explains why I've seemingly randomly had more severe connection issues... I have many connections going through the modem and the state table in my firewall always has more than 1024 entries. It has much more relaxed timeouts than the 1 second the puma 6 uses.
Whenever more than 1000 of those flows are sending packets during the same second I'm effectively running the same DoS attack against my modem - except entirely through normal usage. Thinking about an NTP Server, there are other protocols that behave in a similar fashion. Consider a bittorrent client with DHT enabled, it's going to have a ton of UDP flows to different hosts/ports. An insignificant (usually 20-30Kbps) amount of traffic, on average, but a ton of different flows, coming and going. Consider someone that chooses to run their own recursive DNS server. I've done this for over a decade, ever since ISPs started mucking with NXDOMAIN responses. Even a simple lookup (www.google.com) will generate multiple flows, first to the root servers, then to Google's authoritative servers. Google is a "easy" lookup; most outfits these days use CDNs, meaning you'll see multiple CNAME's, each pointing to a new domain, requiring a new query to the root servers, new query to the authoritative server, rinse and repeat.... Now consider the number of DNS queries required to load a modern webpage. It's not just www.whoever.com. Think about the burst of DNS traffic you'll see from a simple netstat command with reverse resolution enabled..... Hell, looking at the VM I'm typing this on, I have a VPN client installed. When launched it sends out pings to every server in its list (nearly one thousand) so it can sort them by latency. Is the SB6190 tracking ICMP flows? If so, I just blew out its buffer, through the action of launching a single piece of software on a single client machine behind my connection..... This is beyond disappointing. |
|
xymox1 Premium Member join:2008-05-20 Phoenix, AZ ARRIS SB8200 MikroTik CCR1036-8G-2S+
4 recommendations |
xymox1
Premium Member
2017-Apr-26 10:17 am
said by Tchaika:Consider someone that chooses to run their own recursive DNS server. I've done this for over a decade, ever since ISPs started mucking with NXDOMAIN responses. Even a simple lookup (www.google.com) will generate multiple flows, first to the root servers, then to Google's authoritative servers. Google is a "easy" lookup; most outfits these days use CDNs, meaning you'll see multiple CNAME's, each pointing to a new domain, requiring a new query to the root servers, new query to the authoritative server, rinse and repeat....
Now consider the number of DNS queries required to load a modern webpage. It's not just www.whoever.com. Think about the burst of DNS traffic you'll see from a simple netstat command with reverse resolution enabled..... *I* run a rDNS server at home.. |
|
1 recommendation |
to Tchaika
said by Tchaika:Hell, looking at the VM I'm typing this on, I have a VPN client installed. When launched it sends out pings to every server in its list (nearly one thousand) so it can sort them by latency. Is the SB6190 tracking ICMP flows? If so, I just blew out its buffer, through the action of launching a single piece of software on a single client machine behind my connection.....
This is beyond disappointing. While it's possible, I never noticed any issues with DNS or VPN usage when I owned a 6190 for almost two years. While others here did see issues, I literally saw zero difference in experience when swapping the 6190 out for a Netgear CM1000. (In fact, the CM1000 was worse until a firmware update fixed an issue where it would spontaneously go offline every week or so; the 6190 was always rock solid for me.) The fact that the 6190 failed the Puma6 test and the CM1000 did not had literally zero impact on my Internet usage, including a lot of time connected to VPNs. I however do not game nor do I use VOIP, so those users may see more of an issue. |
|
Tchaika join:2017-03-20 New Orleans, LA
4 recommendations |
to xymox1
said by xymox1:*I* run a rDNS server at home.. I would hazard a guess that most technically inclined people do. You've basically got three choices: 1) Use your ISP's servers, with or without NXDOMAIN hijacking, allowing them to hoover up all your queries for marketing purposes. 2) Use Google Public DNS, or a similar service, allowing them to hoover up..... 3) Roll your own rDNS server. #3 is child's play on Linux and Windows Server. |
|
10 recommendations |
to kucharsk
said by kucharsk:You can jump up and down and demand a recall all you want, but if the DoS vulnerability is fixed it still remains most people are thrilled with their 6190s. You keep saying "considered defective" - by whom? Which of the specs on the box is it not meeting? Intellectually, and to people knowledgeable about networking, sure. To the vast majority of customers, not really. ("So, if a bad guy is trying to attack my modem to shut off my Internet service, I might see slower than usual responses. This affects me how, precisely?")
In all seriousness, except for the need to issue firmware that deals with the DoS, the Puma6 devices are no worse off today than they were yesterday and will continue to sell briskly until the DOCSIS 3.1 rollout is complete.
Then again since I can still get a SB6183, SB6190s may continue to be sold for years. I surely wouldn't buy an SSD from you with that attitude. |
|
1 recommendation |
Look, I'm not trying to be an apologist for Arris or Intel here, but the truth is most people seem to be pretty happy with their 6190s, this thread notwithstanding.
As I've said before, if you notice issues, I feel for you and something should be done. |
|
Tchaika join:2017-03-20 New Orleans, LA
14 recommendations |
Tchaika
Member
2017-Apr-26 12:18 pm
said by kucharsk:Look, I'm not trying to be an apologist for Arris or Intel here And yet, here you are, doing exactly that, for reasons unclear to me and I presume others.... :/ |
|
xymox1 Premium Member join:2008-05-20 Phoenix, AZ ARRIS SB8200 MikroTik CCR1036-8G-2S+
8 recommendations |
to kucharsk
said by kucharsk:Look, I'm not trying to be an apologist for Arris or Intel here, but the truth is most people seem to be pretty happy with their 6190s, this thread notwithstanding.
As I've said before, if you notice issues, I feel for you and something should be done. So you don't consider a zero day DoS exploit that's in the wild as something every owner should be concerned about ?! And if it's a architecture issue that can never be fully addressed you don't consider that something every user would feel was significant ?! Listening to you I keep picturing Kelly Ann Conway and hearing alternate facts. It will be interesting to see what CERT thinks. Right now people might be trying out other exploit tools like malformed packets. Maybe we can find a packet of death or cause a full compromise. It's clear this modem is a POS and must have other serious issues. Your defense of the Intel puma 6 is singular and it would seem misguided. Sales must stop and the only mitigation for the zero day is a Broadcom based device. There won't be a firmware fix anytime soon, if ever. The puma 6 and maybe 7 should be thrown out and replaced. I can see script kiddie running this against blocks of IPs and just knocking people and companies offline for fun. You don't think this is a issue ? If your a company, it's OK to leave this modem connected ? |
|
9 recommendations |
to Tchaika
said by Tchaika:said by kucharsk:Look, I'm not trying to be an apologist for Arris or Intel here And yet, here you are, doing exactly that, for reasons unclear to me and I presume others.... :/ Indeed. I have just swapped out my 6190 for the CM600 for the last time. I swapped it back in to test the new firmware, but with these latest results its clear that anything at this point is just Band-Aids. Sales of any Puma 6 modem should halt immediately. |
|
4 recommendations |
to xymox1
I had a 6190 but I could definitely tell the difference in response times when gaming. Just wanted to point out Arris to their credit, did offer me a refund, I'm sure they didn't have to technically but I am glad they did and will keep this in mind for my next upgrade, of course next time I will make sure to only upgrade once a product is field tested and garnered good reviews |
|
6 recommendations |
to xymox1
Also would like to see how he seems to know that there are "a ton" of happy users out there. Also were is his proof and testing results as well. He asked for them. Why not post his proof. |
|
ARRIS SB6183
1 recommendation |
to xymox1
It's only a matter of time at this point. People are going to be running large scale DDoS attacks with only 1Gbps ports if they can do it with just 1.5Mbps... Sheesh, this whole thing is a disaster, and it seems as if its mostly been swept under the rug. |
|
train_wreckslow this bird down join:2013-10-04 Antioch, TN Cisco ASA 5506 Cisco DPC3939
4 recommendations |
to josefl710
said by josefl710:Just wanted to point out Arris to their credit, did offer me a refund, I'm sure they didn't have to technically but I am glad they did and will keep this in mind for my next upgrade Good on them. Always glad to hear when a company makes it right for the customer, particularly in this case where it really wasn't their direct fault. |
|
6 recommendations |
Anond3efc to xymox1
Anon
2017-Apr-26 1:10 pm
to xymox1
said by xymox1:I can see script kiddie running this against blocks of IPs and just knocking people and companies offline for fun. You don't think this is a issue ? If your a company, it's OK to leave this modem connected ? I don't think any provider with a lot of Puma6 devices on their network is going to like this. What's to stop someone with a botnet from taking down most of their subscriber CPE by exploiting this flaw. All it would take is someone with a chip on their shoulder or someone that wouldn't mind making a few bitcoins by extortion. |
|