dslreports logo
 
    All Forums Hot Topics Gallery
spc
uniqs
10320

train_wreck
slow this bird down
join:2013-10-04
Antioch, TN
Cisco ASA 5506
Cisco DPC3939

3 recommendations

train_wreck to mackey

Member

to mackey

Re: SB6190 Puma6 TCP/UDP Network Latency Issue Discussion

said by mackey:

After disabling the packet processor with # echo "disable" > /proc/net/ti_pp iptraf-ng shot up to 5100 pps +/- 100 to match the server and web browsing went back to normal.

Just for curiosity, when you disable the processor how much of a total performance drop do you see?

mackey
Premium Member
join:2007-08-20

4 recommendations

mackey

Premium Member

Speed test maxes out at ~60 mbps instead of my normal ~350 mbps. The 6190's CPU goes to 100% as well.

train_wreck
slow this bird down
join:2013-10-04
Antioch, TN

7 recommendations

train_wreck

Member

Yeah figured as much. Weak CPU, with hardware assisted ASICs which come with many of their own "limitations".

Squishy Tia
join:2016-05-16

6 recommendations

Squishy Tia

Member

I think this pretty much seals the deal with the Puma 6 devices. The CPU can't handle the traffic on its own, and the packet processor can't handle more than a tiny amount of traffic before it becomes overloaded. And together performance is still well below any reasonable expectations.

And by the nomenclature in the command line, this is a Texas Instruments packet processor, yes?

What this set of tests confirms is that the Puma 6 can be perfectly fine at ~60 Mbit/sec or below regardless of traffic, or mediocre due to packet loss at any amount of traffic if the packet processor is left on. That's one hell of a design flaw and indicates these need to be recalled now rather than later.

Now the next phase of these tests: ≤24 bonded channels vs. >24 bonded channels using the same test methods. The reason I bring this up is because baseline latency doubles on puma 6 devices once you surpass 24 bonded channels on the downstream side, indicating a rather significant bottleneck occuring, presumably in either the ARM CPU or the packet processor, or both.

I'm very interested to see whether or not number of bonded channels affects either of these processors when they're isolated.
kucharsk
join:2001-03-30
Louisville, CO

kucharsk

Member

I still think the odds of a recall are infinitesimal.

A wild swag at what will probably happen is a software patch that will be more aggressive about just dropping packets destined for new destination ports if CPU usage is above 80% or so.

Squishy Tia
join:2016-05-16

7 recommendations

Squishy Tia

Member

said by kucharsk:

I still think the odds of a recall are infinitesimal.

A wild swag at what will probably happen is a software patch that will be more aggressive about just dropping packets if CPU usage is above 80% or so.

If they do that, then the device no longer falls within any respectable standard and would be deemed ultimately defective and would end up recalled. I'm sure you can imagine the support nightmare that would ensue when customers and businesses show constant dropped packets, especially for gaming, VOIP, and VPNs. No, unless there's some magic bullet that can strike the components of puma 6 and fuse them together into a lean mean networking machine, it's done for and a recall is very likely.

This is all going to end up as mounting evidence against Intel's puma 6 architecture, that much you can bank on.

The reason the architecture and thus the devices would be considered defective is that if you have to drop packets before even reaching max load, these devices are defective for the purposes of their intended markets, which are high speed/low latency applications. It certainly makes them unfit for anything higher than ~75 Mbit/sec.

And if you still believe a recall can't happen, take a quick look back at the Canadian MSOs which have already done recalls of puma 6 devices (prioritizing their gigabit tier customers first). The dominos keep falling, and they aren't falling in Intel's favor.
kucharsk
join:2001-03-30
Louisville, CO

1 recommendation

kucharsk

Member

You can jump up and down and demand a recall all you want, but if the DoS vulnerability is fixed it still remains most people are thrilled with their 6190s. You keep saying "considered defective" - by whom? Which of the specs on the box is it not meeting? Intellectually, and to people knowledgeable about networking, sure. To the vast majority of customers, not really. ("So, if a bad guy is trying to attack my modem to shut off my Internet service, I might see slower than usual responses. This affects me how, precisely?")

Intel has already moved on and so likely doesn't care if Puma6 sales fall to zero tomorrow.

Cable companies can recall devices all they want - doesn't mean they'll see a penny back from the manufacturer, and it doesn't mean the manufacturer will issue one. In fact it would likely be a matter of Intel cutting them a deal on Puma7s or 8s or some other under-the-table payment with zero admission of "fault."

In all seriousness, except for the need to issue firmware that deals with the DoS, the Puma6 devices are no worse off today than they were yesterday and will continue to sell briskly until the DOCSIS 3.1 rollout is complete.

Then again since I can still get a SB6183, SB6190s may continue to be sold for years.
kucharsk

1 recommendation

kucharsk to train_wreck

Member

to train_wreck
said by train_wreck:

Yeah figured as much. Weak CPU, with hardware assisted ASICs which come with many of their own "limitations".

While true, that makes for good system design; there's no sense in selling an engineered solution with more computing horsepower than it "needs" according to the design.

Yes, this falls apart in cases like this where a hardware processor doesn't seem to be performing as it should, but in most cases it saves a huge amount of money.

nallar
join:2016-12-07

5 recommendations

nallar

Member

Exactly. Hardware acceleration is fine if engineered correctly and we shouldn't be asking manufacturers to never use it.

The problem here is that it doesn't work properly.

The tiny lookup table issue explains why I've seemingly randomly had more severe connection issues... I have many connections going through the modem and the state table in my firewall always has more than 1024 entries. It has much more relaxed timeouts than the 1 second the puma 6 uses.

Whenever more than 1000 of those flows are sending packets during the same second I'm effectively running the same DoS attack against my modem - except entirely through normal usage.

mackey
Premium Member
join:2007-08-20

3 recommendations

mackey to Squishy Tia

Premium Member

to Squishy Tia
said by Squishy Tia:

What this set of tests confirms is that the Puma 6 can be perfectly fine at ~60 Mbit/sec or below regardless of traffic

Not sure how you got to that conclusion, even with the packet processor disabled it still has the "Puma 6" syndrome with delayed or lost packets (i.e. the DSLR Puma 6 test looks the same). In fact it's worse as with the PP disabled even established connections see the packet loss.
mackey

3 recommendations

mackey

Premium Member

First run, PP Enabled:


2nd run, PP Disabled:


3rd run, PP still Disabled:


4th run, PP re-Enabled:

Datalink
Premium Member
join:2014-08-11
Ottawa ON

5 recommendations

Datalink

Premium Member

Taking into consideration the discussion over the last three pages of posts, have a look at a couple of previous posts by college IT admins. What has been said over the last three pages reminded me of what the admins were seeing. I haven't reread thru all of the posts below, just the first half page or so.

»[Equip] DPC3941B Modem Capacity

»[Connectivity] Business 100/20 High Latency/High Loss when 400+ users on

Comments, given the discussion in this thread and the fact that both threads use a Puma 6 modem?

mackey
Premium Member
join:2007-08-20

3 recommendations

mackey

Premium Member

Yeah, it really looks like they hit this Puma 6 SPS (sessions per second) issue.

Anon07dd2
@2600:387.x

1 recommendation

Anon07dd2 to Squishy Tia

Anon

to Squishy Tia
Problem with this is that 5000+ UDP pps is not typical for a 90% of packets would realistically reside. The standards speak to throughput Mbps throughput, not PPS; so, I am interested in what testing of real traffic mix shows, 768 to 1500 byte packets both UDP and TCP, vs academia type testing of unrealistic traffic under the guise of "normal." Is this shows to be fine, let's call it what it is, DoS vulnerable, not flat out broken.

mackey
Premium Member
join:2007-08-20

6 recommendations

mackey

Premium Member

said by Anon07dd2 :

Problem with this is that 5000+ UDP pps is not typical for a 90% of packets would realistically reside.

At max length (1500 bytes), 5000 PPS is only 7.5 mbps. I have a few IP security cameras which spit out more then that.
said by Anon07dd2 :

The standards speak to throughput Mbps throughput, not PPS

Every router I've ever seen which is worth anything has both mbps *and* PPS throughput listed in the specs. Heck, even Ubnt's ERL has the PPS plastered all over their website.
kucharsk
join:2001-03-30
Louisville, CO

1 recommendation

kucharsk

Member

said by mackey:

Every router I've ever seen which is worth anything has both mbps *and* PPS throughput listed in the specs. Heck, even Ubnt's ERL has the PPS plastered all over their website.

You have to talk about what class of router you are talking about.

Certainly I haven't seen an everyday home usage router, say your everyday WiFi router list such specs.
Tchaika
join:2017-03-20
New Orleans, LA

5 recommendations

Tchaika to nallar

Member

to nallar
said by nallar:

The tiny lookup table issue explains why I've seemingly randomly had more severe connection issues... I have many connections going through the modem and the state table in my firewall always has more than 1024 entries. It has much more relaxed timeouts than the 1 second the puma 6 uses.

Whenever more than 1000 of those flows are sending packets during the same second I'm effectively running the same DoS attack against my modem - except entirely through normal usage.

Thinking about an NTP Server, there are other protocols that behave in a similar fashion. Consider a bittorrent client with DHT enabled, it's going to have a ton of UDP flows to different hosts/ports. An insignificant (usually 20-30Kbps) amount of traffic, on average, but a ton of different flows, coming and going.

Consider someone that chooses to run their own recursive DNS server. I've done this for over a decade, ever since ISPs started mucking with NXDOMAIN responses. Even a simple lookup (www.google.com) will generate multiple flows, first to the root servers, then to Google's authoritative servers. Google is a "easy" lookup; most outfits these days use CDNs, meaning you'll see multiple CNAME's, each pointing to a new domain, requiring a new query to the root servers, new query to the authoritative server, rinse and repeat....

Now consider the number of DNS queries required to load a modern webpage. It's not just www.whoever.com. Think about the burst of DNS traffic you'll see from a simple netstat command with reverse resolution enabled.....

Hell, looking at the VM I'm typing this on, I have a VPN client installed. When launched it sends out pings to every server in its list (nearly one thousand) so it can sort them by latency. Is the SB6190 tracking ICMP flows? If so, I just blew out its buffer, through the action of launching a single piece of software on a single client machine behind my connection.....

This is beyond disappointing.

xymox1
Premium Member
join:2008-05-20
Phoenix, AZ
ARRIS SB8200
MikroTik CCR1036-8G-2S+

4 recommendations

xymox1

Premium Member

said by Tchaika:

Consider someone that chooses to run their own recursive DNS server. I've done this for over a decade, ever since ISPs started mucking with NXDOMAIN responses. Even a simple lookup (www.google.com) will generate multiple flows, first to the root servers, then to Google's authoritative servers. Google is a "easy" lookup; most outfits these days use CDNs, meaning you'll see multiple CNAME's, each pointing to a new domain, requiring a new query to the root servers, new query to the authoritative server, rinse and repeat....

Now consider the number of DNS queries required to load a modern webpage. It's not just www.whoever.com. Think about the burst of DNS traffic you'll see from a simple netstat command with reverse resolution enabled.....

*I* run a rDNS server at home..
kucharsk
join:2001-03-30
Louisville, CO

1 recommendation

kucharsk to Tchaika

Member

to Tchaika
said by Tchaika:

Hell, looking at the VM I'm typing this on, I have a VPN client installed. When launched it sends out pings to every server in its list (nearly one thousand) so it can sort them by latency. Is the SB6190 tracking ICMP flows? If so, I just blew out its buffer, through the action of launching a single piece of software on a single client machine behind my connection.....

This is beyond disappointing.

While it's possible, I never noticed any issues with DNS or VPN usage when I owned a 6190 for almost two years.

While others here did see issues, I literally saw zero difference in experience when swapping the 6190 out for a Netgear CM1000.

(In fact, the CM1000 was worse until a firmware update fixed an issue where it would spontaneously go offline every week or so; the 6190 was always rock solid for me.)

The fact that the 6190 failed the Puma6 test and the CM1000 did not had literally zero impact on my Internet usage, including a lot of time connected to VPNs.

I however do not game nor do I use VOIP, so those users may see more of an issue.
Tchaika
join:2017-03-20
New Orleans, LA

4 recommendations

Tchaika to xymox1

Member

to xymox1
said by xymox1:

*I* run a rDNS server at home..

I would hazard a guess that most technically inclined people do. You've basically got three choices:

1) Use your ISP's servers, with or without NXDOMAIN hijacking, allowing them to hoover up all your queries for marketing purposes.
2) Use Google Public DNS, or a similar service, allowing them to hoover up.....
3) Roll your own rDNS server.

#3 is child's play on Linux and Windows Server.
ByteEnable
join:2017-04-20
Houston, TX

10 recommendations

ByteEnable to kucharsk

Member

to kucharsk
said by kucharsk:

You can jump up and down and demand a recall all you want, but if the DoS vulnerability is fixed it still remains most people are thrilled with their 6190s. You keep saying "considered defective" - by whom? Which of the specs on the box is it not meeting? Intellectually, and to people knowledgeable about networking, sure. To the vast majority of customers, not really. ("So, if a bad guy is trying to attack my modem to shut off my Internet service, I might see slower than usual responses. This affects me how, precisely?")

In all seriousness, except for the need to issue firmware that deals with the DoS, the Puma6 devices are no worse off today than they were yesterday and will continue to sell briskly until the DOCSIS 3.1 rollout is complete.

Then again since I can still get a SB6183, SB6190s may continue to be sold for years.

I surely wouldn't buy an SSD from you with that attitude.
kucharsk
join:2001-03-30
Louisville, CO

1 recommendation

kucharsk

Member

Look, I'm not trying to be an apologist for Arris or Intel here, but the truth is most people seem to be pretty happy with their 6190s, this thread notwithstanding.

As I've said before, if you notice issues, I feel for you and something should be done.
Tchaika
join:2017-03-20
New Orleans, LA

14 recommendations

Tchaika

Member

said by kucharsk:

Look, I'm not trying to be an apologist for Arris or Intel here

And yet, here you are, doing exactly that, for reasons unclear to me and I presume others.... :/

xymox1
Premium Member
join:2008-05-20
Phoenix, AZ
ARRIS SB8200
MikroTik CCR1036-8G-2S+

8 recommendations

xymox1 to kucharsk

Premium Member

to kucharsk
said by kucharsk:

Look, I'm not trying to be an apologist for Arris or Intel here, but the truth is most people seem to be pretty happy with their 6190s, this thread notwithstanding.

As I've said before, if you notice issues, I feel for you and something should be done.

So you don't consider a zero day DoS exploit that's in the wild as something every owner should be concerned about ?! And if it's a architecture issue that can never be fully addressed you don't consider that something every user would feel was significant ?!

Listening to you I keep picturing Kelly Ann Conway and hearing alternate facts.

It will be interesting to see what CERT thinks.

Right now people might be trying out other exploit tools like malformed packets. Maybe we can find a packet of death or cause a full compromise. It's clear this modem is a POS and must have other serious issues.

Your defense of the Intel puma 6 is singular and it would seem misguided. Sales must stop and the only mitigation for the zero day is a Broadcom based device. There won't be a firmware fix anytime soon, if ever. The puma 6 and maybe 7 should be thrown out and replaced.

I can see script kiddie running this against blocks of IPs and just knocking people and companies offline for fun. You don't think this is a issue ? If your a company, it's OK to leave this modem connected ?

BillRoland
Premium Member
join:2001-01-21
Ocala, FL

9 recommendations

BillRoland to Tchaika

Premium Member

to Tchaika
said by Tchaika:

said by kucharsk:

Look, I'm not trying to be an apologist for Arris or Intel here

And yet, here you are, doing exactly that, for reasons unclear to me and I presume others.... :/

Indeed.

I have just swapped out my 6190 for the CM600 for the last time. I swapped it back in to test the new firmware, but with these latest results its clear that anything at this point is just Band-Aids.

Sales of any Puma 6 modem should halt immediately.
josefl710
join:2000-11-04
Houston, TX

4 recommendations

josefl710 to xymox1

Member

to xymox1
I had a 6190 but I could definitely tell the difference in response times when gaming. Just wanted to point out Arris to their credit, did offer me a refund, I'm sure they didn't have to technically but I am glad they did and will keep this in mind for my next upgrade, of course next time I will make sure to only upgrade once a product is field tested and garnered good reviews

BimmerE38FN
join:2002-09-15
Boise, ID

6 recommendations

BimmerE38FN to xymox1

Member

to xymox1
Also would like to see how he seems to know that there are "a ton" of happy users out there. Also were is his proof and testing results as well. He asked for them. Why not post his proof.
joshh20
join:2012-12-19
ARRIS SB6183

1 recommendation

joshh20 to xymox1

Member

to xymox1
It's only a matter of time at this point. People are going to be running large scale DDoS attacks with only 1Gbps ports if they can do it with just 1.5Mbps... Sheesh, this whole thing is a disaster, and it seems as if its mostly been swept under the rug.

train_wreck
slow this bird down
join:2013-10-04
Antioch, TN
Cisco ASA 5506
Cisco DPC3939

4 recommendations

train_wreck to josefl710

Member

to josefl710
said by josefl710:

Just wanted to point out Arris to their credit, did offer me a refund, I'm sure they didn't have to technically but I am glad they did and will keep this in mind for my next upgrade

Good on them. Always glad to hear when a company makes it right for the customer, particularly in this case where it really wasn't their direct fault.

Anond3efc
@comcast.net

6 recommendations

Anond3efc to xymox1

Anon

to xymox1
said by xymox1:

I can see script kiddie running this against blocks of IPs and just knocking people and companies offline for fun. You don't think this is a issue ? If your a company, it's OK to leave this modem connected ?

I don't think any provider with a lot of Puma6 devices on their network is going to like this.
What's to stop someone with a botnet from taking down most of their subscriber CPE by exploiting this flaw.
All it would take is someone with a chip on their shoulder or someone that wouldn't mind making a few bitcoins by extortion.