dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
559598

mackey
Premium Member
join:2007-08-20

16 recommendations

mackey to train_wreck

Premium Member

to train_wreck

Re: SB6190 Puma6 TCP/UDP Network Latency Issue Discussion

Well this is interesting. Your post yesterday got me thinking about UDP and how it handles it, and I was wondering if it really creates a LUT entry for every UDP packet and if so, exactly how resource intensive that process is. I.E. can you DoS one of these with a low bandwidth stream of UDP packets going to different ports.

So, I ran the Puma 6 test while trying that. First up, 2.5 mbps of UDP traffic:



Hmmm, not quite offline. Web pages were still coming up, but taking a good 20-30 seconds for even basic pages. Lets up that to 3.5 mbps:



And there it is.

For reference, a couple seconds after I stopped the test:



Completely offline at 1% of my provisioned download bandwidth? Stick a fork in it, these things are done.

xymox1
Premium Member
join:2008-05-20
Phoenix, AZ

3 recommendations

xymox1 to NetDog

Premium Member

to NetDog
Hahahahaha... oops....

Just wow... Good test...

BillRoland
Premium Member
join:2001-01-21
Ocala, FL

1 recommendation

BillRoland to mackey

Premium Member

to mackey
said by mackey:

Well this is interesting. Your post yesterday got me thinking about UDP and how it handles it, and I was wondering if it really creates a LUT entry for every UDP packet and if so, exactly how resource intensive that process is. I.E. can you DoS one of these with a low bandwidth stream of UDP packets going to different ports.

So, I ran the Puma 6 test while trying that. First up, 2.5 mbps of UDP traffic:

Hmmm, not quite offline. Web pages were still coming up, but taking a good 20-30 seconds for even basic pages. Lets up that to 3.5 mbps:

And there it is.

For reference, a couple seconds after I stopped the test:

Completely offline at 1% of my provisioned download bandwidth? Stick a fork in it, these things are done.

Oh man...

BimmerE38FN
join:2002-09-15
Boise, ID

1 recommendation

BimmerE38FN to mackey

Member

to mackey
Hope NetDog sees this.
ByteEnable
join:2017-04-20
Houston, TX

1 recommendation

ByteEnable to mackey

Member

to mackey
said by mackey:

Well this is interesting. Your post yesterday got me thinking about UDP and how it handles it, and I was wondering if it really creates a LUT entry for every UDP packet and if so, exactly how resource intensive that process is. I.E. can you DoS one of these with a low bandwidth stream of UDP packets going to different ports.

Completely offline at 1% of my provisioned download bandwidth? Stick a fork in it, these things are done.

Excellent experiment! I have three XBox's in my house. This modem would fall to it's knee's with FPS games.

train_wreck
slow this bird down
join:2013-10-04
Antioch, TN

1 recommendation

train_wreck to mackey

Member

to mackey
So I take it that most other modems don't suffer from this problem because they don't use accelerators? (e.g., they have a CPU that processes everything, and is powerful enough to handle it)

KoRnGtL15
Premium Member
join:2007-01-04
Grants Pass, OR

4 recommendations

KoRnGtL15 to mackey

Premium Member

to mackey
You cant even laugh any more. Its just beyond pathetic at this point for this modem. Just recall it already jeebus!
Datalink
Premium Member
join:2014-08-11
Ottawa ON

2 edits

1 recommendation

Datalink to mackey

Premium Member

to mackey
@mackey, how did you generate the UDP traffic, running an Iperf download test? Thats one test that I haven't tried, a UDP Iperf download and Puma 6 test at the same time.

Your test is a good intro into the following post from a Rogers engineer. The graphs were run with a program titled FLENT, The Flexible Network Tester. Looks like it runs on anything but Windows. In any event, this runs a mixture of downloads and uploads and ping test simultaneously while collecting the data for the plot. Similar in idea I think to your test, run a UDP download and Puma 6 TCP test at the same time

»communityforums.rogers.c ··· 7#M44896

Those plots represent the results for a CODA-4582 Puma 7 modem, running firmware version 2.0.10.26T2, and then 2.0.10.27 which has just been released to the test group. Note the very large reduction in ping time from .26T2 to .27.

It would be interesting for anyone who is running Ubuntu, Debian, Arch Linux, or OSX with Macbrew to run this test on a Puma 6 modem. If you have gigabit service, I would be very interested in the results.

»flent.org/

xymox1
Premium Member
join:2008-05-20
Phoenix, AZ
ARRIS SB8200
MikroTik CCR1036-8G-2S+

xymox1 to NetDog

Premium Member

to NetDog
I reproduced this result. I did a iPerf UDP and ran the Puma 6 test and complete DoS.. I will do better tests shortly. But i should be abel to do Pingplotter TCP to google and then run iPerf in UDP and it should kill off connectivity.

I would like to know as well what you used for the UDP stream.

justin
..needs sleep
Mod
join:1999-05-28
2031

justin

Mod

Does it ddos if the udp is from outside unsolicited?
Datalink
Premium Member
join:2014-08-11
Ottawa ON

Datalink

Premium Member

Yet another interesting question. That would be pretty deadly as it wouldn't take much in resources to pull that off.

NetDog
Premium Member
join:2002-03-04
Hollywood, FL

4 recommendations

NetDog to BimmerE38FN

Premium Member

to BimmerE38FN
said by BimmerE38FN:

Hope NetDog sees this.

Oh you know it but I can't confirm it or test it till Thursday in San Jose right now @IPv6 Conference..

BimmerE38FN
join:2002-09-15
Boise, ID

1 recommendation

BimmerE38FN

Member

Will be interesting to see what you find out when you get back. Enjoy SJ.
said by NetDog:

said by BimmerE38FN:

Hope NetDog sees this.

Oh you know it but I can't confirm it or test it till Thursday in San Jose right now @IPv6 Conference..


isp10002
join:2012-10-24

isp10002 to NetDog

Member

to NetDog
This should be on major news networks so we can fry Arris and and intel for this

mackey
Premium Member
join:2007-08-20

2 recommendations

mackey to xymox1

Premium Member

to xymox1
Even though it's trivial I'd still prefer not to post my exact commands, so see PM.

For the record it works with TCP as well, and is spoofable as it does not require completing the TCP 3-way handshake.
mackey

2 recommendations

mackey to justin

Premium Member

to justin
said by justin:

Does it ddos if the udp is from outside unsolicited?

Yes, that's what I've been testing with. Also works with (outside unsolicited) spoofed TCP packets.
Tchaika
join:2017-03-20
New Orleans, LA

3 recommendations

Tchaika to justin

Member

to justin
said by justin:

Does it ddos if the udp is from outside unsolicited?

I host an NTP server in the NTP Pool, serving an average of 20 to 25 requests per second, with nearly all of these coming from unique hosts. There are currently 971 flow entries in my IPv4 table and 66 in IPv6, on Linux, with the default (30 seconds in nf_conntrack_udp_timeout) settings, nearly all of them related to the NTP server.....

..... I ran this server while I had my SB6190. I wonder how much that may have contributed to my problems with that modem?

nallar
join:2016-12-07

1 edit

5 recommendations

nallar to mackey

Member

to mackey
Confirmed. Wrote a little tool to send lots of small UDP packets over a range of ports and seeing similar awful behaviour.

Still can't change modem due to my provider refusing and a monopoly.

»github.com/nallar/Puma6F ··· l#readme



mackey
Premium Member
join:2007-08-20

1 recommendation

mackey

Premium Member

After a bit more testing I can't get it to *completely* fall over, but even with as little as 800 kbps of traffic web browsing starts becoming painful and by 1.6 mbps you have a 50/50 chance of webpages loading.
AnthonyD90
join:2017-03-19

3 recommendations

AnthonyD90

Member

Do you have the resources to try this test on a non puma6 modem as a control? I'm curious to see how a "normal" modem handles a situation like this.

xymox1
Premium Member
join:2008-05-20
Phoenix, AZ
ARRIS SB8200
MikroTik CCR1036-8G-2S+

2 recommendations

xymox1 to NetDog

Premium Member

to NetDog
Just wow.... Thats stunning...

Also as it has not been posted here yet..

»www.prnewswire.com/news- ··· 897.html
kucharsk
join:2001-03-30
Louisville, CO

1 recommendation

kucharsk to mackey

Member

to mackey
I'd like to see:

1) What are the results with other cable modems - both modems like the CM1000/SB8200 and modems like the SB6183.

2) Is this UDP traffic originating from inside or outside the LAN?

Surely someone with a botnet couldn't just send UDP traffic to IP addresses in the Xfinity range and cause most of their users to drop off.
kucharsk

kucharsk to xymox1

Member

to xymox1
Was this from another site on the Internet to your local host or across your LAN?

I don't consider LAN to WAN tests relevant simply because there are any number of things you could do from within your LAN to DoS most any simple router like a cable modem (or even more complex ones, as I've seen a flaky network interface take out an entire corporate subnet before.)

mackey
Premium Member
join:2007-08-20

3 recommendations

mackey to kucharsk

Premium Member

to kucharsk
Don't have time to try #1 right this second even though I have both a 6183 and an 8200, and am nervous about trying as well as I just know Charter will "accidentally" move me to a Spectrum plan thereby breaking the grandfathering and adding $30/mo to my bill.

2) Outside.

And yes, they can.
kucharsk
join:2001-03-30
Louisville, CO

1 recommendation

kucharsk

Member

I know you can't do this test, but someone surely can as at present we have no way of knowing whether this DoS effect is occurring at the cable modem level or higher up at the CMTS.

There are enough people here with SB6183s, CM1000s or SB8200s who should be able to run the same WAN to LAN test and post their results if you could write up your test procedure; that way we could know for sure whether it's the SB6190/Puma6 or just something that happens when the link gets flooded with UDP packets.

mackey
Premium Member
join:2007-08-20

3 recommendations

mackey

Premium Member

said by kucharsk:

at present we have no way of knowing whether this DoS effect is occurring at the cable modem level or higher up at the CMTS.

As the CPU goes to 100% and the load average triples I think it's safe to say it's at the cable modem.

Before:
# top -n1
Mem: 70172K used, 51520K free, 0K shrd, 717864628K buff, 717621460K cached
CPU:  19% usr  23% sys   0% nic  52% idle   0% io   0% irq   4% sirq
Load average: 1.27 1.32 1.36 2/122 908
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  908   526 root     R     1152   1%  33% top -n1
   46     2 root     RWN      0   0%   5% [irq/24-TX Compl]
  359   115 root     S    59032  48%   0% /usr/sbin/snmp_agent_cm -c /etc/agent
  362   115 root     S    57748  47%   0% /usr/sbin/dmg_provisioning
  338   115 root     S    57552  47%   0% /usr/sbin/docsis_mac_manager
  300   115 root     S    57540  47%   0% /usr/sbin/downstream_manager 24 28
  305   115 root     S    57540  47%   0% /usr/sbin/downstream_manager 19 23
  306   115 root     S    57540  47%   0% /usr/sbin/downstream_manager 18 22
  301   115 root     S    57540  47%   0% /usr/sbin/downstream_manager 23 27
  307   115 root     S    57540  47%   0% /usr/sbin/downstream_manager 17 21
  311   115 root     S    57540  47%   0% /usr/sbin/downstream_manager 13 17
  304   115 root     S    57540  47%   0% /usr/sbin/downstream_manager 20 24
  302   115 root     S    57540  47%   0% /usr/sbin/downstream_manager 22 26
  308   115 root     S    57540  47%   0% /usr/sbin/downstream_manager 16 20
  313   115 root     S    57540  47%   0% /usr/sbin/downstream_manager 11 15
  309   115 root     S    57540  47%   0% /usr/sbin/downstream_manager 15 19
  303   115 root     S    57540  47%   0% /usr/sbin/downstream_manager 21 25
  314   115 root     S    57540  47%   0% /usr/sbin/downstream_manager 10 14
  312   115 root     S    57540  47%   0% /usr/sbin/downstream_manager 12 16
  316   115 root     S    57540  47%   0% /usr/sbin/downstream_manager 8 12
# grep Detected /proc/net/pp/LUT2/busy
Detected 8 sessions in packet processor
# 
 

During:
# top -n1
Mem: 70308K used, 51384K free, 0K shrd, 717766324K buff, 717523156K cached
CPU:  19% usr  68% sys   0% nic   0% idle   0% io   0% irq  11% sirq
Load average: 4.09 1.87 1.53 2/122 910
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  910   526 root     R     1152   1%  23% top -n1
    4     2 root     RWN      0   0%  11% [kworker/0:0]
  313   115 root     S    57540  47%   3% /usr/sbin/downstream_manager 11 15
  318   115 root     S    57540  47%   3% /usr/sbin/downstream_manager 6 10
  321   115 root     S    57540  47%   3% /usr/sbin/downstream_manager 3 7
    6     2 root     SW       0   0%   3% [rcu_kthread]
  359   115 root     S    59032  48%   2% /usr/sbin/snmp_agent_cm -c /etc/agent
  306   115 root     S    57540  47%   2% /usr/sbin/downstream_manager 18 22
  307   115 root     S    57540  47%   2% /usr/sbin/downstream_manager 17 21
  304   115 root     S    57540  47%   2% /usr/sbin/downstream_manager 20 24
  302   115 root     S    57540  47%   2% /usr/sbin/downstream_manager 22 26
  303   115 root     S    57540  47%   2% /usr/sbin/downstream_manager 21 25
  312   115 root     S    57540  47%   2% /usr/sbin/downstream_manager 12 16
  315   115 root     S    57540  47%   2% /usr/sbin/downstream_manager 9 13
  317   115 root     S    57540  47%   2% /usr/sbin/downstream_manager 7 11
  319   115 root     S    57540  47%   2% /usr/sbin/downstream_manager 5 9
  320   115 root     S    57540  47%   2% /usr/sbin/downstream_manager 4 8
  322   115 root     S    57540  47%   2% /usr/sbin/downstream_manager 2 6
  242   115 root     S    56336  46%   2% /usr/sbin/hal_tuner_mgr
   28     2 root     SW       0   0%   2% [irq/4-TI IIC]
# grep Detected /proc/net/pp/LUT2/busy
Detected 2032 sessions in packet processor
# 
 

This is with a 1500 kbps / 5500 pps stream from a single host.
kucharsk
join:2001-03-30
Louisville, CO

1 recommendation

kucharsk

Member

Of course the load average skyrockets, but we have no proof it's just the 6190/Puma 6, nor whether the DoS is happening at the CMTS level or not (we know it's busy processing packets, but without sniffing them we have no idea of what packets they are.)

If you can easily write up your test procedure, people here with other modems could quickly report back whether they see the same issues.

xymox1
Premium Member
join:2008-05-20
Phoenix, AZ
ARRIS SB8200
MikroTik CCR1036-8G-2S+

6 recommendations

xymox1

Premium Member

said by kucharsk:

Of course the load average skyrockets, but we have no proof it's just the 6190/Puma 6, nor whether the DoS is happening at the CMTS level or not (we know it's busy processing packets, but without sniffing them we have no idea of what packets they are.)

If you can easily write up your test procedure, people here with other modems could quickly report back whether they see the same issues.

We already confirmed it. I tested it. 2 others also tested it.. Its confirmed.
xymox1

6 recommendations

xymox1 to NetDog

Premium Member

to NetDog
I submitted a CERT report. This is a serious issue, trivial and exploitable externally. This is serious.. Anyone can now knock anyone, or any company, with a Puma 6 offline. Full DoS.. This is ALOT worse then anything we have stumbled across so far. As Mackey said "You can stick a fork in it, its done"...

Initial CERT report VRF#17-04-PFCGX

mackey
Premium Member
join:2007-08-20

mackey to kucharsk

Premium Member

to kucharsk
Edit: removed for now