mackey Premium Member join:2007-08-20
16 recommendations |
to train_wreck
Re: SB6190 Puma6 TCP/UDP Network Latency Issue DiscussionWell this is interesting. Your post yesterday got me thinking about UDP and how it handles it, and I was wondering if it really creates a LUT entry for every UDP packet and if so, exactly how resource intensive that process is. I.E. can you DoS one of these with a low bandwidth stream of UDP packets going to different ports. So, I ran the Puma 6 test while trying that. First up, 2.5 mbps of UDP traffic:
Hmmm, not quite offline. Web pages were still coming up, but taking a good 20-30 seconds for even basic pages. Lets up that to 3.5 mbps:
And there it is. For reference, a couple seconds after I stopped the test:
Completely offline at 1% of my provisioned download bandwidth? Stick a fork in it, these things are done.
|
|
xymox1 Premium Member join:2008-05-20 Phoenix, AZ
3 recommendations |
to NetDog
Hahahahaha... oops....
Just wow... Good test... |
|
1 recommendation |
to mackey
said by mackey:Well this is interesting. Your post yesterday got me thinking about UDP and how it handles it, and I was wondering if it really creates a LUT entry for every UDP packet and if so, exactly how resource intensive that process is. I.E. can you DoS one of these with a low bandwidth stream of UDP packets going to different ports.
So, I ran the Puma 6 test while trying that. First up, 2.5 mbps of UDP traffic:
Hmmm, not quite offline. Web pages were still coming up, but taking a good 20-30 seconds for even basic pages. Lets up that to 3.5 mbps:
And there it is.
For reference, a couple seconds after I stopped the test:
Completely offline at 1% of my provisioned download bandwidth? Stick a fork in it, these things are done. Oh man... |
|
1 recommendation |
to mackey
Hope NetDog sees this. |
|
1 recommendation |
to mackey
said by mackey:Well this is interesting. Your post yesterday got me thinking about UDP and how it handles it, and I was wondering if it really creates a LUT entry for every UDP packet and if so, exactly how resource intensive that process is. I.E. can you DoS one of these with a low bandwidth stream of UDP packets going to different ports.
Completely offline at 1% of my provisioned download bandwidth? Stick a fork in it, these things are done. Excellent experiment! I have three XBox's in my house. This modem would fall to it's knee's with FPS games. |
|
train_wreckslow this bird down join:2013-10-04 Antioch, TN
1 recommendation |
to mackey
So I take it that most other modems don't suffer from this problem because they don't use accelerators? (e.g., they have a CPU that processes everything, and is powerful enough to handle it) |
|
KoRnGtL15 Premium Member join:2007-01-04 Grants Pass, OR
4 recommendations |
to mackey
You cant even laugh any more. Its just beyond pathetic at this point for this modem. Just recall it already jeebus! |
|
|
Datalink Premium Member join:2014-08-11 Ottawa ON 2 edits
1 recommendation |
to mackey
@mackey, how did you generate the UDP traffic, running an Iperf download test? Thats one test that I haven't tried, a UDP Iperf download and Puma 6 test at the same time. Your test is a good intro into the following post from a Rogers engineer. The graphs were run with a program titled FLENT, The Flexible Network Tester. Looks like it runs on anything but Windows. In any event, this runs a mixture of downloads and uploads and ping test simultaneously while collecting the data for the plot. Similar in idea I think to your test, run a UDP download and Puma 6 TCP test at the same time » communityforums.rogers.c ··· 7#M44896Those plots represent the results for a CODA-4582 Puma 7 modem, running firmware version 2.0.10.26T2, and then 2.0.10.27 which has just been released to the test group. Note the very large reduction in ping time from .26T2 to .27. It would be interesting for anyone who is running Ubuntu, Debian, Arch Linux, or OSX with Macbrew to run this test on a Puma 6 modem. If you have gigabit service, I would be very interested in the results. » flent.org/ |
|
xymox1 Premium Member join:2008-05-20 Phoenix, AZ ARRIS SB8200 MikroTik CCR1036-8G-2S+
|
to NetDog
I reproduced this result. I did a iPerf UDP and ran the Puma 6 test and complete DoS.. I will do better tests shortly. But i should be abel to do Pingplotter TCP to google and then run iPerf in UDP and it should kill off connectivity.
I would like to know as well what you used for the UDP stream. |
|
justin..needs sleep Mod join:1999-05-28 2031 |
justin
Mod
2017-Apr-25 5:46 pm
Does it ddos if the udp is from outside unsolicited? |
|
Datalink Premium Member join:2014-08-11 Ottawa ON |
Datalink
Premium Member
2017-Apr-25 5:47 pm
Yet another interesting question. That would be pretty deadly as it wouldn't take much in resources to pull that off. |
|
NetDog Premium Member join:2002-03-04 Hollywood, FL
4 recommendations |
to BimmerE38FN
Oh you know it but I can't confirm it or test it till Thursday in San Jose right now @IPv6 Conference.. |
|
1 recommendation |
Will be interesting to see what you find out when you get back. Enjoy SJ. said by NetDog:Oh you know it but I can't confirm it or test it till Thursday in San Jose right now @IPv6 Conference.. |
|
|
to NetDog
This should be on major news networks so we can fry Arris and and intel for this |
|
mackey Premium Member join:2007-08-20
2 recommendations |
to xymox1
Even though it's trivial I'd still prefer not to post my exact commands, so see PM.
For the record it works with TCP as well, and is spoofable as it does not require completing the TCP 3-way handshake. |
|
mackey
2 recommendations |
to justin
said by justin:Does it ddos if the udp is from outside unsolicited? Yes, that's what I've been testing with. Also works with (outside unsolicited) spoofed TCP packets. |
|
Tchaika join:2017-03-20 New Orleans, LA
3 recommendations |
to justin
said by justin:Does it ddos if the udp is from outside unsolicited? I host an NTP server in the NTP Pool, serving an average of 20 to 25 requests per second, with nearly all of these coming from unique hosts. There are currently 971 flow entries in my IPv4 table and 66 in IPv6, on Linux, with the default (30 seconds in nf_conntrack_udp_timeout) settings, nearly all of them related to the NTP server..... ..... I ran this server while I had my SB6190. I wonder how much that may have contributed to my problems with that modem? |
|
1 edit
5 recommendations |
to mackey
Confirmed. Wrote a little tool to send lots of small UDP packets over a range of ports and seeing similar awful behaviour. Still can't change modem due to my provider refusing and a monopoly. » github.com/nallar/Puma6F ··· l#readme
|
|
mackey Premium Member join:2007-08-20
1 recommendation |
mackey
Premium Member
2017-Apr-25 7:43 pm
After a bit more testing I can't get it to *completely* fall over, but even with as little as 800 kbps of traffic web browsing starts becoming painful and by 1.6 mbps you have a 50/50 chance of webpages loading. |
|
3 recommendations |
Do you have the resources to try this test on a non puma6 modem as a control? I'm curious to see how a "normal" modem handles a situation like this. |
|
xymox1 Premium Member join:2008-05-20 Phoenix, AZ ARRIS SB8200 MikroTik CCR1036-8G-2S+
2 recommendations |
to NetDog
Just wow.... Thats stunning... Also as it has not been posted here yet.. » www.prnewswire.com/news- ··· 897.html |
|
1 recommendation |
to mackey
I'd like to see:
1) What are the results with other cable modems - both modems like the CM1000/SB8200 and modems like the SB6183.
2) Is this UDP traffic originating from inside or outside the LAN?
Surely someone with a botnet couldn't just send UDP traffic to IP addresses in the Xfinity range and cause most of their users to drop off. |
|
kucharsk |
to xymox1
Was this from another site on the Internet to your local host or across your LAN?
I don't consider LAN to WAN tests relevant simply because there are any number of things you could do from within your LAN to DoS most any simple router like a cable modem (or even more complex ones, as I've seen a flaky network interface take out an entire corporate subnet before.) |
|
mackey Premium Member join:2007-08-20
3 recommendations |
to kucharsk
Don't have time to try #1 right this second even though I have both a 6183 and an 8200, and am nervous about trying as well as I just know Charter will "accidentally" move me to a Spectrum plan thereby breaking the grandfathering and adding $30/mo to my bill.
2) Outside.
And yes, they can. |
|
1 recommendation |
I know you can't do this test, but someone surely can as at present we have no way of knowing whether this DoS effect is occurring at the cable modem level or higher up at the CMTS.
There are enough people here with SB6183s, CM1000s or SB8200s who should be able to run the same WAN to LAN test and post their results if you could write up your test procedure; that way we could know for sure whether it's the SB6190/Puma6 or just something that happens when the link gets flooded with UDP packets. |
|
mackey Premium Member join:2007-08-20
3 recommendations |
mackey
Premium Member
2017-Apr-25 11:29 pm
said by kucharsk:at present we have no way of knowing whether this DoS effect is occurring at the cable modem level or higher up at the CMTS. As the CPU goes to 100% and the load average triples I think it's safe to say it's at the cable modem. Before: # top -n1
Mem: 70172K used, 51520K free, 0K shrd, 717864628K buff, 717621460K cached
CPU: 19% usr 23% sys 0% nic 52% idle 0% io 0% irq 4% sirq
Load average: 1.27 1.32 1.36 2/122 908
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
908 526 root R 1152 1% 33% top -n1
46 2 root RWN 0 0% 5% [irq/24-TX Compl]
359 115 root S 59032 48% 0% /usr/sbin/snmp_agent_cm -c /etc/agent
362 115 root S 57748 47% 0% /usr/sbin/dmg_provisioning
338 115 root S 57552 47% 0% /usr/sbin/docsis_mac_manager
300 115 root S 57540 47% 0% /usr/sbin/downstream_manager 24 28
305 115 root S 57540 47% 0% /usr/sbin/downstream_manager 19 23
306 115 root S 57540 47% 0% /usr/sbin/downstream_manager 18 22
301 115 root S 57540 47% 0% /usr/sbin/downstream_manager 23 27
307 115 root S 57540 47% 0% /usr/sbin/downstream_manager 17 21
311 115 root S 57540 47% 0% /usr/sbin/downstream_manager 13 17
304 115 root S 57540 47% 0% /usr/sbin/downstream_manager 20 24
302 115 root S 57540 47% 0% /usr/sbin/downstream_manager 22 26
308 115 root S 57540 47% 0% /usr/sbin/downstream_manager 16 20
313 115 root S 57540 47% 0% /usr/sbin/downstream_manager 11 15
309 115 root S 57540 47% 0% /usr/sbin/downstream_manager 15 19
303 115 root S 57540 47% 0% /usr/sbin/downstream_manager 21 25
314 115 root S 57540 47% 0% /usr/sbin/downstream_manager 10 14
312 115 root S 57540 47% 0% /usr/sbin/downstream_manager 12 16
316 115 root S 57540 47% 0% /usr/sbin/downstream_manager 8 12
# grep Detected /proc/net/pp/LUT2/busy
Detected 8 sessions in packet processor
#
During: # top -n1
Mem: 70308K used, 51384K free, 0K shrd, 717766324K buff, 717523156K cached
CPU: 19% usr 68% sys 0% nic 0% idle 0% io 0% irq 11% sirq
Load average: 4.09 1.87 1.53 2/122 910
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
910 526 root R 1152 1% 23% top -n1
4 2 root RWN 0 0% 11% [kworker/0:0]
313 115 root S 57540 47% 3% /usr/sbin/downstream_manager 11 15
318 115 root S 57540 47% 3% /usr/sbin/downstream_manager 6 10
321 115 root S 57540 47% 3% /usr/sbin/downstream_manager 3 7
6 2 root SW 0 0% 3% [rcu_kthread]
359 115 root S 59032 48% 2% /usr/sbin/snmp_agent_cm -c /etc/agent
306 115 root S 57540 47% 2% /usr/sbin/downstream_manager 18 22
307 115 root S 57540 47% 2% /usr/sbin/downstream_manager 17 21
304 115 root S 57540 47% 2% /usr/sbin/downstream_manager 20 24
302 115 root S 57540 47% 2% /usr/sbin/downstream_manager 22 26
303 115 root S 57540 47% 2% /usr/sbin/downstream_manager 21 25
312 115 root S 57540 47% 2% /usr/sbin/downstream_manager 12 16
315 115 root S 57540 47% 2% /usr/sbin/downstream_manager 9 13
317 115 root S 57540 47% 2% /usr/sbin/downstream_manager 7 11
319 115 root S 57540 47% 2% /usr/sbin/downstream_manager 5 9
320 115 root S 57540 47% 2% /usr/sbin/downstream_manager 4 8
322 115 root S 57540 47% 2% /usr/sbin/downstream_manager 2 6
242 115 root S 56336 46% 2% /usr/sbin/hal_tuner_mgr
28 2 root SW 0 0% 2% [irq/4-TI IIC]
# grep Detected /proc/net/pp/LUT2/busy
Detected 2032 sessions in packet processor
#
This is with a 1500 kbps / 5500 pps stream from a single host. |
|
1 recommendation |
Of course the load average skyrockets, but we have no proof it's just the 6190/Puma 6, nor whether the DoS is happening at the CMTS level or not (we know it's busy processing packets, but without sniffing them we have no idea of what packets they are.)
If you can easily write up your test procedure, people here with other modems could quickly report back whether they see the same issues. |
|
xymox1 Premium Member join:2008-05-20 Phoenix, AZ ARRIS SB8200 MikroTik CCR1036-8G-2S+
6 recommendations |
xymox1
Premium Member
2017-Apr-25 11:39 pm
said by kucharsk:Of course the load average skyrockets, but we have no proof it's just the 6190/Puma 6, nor whether the DoS is happening at the CMTS level or not (we know it's busy processing packets, but without sniffing them we have no idea of what packets they are.)
If you can easily write up your test procedure, people here with other modems could quickly report back whether they see the same issues. We already confirmed it. I tested it. 2 others also tested it.. Its confirmed. |
|
xymox1
6 recommendations |
to NetDog
I submitted a CERT report. This is a serious issue, trivial and exploitable externally. This is serious.. Anyone can now knock anyone, or any company, with a Puma 6 offline. Full DoS.. This is ALOT worse then anything we have stumbled across so far. As Mackey said "You can stick a fork in it, its done"...
Initial CERT report VRF#17-04-PFCGX |
|
mackey Premium Member join:2007-08-20 |
to kucharsk
Edit: removed for now |
|