dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
583
LTCtech
join:2007-06-19
Woodinville, WA

LTCtech

Member

Comcast EDI w/ Ciena LE-311v: iperf Induces Packet Loss

We noticed a peculiarity on our Comcast EDI 50M link which uses a Ciena LE-311v for hand off. This device provided by Comcast is something like a decade old, so maybe that's the cause:
»web.archive.org/web/2005 ··· ions.pdf

Start high bandwidth and high RTT download (doesn’t really matter where from as long as high RTT and has good bandwidth to Comcast):

Aim large UDP datagram 1Mbits iperf at target IP address from any high performance link (nothing on target side has to be listening, in fact 5001 UDP can even be blocked on target side):
iperf -c [TargetIP] -u -t 60 -i 1 -b 1M -l 26k

Here is what happens:
1. HTTP download from link is able to consistently saturate 50M connection
2. While HTTP download is still running, run the iperf command above from a high performance connection
3. Immediately once iperf starts, HTTP download mentioned above falls to 5-10M and stays there
4. Once iperf is turned off, HTTP download recovers to saturate 50M connection

What I think is happening is that the buffers on some device are filling up as they are too small. Datagrams smaller than 26k do not cause this issue. As the UDP streams fills up the buffers it induces packet loss on the circuit. It’s not our router, I have replicated this individually through our pfSense (FreeBSD) firewall, a Cisco ISR, and a Lenovo T530 Laptop with Windows 8.1 hooked up to the Ciena.

High RTT TCP downloads are severely degraded, low RTT downloads are degraded a bit but it’s not really noticeable.

Does anyone have any ideas as to the root cause of this? Can anyone test this on their EDI circuit?
LTCtech

LTCtech

Member

It appears to be a known issue as a post from 2007 describes a similar issue:
»arstechnica.com/civis/vi ··· t=196351
»arstechnica.com/civis/vi ··· p4763011

mackey
Premium Member
join:2007-08-20

mackey to LTCtech

Premium Member

to LTCtech
Not sure what the problem is here. What you are describing is the same principle as a DDoS attack - send more traffic to a target then that target can handle. Unlike TCP, UDP is stateless and will flood a connection even if there is nothing listening on the other end - this is why DNS amplification/reflection attacks have become popular.

You can get the same result if you connect 2 gigabit networks together with a 100 mbit link. If a server attempts to throw a gigabit worth of UDP traffic at a host on the other network, all traffic going through that 100 mbit link is going to suffer (no matter how big of a buffer you have) even if nothing is receiving that UDP traffic.

You will not see this problem with TCP streams due to TCPs "exponential backoff" congestion control algorithm.
LTCtech
join:2007-06-19
Woodinville, WA

LTCtech

Member

After a week of cajoling to upgrade our switch, Comcast came out and installed a brand new Ciena 3916.

Seems the iperf issue is gone and the connection performs a lot better overall. Downloads from high RTT sites are much cleaner. We will do some more testing but it does appear that the Ciena LE-311v or its configuration was at fault.

While we were effectively DOSing our connection, we shouldn't have been able to do it with a 1Mbit stream. The connection should be able to handle 50Mbit, no matter what is thrown at it.