Comcast XFINITY → Adjusting TCP stack for 50mbps - CentOS Linux

Update
Now that I've had a few days to test, I've been able to determine that my upper limit of 18mbps is purely coincidental based on standard TCP window scaling parameters and the fact that every remote device I can use for testing is > 18ms away. I went back and read the DOCSIS3 spec regarding their bonding solution, and they address out of order delivery by having massive buffering capabilities in the modem, which re-orders packets in the correct sequence before dumping them out the Ethernet interface. I spent a bit of time playing around with TCP tuning parameters on one of my boxes I colo @ Gigenet in Chicago, and I've been able to get transfers to hit 50mbps in a single TCP flow.

Server kernel version:

Linux cluster 2.6.18-53.1.21.el5xen #1 SMP Tue May 20 10:31:46 EDT 2008 i686 i686 i386 GNU/Linux
 

Server apache version:

Server version: Apache/2.2.3
Server built:   Jan 15 2008 20:33:30
 

Server TCP modifications:

net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.ipv4.tcp_rmem = 4096        87380   8388608
net.ipv4.tcp_wmem = 4096        87380   8388608
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_abc = 2
 

wget result: (TCP windows didn't grow to hit 6MB/sec until 50MB into the transfer)

--07:17:26--  http://cluster/100mb.bin
Resolving cluster... 
Connecting to cluster|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: `100mb.bin'
 
100%[===========================================================>] 104,857,600 6.04M/s   in 25s    
 
07:17:52 (3.93 MB/s) - `100mb.bin' saved [104857600/104857600]
 

To get a better net result, I created a larger test file. After 50MB into the transfer speeds stayed consistently between 5.5MB/s and 6.2MB/sec.

--08:01:17--  http://cluster/500mb.bin
Resolving cluster... 
Connecting to cluster|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 524288000 (500M) [application/octet-stream]
Saving to: `500mb.bin'
 
100%[===========================================================>] 524,288,000 6.10M/s   in 93s    
 
08:02:50 (5.39 MB/s) - `500mb.bin' saved [524288000/524288000]
 

Conclusion: Amended
So, the 50/5 service really is 50mbps afterall. It just looks like the it takes a lot of work to get TCP to perform at that level. Given the amount of tweaking I had to do to the TCP stack on my Linux box to achieve these results, I can pretty much guarantee that 95+% of Internet servers are not going to be tweaked to provide this level of performance.

Note: these tweaks need to be performed at the SERVER side of the connection; changes on the client side alone will be ineffective at bringing these results!

The most critical components are enabling Appropriate Byte Counting (with aggressive 2*MSS window scaling) to allow the window to scale based on the number of bytes acknowledged with each TCP ACK instead of just based on the number of TCP ACKs, increasing the TCP write memory space for outbound windowing, and changing the TCP Congestion Control Algorithm from 'bic' or 'reno' to 'cubic'.

More interesting stuff there Eric. So for now this would seem more marketing hype rather than actual throughput speeds. At least until more servers are tweaked. So rather than paying for 50mbps down speeds you're really paying for "Up To" in this case. At least the Upstream is much better.

I wonder however even without all the server tweaks wouldn't you still be able to get to that bandwidth by say downloading from multiple sources of higher bandwidth providers? Example: Downloading a very large file from Microsoft and at the same time pulling some Nix distro, etc. Perhaps the combined downloads would be able to hit the 50mbps mark as long as the sources can push at higher levels.

After all I see the problem as not so much how fast Comcast can provide it's more of how fast the data can be sent that seems more of the limitation for now!

Yeah, that works just fine -- it's why I was initially sure that it was per-flow balancing across the channels. If I fired up one session it would hit 2.2MB/sec and never get higher, but I could open up another that would hit 2.2MB/sec, and a 3rd that would hit 1.8-2.2MB/sec. Hitting 50mbps across multiple TCP connections has been quite easy, actually, if I choose the right sources.

Awesome work!

The only time I"ve ever gotten close to 50megs constant was using DAP even for files located in Minneapolis. or at speedtest.net

either way still kind of sucks and really makes you wonder if Comcast is going to address this problem otherwise a lot of people are going to be upset they aren't getting their speeds.