Comcast XFINITY → Re: Comcast is using Sandvine to manage P2P Connections

I wasn't trying to equate them in a technical sense. Most people didn't care about NAT until they figured out they could skirt the "one computer" policy that was previously common with broadband providers. Cable customers played with packet manipulation (albeit a very different form) to their advantage before, now the cable companies are leveraging stupid TCP tricks to serve their agenda.It's adhering to RFC793 which set out the definition of TCP in 1981; you see a RST you have to shut down the connection.

I would agree that it's not the way for a carrier to behave, but I suspect when it comes to Comcast that's where you and I will disagree. Comcast is a residential broadband provider and not a full fledged carrier; the governance of operations is completely different. They're packaging a connection that isn't what you would get from a true carrier; it's a private network that has upstream Internet carrier connectivity. The oversubscription is higher, the ToS/AUP isn't as flexible, but in return you also pay significantly less than you would for a real carrier circuit.

Reset injection is not something all that flashy and new; our 8E6 content filters have been doing this for a couple years now. The key benefits from a network infrastructure standpoint are huge: less devices in-path and simpler firewall rules. While I agree that filtering is a cleaner solution, it's not always the most practical to implement. With the 8E6 filters I can have a simple Checkpoint firewall cluster sitting behind an Internet router with a very simple/easy-to-manage ruleset. Not having to worry about the complexity of a full content filtering ruleset makes life much easier for ongoing firewall management, not to mention the 8E6 can be have signatures updated throughout the day without incurring some of the nasty issues that can result during firewall rule updates. For client traffic filtering I just setup a span session from the Internet router to the 8e6 and it watches for URLs and sends resets on inappropriate content fetches. It stops the connection and I don't have to have another point of failure in my connection path.

Since we're back to talking technical details -- what do you propose for a better solution? Most of the filtering that Comcast does today happens at the cable modem, so the port 137-139 blocks, and the port 25 block if they put it in place happens well before things get upstream. With the dynamic ports used by BitTorrent clearly that isn't a solution.

Even throttling is tricky in that you'd need to identify the traffic so it can be queued appropriately. That means that some device in path would need to be able to recognize P2P traffic and mark the packets appropriately so that the packets could be filtered into the correct throttled queue. That means they can try to make this happen on their existing routing platforms if thats even possible, or they can introduce another box in-line to do the classification and inject another point of failure into the system. Even if they do this they'd have to deal with a significantly more complex queue structure than they have now.

I think if there were easy answers to this problem we wouldn't be 20+ pages into this thread.

-Eric

This is specifically one of the things Sandvine does -- deep packet inspection. The issue described here happens no matter what source or destination TCP port # is used (on either end).

It looks as if Sandvine is analysing established TCP sessions, looking for specific signature bytes (you touched base on this, re: your 8E6). I'm also under the impression that they look for signature bytes in the response packet. Upon matches in both cases (since the inspector is now aware of the TCP state on both ends), injects RST both directions (to the peer/client and the seed/server). That's been confirmed by funchords .

So, based on the methodology they're using for packet analysis, I would say that throttling/rate-limiting would be quite possible. But instead they opted for man-in-the-middle packet injection, which of course, really pisses me off.

Edit: Clarification on port #s

Sure it's possible, but only if the Sandvine box is directly in-line of the conversation path so that it can touch/mark the packets. By doing the reset injection the Sandvine box doesn't have to physically reside in the middle of the communication path, it just needs a span session directed to it so it can see copies of what traffic is flowing through the router and it can issue the resets completely out of band. If the Sandvine box kacks it won't take out the network, only P2P throttling will be broken.

-Eric

That's very true, and something I didn't consider. You're quite right -- rate-limiting would require the Sandvine unit to be sitting in the middle of the network path.

Then you're a bad player, stop doing that! -- There are solutions. Read this informative RFC:

RFC 3360: Inappropriate TCP Resets Considered Harmful

RST abuse is relatively new. The author of that RFC was talking about this:

said by »list.nfr.com/pipermail/f ··· 672.html :

---

Of 24,000 or so web servers that we tested as part of the TBIT project, only 300 or so were behind firewalls that send TCP resets in this case, so clearly most of the world seems to be maintaining reasonably adequate security without sending TCP Resets in this case.

---

Well, let's get one thing perfectly straight: the RST forgery/injection is wrong and must be stopped -- even if there is no other solution to replace it.

But there are solutions:

- Be public about the problem, and enlist the customers' assistance in solving it. "This is a shared service and heavy uploading by one or two customers impacts the entire neighborhood." That's not hard to say -- Wireless ISPs and Satellite ISPs make this fact very clear to their customers. The reason they're not being public about the problem is because they have to compete with DSL and FIOS, which balances a lot more bandwidth across a much larger field of customers. As a result, DSL/FIOS can tolerate a larger percentage of heavy uploaders before their other customers begin to be affected.

- Those that do not cooperatively manage their usage can be put in a penalty box, like the port 25 issue is handled on Comcast. If the account is uploading at a sustained rate over 60%-80% of his tier for two hours, then limit the account to an upload of 128 kbps and send an e-mail to account holder. The account holder gets a Computer-Based Training lesson about about "fair use" of a "shared connection," clicks a link, and he is restored to full service by noon the next day.

"Those that do not cooperatively manage their usage can be put in a penalty box, like the port 25 issue is handled on Comcast. If the account is uploading at a sustained rate over 60%-80% of his tier for two hours, then limit the account to an upload of 128 kbps and send an e-mail to account holder. The account holder gets a Computer-Based Training lesson about about "fair use" of a "shared connection," clicks a link, and he is restored to full service by noon the next day."

Then we can have a 20 page thread about that eh?
Hob

Yeah, exactly! This one, probably: »Comcast Bandwidth Abuse/Limits - Discuss here only

That RFC has very little to do with this discussion. It was drafted largely in response to packets with non-zero reserved bits in the TCP header being rejected by firewalls. Specifically he was concerned with firewalls blocking traffic with hosts that decided to try to implement explicit congestion notification. He did include commentary stating

"We would recommend that the TCP reset not be used as a congestion control mechanism, because this overloads the semantics of the reset message, and inevitably leads to more aggressive behavior from TCP implementations in response to a reset. We would suggest that simply dropping the SYN packet is the most effective response to congestion. The TCP sender will retransmit the SYN packet, using the default value for the Retransmission Timeout (RTO), backing-off the retransmit timer after each retransmit."

There's a bit of an issue with that statement; the goal of Sandvine is to shut down connections, not throttle them. For Sandvine to work transparently it should seem like the host port is closed for connections, and the standard TCP/IP stack response to closed ports is to send a reset! Everybody seems to forget this because nearly everything (including Windows) comes with a firewall these days with a Draconian ruleset that still seems to foster the idea that obscurity has some relation to security. Disable your windows firewall or flush IPtables and try to connect to a closed port -- you'll get a nice RST back indicating the port is not available. From a debugging standpoint this is what you want to see -- some response that will help you determine why things aren't working.

The RFC author's main concern was that TCP implementations would get more aggressive in response to RST packets and start spewing SYNs (he cited the example of a stack that generated 4 connection attempts even after receiving RST responses). It's 5 years later now, and there's no indication that was really a valid concern.

It's important to keep in mind that all RFCs are not standards in and of themselves. Some do gain general acceptance as standards, but anyone can bring forth a document for review. You have to look at RFCs like 1149 or 968 to see that pretty much anyone can submit an RFC about anything, and it doesn't necessarily mean it's right.Is it mean? Sure. Is it tricky? Absolutely. Is it wrong? It depends on how you define wrong. We're talking about using valid TCP constructs to initiate the shutdown of a connection.

If Comcast were a carrier this would be a different discussion, but they're not. Carriers don't have to worry about things like DMCA notices because the responsibility for mitigation falls on the networks that represent the endpoints of the conversation. Comcast doesn't have that same luxury, as they are often one of those end-point networks. This has become more of a problem as options like BitTorrent have drastically lowered the knowledge base required to participate in the distribution of copyrighted material. Others in this thread have argued that P2P applications can indeed be used for legal purposes, but lets be realistic, most of the time that's not the case. Talking about the legal uses of P2P in this thread is like hanging out in a bordello and preaching about the virtues of virginity.The response to the abuse department talking one-on-one with folks that would be the target of this "education initiative" has been to post videos on YouTube or come on forums like this and talk about how Comcast is an evil company that doesn't let users download anything. Let's be realistic, customer education ain't gonna get this done.Pushing configs out to cable modems seems like a kludgy way to deal with this, but it might be workable. Otherwise doing differential throttling would mean more complexity to their existing traffic shaping solution. While that sounds trivial, years of experience in networking has shown me that simple things are easier to manage and break less frequently. In my opinion it's not worth trading overall network path stability to implement a Rube Goldberg system that would punish heavy file sharing users with the network equivalent of a young child's "time out".

-Eric