Level3: Verizon Intentionally Causing Netflix Congestion
by Karl Bode 05:56PM Thursday Jul 17 2014 Tipped by twizlar
While ISPs like Comcast, Verizon and AT&T claim that the latest round of peering and interconnection fights (and poor Netflix performance) are just peering business as usual, Netflix and transit operators continue to accuse ISPs of anti-competitive shenanigans. Level 3 last May proclaimed
that six of the largest ISPs were intentionally creating points of "permanent congestion" by refusing to upgrade their side of transit-operator facing connection links -- only resolved through direct interconnection payments to ISPs.
Verizon recently denied any wrong doing
in a blog post, insisting that they'd done an internal review of their network and found absolutely no congestion. To hear Verizon tell it, the streaming problems plaguing only the customers of certain ISPs are Netflix's fault for intentionally choosing poor transit providers.
In a blog post
today, Level3 VP of Content and Media Mark Taylor says Verizon's recent denial (and Verizon's handy infographic, above) actually proves that they're intentionally throttling connections.
Verizon's infographic shows a red bar where companies like Level3 and Verizon connect to exchange traffic requested by Verizon customers. Taylor points out that Level3 has ample bandwidth on their end of that equation, and -- as mirrored in earlier posts -- accuses Verizon of failing to upgrade capacity on their
end of that connection -- intentionally. Level3 uses the Los Angeles interconnection point in Verizon's example to make their case:
All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.
Verizon has confirmed that everything between that router in their network and their subscribers is uncongested – in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers.
Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion. Why is that? Maybe they can’t afford a new port card because they’ve run out – even though these cards are very cheap, just a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it.
Taylor then proceeds to simply accuse Verizon of using interconnection drive up the costs of a competitor:
But, here’s the other interesting thing also shown in the Verizon diagram. This congestion only takes place between Verizon and network providers chosen by Netflix. The providers that Netflix does not use do not experience the same problem. Why is that? Could it be that Verizon does not want its customers to actually use the higher-speed services it sells to them? Could it be that Verizon wants to extract a pound of flesh from its competitors, using the monopoly it has over the only connection to its end-users to raise its competitors’ costs?
Again, if the FCC's investigation into possible anti-competitive behavior
on the peering and interconnection front is worth anything, it should be able to rather quickly determine if Level3 (or Verizon) is telling the truth, right?
183 comments .. click to read
|reply to catchingup |
Re: Level 3 full of it.
said by catchingup :
Comcast has done the same thing in the past. Don't connect directly to us and we'll route your traffic through our congested to crap TATA links.
They're like the mafia. This is extortion.
AT&T has congestion issues with quite a few transit providers.
You should do some research around Internet routing. The decision of which "crap" link to use is in 100% control of the origin of the traffic (e.g. content providers). ISPs control their forward path back to the origin (which is not a problem) and Content providers control the forward path to the ISPs (which is the problem)
The reason why content providers
will "route your traffic through our congested to crap TATA links" is part of a Peering Playbook strategy called Traffic Manipulation
and designed to cause problems and get people like yourselves to blame the ISP vs the really orchestrators behind these issues.--
"Too often we... enjoy the comfort of opinion without the discomfort of thought." - John F. Kennedy
Boynton Beach, FL
It appears my reply on the Level 3 blog is going through extended moderation time, even though other replies and replies from the author have been posted. For anyone that cares, here it is...
This back and forth is getting really tiresome, neither this post nor the ones from Verizon are giving anywhere close to a complete picture of how the business relationship has been conducted in the past and present. Offering to pay for Verizon to upgrade their ports is like one of your customers offering to pay for the ports / cross connects to your network without any ongoing usage costs. No, its not an exact comparison, but its close enough for this grand-staging effort being conducted by all three levels (content producer, transit, residential ISP).
As to the suggestions that this is an intentional effort by Verizon to thwart NetFlix, the simple questions can be asked
Why are you selling a service (transit) to NetFlix that you know very well in advance that you cannot fully fulfill? If NetFlix is being targeted by Verizon (or Comcast, AT&T) unfairly, why have they and you elected not to pursue any legal remedies to date but rather instead wage a public public relations campaign?
The latest tactic appears to ask the US government to intervene, which may or may not be a bad thing, but remember most consumers only realize their NetFlix experience sucks and doesnt understand or care about peering in any shape or fashion. Regulation may come, but the government is not known for subtle or light gestures, its quite possible all parties involved will find themselves in a much worse position afterwards. Lets also not forget, the regulation may not stop at residential ISPs, interconnections are far more important than any single ISP.
To summarize: I believe all of the parties involved are simply trying to look out for their own interests and profits, rather than any genuine concern about how the Internet works. None of the parties to date have released any meaningful data publicly to support any of their claims, but rather have released highly summarized data that can not be analyzed or verified by any uninvolved party.
Fort Smith, AR
|reply to WhatNow |
Re: Level 3 full of it.
Which never seems to be what a company claims you get, especially if that company is Verizon. Buy 30 megabits, get 30 only on sites they approve of? Bullshit! They get their fees from customers then pout and not deliver the agreed upon product when they can't double dip the content or transit providers. Essentially choosing winners and losers according to how much their non customer content providers pay for the mafia insurance...ummm I mean direct peering.
Verizon: Buy our transit insurance.
Content: Why do I need it? The ISP I pay provides transit and your customers pay you for last mile.
Verizon: Because if you don't, something very bad might happen to your bits if you know what I mean.
|reply to 78036364 | said by 78036364:
Level 3 is mad they are being asked to pay a fee they don't want to pay. Is willing however to buy thousands of port cards that cost several thousand each.
.......what? Level 3 isn't being asked to pay. Netflix is being asked by Verizon to pay. Plugging in those unused ports on verizon's end to level 3 shown in the diagram would fix the problem in no time at all.
Fort Smith, AR
|reply to 78036364 |
Pay a fee for traffic through Verizon that Verizon's paying customers request? Netflix pays Level 3 to deliver the bits and Level 3 gives Netflix enough capacity. But people pay Verizon to deliver the requested content and Verizon fails to give enough capacity? Why should Level 3 pay for capacity that customers pay Verizon to deliver? Why should transit providers be a charity for big rich Verizon? Sheesh do the shills come out when you mention the words Comcast, Verizon, peering, throttling, or congestion