dslreports logo
site
spacer

spacer
 
   
spc
story category
Level 3 Slams Large ISPs For Breaking The Internet
by Karl Bode 10:34AM Wednesday Mar 19 2014
You might recall that Level 3 had plenty to say a few years ago when they were engaged in a traffic interconnection feud with Comcast. At the time, Comcast insisted that this was a simple, run of the mill peering dispute, while Level 3 loudly proclaimed that Comcast was effectively violating net neutrality by forcing companies to pay in order to connect to the Comcast network. Ultimately a confidential deal was struck and level 3 went mute -- until now.

With Comcast now worrying some folks about their new interconnection agreement with Netflix, transit companies (who are losing that business) are pretty clearly a little nervous. Enter Level 3, who in a blog post this week proclaimed again that large incumbent ISPs (the company goes out of the way to avoid naming names) are violating net neutrality by intentionally letting their peering links saturate in a game of chicken designed to net them more money:
quote:
Some ISPs, however, have refused to augment their networks UNLESS the content providers they connect to agree to pay them to do so. Viewed in the light most favorable to these ISPs, they want content suppliers to pay not only for their own increased costs of supplying more robust Internet content, but also for any increased network costs of the ISPs too. This is not only unreasonable on its face, but it is entirely inconsistent with published reports indicating that returns on invested capital for ISPs are excellent, and are expected to improve even further, driving considerable additional growth in economic profits. More cynically, these ISPs simply view these arbitrary tolls as new sources of revenue for their last mile bottleneck monopolies or as a way to unfairly discriminate against content that competes with the content the ISPs themselves supply.

So what if content providers refuse to pay? Some ISPs agree to augment capacity on reasonable terms. But other ISPs try to strong arm the content providers into paying by playing a game of “chicken” with the Internet. These ISPs break the Internet by refusing to increase the size of their networks unless their tolls are paid. These ISPs are placing a bet that because content providers have no other way to get their content to the ISPs subscribers, that they will cave in and start paying them.
To hear Level 3 tell it, giant ISPs are charging new "troll tolls" to access their subscribers. To hear large ISPs tell it, this is simply a traditional peering and interconnection dispute, with both sides jostling for their own best self interests. Level 3 general counsel Michael Mooney proceeds to state in the blog comments that the FCC should expand their currently-in-process new net neutrality rules to make sure they include peering and interconnection agreements:
quote:
One solution, which we have and will continue to be advocating for in Washington, is for the FCC to address these interconnection types of issues as part of its ongoing net neutrality proceedings, and we are optimistic it will. In the same way as last mile ISPs should not be able to discriminate directly against third party provided content, they should not be able to do the same thing indirectly by forcing content companies and intermediary providers into a no win choice of either paying the ISP arbitrary tolls or suffering through lower bitrates and degraded service quality for streaming video.
Consumers have gotten to enjoy the results of these feuds first hand in the form of strangled video connectivity, despite the high price they pay on average for broadband services. The problem is that while there's lots of analysis on what's happening here, few actually are basing their analysis on real data, since the precise structure of these deals remains obfuscated. As such a good place to start would be transparency, so it's precisely clear what, if any, anti-competitive pricing shenanigans are afoot.

view:
topics flat nest 

ieolus
Support The Clecs

join:2001-06-19
Duluth, GA

1 recommendation

Level 3 is correct

What do customers pay ISPs for if not to maintain and upgrade their damn network???
--
"Speak for yourself "Chadmaster" - lesopp
norm

join:2012-10-18
Pittsburgh, PA
Reviews:
·Verizon FiOS

2 recommendations

Re: Level 3 is correct

said by ieolus:

What do customers pay ISPs for if not to maintain and upgrade their damn network???

Access to their walled garden, obviously. I know I personally pay for access to Verizon's network so I can have access to Verizon's own network - assuming I don't have to touch an ALTER.NET hop. Anything else is additional and I should be thankful. /s

I work from home. My work VPN connects to a server in the DFW area that also uses Verizon. It's not uncommon to have packet loss and high latency from PA to TX. Because the issues START at the ALTER.NET hops, which is still Verizon, mind you, they refuse to ever acknowledge the issue.

nothing00

join:2001-06-10
Centereach, NY

3 recommendations

Re: Level 3 is correct

Ah, I see what's happening. If this continues this is what the cable/telcos are after:

Sign up for the Comcast Network today! Includes great services such as:
- Netflix
- Amazon
- Facebook
- NBC
- Comcastic Email
And enjoy unlimited Internet access with 50GB of data a month!

Sign up for the Verizon Network today! Includes great services such as:
- Redbox Instant
- Amazon
- Google
- Twitter
- Verizon Email
And enjoy unlimited Internet access with 75GB of data a month! All at incomparable FiOS speeds up to 500mbps!

Yeah, it's going to hell.
shmerl

join:2013-10-21

Re: Level 3 is correct

Is FiOS really limited to 75 GB a month?
norm

join:2012-10-18
Pittsburgh, PA
Reviews:
·Verizon FiOS

Re: Level 3 is correct

said by shmerl:

Is FiOS really limited to 75 GB a month?

No, nor is Comcast limited to 50GB a month. They were just future theoretical situations proposed by nothing00.

In some markets, Comcast has a 300GB cap. FiOS doesn't have a hard cap but will disconnect heavy users.

brockalee1

@myvzw.com

Re: Level 3 is correct

They will probably place caps on sites not in your "package" .
TBBroadband

join:2012-10-26
Fremont, OH
Reviews:
·AT&T U-Verse
·MegaPath

2 recommendations

Comcast is also a transit company and can seek Netflix as a client if they wish. Liquid Web peers directly with Comcast so why can't NetFlix? L3 is just worrying they're going to lose more customers over them going directly to the ISPs. Google also peers directly with ISPs.
elefante72

join:2010-12-03
East Amherst, NY

Re: Level 3 is correct

You should read up on all the transit battles in the UK and Germany and how the last-mile providers were doing the exact same thing Comcast, Verizon, et al are doing here.

Here's what is happening:

You can drive on the thruway no problem, you just have to wait at the toll booth and pay with cash (Level 3), OR you can get an EZPASS and drive through the "special" toll booth at 60 MPH and pay less with no toll congestion (that doesn't mean the road isn't congested).

I'm not sure they really pay less (they probably do), but driving through the jump point at 60 MPH really makes a difference, doesn't it...

It can also be handled by an inter-web CDN, but I think Netflix should pay for that not offer it for "free", IMHO.

PapaMidnight

join:2009-01-13
Baltimore, MD

Re: Level 3 is correct

Pass through the EZPASS toll booth and still hit the same congested traffic as everyone else. That might actually be the best example of the ludicrous nature of this.
TBBroadband

join:2012-10-26
Fremont, OH

2 recommendations

Comcast is just NOT an ISP. They are peering/transit provider as well. They've been selling peering and transit to other companies for YEARS. This is just a way for L3 to cry about something as they often do when they lose customers.

snsr

join:2008-05-29
00000
Reviews:
·Optimum Online

1 recommendation

Re: Level 3 is correct

"They've been selling peering"

From »en.wikipedia.org/wiki/Peering

The pure definition of peering is settlement-free, "bill-and-keep," or "sender keeps all," meaning that neither party pays the other in association with the exchange of traffic; instead, each derives and retains revenue from its own customers.

cork1958
Cork
Premium
join:2000-02-26
said by ieolus:

What do customers pay ISPs for if not to maintain and upgrade their damn network???

All I can say is, I give up, what do we pay them for?

I think it's for a way for them to figure out how to raise rates, isn't it?
--
The Firefox alternative.
»www.mozilla.org/projects/seamonkey/

level3fail

@qwest.net
Level3 has been horrible for our company in the last few months. We used(last 5 years) to never have problems with our upstream peers, being L3, XO, TW, etc - our data center is actually a pop for L3, and is maybe 150ft from it to our equipment. We have had to let our data center network team know that we are seeing large amounts of packet loss at times, sometimes lasting minutes, sometimes hours. They finally were able to lower the priority of L3 to us, and we are now running as were before, with no major problems(terminating thousands of VPN connections kinda require a good pipe).

Mr Fel
Premium
join:2008-03-17
Louisville, KY
Reviews:
·AT&T U-Verse
·AT&T DSL Service

1 edit

I agree with Level 3 but...

The timing of the post is highly convenient for Level 3 during a time of higher scrutiny for Comcast while it's working on getting the merger with TWC approved. I wonder what type of response Level 3 is hoping to receive, if there ever is one, from Comcast while it's trying to avoid any negative press much like this. I'm surprised the only ISP they named was just AT&T.
--
Change the scheme, alter the mood! Electrify the boys and girls if you'd be so kind.
TBBroadband

join:2012-10-26
Fremont, OH

Re: I agree with Level 3 but...

If L3 even cared about this they would have complained before when Comcast was selling transit across their network for peering. They just want to bring an uproar in the IT industry over this. Nothing going on here with them.
silbaco
Premium
join:2009-08-03
USA

1 recommendation

Re: I agree with Level 3 but...

A merger of TWC and Comcast would make Comcast an even bigger transit provider. A bigger potential threat to Level3. I can't imagine they want this merger to go through.
TBBroadband

join:2012-10-26
Fremont, OH

Re: I agree with Level 3 but...

Which is why they are crying. This is no different than VZ and MCI/UUNet/WorldCom. They created one of the largest backbones combined.

Simba7
I Void Warranties

join:2003-03-24
Billings, MT

Re: I agree with Level 3 but...

..and we all seen what happened with MCI/UUNet/WorldCom. Could this happen again?

tshirt
Premium,MVM
join:2004-07-11
Snohomish, WA
kudos:4
Reviews:
·Comcast

1 recommendation

Change of position?

seem opposite of the position Level3 took when negoeating a dispute with Cogent
»www.prnewswire.com/news-releases···437.html
.
remember before Cogent, Level3 was Netflix's "discount" CDN and was caught trying to slip CDN traffic in with transit.
Maybe it just depends which end of the stick you are grabbing.
InvalidError

join:2008-02-03
kudos:5

1 recommendation

Re: Change of position?

I bet the prospect of losing major customers like Netflix to direct peering agreements would be scary for any CDN and transit provider: they built tens if not hundreds of Gbps across their network and CDN nodes at a cost of several million dollars to support streaming media and now large chunks of that revenue stream are going away.
openbox9
Premium
join:2004-01-26
Germany
kudos:2

1 recommendation

Re: Change of position?

Perhaps the CDNs providing transit to end users should've been a little more competitive and they would have less to fear. Both Comcast and Netflix are on the record stating the issue wasn't "Comcast allowing the connection to saturate". So, who's left at fault? I'd wager on the CDNs/ISPs that Netflix chose to do business with. Netflix learned this the hard way and decided to fix it by arranging an interconnection agreement with Comcast; and soon several other major ISPs
TBBroadband

join:2012-10-26
Fremont, OH
Don't put all your eggs in one basket as the saying goes, and they did just that.
axus

join:2001-06-18
Washington, DC
Reviews:
·Comcast

The internet still works

I haven't had any sites blocked due to ISP shenanigans. In the case of streaming video, you wait for it to buffer and the packets come through. They probably aren't even violating network neutrality, if the connection is dropping packets randomly instead of based on rules targeting certain endpoints.

The real problem is how much negotiating power Comcast has to make these one-sided deals. Comcast/Time Warner merger decreases competition in the ISP / transit company market, which would be good grounds to block it.

Here's how things would work in an ideal world: Level 3 / Netflix / Whoever wouldn't enter any unfair agreements with Comcast. If Comcast balks, their customers have slow connections with dropped packets to some content. Then those customers switch to the competing local ISP.

The problem arises when there is not good alternative for the customer than Comcast, so they stay there and cancel their Netflix because it's not usable, and switch to someone else who did pay Comcast.
openbox9
Premium
join:2004-01-26
Germany
kudos:2

Re: The internet still works

said by axus:

The real problem is how much negotiating power Comcast has to make these one-sided deals.

You assume the Comcast/Netflix deal is "one-sided". Not sure how you can know that given the details aren't public. Add in that service quality improved and well, seems beneficial all around.

Probitas

@teksavvy.com

similar to rent

I've heard some landlords charge premiums on top of rental space in some malls. Your business pays X, and then you pay some percent of $ over that based on how successful (profit) you are. Sounds like gouging to me, as they already pay rent. Seems highly unfair to attempt to tell some business that happens to use your network because their customers (let's remember the ISP customers are trying to use their paid service to access a third party site) want to use the service it's going to cost them MORE to let their own customers have access to another site. Sounds like the mob to me. Quite a racket.

tshirt
Premium,MVM
join:2004-07-11
Snohomish, WA
kudos:4

Re: similar to rent

That's how big malls have always worked rent + tenant improvements + x%.
most franchise s for stores and other business are similar.

David
I start new work on
Premium,VIP
join:2002-05-30
Granite City, IL
kudos:101
Reviews:
·DIRECTV
·AT&T Midwest
·magicjack.com
·Google Voice

3 edits

1 recommendation

I guess it sucks for level 3.

After all if I can avoid paying the middleman and his prices and can contact the ISP's directly, exactly what do I need level 3/middleman for?

I wonder if they will become the floppy disks of the industry as USB drives (direct connection) takes hold.

why pay the middle man's cut when I can just pay the provider directly? Doesn't sams club /wal-mart kind of do that already?

--
02/24/14- My hours recently changed. I work 11:30 A.M. to 10 P.M. central time. I am not in the office on friday, saturday, or sunday. Thanks-David
InvalidError

join:2008-02-03
kudos:5

3 recommendations

Re: I guess level 3 does have a point...

Not every company and over-the-top service provider is large enough to afford the infrastructure necessary to peer directly with ISPs and there are also limits to how small one of those can be before putting them on-net is worth the trouble for ISPs. For all those not-large-enough cases, some sort of middleman to aggregate traffic is necessary.

There will always be a need for middleman networks. What changes is that now traditional transit providers have to get used to the idea that some of their larger clients may end up bypassing them when they become large enough for ISPs to be interested in direct peering with.

In other words, L3 wants to protect their traditional business model against new trends that may significantly alter the balance of power and the way they have to do business. Sounds awfully similar to what the RIAA/MPAA and related entities have been doing for the past 10-15 years. L3 is losing their ability to impose peering terms and they do not like it, just like the *AAs do not like their slipping grips on copyright enforcement, retail markups and independent artists/producers who refuse to become members.
sonicmerlin

join:2009-05-24
Cleveland, OH
kudos:1

Re: I guess level 3 does have a point...

You actually *believe* him? How do you manage to miss every single conversation on this issue only to become "clear" based on the rantings of an irritating corporatist?
guppy_fish
Premium
join:2003-12-09
Lakeland, FL
kudos:1
Reviews:
·Verizon FiOS

2 recommendations

Caught with hand in cookie jar

Cognet, L3, XO and a dozen others thought they had it rigged, get Teir one Status, the sign up Netflix, underbid to get the $$ and flood the peering points

Gig is up dudes, and the need for these networks will go back to doing what they did before, which is they get paid for the tiny part of the network they actually are

battleop

join:2005-09-28
00000

1 recommendation

This is about Level3 protecting their revenue stream.

They are pissed that Netflix just cut big chunk of their revenue by bypassing them and going directly to Comcast.

Let's say Netflix is paying Level3 for 100Gb of traffic to the internet and 50Gb of that traffic was headed to Comcast's network. IF they were paying $1000/GB (just an exampe) and 40Gb of that traffic was headed to Comcast then the Comcast/Netflix peering arrangement just took $40k/mo out of Level3's pocket.

This is what they are whining about.
--
I do not, have not, and will not work for AT&T/Comcast/Verizon/Charter or similar sized company.
XeenFalcon

join:2006-09-13
Salt Lake City, UT

3 recommendations

The Internet in the US is becoming a closed network

When I ran my dial-up ISP back in the late 90's the business model was to pay for a backbone connection or multiple back bone connections with multiple providers like Level 3. The priority was to provide the possible routing to the customers. Today the large ISP's like Comcast and Verizon are essentially back bone providers themselves. Now priorities have changed from giving the customers the best routing possible to the internet to charging the content providers for the best routing possible to the customer. So what if the peering between Comcast/VZ and Level 3/Cogent/XO became lop sided in one direction it’s because we the paying subscriber wanted that data (aka Netflix) provided by them. If a peering point becomes saturated in one direction because the ISP's paying customers are wanting that data the ISP should be obligated to free up that congestion not charge the backbone provider more for access to their customers. It’s a matter of customer service to apply the best routing possible for your customers. It turns that some users were getting better throughput to netflix by using a VPN that put them straight on to cogent or level 3 bypassing the choke point that says it all. If a customer has to use a VPN to essentially change their own routing to get to the content they desire, this blows me away. As this keeps happening backbone providers like Level 3, XO and Cogent will fade away and all will be left with is Comcast and VZ backbone controlling everything and we will be left with is a closed network.

tshirt
Premium,MVM
join:2004-07-11
Snohomish, WA
kudos:4
Reviews:
·Comcast

Re: The Internet in the US is becoming a closed network

said by XeenFalcon:

As this keeps happening backbone providers like Level 3, XO and Cogent will fade away

That isn't going to happen.
Traffic is multiplying at an increasingly rapid pace, and there is plenty for everyone.
In fact Level3 and Cogent still carry Netflix to the caches just outside Comcast network entry points in the new agreements.
McBane

join:2008-08-22
Plano, TX
It's sad they manipulated congress to kill off the mom and pop ISPs of the 90s and now they're doing the same thing to kill off backbone provider competition. The new age of monopolies is upon us. We're living in it now just like it was at the turn of the century. They're slowly taking control of everything.
MrBungle87

join:2013-01-18
Durham, NC
Reviews:
·Frontier Communi..
·Time Warner Cable

1 recommendation

None of it makes any difference

Until ISPs are compelled (forced) to prove that they're NOT intentionally letting their peering points become oversaturated, speculation that Level3 is the bad guy here and Comcast is the good guy is stupid. Give me proof, show me some numbers to back up these ridiculous claims. When people on this site go to bat for Comcast with NO SOLID EVIDENCE of anything they've done (back room deals and vague PR statements about how "robust" their network is don't count), it makes me wonder how many Comcast employees and shareholders we really have here.

•••••

telcodad

join:2011-09-16
Lincroft, NJ
kudos:5

Cogent now offering to pay for peering upgrades

Cogent is now offering to pay the costs for major ISPs to upgrade their connections to them:

Cogent CEO: We’ll Pay For Peering Upgrades
CEO Offers To ‘Resolve The Impasse’ Between Internet Transit Provider & Major ISPs

By Jeff Baumgartner, Multichannel News - March 21, 2014
»www.multichannel.com/distributio···s/149015

The Cogent press release: »www.prnewswire.com/news-releases···731.html