Level3: Verizon Intentionally Causing Netflix Congestion

If the damn greedy customers would just STOP subscribing to netflix, this wouldn't be a problem. Sure they are paying for INTERNET access, but watching TV isn't really internet access, it's streaming video. Netflix should just roll over and pay however much verizon want's to fix the problem, or just cancel the subscriptions of anyone who uses verizon. That's the only real long term fix. If people would just realize, it's a series of tubes, and video is like trying to fit a dump truck down those tubes. Sometimes, it takes the internets a long time to send me an e-mail because all those greedy customers are filling up the trucks. Verizon shouldn't have to do anything, it's those pesky customers that are causing the problem.

Almost failed to detect the sarcasm for a minute.

I did fail :S

yea, that commentitis is a bit of a pain at times.

but here's the sad part, the guys that run the show actually think like that.

The scary part, is some people didn't understand the sarcasm. I hope the 'series of tubes' part gave it away, but, again some people truly believe this to be true. For the record, it's Verizon's fault. Period. Verizon "refuses to upgrade their side of transit-operator facing connection links". Guess what VERIZON, if all your customers are REQUESTING information from a particular transit link, it's YOUR responsibility to upgrade it. Don't ever, ever think that Level-3 is REFUSING to upgrade their links, they are MORE THAN HAPPY to upgrade the link. Let's agree to this : THEY don't have CONGESTION on THEIR network, YOU don't have congestion on your network. Your CUSTOMERS WANT DATA from THEIR NETWORK, but YOU don't want to upgrade your link to them. Guess what, the CUSTOMER doesn't need to think about this, YOU, VERIZON, YOU need to fix the problem, not Level-3, not netflix, YOU. It's YOUR CUSTOMERS who want the data, and they pay you a LOT of money to provide the service, which YOU are INTENTIONALLY letting get congested so YOU can CHARGE SOMEONE ELSE (i.e. double dip). That's greedy, and wrong, and you should be punished.

My hat is in the ring for a class action suit. I'm sick of Verizon's greasy unethical business model.

You must work for Verizon. Then what is the higher speed tiers people PAY for really for? Sending e-mails 0.5 seconds faster for a fortune more per month? In that case I would take the 2 megabit tier and deny greedy Verizon and company the extra revenue for a higher tier of service that would be of no meaningful value.

We all know this is going on, glad these companies are publicly putting the heat on these guys to continue to bring attention to this...

I have a couple of Twinax sitting around. Shall I send them to Verizon...

Anyone with any real intelligence should be able to see what's going on, however just like the Governments of the world, the corporation's such as Verizon rely on the incompetent brain dead to believe their story and happily pay the bills while not getting the product which is already paid for on two fronts. I absolutely hate government and corporations who use the combined power of stupid people (or should I say stupid sheep) to push their own agenda.

So, yeah let me rephrase the title of this post, "Blame the stupid customers" for not demanding that the truth come out.

Agreed!

Yea! Those stupid greedy Consumers who actually want to use their broadband! What idiots! America's mega-corps can't afford for consumers to ACTUALLY use services they are paying for, that would cut profits down. We should just block these services and websites like Netflix and tell the consumers to STFU and pay more.

Sheesh, how stupid. Damn customers. Poor Verizon!

The infrastructure is and has been paid for itself many times over.. Verizon is playing games which will result in the loss of their franchise incumbency in the fios footprint. Now, is Netflix willing to pony up the money to become a last mile ISP?

You sure post a lot of crazy crap.

If you knew the issues, this is not crazy. Look at the BIG picture. 1. Verizon offloads unprofitable geographies. 2. Verizon attemps to push customers onto expensive wirless from wireline. 3. Verizon works to Comcast to be anti-competitive. 4. Verizon raises prices (or effectively eliminating discounts) 5. Verizon lets peering points of entry to their network saturate due to customer demand on bandwidth THEY ARE PAYING FOR AT THE CURRENT MARKET RATES. 6. Verizon lobbys and sues to get net neutrality laws passed to gain financial advantage over the peering issue to bolster profitability. 7. Verizon has a lobbying and corporate history of keeping competitors out of its markets (the whole let's not build or let anybody else enter the market to build decades in the making).

Crazy? Telcos have alot to answer for & their Cable co-conspirators!

I think the crazy comment was in reference to your suggestion that Verizon is somehow going to lose its franchises.

It very much was.

Try reading the quoted text. This just proves the crazy.

Doubtful, but for the sake of argument, let me ask: you're not suggesting that just because the infrastructure is "paid for" that services over the infrastructure be provided for free? Recovery of the invested capital is only part of the price you pay for service. There's also the expense of providing the service (right-of-way fees, property taxes, salaries of technicians, cost of replacement parts and repairs to equipment, building rents, cost of electricity, etc.)

So here's a scenario, Pennsylvania worked out a deal with Verizon, the deal was to wire the entire state with fiber(deal was made in 1994) in exchange of over $2billion worth of tax breaks, the tax breaks were to fund the installation of the systems across the state and would open them up to the entire state as a market.

Verizon got their tax breaks(is likely still getting them), Pennsylvania still isn't wired, they lied and got away with it.

NYC made the same deal in 2008 and the said the entire city would be wired by 2014, Karl Bode(you know, the guy in charge) warned about the deal, and guess what he was right, they lied and got away with it a second time.

Verizon has a history of pulling bullshit, and this is no different, VPN's magically fix the issues seen with netflix, redbox instant works perfectly fine for Verizon customers that right there REEKS of traffic shaping to artificially neuter a competing service and last i checked it is entirely against the whole concept of a free market.

Imagine the freaking out if Netflix worked out a deal with companies that use open connect to route redbox traffic onto saturated transits and degrade it's performance, Verizon would be losing their shit.

Not the way the internet works. Netflix chooses its transit providers but Verizon chooses, in part, the number of ports they have with X or Y transit provider that traffic reaches them on. I do believe Netflix is playing politics with its customers by constantly changing routes with their heavy traffic for free or low cost peering. But I do feel Verizon is just as wrong pretty much saying customers have to live with it though they pay for certain speeds. That is why last miles should not also be transit providers. Sort of conflict of interest where problems in their transit business affect their residential internet business customers. Remember Comcast customers had the same issue. Yes, Comcast is a cable company but they don't offer Redbox, etc. Many other providers without issues are cable companies too. They do, however, offer transit unlike some cable companies without issue which makes them more like a Verizon.

Netflix can't alter where the traffic enters the Verizon network, the Verizon internal routing dictates where the traffic exits and enters their network.

The data request originates on Verizons network, leaves the network through a pre-selected node based on their routing table and on to L3's network, that request is received by Netflix and a response follows the same route back to the customer, the congestion is when Netflix traffic transfers back onto Verizons network following the same path that the request packet.

by altering the route off Verizons network by use of a VPN it completely fixes the issue, yet if the client directly sends a Netflix request through Verizons default routing it ends up going through a congested node.

Netflix may be playing politics, but Verizon has a personal interest for the service to run poorly on their network because they are in partnership with a competing service, Verizon and Netflix signed a deal in April of this year to resolve issues, and Verizon hasn't held up their part of the bargain, just like they didn't in NYC and PA.

Netflix mainly just chooses transit providers and can set up a routing table on their internal infrastructure to choose to send it over 1 or the other they directly peer with. Netflix could conceivably alter this table to send traffic over X provider with Y capacity into the Verizon network. Sure, Verizon can choose who to send requests outbound among their providers, but if no direct route is advertised to that provider, then traffic has to be handed off to another network. This is known as traffic manipulation. So if the Netflix data center has direct connection to Cogent and Level3, they are capable of determining which ISP handles routing the traffic to Verizon. They could distribute it evenly, or in a predatory move, distribute it all on 1 path. What we have is a bunch of finger pointing and not much facts. But is it not just as possible that Netflix is using customers as pawns by shaming certain ISPs and selling Open Connect? Aka virtually free peering that would give them a distinct competitive advantage both in transit costs and performance over other streaming hosts? Personally I don't trust Verizon or Reed Hastings. Just because everyone hates Verizon does not make them 100% guilty. It might be suspect that Verizon and other troublesome ISPs have video services they may want to protect(but just as many not troublesome ISPs do). But it is just as suspect that the problem only surfaces after Netflix drops traditional CDNs at about the time this performance issue happened, then goes to the press in a shame game, all the while essentially touting free direct peering(OpenConnect) as a solution to a problem that never before existed until they took over streaming. Yes, Netflix can choose which IP the stream originates from(read up on how CDNs work) and conceivably, the IP chosen(by DNS propagation from the root servers) can be set on the poor network path on the congested link. Then use public backlash to make the ISP cave to a deal nobody else can get.

Here is the thing though, Netflix signed a deal with Verizon already to solve congestion issues, and it may be tweaking things on their end, but it's already been proven that other ISP's have been the ones manipulating the situation(see how fast Comcast got after they were paid), and after they got Netflix to sign a deal the flood gates were opened and people were happy, but this doesn't seem to be the case with Verizon.

Definitely not a Verizon fan here, but Verizon does everything at a glacial pace since the beginning of time. Comcast likely gave Netflix a sweet deal to shut the Netflix people up during merger time. A very small price for them to pay if they get TWC. I agree that Verizon would do well to make customers happy and rush along on fulfilling their signed deal with Netflix. But definitely not giving Netflix free peering(see OpenConnect) like they wanted before. It is, after all just as much Netflix's responsibility to responsibly route traffic from their end as it is Verizon's to ensure a decent customer experience.

Verizon isn't very high on the list of customer experiences, unless it's negative ones.

I agree. I have stated numerous times at different opportunities that I would never deal with Verizon ever again, if it can, in any way, be helped. They are the bottom of my list plain and simple. I still try and keep my insights and comments fair and balanced. As I see it, while Verizon may share a chunk of blame and be acting shady as usual, things don't look that much better for Netflix's end when you put aside your like or dislike of the 2 companies and consider the possibilities and likelihoods. After all, that is all we got since both companies are hush hush as to hard technical details but loose lipped with finger pointing.

I've heard this urban myth more than once. Do you have a cite for these alleged "tax breaks"?

»Picture Perfect Deal
it was reported and covered right on this site.

Thanx for the link. As I suspected it relies on a Bruce Kushnick report which I've debunked before.

I don't suppose you actually read the 45 page report that this article links to? If you did, you'd see the alleged "tax breaks" are nothing more than accelerated depreciation which is a standard feature of the FEDERAL tax code, not a special "gift" from the state of Pennsylvania as Bruce would have you believe; states don't have the authority to authorize such deductions, unless it is a state deduction against state income taxes (I don't know if PA has a state income tax, but nowhere in his 45 pages does Bruce say whether this is for state or federal purposes. If it was for state, I'm sure he would have said so, as that would support his case. But I suspect this is actually a federal deduction and the reason he hides that little fact is that it does not support his case.

And if you know anything about accounting, you'd know that SFAS (Statement of Financial Accounting standards) No. 71 is an accounting standard that publicly traded companies like VZ must follow by law. Bruce complains that VZ derived some benefit from following SFAS 71; but, if the law requires them to follow it, then they must follow it regardless.

But adherence to SFAS 71 has nothing to do with VZ's network plans in PA; it is an accounting standard that all companies must follow if it applies to them, but of course Bruce never mentions that, either.

As usual Bruce plays fast and loose with his facts, and people too eager to believe his B.S. (like you and Karl Bode) don't look beyond the numbers to see if they actually mean anything.

Bottom line: Pennsylvania did NOT give Bell of PA "over $2B worth of tax breaks" in exchange for a statewide fiber network. That came from following the Federal tax code and a generally accepted accounting principle (GAAP)

Thanx again for the link.

But really, you pick a Karl Bode article from eleven years ago and act like I should've been well aware of it? I do try to read most articles here but you'll have to forgive me if that particular one didn't come immediately to mind.

Still, I asked for a link and you gave it, so thanx.

and YOU know the price and cost of building and maintaining said infrastructure?

Level 3 is mad they are being asked to pay a fee they don't want to pay. Is willing however to buy thousands of port cards that cost several thousand each.

I'm on Verizon DSL, when I use my connection which shows speeds of 14/1 on all speed tests with very low latency (32 ms) with youtube or Netflix I see constant buffering with youtube and only get standard def on netflix. However, when I fire up my trusty Private Internet Access VPN and aim it their East Coast node my youtube buffering problems disappear and my Netflix connection goes to HD. Verizon like most corporations is in business to give the least possible service for the highest price.

Somebody sneak in there with a few 2ft cat6 cables already.

found this

"Do you have LOADING ISSUES? Is VERIZON your INTERNET SERVICE PROVIDER? #TheDirtFarmer team has found a connection among the two and it is our firm belief that (some) Verizon customers (specifically those in Maryland, Virginia and Pennsylvan...ia) are unable to load their farms because of it. Due to Verizon's recent war of words with Netflix over streaming speeds and fees and one company blaming the other prompting an FCC investigation, it is our belief that in some regions, Verizon is throttling (limiting) streaming/flash speeds to a point where players are unable to load their farms. " this is about farmville but the listed fixes say to not use VZ dns.
using another dns might fix netflix issues too. 4.2.2.1, 4.2.2.2 are good fast free servers
-
looks like VZ wants a sweet deal like comcast got.

It appears my reply on the Level 3 blog is going through extended moderation time, even though other replies and replies from the author have been posted. For anyone that cares, here it is...

--

This back and forth is getting really tiresome, neither this post nor the ones from Verizon are giving anywhere close to a complete picture of how the business relationship has been conducted in the past and present. Offering to pay for Verizon to upgrade their ports is like one of your customers offering to pay for the ports / cross connects to your network without any ongoing usage costs. No, its not an exact comparison, but it’s close enough for this grand-staging effort being conducted by all three levels (content producer, transit, residential ISP).

As to the “suggestions” that this is an intentional effort by Verizon to thwart NetFlix, the simple questions can be asked… Why are you selling a service (transit) to NetFlix that you know very well in advance that you cannot fully fulfill? If NetFlix is being targeted by Verizon (or Comcast, AT&T) unfairly, why have they and you elected not to pursue any legal remedies to date but rather instead wage a public “public relations” campaign?

The latest tactic appears to ask the US government to intervene, which may or may not be a bad thing, but remember most consumers only realize their NetFlix experience sucks and doesn’t understand or care about peering in any shape or fashion. Regulation may come, but the government is not known for subtle or light gestures, its quite possible all parties involved will find themselves in a much worse position afterwards. Let’s also not forget, the regulation may not stop at residential ISPs, interconnections are far more important than any single ISP.

To summarize: I believe all of the parties involved are simply trying to look out for their own interests and profits, rather than any genuine concern about how the Internet works. None of the parties to date have released any meaningful data publicly to support any of their claims, but rather have released highly summarized data that can not be analyzed or verified by any uninvolved party.

Government steps in, determines internet access and connectivity is necessary for the continued growth and prosperity of the US, gets rid of all middle men and properly connects the completely insane # of providers to one GIANT network, and says there you go, no more peering issues. And instead of competition you have one provider run by the US government, who is tired of watching business get in the way of business. All fixed now.

About Verizon not being able to afford stuff.

This is good that such a big, well respected company has come out in support of getting the truth about Verizon out there.

I imagine that Netflix can easily saturate one or two tier1 links as described above. IPSs reroutes "normal" traffic somewhere else and don't care much about what happens for that particular link.

I understand that Netflix would greatly prefer IPSs to play their game and install their caching server everywhere.

But couldn't Netflix make the effort to over diversify their peering facing these reluctant ISPs ? And to smooth things out, develop a load balancing scheme in function of the perceived congestion between their servers and clients (real time analysis)?

The solution I outlined might be the business equivalent of letting yourself distracted by bully tactics. Incumbents are practically monopoles after all.

Direct peering makes much more sense of course unless the arrangement offered are truly unfair to content providers.

Are other tier1 providers scared of doing business with Netflix in the current situation(i.e. at a fair price)?

Otherwise over peering seems obvious (to me lol, am not a Network Engineer though!)

The other side of the coin could be that by playing the victim, Netflix hopes to finally push that ultimate edge in their market: their OWN cache servers everywhere, instead of simply fair direct peering. Who will be able compete with that.

The more I think about it, the more I sense that no one, including tier1 providers want to tell "the whole truth". And this is the reason this greek tragedy has lasted for so long without catharsis.

I would usually side with the fact that any fight against the ISP oligopoly is a good fight. But what if in this case it's mostly going to help create monopoly with a very strong technical edge. Maybe we should expect better answers from the parties involved rather than taking side this easily. Me guilty first until the over peering solution came to me.

If anyone was paying attention that's all Comcast (and likely Verizon) have been asking for ..."Deliver to loading dock A and unload"

McWireless can't be bothered by those pesky FiOS users. He long ago stopped upgrading users to FiOS. Why would he make any upgrades to help current FiOS users watch Netflix?

Would this not affect all Verizon customers?

No one is getting 50 or 75 down on LTE.

Gee, if only there were this little thing called competition...then maybe ISPs would strive to have the best network. That's why i think that all internet access (cell, fiber, etc.) should be taxed and the taxes should be used to pay for a national fiber network that connects every home and business. The ISPs could then resell that fiber access to end users.

From your lips to G-d's ear!

Riight.

My taxes already go to the government which has built a redundant fiber service, that has run past our front door for the past decade, but only the elites are permitted access.

Meanwhile, the duopoly continues to invest in their plant, and offers greater speeds for less money each successive year, while I am assaulted with new and higher taxes.

Denial is a way of life for ISPs.

I used to use a dialup ISP called Localnet. When I first signed up, they were great. I always got 6K/s on a single connection, which is about the maximum dialup can do. Then literally overnight my speed went down to 2-3K/s. They claimed this was normal and that line noise was to blame. I had my line tested, no noise. They blamed general internet congestion. I had an IT expert test this by connecting to the same local phone number I used. He found massive packet loss happening on their networks. They ignored this as well and continued to insist that there was nothing wrong at their end despite the fact that they admitted they had not actually tested the lines I was connecting to.

I made many calls to their tech support and just once was I able to get an engineer on the line who said he would look into this and call me back with a definite answer. He never called and when I called him, he was unavailable, but someone else supposedly spoke to him and then gave me the same answer I'd gotten numerous times before; Nothing wrong here, it must be on your end.

title 2 government tax breaks and title 1 prices causes verizon to realize lets just shake hands with comcast and not really upgrade. Who else could come in now?

We all knew this was the case.

They want you do buy pay tv even though it's 10 times or more the price of Netflix. Sickos.

David Young is the Verizon exec, not the Level 3 exec. (That's Mark Taylor).

I take the congestion as intentional.

I had no problems on a 3.5mbs DSL through Atlantic Nexus. Switched to comcast 25/5 and now youtube and Netflix get buffering issues.

Changed to non-comcast DNS and all of a sudden, everything is fine. Netflix, youtube, Hulu will give a the closet sever via their DNS. Looks like comcast is just giving out one DNS so all their data gets throttled through one connection.

Isn't it odd that Level3 traffic to Verizon is now experiencing 9.09% packet loss. This according to internetpulse.net it was even worse earlier, up to 15%, I believe is what I saw.

From "The Art of Peering: The Peering Playbook"

quote:

---

2. Threatening litigation and government intervention often shuts down the conversation between Peering Coordinators .
3. Public Name Calling and badgering in public forums proves to bring personality conflicts into play and often results in doors being closed that should be open.

---

Whining to the government? Public shaming?
This guy just described Netflix to a "T".