dslreports logo
site
spacer

spacer
 
   
spc
story category
Sandvine: Netflix, Google ISP Rankings 'Conflicting,' Inaccurate
by Karl Bode 11:43AM Friday Jun 06 2014
Network gear manufacturer Sandvine apparently isn't a big fan of both Netflix's and YouTube's new ISP streaming performance rankings, insisting that the data collected by both is unreliable and conflicting. In a blog post, Sandvine points out that ISPs deemed "HD Verified" by Google's new ISP ranking (discussed by us here) are sometimes categorized as under-performers in Netflix's rankings, and vice-versa:
quote:
Click for full size
Google is essentially saying Rogers’ customers who use YouTube are capable of regularly experiencing HD streams, while Netflix is saying Rogers’ subscribers are experiencing the worst quality of Netflix streaming in the country. At the same time Netflix is saying Bell Canada’s DSL subscribers are capable of experiencing HD streams when using Netflix, but Google is saying Bell non-Fibe (DSL) are not YouTube HD-verified and “should be able to watch YouTube videos in Standard Definition (at least 360p) with moderate load times."
I've already pointed out numerous times that Netflix's streaming rankings are made less valuable by the fact that Netflix CDN partners do better in the rankings than those who refuse to participate in the free service. Similarly I've noted Google and YouTube's new ISP rankings need some work as well, as navigation is clunky and some ISPs are listed as HD verified even in markets they don't even provide service.

With that said, it's worth noting that the lion's share of Sandvine's money comes courtesy of the nation's largest ISPs, and as such Sandvine has traditionally been reluctant to admit fault on the ISP end of the equation. The company recently proclaimed that YouTube streaming issues that have plagued most broadband users are entirely YouTube's fault, even if data tends to suggest there's plenty of blame to go around.

It also seems obvious that ranking YouTube and Netflix streaming performance would be "conflicting," given those rankings are testing entirely different content services, taking entirely different routes to the end user. Routes can be even more different now that companies like Netflix are (begrudgingly) striking direct interconnection deals with ISPs like Comcast and Verizon.

While both Google and Netflix efforts need work (and will hopefully improve), they're at least attempting to shed a little more light on the buffering problems that have plagued the industry the last few years. That's more than can be said for most ISPs, which have long hidden performance data (unless it's to cherry pick it to justify policies like usage caps, deregulation, etc.). If ISPs want to contest YouTube and Netflix's findings, they could simply release raw performance data of their own.

At a time when transit companies are claiming these slowdowns are intentional to make an extra buck, it seems like every shred of additional performance data is helpful if we're interested in the truth.

view:
topics flat nest 
norm

join:2012-10-18
Pittsburgh, PA
kudos:1

1 recommendation

Verizon

Verizon is HD verified with Google but performs poorly with Netflix. Verizon also uses Google Global Cache...
existenz

join:2014-02-12
kudos:2

Re: Verizon

These reports really don't have much substance given that many ISP's offer service at 5Mbps or less, including Google Fiber.
rebus9

join:2002-03-26
Tampa Bay

1 recommendation

And it doesn't address peering-specific deficiencies. My FIOS connection runs great-- except for certain things like Netflix, and that's a direct result of VZ refusing to upgrade their peering links with Cogent.
norm

join:2012-10-18
Pittsburgh, PA
kudos:1
Reviews:
·Verizon FiOS

Re: Verizon

said by rebus9:

And it doesn't address peering-specific deficiencies. My FIOS connection runs great-- except for certain things like Netflix, and that's a direct result of VZ refusing to upgrade their peering links with Cogent.

And that's a direct result of VZ refusing to update their peering links with Level 3, NTT, Tata, Cogent, TeliaSonera, and virtually everybody else
openbox9
Premium
join:2004-01-26
Germany
kudos:2

1 recommendation

Re: Verizon

Peering is a two-, or more, party arrangement
BiggA

join:2005-11-23
EARTH
Exactly.

fuziwuzi
Not born yesterday
Premium
join:2005-07-01
Atlanta, GA

3 recommendations

Really?

Sandvine, the obvious shill for the ISPs that pay it.
--
Teabaggers: Destroying America is Priority #1
dfxmatt

join:2007-08-21
Evanston, IL

Re: Really?

It's amazing how fast people forget that sandvine worked effectively hands on with comcast to screw people over with the reset flags issue from 2007.

They have always been explicitly behind ISP's and government spying while remaining explicitly against companies like netflix.

batman

@50.182.54.x
said by fuziwuzi:

Sandvine, the obvious shill for the ISPs that pay it.

Sandvine is for hire BY ANYONE who will pay their fees. They are not a captive subsidiary of ISPs.
dfxmatt

join:2007-08-21
Evanston, IL

Re: Really?

Who do you think is paying them most commonly? security agencies and ISPs. Almost nobody else even needs deep packet inspection; it simply adds more data to something that can be accomplished without deep packet inspection in the first place, through proper networking gear such as riverbeds.

It's also really, really expensive.

I have sat in on conversations with major providers of DPI gear (dell, HP). It's prohibitively expensive and is actually a glorified winpcap capture.

PapaMidnight

join:2009-01-13
Baltimore, MD

1 edit

It makes sense

Considering that either company may have partnered with individual ISPs to directly peer with them, and/or that subscribers to any of the ISPs on that list could take any number of routes to reach either YouTube or Netflix, it indeed makes sense that the data each has could be conflicting.

While someone's packets on Verizon could theorectically take congested routes to get to Netflix, they may have great routing to YouTube.

Likewise, while someone's packets on Comcast could take congested routes to get to YouTube, they'll likely have a great experience with Netflix (as of current).

All I see is SandVine starting a whole lot of something over a whole lot of nothing.

Edit: Syntax error corrected. "someone" corrected to "someone's" in second paragraph.

WildBill

@95.130.9.x

Re: It makes sense

said by PapaMidnight:

Considering that either company may have partnered with individual ISPs to directly peer with them, and/or that subscribers to any of the ISPs on that list could take any number of routes to reach either YouTube or Netflix, it indeed makes sense that the data each has could be conflicting.

I agree with most of your points. It's not even that easy, however, as most of Comcast's peering paths are congested.

nothing00

join:2001-06-10
Centereach, NY

4 recommendations

Sandvine Confirms ISPs Not Treating Traffic Equally

So, how is that for an alternate headline?

Seems like Sandvine stuck their foot in it this time.

Killa200
Premium
join:2005-12-02
Southeast TN

1 recommendation

Re: Sandvine Confirms ISPs Not Treating Traffic Equally

Or maybe even "YouTube and Netflix not treating pay and free customers equally by blending peering traffic to optimize routes"

Not quite as good of a ring to it, but just as true.

nothing00

join:2001-06-10
Centereach, NY

Re: Sandvine Confirms ISPs Not Treating Traffic Equally

Well, let's go through this. The free vs paid is a red herring. One ISP sees better results on "free" and the other on "paid". Last on this topic, each service is a business and making money whether the end user pays directly or not.

The suggestion that content providers are responsible for coordinating access to optimize throughput and maintaining an ISP's network is pretty bold.

So no, that headline wouldn't be just as true.

Would you have preferred headlines such as:
Sandvine Accuses its Customers of Gross Network Neutrality Violations
edit: can't actually use that last one

Killa200
Premium
join:2005-12-02
Southeast TN
Reviews:
·Charter

2 edits

1 recommendation

Re: Sandvine Confirms ISPs Not Treating Traffic Equally

With as large a content company as Netflix is, and with their ability to gather enough metrics to see exactly where congestion is happening (or that is at least something they say they can do with all this new fangled isp blaming) then I would say they certainly could lend a hand coordinating their services, aside from their CDN and open peering options, which I don't consider to be anything short of band-aid or a scapegoat on their obligation to buy the transit needed to operate their business.

Any kind of peer blending past what they do now certainly couldn't hurt. Obviosly they are only using a couple of teir1 peers if their traffic outbound is enough to overload peers at the largest of providers that never had issues before. Id say they have balancing issues of their own that could be fix not only with these "evil isp peering arrangements" but with some plain ole "stop picking one cheap outbound provider to put all your crap on". Yeah that costs money, but maybe they can raise the cost of netflix to fix their issues,

TL:DR : When your content company is so big that you are a significantly measurable amount of the entire internet's daily traffic, you have an obligation to make sure your product gets to your viewers the best way possible, with something more effort wise than blaming everyone else for the problems you caused.

nothing00

join:2001-06-10
Centereach, NY

Re: Sandvine Confirms ISPs Not Treating Traffic Equally

Ah, see, from your context it sounded like you were suggesting that Netflix and YouTube coordinate amongst themselves to maximize ISP bandwidth, thus working around and attempting to fix the ISP's problem for them.

Sure, coordinating a large amount of bandwidth with an ISP makes sense. However residential ISPs have recently confused the word "coordinate" with "pay".

But it's not Netflix that created the problem, residential ISPs such as Verizon and Comcast created the problem by not upgrading their infrastructure for their paying customers. Netflix's ISPs have no problem delivering the bits and are getting paid far far less then Comcast and Verizon.

Killa200
Premium
join:2005-12-02
Southeast TN
Reviews:
·Charter

1 edit

1 recommendation

Re: Sandvine Confirms ISPs Not Treating Traffic Equally

Oh no, I fully believe Netfix / Youtube / Anyone else should be paying any ISP to interconnect with them if said content companies wants to use an ISP as a peer.

The absolute is content companies have to pay for their transit, as they have nothing to "give" to balance out into a peering agreement. Any ISP could care less unless their small fries if any content companies traffic makes it into their network. Assuming "better access to x content company's content" doesn't count as something to give, thats something to be expected if content companies want to stay in business.

The thing people of you fail to realize is agreements like this are in no way nor never can be classified as "a failure of net neutrality". Net Neutrality has squat to do with backbone transit. You can't keep looking at Netflix, Youtube, and the huge providers of content like you do DSLReports, or Reddit, or whatever small fries website in comparison to big content company's traffic.

As an example, Netflix puts outbound over 1,000,000 mbit/s of traffic ( Link to back that up). That is more than most Teir 2 transit companies touch bi-directionally per second to all of their customers combined. That is a single content company, creating more traffic than a company that provides backbone access for several thousand small companies. Or as another example, thats 10x more than the amount of traffic handled by popular Canadian ISP TekSavvy (Link again)

Its easy for a content company this size to overwhelm a single peering point of any sized ISP. And with a content provider playing pick and choose instead of spreading load, all this traffic never sees half the other peer points of an ISP.

Should the ISP upgrade the peer? Hell no! They have tens of peering points in that location already that handle just fine aside from this one, hence why you see all these reports of EVERY OTHER SERVICE BESIDES NETFLIX working fine simultaneously to Netflix bogging down.

If Netflix were to say "hey we need to diversify and spread all this traffic into several Teir1 peering points, with people such as Level3, Cogent, XO, Global Crossing, HE, etc" at their peering points, then you'd probably never see this happen with any ISP, at all. Netflix's traffic would travel over multiple peering points as it hit the internet, and would have several routes of priority into any ISP with their already established multiple links. That takes money though, and from the ISP's eyes they already have multiple peers to handle this for every other company not sending out traffic in the order of magnitude Netflix does.

You don't see this in smaller ISP's networks because they only have one or two peering points, or may not even be multi homed at all. When they saturate, they saturate for everything. This is why smaller ISPs that can actually attach to a Netflix POP tend to do this "open connect" agreement. They get to not have to upgrade their 1 or 2 backbone connections to the world if they can hook a large line for free into a Netflix POP. Generally this ends up not being free either, as the ISP has to build to a pop, or upgrade a backbone to a POP, but at least they don't have to pay for DIA on that part of the connection, as Netflix is giving ti away.

This Netflix fiasco of linking with providers directly is one of many ways Netflix, Youtube, or any other big player could fix this congestion. Netflix could have chosen to scale back their peering with who they use now, and instead open peer point with the 10's of other Teir1 providers they don't use at all their pop's, and this would fix for everyone. Instead the biggest of the ISP's with the largest of capacity problems to Netflix have chosen to go into agreement with each other. This sets a standard for every other provider that doesn't also peer to Verizon of Comcast to need to do the same to fix their issue with Netflix.
rradina

join:2000-08-08
Chesterfield, MO

Re: Sandvine Confirms ISPs Not Treating Traffic Equally

We can just as easily say customers PULL from Netflix as we can Netflix "puts outbound".

This "balance" crap is old Telco thinking from terminating vs. originating calls (did you call me vs. I called you crap).

QuietRiot

@50.170.133.x
Well, they do offer boxes that act as caches to prevent peering traffic in the first place... It would be nice if most of those few hundred very large but often repeated streams only had to be downloaded through peering arrangements a single time, dontchthink?!

Killa200
Premium
join:2005-12-02
Southeast TN
Reviews:
·Charter

1 edit

Re: Sandvine Confirms ISPs Not Treating Traffic Equally

Id agree if they offered them to anyone who wanted them. We tried signing up for one in our system and they refuse to offer the boxes for any isp that had peaks less than a consistant 5000mbit/sec of netflix traffic alone. They said we were welcome to build out to the nearest peering point 150 miles away and peer or find another provider that would get a cache box for us.

Some "open connect" agreement eh?

IowaCowboy
Iowa native
Premium
join:2010-10-16
Springfield, MA
kudos:1

1 recommendation

Conflict of interest

Sandvine's biggest customers are the big telcos and the cablecos.
ITGeeks

join:2014-04-20
Cleveland, OH

Re: Conflict of interest

And Cogents largest customer is Netflix. so?

brucecr0ft

@141.212.108.x

In research forums too

I've seen them try to pass off their cable/telco inspired "research" as facts at multiple tech forums like NANOG and IETF. It is absolutely disgusting. Karl is right to call them out.

victorgee

@198.12.76.x

Re: In research forums too

said by brucecr0ft :

I've seen them try to pass off their cable/telco inspired "research" as facts at multiple tech forums like NANOG and IETF. It is absolutely disgusting. Karl is right to call them out.

Yes, they badmouthed Netflix non stop at the Atlanta NANOG conference pretending it was fact based research. Disgusting indeed.
openbox9
Premium
join:2004-01-26
Germany
kudos:2
Is the data collected by Sandvine not factual? Is the data collected by Google not factual? Is the data collected by Netflix not factual? Statistics are simply an avenue for telling a story. They can show just about anything you want.
bn1221

join:2009-04-29
Cortland, NY

Interesting idea

Sandvine to rankings....pot, meet kettle
ITGeeks

join:2014-04-20
Cleveland, OH

Re: Interesting idea

Would be better to be seen by the actual network hardware manufacturer then some Content company. Who is only getting their numbers based upon the quality of the video going to the customer.
BrianON

join:2011-09-30
Ottawa, ON
Reviews:
·TekSavvy Cable

Netflix and YouTube never said they are general rankings anyway.

The Netflix and Youtube rankings are based on the speeds and therefore video quality that users are actually getting when they watch video from each service.

An ISP low in the rankings of Netflix means lower average video quality while watching Netflix. An ISP low in the rankings of YouTube means lower average video quality while watching YouTube. Therefore the rankings make it easier for end users to see how their ISP compares to others in this regard for each service.

That they aren't "Internet Video Rankings" is obvious so i am not sure what Sandvine is whining about.

nekkidtruth
YISMM
Premium
join:2002-05-20
London, ON
kudos:2
Reviews:
·WIND Mobile
·Rogers Hi-Speed

I don't believe the ISP's for a second...

When Bell and Rogers were forced to show the CRTC the reason for their throttling, they made themselves look like absolute morons. Their networks were practically wide open highways with absolutely zero congestion. They were throttling simply because they felt like it.

I'll take Netflix and Level3's word over the ISP's any day.
--
Weeeeeee
ITGeeks

join:2014-04-20
Cleveland, OH

Re: I don't believe the ISP's for a second...

Level3 is an ISP as well.

nekkidtruth
YISMM
Premium
join:2002-05-20
London, ON
kudos:2

Re: I don't believe the ISP's for a second...

Anyone who provides Internet is an ISP. You know I'm clearly referring to Tier 2 ISP's.
--
Weeeeeee
MrBungle87

join:2013-01-18
Durham, NC

Lies, damned lies, and statistics.

You can bet that anything Sandvine says is a lie or a distortion. They'll white knight for ISPs, governments, or any corrupt third party as long as they get a big payday.

linicx
Caveat Emptor
Premium
join:2002-12-03
United State
Reviews:
·TracFone Wireless
·CenturyLink

Yeah

He said. She said. I get it. It is a group of undisciplined and unregulated asses that are fighting among themselves like dogs over a bone.

FCC may, or may not, care. It makes little difference as Congress will continue in its present gridlock for at least another 18 months during which time there will be one or more price increases.

I personally say no to all on-remand-movies. It's too expensive. I spend locally to help keep our little movie store open.

--
Mac: No windows, No Gates, Apple inside