dslreports logo
site
   
spc
Related:
story category
Sandvine: YouTube Issues Google's Fault, Not ISP Related
by Karl Bode 08:58AM Monday Sep 30 2013
Despite the faster speeds now being pushed through fiber and DOCSIS 3.0, there's many users who continue to suffer from the inability to quickly and consistently stream YouTube videos. Spend a few minutes in any of our forums and you'll find this is a universal problem with many carriers, including AT&T U-Verse, Verizon FiOS and Time Warner Cable.

Is it due to CDN problems? Peering imbalances? DNS issues? Is it the ISPs' fault or is the problem with Google's caching servers stored on the ISP network (which ISPs have no access to)? Most users blame their ISP, but according to Sandvine, the culprit for YouTube buffering lag is quite simply overloaded Google servers:
quote:
...we can conclude that the root cause of the degradation in quality is likely occurring because of an oversubscription in the Google server farm (where YouTube is hosted) which makes YouTube unable to meet high lunch time and evening video demand. This oversubscription would result from a commercial decision by YouTube to regarding how much capital they wanted to invest in server capacity to maintain quality.

So the next time you try to watch a YouTube video and it buffers, don’t automatically blame your ISP. Google may very well have made a commercial decision that limits the ISPs ability to improve quality and caused the tens of thousands of users, like me, who would otherwise like to spend their lunch hour watching videos on YouTube, return to more productive activities.
Sandvine's Dan Deeth also directs users to this Youtube page that can help users compare their YouTube performance to that of their region, or their overall ISP. Keep in mind of course that the majority of Sandvine cash comes from the largest ISPs, so it's unlikely they're going to criticize carriers. Example A is the fact that Sandvine chooses to ignore that ISPs share some part of the blame as we've seen with an endless stream of peering fights.

view:
topics flat nest 
Bengie25

join:2010-04-22
Wisconsin Rapids, WI

My Anecdote

I haven't had YouTube buffering issues in a long while. Even yesterday I set a video to 1440p and was pulling down 40mb/s for a good 20 seconds and it did not buffer even then, was instant viewing.
bookertdub

join:2012-10-08
San Diego, CA

Re: My Anecdote

1440p?
Bengie25

join:2010-04-22
Wisconsin Rapids, WI

Re: My Anecdote

2560x1440. Some YouTube videos are that high, there are even a few 4k videos.

hello123454
Premium
join:2002-02-02
Wilmington, DE
kudos:1

Oversubscription in the Google server farm?

Let's assume this is the big issue...server capacity. You mean to tell me a company like Google isn't pro active enough to invest in their own architecture? I just find that hard to believe. If people are really having legitimate issues what's the problem with investing more money to resolve all of this?

yotube

@comcast.net

1 recommendation

Re: Oversubscription in the Google server farm?

said by hello123454:

You mean to tell me a company like Google isn't pro active enough to invest in their own architecture? I just find that hard to believe. If people are really having legitimate issues what's the problem with investing more money to resolve all of this?

For Google Search, Google Ads (doubleclick), etc, I see huge proactive investment. On YouTube the P&L may not be as great and for the most part everyone blames the ISP so who cares....
mario44222

join:2003-11-22
Cliffside Park, NJ

1 recommendation

Youtube Beta?

probably because youtube is still beta just like everything google is lol
Bengie25

join:2010-04-22
Wisconsin Rapids, WI

Re: Youtube Beta?

+9001 Funny. You sir win one internet.
SunnyD

join:2009-03-20
Madison, AL

1 recommendation

Like Sandvine is going to bite the hands that feed them.

What else is Sandvine supposed to say other than glowing commentary about their top tier customers?

yotube

@comcast.net

Re: Like Sandvine is going to bite the hands that feed them.

I don't see glowing commentary about anyone. Did you even look at the analysis in the report? Can you explain, or have any facts to understand how ll ISPs he same issue?

Bob_in_LA

@myvzw.com

1 recommendation

cogent vz peering congestion is real

There is an mlab3 test server here in the same city as me. It is connected to cogent. I am on Fios 500. That server tests at 500+Mbps during periods of low network usage. During evening peak hours, download speed tests from that server are as low as 522 KILObps. Fios itself works great even then, delivering 500 MEGA bps plus from VZ and some 3rd party test servers. (But not all: some less dramatic peak hour congestion is visible on peering to NTT and others).

High quality last mile is very important. But, it alone is not sufficient to deliver good service. Comparable high quality (zero congestion) peering relationships are essential.

If a content supplier delivers all the way to the local market of the last mile customer who is requesting the content, the last mile ISP has no morally justifiable grounds to impair delivery of that content to its customer. The customer is already paying the ISP for service and the content originator has borne the cost of WAN or CDN.

VZ and other last mile ISPs need to stop their attempts to double dip. The FCC or Congress should compel last mile ISPs to accept traffic requested by their customers if it is delivered to the market. If a last mile ISP fails to perform this simple function then what exactly is it selling to its last mile customers: empty promises.
Bengie25

join:2010-04-22
Wisconsin Rapids, WI
Reviews:
·Solarus

Re: cogent vz peering congestion is real

"Comparable high quality (zero congestion) peering relationships are essential."

I love how Verizon and Comcast call themselves Tier 1 ISPs, then they have congestion.

On Level 3, I have yet to see a link have congestion except for transient problems that may crop-up a few times during the year.

L3 has a strict, upgrade the link when it when the 95% percentile reaches 50% of the link's capacity. Obviously this only applies to internal links and peering links, as customers need to pay to have their links upgraded. Even then, L3 sells bandwidth cheap enough that most of their customers can afford to keep ahead of demand.

Bob_in_LA

@verizon.net

1 recommendation

Re: cogent vz peering congestion is real

Click for full size
Click for full size
hi:
In this particular case, the test server is HERE IN LA. The rtt at this time of day is 5.9 mSec. I don't think there's much of Cogent or Verizon's wide area backbone's involved. The Los Angeles peering connection between the two of them has not been upgraded to handle the offered-load during peak-usage evening hours. This is pure-and-simple an end-run around Net Neutrality. This speed-test server (and cogent) are delivering traffic right here to the destination market of the requester, and the last-mile provider (Verizon) is not conveying the traffic that I requested to me, their paying customer. (And you can imagine what I pay for 500Mbps service!)

I miss Seidenberg. He seemed to want to do something good for the country. I imagine there are others inside Verizon who also want to do the right thing too. But they're not in charge.

Here's the traceroute at 10AM. A screen-cap of the speed-test (now at 10AM) is attached. Works fine when there's low demand.
But at 9PM, when lots of FiOS customers request content that is sourced from servers over on Cogent, the congestion on Verizon's peeing circuit allows as little as 522 KILO bps of the requested data to be delivered to Verizon's own paying customers. A screen-cap of a 9PM speed-test is also attached. (Keep in mind FiOS itself works great, even at 9PM. This is a defect in morality (greed), not a defect in FiOS.

C:\Windows\system32>tracert 38.98.51.45

Tracing route to 38.98.51.45 over a maximum of 30 hops

1 1 ms 1 ms 1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 2 ms 1 ms 2 ms L100.LSANCA-VFTTP-157.verizon-gni.net [71.189.7.1]
3 3 ms 3 ms 3 ms G0-11-4-2.LSANCA-LCR-21.verizon-gni.net [130.81.175.86]
4 2 ms 3 ms 3 ms so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net [130.81.151.246]
5 3 ms 3 ms 3 ms 0.xe-4-1-0.BR1.LAX15.ALTER.NET [152.63.112.229]
6 3 ms 3 ms 4 ms te0-0-0-2.ccr21.lax05.atlas.cogentco.com [154.54.13.85]
7 4 ms 4 ms 4 ms be2025.mpd21.lax01.atlas.cogentco.com [154.54.30.189]
8 4 ms 4 ms 4 ms te7-1.mag01.lax01.atlas.cogentco.com [154.54.28.142]
9 3 ms 3 ms 3 ms te4-2.ccr01.lax09.atlas.cogentco.com [154.54.31.90]
10 3 ms 4 ms 4 ms 38.98.51.45

Trace complete.
smcallah

join:2004-08-05
Home

1 recommendation

IPv6

I stopped seeing YouTube buffering issues after I started using it exclusively over IPv6 at home.

Anonuser

join:2003-01-03
Milwaukee, WI

Peering Points

Every time I have a issue with youtube (I'm on it constantly as I am a Content Provider/Partner), their is some major peering issue at the handoff point. Have NO issues with my Charger internet connection, though, my FASTER time warner connection ALWAYS has issues. I get 3 or 4 hops on the routing, then latency spikes from 20ms up to about 400-1000ms, then another 4 or 5 hops to Youtube. It's always at the peering point. Which appears overloaded.
--
»KmanScooters.com Home of Wisconsin's Most Affordable Cars, Motorcycles and Scooters
"Yes, I'm a Freemason
No, I don't know where the Grail is
No, I don't know where the Ark is
And don't even ask me about
"The Da Vinci Code."
Bengie25

join:2010-04-22
Wisconsin Rapids, WI

Re: Peering Points

Yeah, "peering issues".

Too bad calling TWC and telling them to stop messing around, isn't going to do any good :*(
Chubbysumo

join:2009-12-01
Superior, WI
Reviews:
·Charter

1 recommendation

This is misleading

This article and Sandvine is misleading, and this was likely pushed for by a major ISP trying to pass off blame. The blame does lie with ISPs, but its a really simple fix for them. ISPs host Youtube and google caching servers on their own networks, to reduce the amount of data that crosses their peering points. This is logical to do. The problem comes in when ISPs refuse to upgrade those Caching servers, or in the case of YT servers(which are maintained by google), they refuse to feed those servers more external and internal bandwidth. The effect is exactly what we see happening, where YT buffers on all but Google direct servers(never heard anyone complain of googles own YT servers having bandwidth issues). The ISPs do not give their own in network caching servers more bandwidth to use both externally and internally, and thus, it becomes the bottlneck that everyone experiences, and it effectively does the same thing as throttling YT without ever having to do any active network management, and has a different legal definition from throttling, so they are not likely to ever get sued over it. The ISPs could very easily fix nearly all YT issues by simply upgrading the bandwidth both internally(to their own customers) and externally(loading stuff that is not cached, crosses peering point), but why would they do this when it naturally limits competition to their television services without the same legal definition.

I blocked my ISPs caching servers because charters were terrible. My YT experience is now loading from somewhere on the west coast(likely googles own MTView complex, according to GEOIP anyways), with zero buffering issues, and lower latency than even charters own YT caching server that is only about half the distance away(STL, MO is where the GEOIP record shows, but it could be closer). The fix is easy, but the ISPs hate competition.
Bengie25

join:2010-04-22
Wisconsin Rapids, WI
Reviews:
·Solarus

Re: This is misleading

I wondered about this when I was a Charter customer.

I actually did not have many buffering issues during peak hours, but some times I would get these strange issues where 720p would have a ton of buffering issues, but 1080p was just fine, or some other combination.

Or I would try to view something that was not as popular, like a video about FreeBSD, and I would get buffering with 480p, but had no issues watching the newest pop-music video at 1080p.

Now it all makes sense. a Mix of caching and congested peering could give these effects.

My new ISP does not have issues buffering almost at all, but if it does, it affects all of my videos equally.
FactChecker
Premium
join:2008-06-03
Can't google also upgrade their network capacity? I think in most cases they are directly connected to the ISPs... See this thread which talked about why different things fix the issue with FiOS

»Re: Horrible youtube speeds

said by Jcink:

I don't have all the details so take this with a grain of salt but if they're paying for dedicated routes and they're saturated then they have two options: buy more at the normal price, get verizon to increase it for free or at a lower cost. If Google chooses the latter and Verizon won't budge, this is where we all stand now.

Google probably thinks they can hurt Verizon and force them into getting what they want by allowing FiOS users to suffer until Verizon gives in. It hasn't worked yet but I can't help but wonder how many calls they get from people saying "the internet is being slow" when youtube doesn't load. And customers who have dropped the service over youtube and netflix being too slow.

--
"Too often we... enjoy the comfort of opinion without the discomfort of thought." - John F. Kennedy
Bengie25

join:2010-04-22
Wisconsin Rapids, WI
Reviews:
·Solarus

Re: This is misleading

"Can't google also upgrade their network capacity?"

What he was getting after was that the most common issue is that it is not Google's network or their servers, it is entirely the ISP doing something or not doing something.

Case 1: ISP has peering with Google, but doesn't upgrade their link so customers get slow Google

Case 2: ISP gets cache server from Google, but doesn't ask Google for more as demand grows or doesn't supply the local cache with enough bandwidth, so the customer gets slow Google.

Case 3: Customer by passes peering link and cache servers and gets fast Google, assuming their ISP isn't otherwise congested for other reasons.

Slow YouTube isn't always the ISP's fault, but neither is a headache always late night drinkings fault, but if someone is up all night drinking, it's a great first target to blame.
FactChecker
Premium
join:2008-06-03

Re: This is misleading

ok. I just found it was interesting that the issue was across most ISPs and not with other services.... Kinda like blaming all liquor companies for your hang over. I also found it interesting that there are ways to avoid it (manually selecting servers vs Googles choice to give your ISP) over the same ISP that is at blame

--
"Too often we... enjoy the comfort of opinion without the discomfort of thought." - John F. Kennedy
Bengie25

join:2010-04-22
Wisconsin Rapids, WI
Reviews:
·Solarus

Re: This is misleading

"ok. I just found it was interesting that the issue was across most ISPs"

That does make a difference.

The graph on the linked site does only show a dip of 5mb down to 4mb. Most people that I see with performance issues are complaining about sub 1mb speeds.

It is interesting to see such a widely-observed blatant dip, even if it isn't a huge amount.

The other thing I find interesting is what does "5mb" actually mean. When I watch YouTube, I don't get constant streaming, I get 40mb bursts every 5-8 seconds, except for the initial buffering or seeking, then I just get a 40mb burst until their is about a minute of buffer.
Chubbysumo

join:2009-12-01
Superior, WI
Reviews:
·Charter
well at least someone else gets it. Google does not control how much bandwidth the IPSs give to their servers, or how many the ISP takes. If the ISP wants more, google will pay for and maintain them(it actually reduces googles costs with peering as well), but if the ISP decides to short the server on bandwidth, or simply let a server get overrun with traffic when they should be getting more, it will be the same damn effect. Once I blocked all of charters servers, I got better speed and latency from a server that is over twice the distance away.

yotube

@69.241.19.x
said by Chubbysumo:

The blame does lie with ISPs, but its a really simple fix for them.

Why does do some youtube servers work well and some do not (see the reddit post). If I get great performance from by broadband, and hide (via proxy) or block some servers does youtube do well?

Sounds like the performance is controlled more by Google than my ISP. They can route around (or create) these congested peering points
Chubbysumo

join:2009-12-01
Superior, WI
Reviews:
·Charter

Re: This is misleading

said by [yotube :

]

said by Chubbysumo:

The blame does lie with ISPs, but its a really simple fix for them.

Why does do some youtube servers work well and some do not (see the reddit post). If I get great performance from by broadband, and hide (via proxy) or block some servers does youtube do well?

Sounds like the performance is controlled more by Google than my ISP. They can route around (or create) these congested peering points

engrish is grate!

On a more serious note, your ISP might have more than one caching server, as most do, and you might need to block quite a few before you are actually loading from an actual google server. All google servers are fed with much more bandwidth than what they need, and have "burstable" speed capacity so in the event that the server gets overrun with more people than it can handle, their ISP will turn on more bandwidth(for a cost) for a little while to handle the increase in traffic. I had to block 19 ISP based caching server IP addresses before I got out of charters network. sometimes a VPN only points you back to an already overloaded ISP caching server, and not an actual google server.
InvalidError

join:2008-02-03
kudos:5

There is a simple fix for overloaded Youtube servers

Stop resetting the "Never play higher quality" option.

About once a week, Youtube forgets my account Playback settings. I wish I could tell it to stream 480p (instead of 360p default) and not have to periodically bother with turning on the low-bandwidth option again for the 30+th time.

Tamarisk

@arkomnet.eu

No Problems in Europe

Moved to Poland a month ago to be with family. Had TWC in Ventura, CA and suffered slow YouTube loading on a daily basis. Using a 5Mbps connection (not the fastest available, but I'm in a rural area so it has to be through wireless) here it's silky smooth so I can now honestly say something was wrong with the way YouTube was served through Time Warner.

JakCrow

join:2001-12-06
Palo Alto, CA
Reviews:
·SONIC.NET

Lack of consistent buffering

IMO, Youtube doesn't buffer videos as well as it once did either. The current player will even reset the buffer if you back up a video. And don't get me started on the various clients for mobile and media devices, with their almost universal lack of buffering.

IowaCowboy
Iowa native
Premium
join:2010-10-16
Springfield, MA
kudos:1
Reviews:
·Verizon Broadban..
·Comcast

We should get together and hire a consultant

We should get together and hire a networking engineer (who is disinterested in ISPs or Google/YouTube) to give us a impartial opinion on why YouTube sucks so much. I'm willing to bet it would boil down to the last mile ISPs.

The minimum qualifications for this specialist would be a bachelor's degree in engineering with a background in computer science/networking and not have any financial interest in broadband ISPs or Google/YouTube.

My hometown of Cedar Rapids IA is home to a lot of people with engineering degrees as they are home to Rockwell Collins, which is a government military aerospace contractor.
--
I've experienced ImOn (when they were McLeod USA), Mediacom, Comcast, and Time Warner and I currently have DirecTV. They are much better than broadcast TV.

I have not and will not cut the cord.
brad152

join:2006-07-27
Phoenix, AZ

CenturyLink does not have this problem in Phoenix

I never have an issue on my CenturyLink 40/5 VDSL2 connection, but when i used Cox here, it was quite normal to have YouTube loading issues during peak times. I can even watch 1080i videos no problem at anytime of day.
tommy2text0

join:2011-05-24
Stratford, CT

Re: CenturyLink does not have this problem in Phoenix

I stream You Tube with no buffering ever on an Optimum 101/35 connection. The only time I ever had buffering issues was on older computers. All my devices stream You Tube without ever having a glitch and I never blocked any caching servers.

David
I start new work on
Premium,VIP
join:2002-05-30
Granite City, IL
kudos:101
Reviews:
·DIRECTV
·AT&T Midwest
·magicjack.com
·Google Voice

I could have told them that for much cheaper...

I knew it was a problem with youtube a long, long, long, long, long, long, long, long, long, long, long, long time ago.

I figured it out without smartvideo extentions for google chrome and firefox. I even figured it out before everyone else started talking about it.

How did I figure it out?

At the time used Atube catcher- I knew when I saw 600kbps download at full speed with my test WinXP machine in the basement, it wasn't a ADSL, peering or really a line problem. I could watch the entire video just by downloading it locally and streaming it, and ironically that way was much faster than waiting for youtube's video window to fire up and buffer.
--
If you have a topic in the direct forum please reply to it or a post of mine, I get a notification when you do this.
Koetting Ford, Granite City, illinois... YOU'RE FIRED!!
DasBubbicus

join:2013-07-21
Brockville, ON

Another anecdote...

All of my YouTube issues disappeared once I started routing everything through a VPN tunnel. YMMV.

Jon Geb
Wal-Mart Sucks

join:2001-01-09
Howell, MI

AT&T and Youtube

This was a big problem for me on Uverse.... YouTube was heavily throttled.

I have roughly the same speed on Comcast today and all issues are gone.

Koil
Premium
join:2002-09-10
Irmo, SC
kudos:2

YT is a flat out joke.

TWC here and YT is about useless. I don't know where the blame lies, but I do know that YT vids are more of a pain in the ass than they're worth AND the players functionality is a joke. Straight out of 1995.

Maybe *I* am the problem here, so let me ask:

WHY can't the goddamn video that I've ALREADY DOWNLOADED AND WATCHED remain on my PC until I close out the browser, or move to another video or whatever. WHY DOES IT HAVE TO RELOAD FROM SCRATCH if I wanna to jump back 4 seconds? This infuriates me.

Why don't the ads from YT ever stop and buffer? They seem to play perfect everytime.

Why doesn't pausing let the video buffer all the time? This seems very hit and miss.

Why can't I click something that allows the video to preload into a buffer, so it can load while I'm doing something else, and then come back to it and watch without having to download as I watch it?

Personally, I find YT to be a lower tier type of service. Along the lines of a MySpace...it had potential, and was great for a while, but now has gotten to the point I'd rather avoid it.
--
I am only one, but I am one. I cannot do everything, but I can do something. And I will not let what I cannot do interfere with what I can do.- Edward Everett Hale

My Blog - Raising Connor