Sandvine: YouTube Issues Google's Fault, Not ISP Related
Despite the faster speeds now being pushed through fiber and DOCSIS 3.0, there's many users who continue to suffer from the inability to quickly and consistently stream YouTube videos. Spend a few minutes in any of our forums and you'll find this is a universal problem with many carriers, including AT&T U-Verse
, Verizon FiOS
and Time Warner Cable
Is it due to CDN problems? Peering imbalances? DNS issues? Is it the ISPs' fault or is the problem with Google's caching servers stored on the ISP network (which ISPs have no access to)? Most users blame their ISP, but according to Sandvine
, the culprit for YouTube buffering lag is quite simply overloaded Google servers:
...we can conclude that the root cause of the degradation in quality is likely occurring because of an oversubscription in the Google server farm (where YouTube is hosted) which makes YouTube unable to meet high lunch time and evening video demand. This oversubscription would result from a commercial decision by YouTube to regarding how much capital they wanted to invest in server capacity to maintain quality.
So the next time you try to watch a YouTube video and it buffers, don’t automatically blame your ISP. Google may very well have made a commercial decision that limits the ISPs ability to improve quality and caused the tens of thousands of users, like me, who would otherwise like to spend their lunch hour watching videos on YouTube, return to more productive activities.
Sandvine's Dan Deeth also directs users to this Youtube page
that can help users compare their YouTube performance to that of their region, or their overall ISP. Keep in mind of course that the majority of Sandvine cash comes from the largest ISPs, so it's unlikely they're going to criticize carriers. Example A is the fact that Sandvine chooses to ignore that ISPs share some part of the blame as we've seen with an endless stream of peering fights
Re: Oversubscription in the Google server farm?
said by hello123454:For Google Search, Google Ads (doubleclick), etc, I see huge proactive investment. On YouTube the P&L may not be as great and for the most part everyone blames the ISP so who cares....
You mean to tell me a company like Google isn't pro active enough to invest in their own architecture? I just find that hard to believe. If people are really having legitimate issues what's the problem with investing more money to resolve all of this?
Youtube Beta? probably because youtube is still beta just like everything google is lol
Like Sandvine is going to bite the hands that feed them. What else is Sandvine supposed to say other than glowing commentary about their top tier customers?
cogent vz peering congestion is real There is an mlab3 test server here in the same city as me. It is connected to cogent. I am on Fios 500. That server tests at 500+Mbps during periods of low network usage. During evening peak hours, download speed tests from that server are as low as 522 KILObps. Fios itself works great even then, delivering 500 MEGA bps plus from VZ and some 3rd party test servers. (But not all: some less dramatic peak hour congestion is visible on peering to NTT and others).
High quality last mile is very important. But, it alone is not sufficient to deliver good service. Comparable high quality (zero congestion) peering relationships are essential.
If a content supplier delivers all the way to the local market of the last mile customer who is requesting the content, the last mile ISP has no morally justifiable grounds to impair delivery of that content to its customer. The customer is already paying the ISP for service and the content originator has borne the cost of WAN or CDN.
VZ and other last mile ISPs need to stop their attempts to double dip. The FCC or Congress should compel last mile ISPs to accept traffic requested by their customers if it is delivered to the market. If a last mile ISP fails to perform this simple function then what exactly is it selling to its last mile customers: empty promises.
Re: cogent vz peering congestion is real hi:
In this particular case, the test server is HERE IN LA. The rtt at this time of day is 5.9 mSec. I don't think there's much of Cogent or Verizon's wide area backbone's involved. The Los Angeles peering connection between the two of them has not been upgraded to handle the offered-load during peak-usage evening hours. This is pure-and-simple an end-run around Net Neutrality. This speed-test server (and cogent) are delivering traffic right here to the destination market of the requester, and the last-mile provider (Verizon) is not conveying the traffic that I requested to me, their paying customer. (And you can imagine what I pay for 500Mbps service!)
I miss Seidenberg. He seemed to want to do something good for the country. I imagine there are others inside Verizon who also want to do the right thing too. But they're not in charge.
Here's the traceroute at 10AM. A screen-cap of the speed-test (now at 10AM) is attached. Works fine when there's low demand.
But at 9PM, when lots of FiOS customers request content that is sourced from servers over on Cogent, the congestion on Verizon's peeing circuit allows as little as 522 KILO bps of the requested data to be delivered to Verizon's own paying customers. A screen-cap of a 9PM speed-test is also attached. (Keep in mind FiOS itself works great, even at 9PM. This is a defect in morality (greed), not a defect in FiOS.
Tracing route to 126.96.36.199 over a maximum of 30 hops
1 1 ms 1 ms 1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 2 ms 1 ms 2 ms L100.LSANCA-VFTTP-157.verizon-gni.net [188.8.131.52]
3 3 ms 3 ms 3 ms G0-11-4-2.LSANCA-LCR-21.verizon-gni.net [184.108.40.206]
4 2 ms 3 ms 3 ms so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net [220.127.116.11]
5 3 ms 3 ms 3 ms 0.xe-4-1-0.BR1.LAX15.ALTER.NET [18.104.22.168]
6 3 ms 3 ms 4 ms te0-0-0-2.ccr21.lax05.atlas.cogentco.com [22.214.171.124]
7 4 ms 4 ms 4 ms be2025.mpd21.lax01.atlas.cogentco.com [126.96.36.199]
8 4 ms 4 ms 4 ms te7-1.mag01.lax01.atlas.cogentco.com [188.8.131.52]
9 3 ms 3 ms 3 ms te4-2.ccr01.lax09.atlas.cogentco.com [184.108.40.206]
10 3 ms 4 ms 4 ms 220.127.116.11
IPv6 I stopped seeing YouTube buffering issues after I started using it exclusively over IPv6 at home.
Peering Points Every time I have a issue with youtube (I'm on it constantly as I am a Content Provider/Partner), their is some major peering issue at the handoff point. Have NO issues with my Charger internet connection, though, my FASTER time warner connection ALWAYS has issues. I get 3 or 4 hops on the routing, then latency spikes from 20ms up to about 400-1000ms, then another 4 or 5 hops to Youtube. It's always at the peering point. Which appears overloaded.
»KmanScooters.com Home of Wisconsin's Most Affordable Cars, Motorcycles and Scooters
"Yes, I'm a Freemason
No, I don't know where the Grail is
No, I don't know where the Ark is
And don't even ask me about
"The Da Vinci Code."
This is misleading This article and Sandvine is misleading, and this was likely pushed for by a major ISP trying to pass off blame. The blame does lie with ISPs, but its a really simple fix for them. ISPs host Youtube and google caching servers on their own networks, to reduce the amount of data that crosses their peering points. This is logical to do. The problem comes in when ISPs refuse to upgrade those Caching servers, or in the case of YT servers(which are maintained by google), they refuse to feed those servers more external and internal bandwidth. The effect is exactly what we see happening, where YT buffers on all but Google direct servers(never heard anyone complain of googles own YT servers having bandwidth issues). The ISPs do not give their own in network caching servers more bandwidth to use both externally and internally, and thus, it becomes the bottlneck that everyone experiences, and it effectively does the same thing as throttling YT without ever having to do any active network management, and has a different legal definition from throttling, so they are not likely to ever get sued over it. The ISPs could very easily fix nearly all YT issues by simply upgrading the bandwidth both internally(to their own customers) and externally(loading stuff that is not cached, crosses peering point), but why would they do this when it naturally limits competition to their television services without the same legal definition.
I blocked my ISPs caching servers because charters were terrible. My YT experience is now loading from somewhere on the west coast(likely googles own MTView complex, according to GEOIP anyways), with zero buffering issues, and lower latency than even charters own YT caching server that is only about half the distance away(STL, MO is where the GEOIP record shows, but it could be closer). The fix is easy, but the ISPs hate competition.
Re: This is misleading
said by [yotube :engrish is grate!
said by Chubbysumo:
The blame does lie with ISPs, but its a really simple fix for them.
Why does do some youtube servers work well and some do not (see the reddit post). If I get great performance from by broadband, and hide (via proxy) or block some servers does youtube do well?
Sounds like the performance is controlled more by Google than my ISP. They can route around (or create) these congested peering points
On a more serious note, your ISP might have more than one caching server, as most do, and you might need to block quite a few before you are actually loading from an actual google server. All google servers are fed with much more bandwidth than what they need, and have "burstable" speed capacity so in the event that the server gets overrun with more people than it can handle, their ISP will turn on more bandwidth(for a cost) for a little while to handle the increase in traffic. I had to block 19 ISP based caching server IP addresses before I got out of charters network. sometimes a VPN only points you back to an already overloaded ISP caching server, and not an actual google server.
| |DavidI start new work on Premium,VIPReviews:
Granite City, IL
I could have told them that for much cheaper... I knew it was a problem with youtube a long, long, long, long, long, long, long, long, long, long, long, long time ago.
I figured it out without smartvideo extentions for google chrome and firefox. I even figured it out before everyone else started talking about it.
How did I figure it out?
At the time used Atube catcher- I knew when I saw 600kbps download at full speed with my test WinXP machine in the basement, it wasn't a ADSL, peering or really a line problem. I could watch the entire video just by downloading it locally and streaming it, and ironically that way was much faster than waiting for youtube's video window to fire up and buffer.
If you have a topic in the direct forum please reply to it or a post of mine, I get a notification when you do this.
Koetting Ford, Granite City, illinois... YOU'RE FIRED!!
YT is a flat out joke. TWC here and YT is about useless. I don't know where the blame lies, but I do know that YT vids are more of a pain in the ass than they're worth AND the players functionality is a joke. Straight out of 1995.
Maybe *I* am the problem here, so let me ask:
WHY can't the goddamn video that I've ALREADY DOWNLOADED AND WATCHED remain on my PC until I close out the browser, or move to another video or whatever. WHY DOES IT HAVE TO RELOAD FROM SCRATCH if I wanna to jump back 4 seconds? This infuriates me.
Why don't the ads from YT ever stop and buffer? They seem to play perfect everytime.
Why doesn't pausing let the video buffer all the time? This seems very hit and miss.
Why can't I click something that allows the video to preload into a buffer, so it can load while I'm doing something else, and then come back to it and watch without having to download as I watch it?
Personally, I find YT to be a lower tier type of service. Along the lines of a MySpace...it had potential, and was great for a while, but now has gotten to the point I'd rather avoid it.
I am only one, but I am one. I cannot do everything, but I can do something. And I will not let what I cannot do interfere with what I can do.- Edward Everett Hale
My Blog - Raising Connor