Youtube, as with most other streaming media, is delivered over a content distribution network (CDN). This keeps the bulk of streaming traffic off the regular Internet and utilizes a preferred connection arrangements to the ISP's internal network. It is ultimately up to your ISP to decide if they want to utilize these preferred private arrangements.
Here is a study a while back about YouTube
Are you using the DNS servers provided by your ISP or running third party DNS? If you are running third party DNS try switching to your ISP's DNS. Third party DNS is not aware of any special arraignments your ISP has (if any) with a particular CDN. They will resolve a valid IP to obtain the data but it may be delivered over a slower or more congested path then if you use your ISP's DNS.
Or it may be the case the videos you are streaming are on servers that are heavily loaded. Or it may be the peering arrangement your ISP is using happens to be congested on the route used to access the particular youtube servers.