dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
2739
share rss forum feed

johnnn

join:2007-01-25
Ypsilanti, MI

[Speed] Congested Comcast Peering with Akamai, Google in Midwest


Tracert to Akamai host
Click for full size
Data transfer plot from Chicago-based Comcast server
Click for full size
Data transfer plot from Akamai host

Tracert to Google (YouTube) host
In Michigan, I'm experiencing persistent congestion even to the largest of CDNs...but every time I do a Speedtest.net or Shaperprobe test, I have no problem maxing out my provisioned rates (Blast 25/4). Attached are screenshots of traceroutes to each directly-connected host, a screenshot of connection utilization to the Akamai host (Google looked similar), and for reference a speedtest (»www.speedtest.net/result/2346965994.png) to one of Comcast's own servers in Chicago.

This is not a new problem. I haven't been able to get sufficient bandwidth for any high-quality video service except Netflix. Does anyone else have this issue?


28619103
Premium
join:2009-03-01
21435

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Can't see any information in the data that supports your conclusion. Can you run mtr or pingplotter with a larger dataset to the servers in question.

Also note Netflix uses Akamai as well as other CDNs and publishes their performance statistics to Comcast and other ISPs


johnnn

join:2007-01-25
Ypsilanti, MI

I would be happy to run pingplotter for longer to said servers, but ICMP latency isn't the issue; throughput is. Do you think the CDNs in question could be the issue?

The trouble with this kind of selective congestion: I can't easily tell if Comcast is running hot ports (something that is often alleged and sometimes backed up, like in the Tata case from awhile back) or if the CDN is performing poorly.

I'm having trouble with providers that *aren't* serving me Netflix (that's always Level3). I'm not a particularly savvy user so I don't know of ways to gather conclusive data that would support my conclusion. Help would be gratefully received, because from what I've observed, there's an issue.



28619103
Premium
join:2009-03-01
21435

1 recommendation

If there were a problem in the network portion you would probably see latency/loss starting at a hop AND CONTINUING all the way to the destination. Seeing a problem at one hop that does not continue is not a problem and seeing no issues along the network path tends to implicate the server you are talking to.


mountainview

join:2011-09-07
united state
reply to johnnn

I am so tired of people saying something on the internet is wrong because you think your ping/traceroute does not look good.

If it's not LAN or a network you completely control, don't even bother... A route can just ignore or deprioritize icmp...
with multiplication involved, GSLB (CDN) your traceroute is pretty much worthless as the path is very likely to change in 60s if not fasters...

If you are not a network operator, you don't need to complaint about someone other than your ISP because you have no control over them and they probably don't give a fark about you....


johnnn

join:2007-01-25
Ypsilanti, MI

Please read the OP. The tracert looks just fine. It's the throughput that's the problem.



JohnInSJ
Premium
join:2003-09-22
Aptos, CA
reply to johnnn

said by johnnn:

In Michigan, I'm experiencing persistent congestion even to the largest of CDNs

Which CDNs. How does "persistent congestion" manifest? What you're attempting to document on your endpoint isn't really possible to document - you have no way of knowing what, if any, point between you and the CDN, or the CDN itself, is limiting the flow of data.

Your only named provider in your post is Nefflix, and you say they have no problem maintaining a 3+mbit stream to you.

So, which CDNs are not? When?
--
My place : »www.schettino.us

johnnn

join:2007-01-25
Ypsilanti, MI

I addressed the issue with documenting this type of congestion in an earlier post: "The trouble with this kind of selective congestion: I can't easily tell if Comcast is running hot ports (something that is often alleged and sometimes backed up, like in the Tata case from awhile back) or if the CDN is performing poorly."

This is a phenomenon I've observed over the life of my ~2 years of Comcast service in my location. The congestion manifests itself in widely variable (but unusably low average) transfer bitrates. I know it's very difficult to document, which is why I'm bringing it up on this forum. I would like better tools (like the Explicit Congestion Notification RFC from 2008), but I don't have them.

The major CDNs that seems to be an issue are Akamai and Google (also mentioned previously in this thread), but sometimes Limelight gives me issues too. HD video streams from NPR music (which use Akamai or Limelight for live broadcasts) have historically been difficult to maintain, and YouTube's 1080p video library is almost never watchable. Recently, I've even seen sporadic trouble with Netflix when it is delivered through Level 3.


mountainview

join:2011-09-07
united state
reply to johnnn

Dunno what the OP is complaining about..Peers/Transit providers can of course deprioritize certain traffic, ISP has QoS too...
What your snapshots are mostly irrelevant to the issue.. And most likely you can't do much about.... Not satisfied? Get another ISP....
From what I have seen, as a residential customer, if your speedtest and ping looks fine to your ISP's test server, in their eyes you are just making some nonsense....


johnnn

join:2007-01-25
Ypsilanti, MI

1 recommendation

I'm well aware of how residential ISPs view their customers. Isn't this the place we're supposed to talk about it and be heard?

If we are to effect change in an environment where the regulatory and business cards are stacked against us, why would we stay silent? If mountainview would rather not effect change, s/he might be better off working for Comcast. You know, they do pay people to toe the corporate line.

Finally, how are my snapshots irrelevant? They illustrate a problem I know from experience to be systemic. I don't have a reason to misrepresent what I'm posting, and I have knowledge of the network architecture. If mountainview would rather not hear about the condition of the residential broadband industry from the viewpoint of the consumer, s/he may want to find another forum to troll, as this one seems dedicated to that exact purpose.



JohnInSJ
Premium
join:2003-09-22
Aptos, CA

1 recommendation

reply to johnnn

said by johnnn:

various observations

But there is no way to verify your observations.

Let's start with some actual facts

- what tier are you on

- what is your home network like

- what load are you putting on your connection (are you uploading while attempting these downloads, for example? I don't care what you're uploading - just tell me if you're using x% of your uplink... do you have 1, 2, 5, 20 devices sharing the connection? Are any of THEM uploading, or downloading at the same time? Can you disconnect everything from the network except one test machine and try to reproduce the problem? Did you verify the machine being used to test is free of malware and rootkits?)

- what devices are you attempting to render HD content streams on, and are those devices themselves having performance issues

- what are your modem signals when you are having these issues?

Comcast has many millions of users. If they were having regular widespread CDN peering issues, we'd see millions of users complaining about it.

Your problem might be one of local congestion on your connection, or even a signal/drop issue. Jumping directly to a local node, neighborhood, or CDN peering congestion issue seems to be sidestepping much more likely problems
--
My place : »www.schettino.us

johnnn

join:2007-01-25
Ypsilanti, MI

Thanks, JohnInSJ, here goes:

-Blast (25/4)
-I have a Motorola SB 6121; 4 downstream, 3 upstream channels bonded.
-I have all cat5e cable between devices (modem-router, router-computers). The router is a TP-Link TL-WR2543ND with 4 gigabit ports.
-I am not loading the connection when I perform tests
-I have run tests with multiple computers hooked directly to the modem
-I have had two tech visits to verify good line quality (both found a very stable plant/no issues using their JDSU equipment)
-All of my computers are free of malware and experience little/no resource contention during testing. I have tested on Windows 7, Windows 8, Ubuntu 12.04.
-I have asked pointed questions about congestion in my area in Comcast Direct ("can you confirm there is congestion between endpoints in my area and certain peering/transit links?") and received nonspecific answers which refrain from disconfirming.



JohnInSJ
Premium
join:2003-09-22
Aptos, CA

I assume when the techs visited (as in when they were there testing your line/modem signals) you were NOT seeing the issue?

Comcast doesn't generally have congestion issues anywhere, in the modern era. This is is why I would think this is something other than congestion - if its intermittent, does it vary in time of day, day of week?
--
My place : »www.schettino.us



NetFixer
Freedom is NOT Free
Premium
join:2004-06-24
The Boro
Reviews:
·Cingular Wireless
·Comcast Business..
·Vonage

1 recommendation

reply to johnnn

I am not going to try to convince you that you are not seeing problems with some sites/services, but what you seem to expect Comcast to do is not going to happen for a residential or small business customer; they are only going to look at the results for the majority of similar customers.

The only way you are going to get any ISP to even bother to look at transit problems that are outside their network (especially if most of their other customers are not seeing/reporting the same problem) is if you are an Enterprise level customer (such as a customer with multiple Metro Ethernet or DS3 connections paying at least 5 digit amounts each month for service).

Sorry, but that is just the way that the industry works.
--
A well-regulated militia, being necessary to the security of a free State, the right of the people to keep and bear arms shall not be infringed.

When governments fear people, there is liberty. When the people fear the government, there is tyranny.


johnnn

join:2007-01-25
Ypsilanti, MI
reply to JohnInSJ

Both visited during the day on weekdays. They didn't perform tests past basic speedtests to the Comcast server near Detroit. The throughput problem is definitely exacerbated during peak times. Recently it was so bad I couldn't stream a 360p YouTube video from beginning to end without pauses (in video info, stream fluctuates between 200-900kbps).

The reason I believe congestion (at some level) to be the issue is this: I could close the page, retry, get the same poor performance, but then run a shaperprobe and get 36mbps down (shaped to 25mbps after Powerboost tokens spent) from whichever servers host the Shaperprobe test files.

NetFixer, I agree with your assessment.



JohnInSJ
Premium
join:2003-09-22
Aptos, CA

said by johnnn:

Both visited during the day on weekdays. They didn't perform tests past basic speedtests to the Comcast server near Detroit. The throughput problem is definitely exacerbated during peak times. Recently it was so bad I couldn't stream a 360p YouTube video from beginning to end without pauses (in video info, stream fluctuates between 200-900kbps).

The reason I believe congestion (at some level) to be the issue is this: I could close the page, retry, get the same poor performance, but then run a shaperprobe and get 36mbps down (shaped to 25mbps after Powerboost tokens spent) from whichever servers host the Shaperprobe test files.

NetFixer, I agree with your assessment.

Are you using Comcast DNS or some other DNS? It's possible you're not getting content from the closest CDN connection point if you're using a non-comcast DNS.
--
My place : »www.schettino.us


depster

join:2001-06-07
united state
reply to johnnn

Is your router capable of IPv6? If so try disabling that capability. I have a netgear router that was in auto mode, and was in 6to4 tunnel mode. Caused my streaming of netflix and youtube to be terrible.


johnnn

join:2007-01-25
Ypsilanti, MI
reply to JohnInSJ

I'm using anycast (75.75.75.75), and my router doesn't support IPv6.



depster

join:2001-06-07
united state

It may be that your local router is being upgraded... in your first post Hop 3 indicated a 'te' interface.

When I just tracerouted to it now, it's showing as a 'xe' interface.

  3    20 ms    10 ms    10 ms  96.120.41.137
  4    12 ms     7 ms     7 ms  xe-8-2-0-sur01.grandrapids.mi.michigan.comcast.net [68.86.140.185]
  5    17 ms    19 ms    27 ms  te-0-2-0-4-ar01.taylor.mi.michigan.comcast.net [69.139.254.29]
  6    18 ms    17 ms    19 ms  xe-7-3-0-sur01.wannarbor.mi.michigan.comcast.net [68.85.218.201]
 


DrDrew
That others may surf
Premium
join:2009-01-28
SoCal
kudos:15

said by depster:

It may be that your local router is being upgraded... in your first post Hop 3 indicated a 'te' interface.

When I just tracerouted to it now, it's showing as a 'xe' interface.

  3    20 ms    10 ms    10 ms  96.120.41.137
  4    12 ms     7 ms     7 ms  xe-8-2-0-sur01.grandrapids.mi.michigan.comcast.net [68.86.140.185]
  5    17 ms    19 ms    27 ms  te-0-2-0-4-ar01.taylor.mi.michigan.comcast.net [69.139.254.29]
  6    18 ms    17 ms    19 ms  xe-7-3-0-sur01.wannarbor.mi.michigan.comcast.net [68.85.218.201]
 

It's a different interface of the same router. OP's being from the edge of the network inbound, yours being from the core of the network outbound.
--
Two is one, one is none. If it's important, back it up... Somethimes 99.999% availability isn't even good enough.

johnnn

join:2007-01-25
Ypsilanti, MI

DrDrew is right. When I do tracerts *to* the edge they travel through an xe interface.



JohnInSJ
Premium
join:2003-09-22
Aptos, CA
reply to johnnn

said by johnnn:

I'm using anycast (75.75.75.75), and my router doesn't support IPv6.

Try using comcast's DNS. Just for grins.
--
My place : »www.schettino.us


EG
The wings of love
Premium
join:2006-11-18
Union, NJ
kudos:10

1 edit

3 recommendations

reply to depster

FWIW, just a point. As Dr.Drew stated, it's the same physical router. This data needs to also be examined when evaluating the rDNS info on hops;

te-8-2-ur02
xe-7-0-sur01

Even though the IP address is the same, the slot, port, and ring data is different.

Some examples of some meanings;

Some examples;

ge = gigabit ethernet.

te / xe= ten gigabit ethernet.

p, po, pos = Packet Over Sonet (an OC-3 or faster connection).

XE / TE = Juniper / Cisco 10GigE Interface
be / ae = Cisco Bundle Ethernet / Juniper Aggregate Ethernet

he = hundred gigabit ethernet.

1-1, 1-2, 1-11, 6-1, 9-1, 2-2, etc. = slot and port.

ar01, cr01, ur01

User Ring (UR), Area Ring (AR), Core Ring (CR) routing.


johnnn

join:2007-01-25
Ypsilanti, MI

EG, you're the best.



EG
The wings of love
Premium
join:2006-11-18
Union, NJ
kudos:10

I doubt that but thanks for your kind words !



NathanO

join:2008-08-21
Moorestown, NJ
reply to JohnInSJ

That is Comcast's anycast DNS



whfsdude
Premium
join:2003-04-05
Washington, DC
Reviews:
·Comcast
reply to EG

said by EG:

User Ring (UR), Area Ring (AR), Core Ring (CR) routing.

The one I've never understood is "SUR." I have yet to see this interface name used outside of Comcast. Happen to know?

johnnn

join:2007-01-25
Ypsilanti, MI

Click for full size
New interface on the
Click for full size
KS route
Click for full size
CA route
Click for full size
Peak time comparison
There appear to be significant changes underway. One of the routers hostnames has reappeared, and I'm on a different interface. A hop prior to Pontiac has been eliminated from the previous configuration.

There was likely some sort of maintenance window on Friday morning as evidenced by the smokeping graphs attached (this makes sense, as it corresponds with the lowest utilization period for the college town I'm in). Comparing the ICMP performance for peak windows between this week and last, there's been a significant improvement.


JohnInSJ
Premium
join:2003-09-22
Aptos, CA
reply to NathanO

said by NathanO:

That is Comcast's anycast DNS

Perhaps using one from here »www.whatsmydns.net/dns/usa/comcast.html

Michigan

Primary DNS: 68.87.77.130

Secondary DNS: 68.87.72.130

Primary IPv6 DNS: 2001:558:1016:C:68:87:77:130

Secondary IPv6 DNS: 2001:558:100E:4:68:87:72:130

Would yield different CDN entry points?
--
My place : »www.schettino.us

grimmwit

join:2010-01-19
New Boston, NH
reply to johnnn

I didn't go through the entire thread thoroughly, but wanted to add an idea.....

I had the same beef with my 105/20 service. I eventually figured out that the cause was using DNS servers other than Comcasts. This resulted in me being connected to the wrong CDN sites. I switched to the Comcast DNS servers being offered to me via DHCP, and my download speeds jumped dramatically. So if you are using alternate DNS, consider giving your primary/secondary Comcast DNS servers a try.