dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
3138
johnnn
join:2007-01-25
Ann Arbor, MI

johnnn

Member

[Speed] Congested Comcast Peering with Akamai, Google in Midwest


Tracert to Akamai host
Click for full size
Data transfer plot from Chicago-based Comcast server
Click for full size
Data transfer plot from Akamai host

Tracert to Google (YouTube) host
In Michigan, I'm experiencing persistent congestion even to the largest of CDNs...but every time I do a Speedtest.net or Shaperprobe test, I have no problem maxing out my provisioned rates (Blast 25/4). Attached are screenshots of traceroutes to each directly-connected host, a screenshot of connection utilization to the Akamai host (Google looked similar), and for reference a speedtest (»www.speedtest.net/result ··· 5994.png) to one of Comcast's own servers in Chicago.

This is not a new problem. I haven't been able to get sufficient bandwidth for any high-quality video service except Netflix. Does anyone else have this issue?
28619103 (banned)
join:2009-03-01
21435

28619103 (banned)

Member

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Can't see any information in the data that supports your conclusion. Can you run mtr or pingplotter with a larger dataset to the servers in question.

Also note Netflix uses Akamai as well as other CDNs and publishes their performance statistics to Comcast and other ISPs
johnnn
join:2007-01-25
Ann Arbor, MI

johnnn

Member

I would be happy to run pingplotter for longer to said servers, but ICMP latency isn't the issue; throughput is. Do you think the CDNs in question could be the issue?

The trouble with this kind of selective congestion: I can't easily tell if Comcast is running hot ports (something that is often alleged and sometimes backed up, like in the Tata case from awhile back) or if the CDN is performing poorly.

I'm having trouble with providers that *aren't* serving me Netflix (that's always Level3). I'm not a particularly savvy user so I don't know of ways to gather conclusive data that would support my conclusion. Help would be gratefully received, because from what I've observed, there's an issue.
28619103 (banned)
join:2009-03-01
21435

1 recommendation

28619103 (banned)

Member

If there were a problem in the network portion you would probably see latency/loss starting at a hop AND CONTINUING all the way to the destination. Seeing a problem at one hop that does not continue is not a problem and seeing no issues along the network path tends to implicate the server you are talking to.
mountainview
join:2011-09-07
united state

mountainview to johnnn

Member

to johnnn
I am so tired of people saying something on the internet is wrong because you think your ping/traceroute does not look good.

If it's not LAN or a network you completely control, don't even bother... A route can just ignore or deprioritize icmp...
with multiplication involved, GSLB (CDN) your traceroute is pretty much worthless as the path is very likely to change in 60s if not fasters...

If you are not a network operator, you don't need to complaint about someone other than your ISP because you have no control over them and they probably don't give a fark about you....
johnnn
join:2007-01-25
Ann Arbor, MI

johnnn

Member

Please read the OP. The tracert looks just fine. It's the throughput that's the problem.

JohnInSJ
Premium Member
join:2003-09-22
Aptos, CA

JohnInSJ to johnnn

Premium Member

to johnnn
said by johnnn:

In Michigan, I'm experiencing persistent congestion even to the largest of CDNs

Which CDNs. How does "persistent congestion" manifest? What you're attempting to document on your endpoint isn't really possible to document - you have no way of knowing what, if any, point between you and the CDN, or the CDN itself, is limiting the flow of data.

Your only named provider in your post is Nefflix, and you say they have no problem maintaining a 3+mbit stream to you.

So, which CDNs are not? When?
johnnn
join:2007-01-25
Ann Arbor, MI

johnnn

Member

I addressed the issue with documenting this type of congestion in an earlier post: "The trouble with this kind of selective congestion: I can't easily tell if Comcast is running hot ports (something that is often alleged and sometimes backed up, like in the Tata case from awhile back) or if the CDN is performing poorly."

This is a phenomenon I've observed over the life of my ~2 years of Comcast service in my location. The congestion manifests itself in widely variable (but unusably low average) transfer bitrates. I know it's very difficult to document, which is why I'm bringing it up on this forum. I would like better tools (like the Explicit Congestion Notification RFC from 2008), but I don't have them.

The major CDNs that seems to be an issue are Akamai and Google (also mentioned previously in this thread), but sometimes Limelight gives me issues too. HD video streams from NPR music (which use Akamai or Limelight for live broadcasts) have historically been difficult to maintain, and YouTube's 1080p video library is almost never watchable. Recently, I've even seen sporadic trouble with Netflix when it is delivered through Level 3.
mountainview
join:2011-09-07
united state

mountainview to johnnn

Member

to johnnn
Dunno what the OP is complaining about..Peers/Transit providers can of course deprioritize certain traffic, ISP has QoS too...
What your snapshots are mostly irrelevant to the issue.. And most likely you can't do much about.... Not satisfied? Get another ISP....
From what I have seen, as a residential customer, if your speedtest and ping looks fine to your ISP's test server, in their eyes you are just making some nonsense....
johnnn
join:2007-01-25
Ann Arbor, MI

1 recommendation

johnnn

Member

I'm well aware of how residential ISPs view their customers. Isn't this the place we're supposed to talk about it and be heard?

If we are to effect change in an environment where the regulatory and business cards are stacked against us, why would we stay silent? If mountainview would rather not effect change, s/he might be better off working for Comcast. You know, they do pay people to toe the corporate line.

Finally, how are my snapshots irrelevant? They illustrate a problem I know from experience to be systemic. I don't have a reason to misrepresent what I'm posting, and I have knowledge of the network architecture. If mountainview would rather not hear about the condition of the residential broadband industry from the viewpoint of the consumer, s/he may want to find another forum to troll, as this one seems dedicated to that exact purpose.

JohnInSJ
Premium Member
join:2003-09-22
Aptos, CA

1 recommendation

JohnInSJ to johnnn

Premium Member

to johnnn
said by johnnn:

various observations

But there is no way to verify your observations.

Let's start with some actual facts

- what tier are you on

- what is your home network like

- what load are you putting on your connection (are you uploading while attempting these downloads, for example? I don't care what you're uploading - just tell me if you're using x% of your uplink... do you have 1, 2, 5, 20 devices sharing the connection? Are any of THEM uploading, or downloading at the same time? Can you disconnect everything from the network except one test machine and try to reproduce the problem? Did you verify the machine being used to test is free of malware and rootkits?)

- what devices are you attempting to render HD content streams on, and are those devices themselves having performance issues

- what are your modem signals when you are having these issues?

Comcast has many millions of users. If they were having regular widespread CDN peering issues, we'd see millions of users complaining about it.

Your problem might be one of local congestion on your connection, or even a signal/drop issue. Jumping directly to a local node, neighborhood, or CDN peering congestion issue seems to be sidestepping much more likely problems
johnnn
join:2007-01-25
Ann Arbor, MI

johnnn

Member

Thanks, JohnInSJ, here goes:

-Blast (25/4)
-I have a Motorola SB 6121; 4 downstream, 3 upstream channels bonded.
-I have all cat5e cable between devices (modem-router, router-computers). The router is a TP-Link TL-WR2543ND with 4 gigabit ports.
-I am not loading the connection when I perform tests
-I have run tests with multiple computers hooked directly to the modem
-I have had two tech visits to verify good line quality (both found a very stable plant/no issues using their JDSU equipment)
-All of my computers are free of malware and experience little/no resource contention during testing. I have tested on Windows 7, Windows 8, Ubuntu 12.04.
-I have asked pointed questions about congestion in my area in Comcast Direct ("can you confirm there is congestion between endpoints in my area and certain peering/transit links?") and received nonspecific answers which refrain from disconfirming.

JohnInSJ
Premium Member
join:2003-09-22
Aptos, CA

JohnInSJ

Premium Member

I assume when the techs visited (as in when they were there testing your line/modem signals) you were NOT seeing the issue?

Comcast doesn't generally have congestion issues anywhere, in the modern era. This is is why I would think this is something other than congestion - if its intermittent, does it vary in time of day, day of week?

NetFixer
From My Cold Dead Hands
Premium Member
join:2004-06-24
The Boro
Netgear CM500
Pace 5268AC
TRENDnet TEW-829DRU

1 recommendation

NetFixer to johnnn

Premium Member

to johnnn
I am not going to try to convince you that you are not seeing problems with some sites/services, but what you seem to expect Comcast to do is not going to happen for a residential or small business customer; they are only going to look at the results for the majority of similar customers.

The only way you are going to get any ISP to even bother to look at transit problems that are outside their network (especially if most of their other customers are not seeing/reporting the same problem) is if you are an Enterprise level customer (such as a customer with multiple Metro Ethernet or DS3 connections paying at least 5 digit amounts each month for service).

Sorry, but that is just the way that the industry works.
johnnn
join:2007-01-25
Ann Arbor, MI

johnnn to JohnInSJ

Member

to JohnInSJ
Both visited during the day on weekdays. They didn't perform tests past basic speedtests to the Comcast server near Detroit. The throughput problem is definitely exacerbated during peak times. Recently it was so bad I couldn't stream a 360p YouTube video from beginning to end without pauses (in video info, stream fluctuates between 200-900kbps).

The reason I believe congestion (at some level) to be the issue is this: I could close the page, retry, get the same poor performance, but then run a shaperprobe and get 36mbps down (shaped to 25mbps after Powerboost tokens spent) from whichever servers host the Shaperprobe test files.

NetFixer, I agree with your assessment.

JohnInSJ
Premium Member
join:2003-09-22
Aptos, CA

JohnInSJ

Premium Member

said by johnnn:

Both visited during the day on weekdays. They didn't perform tests past basic speedtests to the Comcast server near Detroit. The throughput problem is definitely exacerbated during peak times. Recently it was so bad I couldn't stream a 360p YouTube video from beginning to end without pauses (in video info, stream fluctuates between 200-900kbps).

The reason I believe congestion (at some level) to be the issue is this: I could close the page, retry, get the same poor performance, but then run a shaperprobe and get 36mbps down (shaped to 25mbps after Powerboost tokens spent) from whichever servers host the Shaperprobe test files.

NetFixer, I agree with your assessment.

Are you using Comcast DNS or some other DNS? It's possible you're not getting content from the closest CDN connection point if you're using a non-comcast DNS.

depster
join:2001-06-07
Grand Rapids, MI

depster to johnnn

Member

to johnnn
Is your router capable of IPv6? If so try disabling that capability. I have a netgear router that was in auto mode, and was in 6to4 tunnel mode. Caused my streaming of netflix and youtube to be terrible.
johnnn
join:2007-01-25
Ann Arbor, MI

johnnn to JohnInSJ

Member

to JohnInSJ
I'm using anycast (75.75.75.75), and my router doesn't support IPv6.

depster
join:2001-06-07
Grand Rapids, MI

depster

Member

It may be that your local router is being upgraded... in your first post Hop 3 indicated a 'te' interface.

When I just tracerouted to it now, it's showing as a 'xe' interface.
  3    20 ms    10 ms    10 ms  96.120.41.137
  4    12 ms     7 ms     7 ms  xe-8-2-0-sur01.grandrapids.mi.michigan.comcast.net [68.86.140.185]
  5    17 ms    19 ms    27 ms  te-0-2-0-4-ar01.taylor.mi.michigan.comcast.net [69.139.254.29]
  6    18 ms    17 ms    19 ms  xe-7-3-0-sur01.wannarbor.mi.michigan.comcast.net [68.85.218.201]
 

DocDrew
How can I help?
Premium Member
join:2009-01-28
SoCal
Ubee E31U2V1
Technicolor TC4400
Linksys EA6900

DocDrew

Premium Member

said by depster:

It may be that your local router is being upgraded... in your first post Hop 3 indicated a 'te' interface.

When I just tracerouted to it now, it's showing as a 'xe' interface.

  3    20 ms    10 ms    10 ms  96.120.41.137
  4    12 ms     7 ms     7 ms  xe-8-2-0-sur01.grandrapids.mi.michigan.comcast.net [68.86.140.185]
  5    17 ms    19 ms    27 ms  te-0-2-0-4-ar01.taylor.mi.michigan.comcast.net [69.139.254.29]
  6    18 ms    17 ms    19 ms  xe-7-3-0-sur01.wannarbor.mi.michigan.comcast.net [68.85.218.201]
 

It's a different interface of the same router. OP's being from the edge of the network inbound, yours being from the core of the network outbound.
johnnn
join:2007-01-25
Ann Arbor, MI

johnnn

Member

DrDrew is right. When I do tracerts *to* the edge they travel through an xe interface.

JohnInSJ
Premium Member
join:2003-09-22
Aptos, CA

JohnInSJ to johnnn

Premium Member

to johnnn
said by johnnn:

I'm using anycast (75.75.75.75), and my router doesn't support IPv6.

Try using comcast's DNS. Just for grins.

EG
The wings of love
Premium Member
join:2006-11-18
Union, NJ

1 edit

3 recommendations

EG to depster

Premium Member

to depster
FWIW, just a point. As Dr.Drew stated, it's the same physical router. This data needs to also be examined when evaluating the rDNS info on hops;

te-8-2-ur02
xe-7-0-sur01

Even though the IP address is the same, the slot, port, and ring data is different.

Some examples of some meanings;

Some examples;

ge = gigabit ethernet.

te / xe= ten gigabit ethernet.

p, po, pos = Packet Over Sonet (an OC-3 or faster connection).

XE / TE = Juniper / Cisco 10GigE Interface
be / ae = Cisco Bundle Ethernet / Juniper Aggregate Ethernet

he = hundred gigabit ethernet.

1-1, 1-2, 1-11, 6-1, 9-1, 2-2, etc. = slot and port.

ar01, cr01, ur01

User Ring (UR), Area Ring (AR), Core Ring (CR) routing.
johnnn
join:2007-01-25
Ann Arbor, MI

johnnn

Member

EG, you're the best.

EG
The wings of love
Premium Member
join:2006-11-18
Union, NJ

EG

Premium Member

I doubt that but thanks for your kind words !

NathanO
join:2008-08-21
Seattle, WA

NathanO to JohnInSJ

Member

to JohnInSJ
That is Comcast's anycast DNS

whfsdude
Premium Member
join:2003-04-05
Washington, DC

whfsdude to EG

Premium Member

to EG
said by EG:

User Ring (UR), Area Ring (AR), Core Ring (CR) routing.

The one I've never understood is "SUR." I have yet to see this interface name used outside of Comcast. Happen to know?
johnnn
join:2007-01-25
Ann Arbor, MI

johnnn

Member

Click for full size
New interface on the
Click for full size
KS route
Click for full size
CA route
Click for full size
Peak time comparison
There appear to be significant changes underway. One of the routers hostnames has reappeared, and I'm on a different interface. A hop prior to Pontiac has been eliminated from the previous configuration.

There was likely some sort of maintenance window on Friday morning as evidenced by the smokeping graphs attached (this makes sense, as it corresponds with the lowest utilization period for the college town I'm in). Comparing the ICMP performance for peak windows between this week and last, there's been a significant improvement.

JohnInSJ
Premium Member
join:2003-09-22
Aptos, CA

JohnInSJ to NathanO

Premium Member

to NathanO
said by NathanO:

That is Comcast's anycast DNS

Perhaps using one from here »www.whatsmydns.net/dns/u ··· ast.html

Michigan

Primary DNS: 68.87.77.130

Secondary DNS: 68.87.72.130

Primary IPv6 DNS: 2001:558:1016:C:68:87:77:130

Secondary IPv6 DNS: 2001:558:100E:4:68:87:72:130

Would yield different CDN entry points?
grimmwit
join:2010-01-19
Rio Rancho, NM

grimmwit to johnnn

Member

to johnnn
I didn't go through the entire thread thoroughly, but wanted to add an idea.....

I had the same beef with my 105/20 service. I eventually figured out that the cause was using DNS servers other than Comcasts. This resulted in me being connected to the wrong CDN sites. I switched to the Comcast DNS servers being offered to me via DHCP, and my download speeds jumped dramatically. So if you are using alternate DNS, consider giving your primary/secondary Comcast DNS servers a try.