Topic '[Speed] Congested Comcast Peering with Akamai, Google in Midwest' in forum 'Comcast HSI' - dslreports.com

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Tue, 29 Jan 2013 10:31:06 EDT

johnnn posted : So based on the info in this thread, the changes in throughput (things are much improved), and the routing changes I've observed, it looks like Comcast hooked my neighborhood's CMTS (and likely all of wannarbor) into a Super UR whereas before it routed through two regular 10GigE URs on its way to the AR in Pontiac.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Sun, 27 Jan 2013 19:31:55 EDT

makaze posted :

said by depster:

said by whfsdude:

said by EG:

User Ring (UR), Area Ring (AR), Core Ring (CR) routing.

The one I've never understood is "SUR." I have yet to see this interface name used outside of Comcast. Happen to know?

Maybe Switched User Ring?

Nah it's a Super UR. In most areas it's a juniper with multiple AR connections for redundancy.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Sun, 27 Jan 2013 14:25:11 EDT

NetFixer posted :

said by JohnInSJ:

said by nate1234:

That is Comcast's anycast DNS

Perhaps using one from here »www.whatsmydns.net/dns/usa/comcast.html

...

Would yield different CDN entry points?

That is exactly what I do for the forwarding servers in my in-house DNS server. I chose the primary servers for Atlanta and Chicago because most of my routes go through either Atlanta or Chicago.

[att=1]

C:\>dig -x 68.87.68.162 ; <<>> DiG 9.9.2 <<>> -x 68.87.68.162;; global options: +cmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62426;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION:; EDNS: version: 0, flags:; udp: 1280;; QUESTION SECTION:;162.68.87.68.in-addr.arpa.     IN      PTR ;; ANSWER SECTION:162.68.87.68.in-addr.arpa. 1196 IN      PTR     nrcns.s3woodstock.ga.atlanta.comcast.net. ;; Query time: 31 msec;; SERVER: 192.168.9.2#53(192.168.9.2);; WHEN: Sun Jan 27 13:11:54 2013;; MSG SIZE  rcvd: 108 C:\>dig -x 68.87.72.130 ; <<>> DiG 9.9.2 <<>> -x 68.87.72.130;; global options: +cmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26004;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION:; EDNS: version: 0, flags:; udp: 1280;; QUESTION SECTION:;130.72.87.68.in-addr.arpa.     IN      PTR ;; ANSWER SECTION:130.72.87.68.in-addr.arpa. 6112 IN      PTR     nrcns.area4.il.chicago.comcast.net.130.72.87.68.in-addr.arpa. 6112 IN      PTR     chic-cns.area4.il.chicago.comcast.net. ;; Query time: 15 msec;; SERVER: 192.168.9.2#53(192.168.9.2);; WHEN: Sun Jan 27 13:12:24 2013;; MSG SIZE  rcvd: 125

--
A well-regulated militia, being necessary to the security of a free State, the right of the people to keep and bear arms shall not be infringed.

When governments fear people, there is liberty. When the people fear the government, there is tyranny.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Sat, 26 Jan 2013 21:27:08 EDT

depster posted :

said by whfsdude:

said by EG:

User Ring (UR), Area Ring (AR), Core Ring (CR) routing.

The one I've never understood is "SUR." I have yet to see this interface name used outside of Comcast. Happen to know?

Maybe Switched User Ring?

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Sat, 26 Jan 2013 19:41:04 EDT

ArrayList posted :

said by EG:

ge = gigabit ethernet.

te / xe= ten gigabit ethernet.

p, po, pos = Packet Over Sonet (an OC-3 or faster connection).

XE / TE = Juniper / Cisco 10GigE Interface
be / ae = Cisco Bundle Ethernet / Juniper Aggregate Ethernet

he = hundred gigabit ethernet.

1-1, 1-2, 1-11, 6-1, 9-1, 2-2, etc. = slot and port.

ar01, cr01, ur01

User Ring (UR), Area Ring (AR), Core Ring (CR) routing.

holy cow man. Thanks for this information. I know it's pretty trivial for non-comcast network engineers, but it really helps add some light into where my packets go when they leave town. :)

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Sat, 26 Jan 2013 19:12:40 EDT

grimmwit posted : I didn't go through the entire thread thoroughly, but wanted to add an idea.....

I had the same beef with my 105/20 service. I eventually figured out that the cause was using DNS servers other than Comcasts. This resulted in me being connected to the wrong CDN sites. I switched to the Comcast DNS servers being offered to me via DHCP, and my download speeds jumped dramatically. So if you are using alternate DNS, consider giving your primary/secondary Comcast DNS servers a try.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Sat, 26 Jan 2013 13:41:28 EDT

JohnInSJ posted :

said by nate1234:

That is Comcast's anycast DNS

Perhaps using one from here »www.whatsmydns.net/dns/usa/comcast.html

Michigan

Primary DNS: 68.87.77.130

Secondary DNS: 68.87.72.130

Primary IPv6 DNS: 2001:558:1016:C:68:87:77:130

Secondary IPv6 DNS: 2001:558:100E:4:68:87:72:130

Would yield different CDN entry points?
--
My place : »www.schettino.us

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Sat, 26 Jan 2013 13:36:27 EDT

johnnn posted : There appear to be significant changes underway. One of the routers hostnames has reappeared, and I'm on a different interface. A hop prior to Pontiac has been eliminated from the previous configuration.

There was likely some sort of maintenance window on Friday morning as evidenced by the smokeping graphs attached (this makes sense, as it corresponds with the lowest utilization period for the college town I'm in). Comparing the ICMP performance for peak windows between this week and last, there's been a significant improvement.

New interface on the
KS route
CA route
Peak time comparison

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Sat, 26 Jan 2013 11:45:37 EDT

whfsdude posted :

said by EG:

User Ring (UR), Area Ring (AR), Core Ring (CR) routing.

The one I've never understood is "SUR." I have yet to see this interface name used outside of Comcast. Happen to know?

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Sat, 26 Jan 2013 11:21:53 EDT

nate1234 posted : That is Comcast's anycast DNS

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Fri, 25 Jan 2013 16:09:14 EDT

EG posted : I doubt that but thanks for your kind words ! :)

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Fri, 25 Jan 2013 16:06:51 EDT

johnnn posted : EG, you're the best.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Fri, 25 Jan 2013 16:02:07 EDT

EG posted : FWIW, just a point. As Dr.Drew stated, it's the same physical router. This data needs to also be examined when evaluating the rDNS info on hops;

te-8-2-ur02
xe-7-0-sur01

Even though the IP address is the same, the slot, port, and ring data is different.

Some examples of some meanings;

Some examples;

ge = gigabit ethernet.

te / xe= ten gigabit ethernet.

p, po, pos = Packet Over Sonet (an OC-3 or faster connection).

XE / TE = Juniper / Cisco 10GigE Interface
be / ae = Cisco Bundle Ethernet / Juniper Aggregate Ethernet

he = hundred gigabit ethernet.

1-1, 1-2, 1-11, 6-1, 9-1, 2-2, etc. = slot and port.

ar01, cr01, ur01

User Ring (UR), Area Ring (AR), Core Ring (CR) routing.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Fri, 25 Jan 2013 15:52:53 EDT

JohnInSJ posted :

said by johnnn:

I'm using anycast (75.75.75.75), and my router doesn't support IPv6.

Try using comcast's DNS. Just for grins.
--
My place : »www.schettino.us

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Fri, 25 Jan 2013 15:32:21 EDT

johnnn posted : DrDrew is right. When I do tracerts *to* the edge they travel through an xe interface.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Fri, 25 Jan 2013 15:17:40 EDT

DrDrew posted :

said by depster:

It may be that your local router is being upgraded... in your first post Hop 3 indicated a 'te' interface.

When I just tracerouted to it now, it's showing as a 'xe' interface.

  3    20 ms    10 ms    10 ms  96.120.41.137  4    12 ms     7 ms     7 ms  xe-8-2-0-sur01.grandrapids.mi.michigan.comcast.net [68.86.140.185]  5    17 ms    19 ms    27 ms  te-0-2-0-4-ar01.taylor.mi.michigan.comcast.net [69.139.254.29]  6    18 ms    17 ms    19 ms  xe-7-3-0-sur01.wannarbor.mi.michigan.comcast.net [68.85.218.201]

It's a different interface of the same router. OP's being from the edge of the network inbound, yours being from the core of the network outbound.
--
Two is one, one is none. If it's important, back it up... Somethimes 99.999% availability isn't even good enough.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Fri, 25 Jan 2013 15:07:16 EDT

depster posted : It may be that your local router is being upgraded... in your first post Hop 3 indicated a 'te' interface.

When I just tracerouted to it now, it's showing as a 'xe' interface.

  3    20 ms    10 ms    10 ms  96.120.41.137  4    12 ms     7 ms     7 ms  xe-8-2-0-sur01.grandrapids.mi.michigan.comcast.net [68.86.140.185]  5    17 ms    19 ms    27 ms  te-0-2-0-4-ar01.taylor.mi.michigan.comcast.net [69.139.254.29]  6    18 ms    17 ms    19 ms  xe-7-3-0-sur01.wannarbor.mi.michigan.comcast.net [68.85.218.201]

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Fri, 25 Jan 2013 14:05:33 EDT

johnnn posted : I'm using anycast (75.75.75.75), and my router doesn't support IPv6.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Thu, 24 Jan 2013 20:33:24 EDT

depster posted : Is your router capable of IPv6? If so try disabling that capability. I have a netgear router that was in auto mode, and was in 6to4 tunnel mode. Caused my streaming of netflix and youtube to be terrible.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Thu, 24 Jan 2013 19:16:55 EDT

JohnInSJ posted :

said by johnnn:

Both visited during the day on weekdays. They didn't perform tests past basic speedtests to the Comcast server near Detroit. The throughput problem is definitely exacerbated during peak times. Recently it was so bad I couldn't stream a 360p YouTube video from beginning to end without pauses (in video info, stream fluctuates between 200-900kbps).

The reason I believe congestion (at some level) to be the issue is this: I could close the page, retry, get the same poor performance, but then run a shaperprobe and get 36mbps down (shaped to 25mbps after Powerboost tokens spent) from whichever servers host the Shaperprobe test files.

NetFixer, I agree with your assessment.

Are you using Comcast DNS or some other DNS? It's possible you're not getting content from the closest CDN connection point if you're using a non-comcast DNS.
--
My place : »www.schettino.us

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Thu, 24 Jan 2013 18:18:21 EDT

johnnn posted : Both visited during the day on weekdays. They didn't perform tests past basic speedtests to the Comcast server near Detroit. The throughput problem is definitely exacerbated during peak times. Recently it was so bad I couldn't stream a 360p YouTube video from beginning to end without pauses (in video info, stream fluctuates between 200-900kbps).

The reason I believe congestion (at some level) to be the issue is this: I could close the page, retry, get the same poor performance, but then run a shaperprobe and get 36mbps down (shaped to 25mbps after Powerboost tokens spent) from whichever servers host the Shaperprobe test files.

NetFixer, I agree with your assessment.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Thu, 24 Jan 2013 18:11:21 EDT

NetFixer posted : I am not going to try to convince you that you are not seeing problems with some sites/services, but what you seem to expect Comcast to do is not going to happen for a residential or small business customer; they are only going to look at the results for the majority of similar customers.

The only way you are going to get any ISP to even bother to look at transit problems that are outside their network (especially if most of their other customers are not seeing/reporting the same problem) is if you are an Enterprise level customer (such as a customer with multiple Metro Ethernet or DS3 connections paying at least 5 digit amounts each month for service).

Sorry, but that is just the way that the industry works.
--
A well-regulated militia, being necessary to the security of a free State, the right of the people to keep and bear arms shall not be infringed.

When governments fear people, there is liberty. When the people fear the government, there is tyranny.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Thu, 24 Jan 2013 18:10:57 EDT

JohnInSJ posted : I assume when the techs visited (as in when they were there testing your line/modem signals) you were NOT seeing the issue?

Comcast doesn't generally have congestion issues anywhere, in the modern era. This is is why I would think this is something other than congestion - if its intermittent, does it vary in time of day, day of week?
--
My place : »www.schettino.us

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Thu, 24 Jan 2013 17:23:09 EDT

johnnn posted : Thanks, JohnInSJ, here goes:

-Blast (25/4)
-I have a Motorola SB 6121; 4 downstream, 3 upstream channels bonded.
-I have all cat5e cable between devices (modem-router, router-computers). The router is a TP-Link TL-WR2543ND with 4 gigabit ports.
-I am not loading the connection when I perform tests
-I have run tests with multiple computers hooked directly to the modem
-I have had two tech visits to verify good line quality (both found a very stable plant/no issues using their JDSU equipment)
-All of my computers are free of malware and experience little/no resource contention during testing. I have tested on Windows 7, Windows 8, Ubuntu 12.04.
-I have asked pointed questions about congestion in my area in Comcast Direct ("can you confirm there is congestion between endpoints in my area and certain peering/transit links?") and received nonspecific answers which refrain from disconfirming.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Thu, 24 Jan 2013 16:35:43 EDT

JohnInSJ posted :

said by johnnn:

various observations

But there is no way to verify your observations.

Let's start with some actual facts

- what tier are you on

- what is your home network like

- what load are you putting on your connection (are you uploading while attempting these downloads, for example? I don't care what you're uploading - just tell me if you're using x% of your uplink... do you have 1, 2, 5, 20 devices sharing the connection? Are any of THEM uploading, or downloading at the same time? Can you disconnect everything from the network except one test machine and try to reproduce the problem? Did you verify the machine being used to test is free of malware and rootkits?)

- what devices are you attempting to render HD content streams on, and are those devices themselves having performance issues

- what are your modem signals when you are having these issues?

Comcast has many millions of users. If they were having regular widespread CDN peering issues, we'd see millions of users complaining about it.

Your problem might be one of local congestion on your connection, or even a signal/drop issue. Jumping directly to a local node, neighborhood, or CDN peering congestion issue seems to be sidestepping much more likely problems
--
My place : »www.schettino.us

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Thu, 24 Jan 2013 16:17:02 EDT

johnnn posted : I'm well aware of how residential ISPs view their customers. Isn't this the place we're supposed to talk about it and be heard?

If we are to effect change in an environment where the regulatory and business cards are stacked against us, why would we stay silent? If mountainview would rather not effect change, s/he might be better off working for Comcast. You know, they do pay people to toe the corporate line.

Finally, how are my snapshots irrelevant? They illustrate a problem I know from experience to be systemic. I don't have a reason to misrepresent what I'm posting, and I have knowledge of the network architecture. If mountainview would rather not hear about the condition of the residential broadband industry from the viewpoint of the consumer, s/he may want to find another forum to troll, as this one seems dedicated to that exact purpose.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Thu, 24 Jan 2013 16:11:29 EDT

mountainview posted : Dunno what the OP is complaining about..Peers/Transit providers can of course deprioritize certain traffic, ISP has QoS too...
What your snapshots are mostly irrelevant to the issue.. And most likely you can't do much about.... Not satisfied? Get another ISP....
From what I have seen, as a residential customer, if your speedtest and ping looks fine to your ISP's test server, in their eyes you are just making some nonsense....

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Thu, 24 Jan 2013 09:05:37 EDT

johnnn posted : I addressed the issue with documenting this type of congestion in an earlier post: "The trouble with this kind of selective congestion: I can't easily tell if Comcast is running hot ports (something that is often alleged and sometimes backed up, like in the Tata case from awhile back) or if the CDN is performing poorly."

This is a phenomenon I've observed over the life of my ~2 years of Comcast service in my location. The congestion manifests itself in widely variable (but unusably low average) transfer bitrates. I know it's very difficult to document, which is why I'm bringing it up on this forum. I would like better tools (like the Explicit Congestion Notification RFC from 2008), but I don't have them.

The major CDNs that seems to be an issue are Akamai and Google (also mentioned previously in this thread), but sometimes Limelight gives me issues too. HD video streams from NPR music (which use Akamai or Limelight for live broadcasts) have historically been difficult to maintain, and YouTube's 1080p video library is almost never watchable. Recently, I've even seen sporadic trouble with Netflix when it is delivered through Level 3.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Thu, 24 Jan 2013 08:58:42 EDT

JohnInSJ posted :

said by johnnn:

In Michigan, I'm experiencing persistent congestion even to the largest of CDNs

Which CDNs. How does "persistent congestion" manifest? What you're attempting to document on your endpoint isn't really possible to document - you have no way of knowing what, if any, point between you and the CDN, or the CDN itself, is limiting the flow of data.

Your only named provider in your post is Nefflix, and you say they have no problem maintaining a 3+mbit stream to you.

So, which CDNs are not? When?
--
My place : »www.schettino.us

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Thu, 24 Jan 2013 08:49:12 EDT

johnnn posted : Please read the OP. The tracert looks just fine. It's the throughput that's the problem.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Thu, 24 Jan 2013 01:33:30 EDT

mountainview posted : I am so tired of people saying something on the internet is wrong because you think your ping/traceroute does not look good.

If it's not LAN or a network you completely control, don't even bother... A route can just ignore or deprioritize icmp...
with multiplication involved, GSLB (CDN) your traceroute is pretty much worthless as the path is very likely to change in 60s if not fasters...

If you are not a network operator, you don't need to complaint about someone other than your ISP because you have no control over them and they probably don't give a fark about you....

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Sun, 02 Dec 2012 19:23:05 EDT

Sofa King posted : If there were a problem in the network portion you would probably see latency/loss starting at a hop AND CONTINUING all the way to the destination. Seeing a problem at one hop that does not continue is not a problem and seeing no issues along the network path tends to implicate the server you are talking to.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Sun, 02 Dec 2012 18:59:35 EDT

johnnn posted : I would be happy to run pingplotter for longer to said servers, but ICMP latency isn't the issue; throughput is. Do you think the CDNs in question could be the issue?

The trouble with this kind of selective congestion: I can't easily tell if Comcast is running hot ports (something that is often alleged and sometimes backed up, like in the Tata case from awhile back) or if the CDN is performing poorly.

I'm having trouble with providers that *aren't* serving me Netflix (that's always Level3). I'm not a particularly savvy user so I don't know of ways to gather conclusive data that would support my conclusion. Help would be gratefully received, because from what I've observed, there's an issue.

Re: [Speed] Congested Comcast Peering with Akamai, Google in Mid

Sun, 02 Dec 2012 18:35:46 EDT

Sofa King posted : Can't see any information in the data that supports your conclusion. Can you run mtr or pingplotter with a larger dataset to the servers in question.

Also note Netflix uses Akamai as well as other CDNs and publishes their performance statistics to Comcast and other ISPs

[Speed] Congested Comcast Peering with Akamai, Google in Midwest

Sun, 02 Dec 2012 17:02:55 EDT

johnnn posted : In Michigan, I'm experiencing persistent congestion even to the largest of CDNs...but every time I do a Speedtest.net or Shaperprobe test, I have no problem maxing out my provisioned rates (Blast 25/4). Attached are screenshots of traceroutes to each directly-connected host, a screenshot of connection utilization to the Akamai host (Google looked similar), and for reference a speedtest (»www.speedtest.net/result/2346965994.png) to one of Comcast's own servers in Chicago.

This is not a new problem. I haven't been able to get sufficient bandwidth for any high-quality video service except Netflix. Does anyone else have this issue?

Tracert to Akamai host
Data transfer plot from Chicago-based Comcast server
Data transfer plot from Akamai host
Tracert to Google (YouTube) host