Comcast Finally Launches DNS Redirection → Re: Options for Denver

The only gotcha to using options like Level(3)'s resolvers or OpenDNS is that it may affect the performance of geodisbursed services (ie, Google/Youtube, Akamai) .

The problem is that global load balancers will return DNS responses based on the IP of the server doing the query. If you use Level(3)'s servers, it will return a content engine that has the best path / bandwidth to Level(3)'s network. If you use Qwest's DNS servers, it will return the IP of a content engine that has the best route / bandwidth to Qwest.

For example, using Level(3)'s nearest server in Chicago:

traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 40 byte packets
 1  leviathan (192.168.0.1)  0.345 ms  0.319 ms  0.315 ms
 2  173-8-x-x-Minnesota.hfc.comcastbusiness.net (173.8.x.x)  0.785 ms  0.988 ms  1.721 ms
 3  73.205.28.1 (73.205.28.1)  7.762 ms  8.673 ms  12.105 ms
 4  ge-9-11-ur01.brooklynpark.mn.minn.comcast.net (68.85.165.177)  13.051 ms  13.187 ms  13.303 ms
 5  te-0-1-0-7-ar01.roseville.mn.minn.comcast.net (68.87.174.21)  14.088 ms  14.226 ms  14.349 ms
 6  te-0-4-0-0-cr01.chicago.il.ibone.comcast.net (68.86.91.137)  23.664 ms  22.940 ms  23.020 ms
 7  xe-10-0-0.edge1.Chicago2.Level3.net (4.71.248.29)  21.427 ms  20.643 ms  21.364 ms
 8  vlan51.ebr1.Chicago2.Level3.net (4.69.138.158)  21.439 ms  17.752 ms  18.308 ms
 9  ae-6.ebr1.Chicago1.Level3.net (4.69.140.189)  27.739 ms  27.074 ms  27.129 ms
10  ae-11-55.car1.Chicago1.Level3.net (4.68.101.130)  21.758 ms  21.005 ms 
11  vnsc-bak.sys.gtei.net (4.2.2.2)  19.623 ms  18.614 ms  21.966 ms
 

; <<>> DiG 9.3.4-P1 <<>> www.l.google.com @4.2.2.2
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20520
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
 
;; QUESTION SECTION:
;www.l.google.com.              IN      A
 
;; ANSWER SECTION:
www.l.google.com.       252     IN      A       74.125.53.147
www.l.google.com.       252     IN      A       74.125.53.99
www.l.google.com.       252     IN      A       74.125.53.103
www.l.google.com.       252     IN      A       74.125.53.104
 
;; Query time: 20 msec
;; SERVER: 4.2.2.2#53(4.2.2.2)
;; WHEN: Thu Jul  9 23:56:43 2009
;; MSG SIZE  rcvd: 98
 

My traffic ends up going to Google in Seattle:

traceroute to 74.125.53.147 (74.125.53.147), 30 hops max, 40 byte packets
 1  leviathan (192.168.0.1)  0.277 ms  0.266 ms  0.262 ms
 2  173-8-x-x-Minnesota.hfc.comcastbusiness.net (173.8.x.x)  0.864 ms  1.247 ms  1.667 ms
 3  73.205.28.1 (73.205.28.1)  8.632 ms  13.046 ms  14.046 ms
 4  ge-9-11-ur01.brooklynpark.mn.minn.comcast.net (68.85.165.177)  14.220 ms  14.335 ms  14.443 ms
 5  te-0-1-0-7-ar01.roseville.mn.minn.comcast.net (68.87.174.21)  15.051 ms  15.182 ms  15.294 ms
 6  68.86.91.5 (68.86.91.5)  23.834 ms  23.949 ms  24.012 ms
 7  xe-10-1-0.edge1.Chicago2.Level3.net (4.71.248.17)  27.257 ms  16.259 ms  22.470 ms
 8  vlan51.ebr1.Chicago2.Level3.net (4.69.138.158)  23.420 ms  17.044 ms  21.017 ms
 9  ae-3.ebr2.Denver1.Level3.net (4.69.132.61)  50.168 ms  50.565 ms  50.686 ms
10  ae-2.ebr2.Seattle1.Level3.net (4.69.132.53)  64.192 ms  64.425 ms  64.540 ms
11  ae-21-52.car1.Seattle1.Level3.net (4.68.105.34)  63.460 ms  60.671 ms  64.709 ms
12  GOOGLE-INC.car1.Seattle1.Level3.net (4.79.104.74)  83.396 ms  81.343 ms  82.263 ms
13  209.85.249.32 (209.85.249.32)  82.445 ms  79.255 ms  83.490 ms
14  216.239.46.208 (216.239.46.208)  90.365 ms  90.160 ms 72.14.239.12 (72.14.239.12)  92.453 ms
15  209.85.250.146 (209.85.250.146)  93.873 ms 64.233.174.131 (64.233.174.131)  94.810 ms  84.726 ms
16  216.239.48.165 (216.239.48.165)  105.094 ms  89.687 ms 72.14.232.2 (72.14.232.2)  100.294 ms
17  pw-in-f147.google.com (74.125.53.147)  94.271 ms  86.319 ms  90.354 ms
 

If I perform the same query on my local Comcast DNS servers:

traceroute to 68.87.72.130 (68.87.72.130), 30 hops max, 40 byte packets
 1  leviathan (192.168.0.1)  0.377 ms  0.350 ms  0.348 ms
 2  173-8-x-x-Minnesota.hfc.comcastbusiness.net (173.8.x.x)  0.834 ms  1.745 ms  2.014 ms
 3  73.205.28.1 (73.205.28.1)  8.263 ms  12.692 ms  13.687 ms
 4  ge-9-11-ur01.brooklynpark.mn.minn.comcast.net (68.85.165.177)  13.815 ms  13.927 ms  14.056 ms
 5  te-0-1-0-7-ar01.roseville.mn.minn.comcast.net (68.87.174.21)  14.667 ms  14.786 ms  14.950 ms
 6  te-7-3-cr01.chicago.il.cbone.comcast.net (68.86.72.49)  39.951 ms  39.973 ms  40.143 ms
 7  te-0-12-0-2-ar01.area4.il.chicago.comcast.net (68.86.72.34)  40.278 ms  39.450 ms  39.528 ms
 8  pos-0-1-0-0-ar01.elmhurst.il.chicago.comcast.net (68.87.230.238)  40.442 ms  36.522 ms  36.579 ms
 9  te-9-4-ur04.area4.il.chicago.comcast.net (68.87.210.2)  36.986 ms  41.371 ms  41.430 ms
10  chic-cns.area4.il.chicago.comcast.net (68.87.72.130)  39.995 ms  40.311 ms  40.348 ms
 

; <<>> DiG 9.3.4-P1 <<>> www.l.google.com @68.87.72.130
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54812
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
 
;; QUESTION SECTION:
;www.l.google.com.              IN      A
 
;; ANSWER SECTION:
www.l.google.com.       257     IN      A       209.85.225.104
www.l.google.com.       257     IN      A       209.85.225.147
www.l.google.com.       257     IN      A       209.85.225.99
www.l.google.com.       257     IN      A       209.85.225.103
 
;; Query time: 40 msec
;; SERVER: 68.87.72.130#53(68.87.72.130)
;; WHEN: Fri Jul 10 00:00:20 2009
;; MSG SIZE  rcvd: 98
 

I get directed to a closer Chicago Google server:

traceroute to 209.85.225.104 (209.85.225.104), 30 hops max, 40 byte packets
 1  leviathan (192.168.0.1)  0.123 ms  0.103 ms  0.125 ms
 2  173-8-x-x-Minnesota.hfc.comcastbusiness.net (173.8.x.x)  0.590 ms  1.574 ms  1.844 ms
 3  73.205.28.1 (73.205.28.1)  8.292 ms  9.311 ms  12.639 ms
 4  ge-9-11-ur01.brooklynpark.mn.minn.comcast.net (68.85.165.177)  13.645 ms  13.795 ms  13.358 ms
 5  te-0-1-0-7-ar01.roseville.mn.minn.comcast.net (68.87.174.21)  14.031 ms  14.112 ms  13.472 ms
 6  68.86.91.5 (68.86.91.5)  22.752 ms  22.865 ms  22.972 ms
 7  xe-9-3-0.edge1.Chicago2.Level3.net (4.71.248.21)  21.327 ms  22.244 ms  22.378 ms
 8  vlan52.ebr2.Chicago2.Level3.net (4.69.138.190)  22.303 ms  18.098 ms  18.152 ms
 9  ae-5.ebr2.Chicago1.Level3.net (4.69.140.193)  18.142 ms  30.926 ms  31.183 ms
10  ae-21-54.car1.Chicago1.Level3.net (4.68.101.98)  36.754 ms  36.704 ms  36.764 ms
11  GOOGLE-INC.car1.Chicago1.Level3.net (4.79.208.18)  19.757 ms  16.104 ms  19.323 ms
12  209.85.254.122 (209.85.254.122)  20.232 ms  20.184 ms  19.546 ms
13  209.85.241.22 (209.85.241.22)  29.559 ms  28.747 ms  31.927 ms
14  209.85.241.27 (209.85.241.27)  33.081 ms  28.377 ms 209.85.241.29 (209.85.241.29)  32.558 ms
15  72.14.239.18 (72.14.239.18)  42.966 ms 209.85.248.102 (209.85.248.102)  41.763 ms  43.089 ms
16  iy-in-f104.google.com (209.85.225.104)  33.559 ms  33.653 ms  32.667 ms
 

Level3 uses TCP Anycast as does NTT, OpenDNS,UltraDNS, etc. If you like Level3's DNS as they are seperate from Comcast or anyone else aka big brokther affect, then okay but you will not be hurt by performance for using them. Google recently moved all YouTube services on to it's network and shutdown the Youtube network, plain and simple overload.

»who.is/dns/youtube.com/
»webtrace.info/asn/AS15169

1 traceroute www.youtube.com
Translating "www.youtube.com"...domain server (12.129.192.148) [OK]

Type escape sequence to abort.
Tracing the route to youtube-ui.l.google.com (66.249.80.100)

1 12.129.193.236 0 msec 0 msec 0 msec
2 mdf001c7613r0002-gig-12-2.lax1.attens.net (12.129.193.253) 152 msec 0 msec 1
96 msec
3 gar4.la2ca.ip.att.net (12.122.255.69) 4 msec 0 msec 0 msec
4 cr2.la2ca.ip.att.net (12.122.129.170) [MPLS: Label 16566 Exp 0] 68 msec 68 m
sec 68 msec
5 cr1.slkut.ip.att.net (12.122.30.29) [MPLS: Labels 20519/16020 Exp 0] 68 msec
72 msec 68 msec
6 cr2.dvmco.ip.att.net (12.122.30.26) [MPLS: Labels 1047601/16020 Exp 0] 68 ms
ec 68 msec 92 msec
7 cr1.cgcil.ip.att.net (12.122.31.86) [MPLS: Labels 20833/16020 Exp 0] 72 msec
68 msec 68 msec
8 cr2.cgcil.ip.att.net (12.122.2.54) [MPLS: Labels 20637/16020 Exp 0] 72 msec
68 msec 68 msec
9 cr1.n54ny.ip.att.net (12.122.1.1) [MPLS: Labels 0/16020 Exp 0] 72 msec 72 ms
ec 68 msec
10 gar9.n54ny.ip.att.net (12.122.131.77) 72 msec 72 msec 68 msec
11 12.86.0.222 72 msec 68 msec 72 msec
12 72.14.239.248 68 msec 72 msec 68 msec
13 * * *
14 youtube-ui.l.google.com (66.249.80.100) 68 msec 72 msec 68 msec

Again, yes you can potentially face performance issues by using off-network DNS. The DNS servers for Akamai, Google, Yahoo, Limelight/NetFlix, don't return fixed IPs for name resolution -- they return a different IP based on the source of the query.

Unless the DNS server you use is representative of the network you are sourcing your connection from, you aren't allowing the global load balancers to do their job to get you to the best content server.

If you want more detail about how this works, look up product information for products like the F5 Global Traffic Manager.

I understand what you are saying but you are wrong. You didn't read my ultimate response.

"Google recently moved all YouTube services on to it's network and shutdown the Youtube network, plain and simple overload."

YouTube has been running on Google's network infrastructure for well over a year now. They started doing that transition in 2008 after the acquisition in 2006. You can validate this independently through historical rviews data.

The slowdowns recently are due to Youtube going to HQ feeds by default and increasing the bandwidth demands more than was anticipated. People using centralized DNS services and overloading certain content clusters further added to the problem.

I sold 10-G Wavelengths to YouTube, LLC a Google Company and they quit renewing contracts this year. I also follow the stock market and I know the day the acquisition closed and when they started.

I am a technology security contultant for a living.

Youtube.com A record TTL is 5 minutes - check your facts. Today's Anycast DNS operates via OSPF.

Google's network is the problem, the reason you get routed all over the damn place is it's GOOGLE NS pushing idle pipes which is Japan,Texas, Virginia,Mt View, New York, London.

You cannot compare limelight to Akamai as I just did a consulting project for limelight's CDN network acting screwy on download.microsoft.com

I have friends that work in engineering for Level(3) and Cogent -- what the big customers are doing comes up over beers on occasion. Google started rolling the Youtube traffic under AS15169 in '08.I'm a network architect / engineer for a Fortune 100 heathcare organization. The infrastructure managed by my department includes the global traffic managers which distributes traffic to our geographically distributed hosting data centers for both client portal and employee VPN traffic. A 5 minute TTL has absolutely nothing to do with anycasting, and anycast is not protocol dependent. All you're doing is injecting the same route from multiple locations and relying on the underlying routing protocol metric, be it hop count in a distance-vector protocol or path cost in a link-state protocol, to get you to the closest adjacency for that route.The reason you get routed all over hell is if you are using a DNS server that has no clearly identifiable origin. Global load balancers (which are just intelligent DNS servers) can use GeoIP databases to direct you to a particular cache, or more commonly they can take an iBGP feed from your Internet edge routers and leverage AS path information or BGP Community tags on networks such as Level(3) that identify the region of origin.

The only way to do global load balancing is at the DNS level because you need the IP of the specific geographic cluster identified before you start in with the TCP session initiation, because once you get the connection open you can't move it to a new cluster unless you do a redirect to a different hostname and force another DNS query. The problem with using centralized DNS services, especially anycasted DNS services, is that it's damn near impossible for anyone trying to load balance to map out where the heck those servers reside. 4.2.2.2 is a local route pretty much anywhere in the world, so anything trying to do route identification is going to redirect you to content servers all over the place. Intermediate DNS resolvers hide the originating client's IP information, so the smart DNS servers (aka global load balancers) can only use the IP of the DNS resolver itself to know which cache IP to return.CDNs are CDNs. Akamai is obviously at least an order of magnitude bigger than LimeLight, but the principle of their operation is exactly the same.