dslreports logo
    All Forums Hot Topics Gallery
spc
uniqs
1699
Joe Sixpack
join:2015-03-27

Joe Sixpack

Member

WoW Redirecting DNS udp/53 traffic? google.com resovles to WOW IP.

Tonight I had an outage and as I was testing the connection I tried to ping the usual suspect.. google.com

I noticed something weird though.. the Ip that was coming back was what appeared to be a WoW IP block.

I did a nslookup and spit out a bunch of wow ip's
*HUH*?
So I a lookup using google toolbox and of course the real google.com ip list is different.

I do not use WOW's dns, I use google's 8.8.8.8 / 8.8.4.4
So this time just to make sure I specify specifically to query google dns
Again results come back the same.

I tried doing the lookup on the router via ssh, same results.
Only thing I can figure is WoW is intercepting udp53 traffic.

Server:  google-public-dns-a.google.com
Address:  8.8.8.8
 
Non-authoritative answer:
Name:    google.com
Addresses:  2607:f8b0:400d:c06::8b
          65.60.171.155
          65.60.171.152
          65.60.171.176
          65.60.171.181
          65.60.171.144
          65.60.171.185
          65.60.171.170
          65.60.171.159
          65.60.171.166
          65.60.171.163
          65.60.171.154
          65.60.171.174
          65.60.171.187
          65.60.171.177
          65.60.171.148
          65.60.171.165
 
Tracing route to google.com [65.60.171.155]
over a maximum of 30 hops:
 
  1     *Local Hop*
  2     *Local Hop*
  3    21 ms    25 ms    29 ms  10.208.64.1
  4   121 ms    53 ms    45 ms  76-73-166-97.knology.net [76.73.166.97]
  5    26 ms    38 ms    24 ms  76-73-167-214.knology.net [76.73.167.214]
  6    26 ms    31 ms    19 ms  76-73-167-89.knology.net [76.73.167.89]
  7    19 ms    18 ms   207 ms  60-65-155-171.static.col.wideopenwest.com [65.60.171.155]
 
Trace complete.
 
Via google toolbox "dig" real IP's
google.com. 299 IN A 74.125.21.102
google.com. 299 IN A 74.125.21.139
google.com. 299 IN A 74.125.21.100
google.com. 299 IN A 74.125.21.101
google.com. 299 IN A 74.125.21.138
google.com. 299 IN A 74.125.21.113
 

So it's a bit disturbing frankly, They might possibility have a good reason but I don't know what it is.
Perhaps it's some sort of caching proxy? but then again how much static content comes from google.com? not much I wouldn't think and it's not very media rich.. pictures, movies, etc.

anyone know the story with this?

Bill_MI
Bill In Michigan
MVM
join:2001-01-03
Royal Oak, MI
TP-Link Archer C7
Linksys WRT54GS
Linksys WRT54G v4

Bill_MI

MVM

Hi Joe. Here's the thread from last November about WOW's caching servers for Google: »oddness when pining google

Just_Dan See Profile confirms the Google-operated server inside WOW's network. Netflix, too.

Interesting Google's own DNS effectively points to WOW's caching servers but it makes sense for Google DNS to support WOW's caching setup. Trying level3 (4.2.2.1 thru 4.2.2.6) looks more normal here. Do you see the same when you don't use WOW or Google DNS?
Joe Sixpack
join:2015-03-27

Joe Sixpack

Member

Interestingly I do get different results via that NDS.
Server:  a.resolvers.level3.net
Address:  4.2.2.1
 
Non-authoritative answer:
Name:    google.com
Addresses:  2607:f8b0:4009:808::200e
          173.194.46.33
          173.194.46.32
          173.194.46.35
          173.194.46.38
          173.194.46.34
          173.194.46.41
          173.194.46.40
          173.194.46.46
          173.194.46.39
          173.194.46.36
          173.194.46.37
 
Which is actually still different then Googles own lookup on toolbox, but are real google IP's.
My thoughts was that wow was intercepting DNS traffic, since I was using google DNS
but perhaps Google DNS servers are smart enough to return results specific to my IP?
Oh you're on a wow IP, so here's wow cache server.

Otherwise I guess they'd be changing the results on the level3 lookup as well.
Or is it perhaps they intercept and only inspect DNS outgoing for 8.8.8.8 for google lookups?
Im kinda curious how exactly that works.

I dunno, at least it's not as sinister as I feared.
I've only been with wow for a month and after the BPI problem I had no faith in them.

Thanks for the info Bill.

Bill_MI
Bill In Michigan
MVM
join:2001-01-03
Royal Oak, MI
TP-Link Archer C7
Linksys WRT54GS
Linksys WRT54G v4

1 edit

Bill_MI

MVM

I'm curious, too. Google has the clout to do anything they want, and they do, which never made me a fan.

I do know one thing, Google protects their SSL certificates like a mother bear with Chrome calling home to report anything strange. A few weeks ago Google caught a bad cert by CNNIC: »arstechnica.com/security ··· f-trust/

With this reputation in mind, I'm sure Google's caching servers in WOW's network are locked down like a fortress. It definitely does SSL so WOW has little to do with its operation, except give it a WOW IP, so I'm not worried about WOW. I'd worry more about Google.
Sadachara
join:2015-02-21
United State

Sadachara to Joe Sixpack

Member

to Joe Sixpack
Actually, Bill's positive outlook isn't accurate.

WOW has been implicated in hijacking traffic in the past.. Google NebuAD - there were lawsuits involved with them hijacking traffic. WOW is also know to hijack via NXDOMAIN, and PAXFIRE servers.

WOW actually has a history of sniffing/intercepting, or contacting companies to do it.

»www.mediapost.com/public ··· uit.html

aes128
join:2003-12-19
Warren, MI

1 edit

2 recommendations

aes128

Member

said by Sadachara:

Actually, Bill's positive outlook isn't accurate.

WOW has been implicated in hijacking traffic in the past.. Google NebuAD - there were lawsuits involved with them hijacking traffic. WOW is also know to hijack via NXDOMAIN, and PAXFIRE servers.

WOW actually has a history of sniffing/intercepting, or contacting companies to do it.

»www.mediapost.com/public ··· uit.html

I do not believe this has anything at all to do with the OPs issue. We already know that WOW will send you to some advertising page if you use their servers and type in a bad name.

The thing with Google, Netflix, Akamai is different. These all can act as a CDN (Content Delivery Network). So if WOW has some google servers in their network the Google name server looks at the IP to see there the request is coming from. If it is from an AS that contains some Google servers, it returns those IPs which in this case belong to WOW.

When I worked at Chrysler we used Akamai to deliver content so a DNS lookup to a "chrysler.com" IP might get you an Akamai server IP instead of our IP and depending on where in the country you were, it could change. You get the closest one.

Here is my dig output:

[crypto@centmain ~]$ dig google.com +trace

; > DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6_6.2 > google.com +trace
;; global options: +cmd
. 516029 IN NS m.root-servers.net.
. 516029 IN NS e.root-servers.net.
. 516029 IN NS c.root-servers.net.
. 516029 IN NS d.root-servers.net.
. 516029 IN NS f.root-servers.net.
. 516029 IN NS i.root-servers.net.
. 516029 IN NS g.root-servers.net.
. 516029 IN NS l.root-servers.net.
. 516029 IN NS h.root-servers.net.
. 516029 IN NS a.root-servers.net.
. 516029 IN NS j.root-servers.net.
. 516029 IN NS b.root-servers.net.
. 516029 IN NS k.root-servers.net.
;; Received 228 bytes from 127.0.0.1#53(127.0.0.1) in 572 ms

com. 172800 IN NS e.gtld-servers.net.
com. 172800 IN NS d.gtld-servers.net.
com. 172800 IN NS b.gtld-servers.net.
com. 172800 IN NS m.gtld-servers.net.
com. 172800 IN NS a.gtld-servers.net.
com. 172800 IN NS l.gtld-servers.net.
com. 172800 IN NS g.gtld-servers.net.
com. 172800 IN NS h.gtld-servers.net.
com. 172800 IN NS c.gtld-servers.net.
com. 172800 IN NS f.gtld-servers.net.
com. 172800 IN NS i.gtld-servers.net.
com. 172800 IN NS k.gtld-servers.net.
com. 172800 IN NS j.gtld-servers.net.
;; Received 500 bytes from 192.33.4.12#53(192.33.4.12) in 947 ms

google.com. 172800 IN NS ns2.google.com.
google.com. 172800 IN NS ns1.google.com.
google.com. 172800 IN NS ns3.google.com.
google.com. 172800 IN NS ns4.google.com.
;; Received 164 bytes from 192.31.80.30#53(192.31.80.30) in 256 ms

google.com. 300 IN A 67.149.209.237
google.com. 300 IN A 67.149.209.223
google.com. 300 IN A 67.149.209.251
google.com. 300 IN A 67.149.209.224
google.com. 300 IN A 67.149.209.230
google.com. 300 IN A 67.149.209.244
google.com. 300 IN A 67.149.209.210
google.com. 300 IN A 67.149.209.245
google.com. 300 IN A 67.149.209.238
google.com. 300 IN A 67.149.209.231
google.com. 300 IN A 67.149.209.217
google.com. 300 IN A 67.149.209.216
;; Received 220 bytes from 216.239.34.10#53(216.239.34.10) in 51 ms

You see I go to the roots, then to Google and get back WOW IPs. I run my own BIND server.

Netflix/Google/Amazon need to have their content distributed all across the country so you can stream from the closest server else it would never work very well. This is how I understand these things to work if the ISP has local cache servers.

Bill_MI
Bill In Michigan
MVM
join:2001-01-03
Royal Oak, MI
TP-Link Archer C7
Linksys WRT54GS
Linksys WRT54G v4

2 edits

Bill_MI

MVM

I had procrastinated looking up the dig +trace option (thanks!). I'm getting same plus large RRSIG and NSEC3 records from root through the gtld-server. I think that's DNSEC so those are signed. Someday I'll learn to use dig better. This is just the last part...
google.com.172800INNSns2.google.com.
google.com.172800INNSns1.google.com.
google.com.172800INNSns3.google.com.
google.com.172800INNSns4.google.com.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0QFMDQRCSRU0651QLVA1JQB21IF7UR NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20150426045831 20150419034831 33878 com. WeS9ewHMkI8JH8ywDZbIPvWwkBB0Y795xNzl4ROIjVMo/+S++A3Mn8hx 1oJjqHOiu0p1YuUyaMoSe7nzO3g38yjzmJUl1GZz+kuKaFxnQriUc4oG tCFj0e45dgerJAfQbDBkPJXyl60J74ukWh8JYs5g+Zd8mn0cRRAo3gjQ Tpw=
S84AE3BIT99DKIHQH27TRC0584HV5KOH.com. 86400 IN NSEC3 1 1 0 - S84J17P3PT4RKMEJOHNGD73C5Q5NV5S9 NS DS RRSIG
S84AE3BIT99DKIHQH27TRC0584HV5KOH.com. 86400 IN RRSIG NSEC3 8 2 86400 20150428044006 20150421033006 33878 com. jq+pV+B2rwgJstTLcA6e0ziKB55fNFYDblyFzHdcSjzudr2BtwEKyLZ5 iTzhZZQoyI4XV6y3Hhp2oH7OaqZba6vOOIlQlgFwZY9K++pAcA0m6Yfj ub7bFyAY8Cvx01VhQ21SowaD5WlEVepiKoyeLFHhSEM+Gc+KOpFU07vQ zjc=
;; Received 660 bytes from 192.26.92.30#53(c.gtld-servers.net) in 43 ms
 
google.com.300INA67.149.209.224
google.com.300INA67.149.209.230
google.com.300INA67.149.209.244
google.com.300INA67.149.209.245
google.com.300INA67.149.209.251
google.com.300INA67.149.209.216
google.com.300INA67.149.209.237
google.com.300INA67.149.209.223
google.com.300INA67.149.209.238
google.com.300INA67.149.209.210
google.com.300INA67.149.209.231
google.com.300INA67.149.209.217
;; Received 220 bytes from 216.239.36.10#53(ns3.google.com) in 53 ms
 
 
EDIT: formatting sucked off tabs!

aes128
join:2003-12-19
Warren, MI

aes128

Member

Dig is actually a wonderful tool if you have access to it.

When I was working I knew a guy who was a DNS genius, he actually knew Paul Vixie personally. If you do not know who that is, look him up, then you will know.

I used to ask him stuff and tell him I used nslookop. He would look at me and say, "Use dig and come back with your question". I used to give him my BIND configurations to debug.

The "DNS & BIND" books by O'Reilly are the best you can get. I have several of them over the years.

RootWyrm
join:2011-05-09

RootWyrm

Member

said by aes128:

Dig is actually a wonderful tool if you have access to it.

When I was working I knew a guy who was a DNS genius, he actually knew Paul Vixie personally. If you do not know who that is, look him up, then you will know.

If you have access? Ask and ye shall receive.

Bill_MI
Bill In Michigan
MVM
join:2001-01-03
Royal Oak, MI
TP-Link Archer C7
Linksys WRT54GS
Linksys WRT54G v4

Bill_MI to aes128

MVM

to aes128
Unfortunately, I've used it too little over decades for the syntax to ever sink in. I cut my teeth on bind8 but all on my own - wish I'd had a guru. I'm running bind9 with my old bind8 zone files tweaked to work - another one of those projects when I get a "ROUND TUIT".

In the meantime, I did get DNSSEC working last year, best I can tell. Notice the NSEC3 and RRSIG records?

aes128
join:2003-12-19
Warren, MI

aes128 to RootWyrm

Member

to RootWyrm
said by RootWyrm:

said by aes128:

Dig is actually a wonderful tool if you have access to it.

When I was working I knew a guy who was a DNS genius, he actually knew Paul Vixie personally. If you do not know who that is, look him up, then you will know.

If you have access? Ask and ye shall receive.

I think I ran across that site before. I guess I really meant too bad there is not a version of "Dig" for windows. I have never found one. One of the reason I keep a Linux box around is just so I have "Dig" and can run BIND.

Have you seen a version for Windows?
devolved
join:2012-07-11
Rapid City, SD
Ooma Telo

1 edit

devolved

Member

said by aes128:

Have you seen a version for Windows?

Here -> »www.isc.org/downloads/

Click "BIND" and download the latest appropriate version (32 or 64-bit). After downloading, unzip to a directory of your choice.

This blog post has more information on getting dig to work in Win 7 -> »www.danesparza.net/2011/ ··· ndows-7/

aes128
join:2003-12-19
Warren, MI

aes128

Member

said by devolved:

said by aes128:

Have you seen a version for Windows?

Here -> »www.isc.org/downloads/

Click "BIND" and download the latest appropriate version (32 or 64-bit). After downloading, unzip to a directory of your choice.

This blog post has more information on getting dig to work in Win 7 -> »www.danesparza.net/2011/ ··· ndows-7/

Thanks. I don't really want to run BIND on Windows as I already have it on Linux but if it brings dig along, I might try it. It will save me from having to ssh to the Linux box to use it.
devolved
join:2012-07-11
Rapid City, SD
Ooma Telo

devolved

Member

said by aes128:

Thanks. I don't really want to run BIND on Windows as I already have it on Linux but if it brings dig along, I might try it. It will save me from having to ssh to the Linux box to use it.

Welcome. I downloaded it to my Windows 7 box, but I haven't installed it yet. I've got a Macbook Pro and use dig on it. I've had this laptop almost 5 years and recently stumbled on dig about a year ago. LOL

aes128
join:2003-12-19
Warren, MI

aes128

Member

Interesting we have heard nothing from the OP in several days. So are you OK with the comments here?

People post questions here, they get answers and never respond. Oh well, I hope we have helped.

WOW_James
Premium Member
join:2014-12-04
Woodland, GA

WOW_James to Joe Sixpack

Premium Member

to Joe Sixpack
Just to add to the convo a bit.

We now have caching servers from:
Netflix
Google
Akamai
and new to the stable......
Facebook

So... You will see a lot of the traffic to these domains hitting WOW IPs.

Thanks
James

aes128
join:2003-12-19
Warren, MI

aes128

Member

Thanks James for that.

I get it about how CDN cache servers work but many do not. When I was at Chrysler we never had CDN servers on our premisses since we are not an ISP but as I said we farmed out stuff all the time to Akamai. We could not handle the traffic for Superbowl commercials and such.

There was a time when we also farmed out web server traffic but because of poor service on the part of the people we farmed it out to, we brought most of our web sites back in-house to our own data centers.
Joe Sixpack
join:2015-03-27

Joe Sixpack to aes128

Member

to aes128
Oh ya, I didn't have anything to add, I thought we got to the bottom of it already.. google operated cache server.

Thanks for your guys help.
Joe Sixpack

Joe Sixpack to WOW_James

Member

to WOW_James
Thanks for clearing that up James
HolyRoses
join:2002-10-09
Rochester, MI

HolyRoses to WOW_James

Member

to WOW_James
What happens if we are using google public dns? Do we need to be using WoW's DNS to take advantage of these caching servers?

WOW_James
Premium Member
join:2014-12-04
Woodland, GA

WOW_James

Premium Member

Nope,
Googles DNS is what does the redirecting. not WOW. Our servers just cache the Google authoritative response to the query.
It shouldn't matter who's DNS you use. But I would recommend ours (for the lower Latency) or Google's. As google figured out the load balancing between the cache servers and the peering links, they will send a DNS response they feel is most appropriate.

Hope that helps

James
Joe Sixpack
join:2015-03-27

Joe Sixpack to HolyRoses

Member

to HolyRoses
I use google public DNS when I noticed this, Oddly enough I never checked to see what WOW dns hands out cause I never use it *goes off to check*

- GOOGLE DNS -
nslookup google.com 8.8.8.8
Server:  google-public-dns-a.google.com
Address:  8.8.8.8
 
Non-authoritative answer:
Name:    google.com
Addresses:  2607:f8b0:400d:c08::8b
          65.60.171.144
          65.60.171.165
          65.60.171.163
          65.60.171.152
          65.60.171.170
          65.60.171.176
          65.60.171.185
          65.60.171.148
          65.60.171.166
          65.60.171.174
          65.60.171.154
          65.60.171.181
          65.60.171.187
          65.60.171.159
          65.60.171.177
          65.60.171.155
 
- WOW DNS -
nslookup google.com 64.233.222.2
Server:  col11-dns1.col.wideopenwest.com
Address:  64.233.222.2
 
Non-authoritative answer:
Name:    google.com
Addresses:  2607:f8b0:4006:80c::1001
          173.194.123.78
          173.194.123.72
          173.194.123.67
          173.194.123.68
          173.194.123.66
          173.194.123.69
          173.194.123.65
          173.194.123.71
          173.194.123.73
          173.194.123.70
          173.194.123.64
 

Ok.. now that IS weird..
Google public DNS returns wow ip, Im not sure what mechanism does this.. does the google DNS see im on wow and return a wow ip? (This is my theory)

Weirdly WOW dns returns a google IP, so if you're using wow dns normally how would you ever contact the google cache server? unless maybe wow redirects to the internal cache transparently and you'd never know it, except the google DNS specifically names the cache server.

I did a trace route and it looks like it does leave WoW's network, 198.32.118.39 seems to be a Equinix datacenter ip.
On the other hand the IP that the Google Pub DNS is handing out to me is a even shorter distance and stays within wow, it's kinda odd.. Im actually better off with using google DNS then wow's?

- Trace Route, Looks like it goes straight from WoW -> Google
 
Tracing route to lga15s48-in-f14.1e100.net [173.194.123.78]
over a maximum of 50 hops:
 
  1     *Local Hop*
  2     *Local Hop*
  3    17 ms    17 ms    15 ms  10.208.64.1
  4    14 ms    13 ms    17 ms  76-73-166-93.knology.net [76.73.166.93]
  5    13 ms    16 ms    14 ms  76-73-167-214.knology.net [76.73.167.214]
  6    13 ms    13 ms    17 ms  76-73-167-89.knology.net [76.73.167.89]
  7    39 ms    34 ms    41 ms  core1-0-0-8.lga.net.google.com [198.32.118.39]
  8    65 ms    34 ms    37 ms  209.85.248.178
  9    37 ms    33 ms    36 ms  72.14.252.27
 10    35 ms    35 ms    37 ms  lga15s48-in-f14.1e100.net [173.194.123.78]
 
- Trace Rotue, Stays within WOW
Tracing route to 60-65-144-171.static.col.wideopenwest.com [65.60.171.144]
over a maximum of 50 hops:
 
  1     *Local Hop*
  2     *Local Hop*
  3    13 ms    16 ms    13 ms  10.208.64.1
  4    16 ms    13 ms    17 ms  76-73-166-93.knology.net [76.73.166.93]
  5    22 ms    15 ms    14 ms  76-73-167-214.knology.net [76.73.167.214]
  6    14 ms    15 ms    16 ms  76-73-167-89.knology.net [76.73.167.89]
  7    15 ms    16 ms    16 ms  60-65-144-171.static.col.wideopenwest.com [65.60.171.144]
 
Joe Sixpack

Joe Sixpack to WOW_James

Member

to WOW_James
@James
Ah that's what I suspected, that the google DNS was handing out wow IP cause im on wow.
I just checked wow dns and oddly it gives me a IP registered to Google, not wow.

It returned a ip block in the same range as what I got from level3.
So it's kinda weird that this whole thing started because google dns.

somewhat confused here.
HolyRoses
join:2002-10-09
Rochester, MI

HolyRoses to Joe Sixpack

Member

to Joe Sixpack
it some smart DNS server they are using, not normal.

WOW_James
Premium Member
join:2014-12-04
Woodland, GA

WOW_James to Joe Sixpack

Premium Member

to Joe Sixpack
Google looks at the IP the request comes from. It has each subnet it sees on the internet mapped to a specific server cluster.

Many WOW IPs are mapped to WOW server clusters, that we host for google at out Hubs and Pops. But some IP blocks show better performance going to a Google IP (Usualy over a peering connection with Google at one of several peering points). If the DNS server you are hitting (Wow DNS server) has better performance to one of the google peering points, that's where a query from that server will send you. but Google own DNS looks at your home IP and bases your forwarding on that.

Does that make sense? Anything I should clarify?

James
Joe Sixpack
join:2015-03-27

Joe Sixpack

Member

That makes sense Just would assume internal cache server in wow's network would be typically faster then even a peering connection with google.

I know it's fewer hops internal for me that's for sure.

Inflex
join:2002-09-05

Inflex to WOW_James

Member

to WOW_James
said by WOW_James:

Just to add to the convo a bit.

We now have caching servers from:
Netflix
Google
Akamai
and new to the stable......
Facebook

So... You will see a lot of the traffic to these domains hitting WOW IPs.

Thanks
James

James,
Any plans to add Amazon?

Thanks,
Doug
Inflex

Inflex to WOW_James

Member

to WOW_James
said by WOW_James:

Google looks at the IP the request comes from. It has each subnet it sees on the internet mapped to a specific server cluster.

Many WOW IPs are mapped to WOW server clusters, that we host for google at out Hubs and Pops. But some IP blocks show better performance going to a Google IP (Usualy over a peering connection with Google at one of several peering points). If the DNS server you are hitting (Wow DNS server) has better performance to one of the google peering points, that's where a query from that server will send you. but Google own DNS looks at your home IP and bases your forwarding on that.

Does that make sense? Anything I should clarify?

James

Networking has gotten much more complicated since the days I was in the biz.

WOW_James
Premium Member
join:2014-12-04
Woodland, GA

WOW_James to Inflex

Premium Member

to Inflex
Amazon doesn't have a program for this type of thing. We peer with them in several locations, but at the moment, all of there servers are in their own or rented facilities. No cache deployments with carriers/ISPs.

Thanks
James