dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
72257
atreznik
join:2004-08-30
USA

atreznik to scottwed0

Member

to scottwed0

Google & Hamachi VPN timeouts

Comcastic HSI in detroit area. scottwed, I'm sitting in the hotel in miami and I wanted to reply so you knew you weren't the only one. Last thursday night (10/25) was when I first noticed issues with the google domain and also VPN connections via hamachi. Fairly similar symptoms; I had it up already (hamachi) and it seemed to work fine, however when Google stopped working I tried restarting my machine and then Hamachi couldn't resolve its servers and the game was over (literally).

I didn't check for reset packets but that theory seems to make sense. I had to go out of town abruptly for work to trinidad and puerto rico and didn't get a chance to flushdns after I changed DNS servers for my router. By the way I agree with you, I have several computers behind a Dlink 1310 with TKIP and wired or wireless, it makes no difference, encryption or otherwise, not that it should. Will try a proxy when I get back as well. BTW, I like your theory about a battle between Comcast/ATT and Google.

I am interested to see someone get to the bottom of this...

Apologies for the incoherent style I'm near exhaustion. Trust me though I'm usually right on the ball.

I'm anxious to get home and see everything working okay and get Hamachi back up. That would be just Comcastic.
scottwed0
join:2005-06-23
Lehi, UT

scottwed0

Member

Thanks Atreznik, it's good to know that my symptoms are reproducible at other locations. From my location, the routes to Google and Hamachi servers are drastically different once they leave Comcast (Level3 vs ATT). I did not see the issue at all this Sunday.

So I had another thought today, this one doesn't depend upon any paranoia about private company battles. What if the packet shaper simply has a total daily traffic limit per domain? Everyone gets home from work, and starts surfing, the limit gets hit on many, but not all nights.

I'm still trying to identify when the problem starts each night, so if there is a consistent start period, then I'm going right back to 100% paranoia

One partial argument against the sloppy Sandvine configuration theory; I've noticed that Yahoo is still available throughout the downtime. It also doesn't explain why Hamachi is affected.

If you want full-blown paranoia, one side or the other is trying to force the issue of a "net-neutrality" bill, something that I think is well intentioned but dangerous for the Internet.

MrDaBomb
@comcast.net

MrDaBomb to xfezz2

Anon

to xfezz2

Re: Problems loading google (DNS issues possibly?)

I'm sure it isn't a DNS issue. Google's web servers respond to straight IP requests. For instance, right now www.google.com resolves as 72.14.253.103 for me (its different depending on your geographical location). If you go to »72.14.253.103/ you'll see the google start page. When Comcast is blocking Google, I get HTTP reset errors from that site, but when I goto another location (SSH Proxy tunnel through a server thats connected through Global Crossing) I can bring up that IP's website fine.

I totally agree its something with Comcast's packet filter. Whether it is intentional or not, it shouldn't be happening, and Comcast should be a LOT faster about fixing this kind of problem.
svideo
join:2007-10-30

svideo to xfezz2

Member

to xfezz2
I'm seeing the same thing on Comcast in Grand Rapids, MI. DNS does not appear to be the problem. I have packet captures and a detailed description posted up here. The short version is that I'll see TCP RSTs against google but nowhere else, across 2 different IPs (my TTL had expired while testing). All other traffic works as expected, including pings to the google hosts.

artlogic
@comcast.net

artlogic to xfezz2

Anon

to xfezz2
I live in a small town, South of Lansing, Michigan, and this exact issue was happening to me about a week ago. I lost connection to certain Google sites for 3 hours or so. I started investigating it, because I could access it from other locations. Here's what I found:

* Other folks with Comcast in Lansing proper were not having this issue.
* WireShark captures (which I saved) show what appear to be TCP RST packets being injected
* Whatever was doing this seemed to be looking for a HTTP header, specifically: "Host: google.com" or some variant - I could connect by IP - in fact I just used the IP I got from pinging google.com which worked.

IMHO, it appears as if Comcast is filtering HTTP traffic specifically - and looking for google.com in the host header. I'm not sure if someone if maliciously screwing with their bittorrent filtering software (which officially doesn't exist) or if Comcast is doing limited testing of web filtering technology, but I can say that the issue I experience was NOT a DNS issue. If anyone would like to see my packet captures, let me know.

greeneg
@comcast.net

greeneg

Anon

Not sure if this violates posting policy, but for those that are trying to debug this can you please report your findings to »www.google.com/support/b ··· x=answer so the members of traffic team inside Google can either pressure Comcast to fix this issue or double check the settings on Google's infrastructure to assure that this issue is not being caused by any unintentional changes. Thanks.

Comspastic
join:2007-10-22
Seattle, WA

Comspastic to telcolackey5

Member

to telcolackey5
Google hasn't changed anything. He's saying it's related to root and caching servers, and it doesn't have to do with google at all. Sorry i don't know that much about DNS.
nosprings
join:2007-10-30
Englewood, CO

nosprings to artlogic

Member

to artlogic
I would like to see the captures
nosprings

nosprings

Member

Can anyone post traces showing the resets? I want to see if I was having the same problem
clayjar
join:2007-10-30
Plainfield, IL

clayjar

Member

dump.txt
7,628 bytes
tcpdump from port 80
said by nosprings:

Can anyone post traces showing the resets? I want to see if I was having the same problem
Here's a dump of data on port 80 on my box while trying to connect to www.google.com. Hope that helps.

FromSnowLand
@comcast.net

FromSnowLand to xfezz2

Anon

to xfezz2
I have been having the same problem for over a week. Here is what I have found:
1) it only happens between 7 pm and midnight
2) only happens if I have been downloading for over an hour (news-reader - no impact on downloads)
3) only impacts iGoogle, Google search & Google Reader (gmail & google news work fine)

I did some testing and if I go through a HTTP Proxy service, I can access the 3 google services just fine, even though at the same time I get the connection resets directly.

I have heard the Comcast is using the Sandvine service to manage traffic, that these devices are internal to the Comcast network (not at the edge) and are responsible for the resets. I suspect that Sandvine has determined certain Google services are p2p and is issuing resets to slow them down, but since they are really web applications, they die when reset (conjecture on my part).


Nubiatech
soy capitan
join:2007-09-02
Chicago, IL

Nubiatech to xfezz2

Member

to xfezz2
Thanks clayjar See Profile and svideo See Profile for providing packet captures.
Here are my observations:

1. After google recieves an SYN from clayjar See Profile 's computer, it replies with SYN/ACK
17:24:50.305541 IP (tos 0x20, ttl 53, id 57627, offset 0, flags [none], proto TCP (6), length 60) py-in-f147.google.com.http > 192.168.12.3.36519: S, cksum 0x1a42 (correct), 1211513156:1211513156(0) ack 4243223793 win 5672 <mss 1430,sackOK,timestamp 
28289791 2587334990,nop,wscale 0>
 
Pay attention to the following:
-- time to live (ttl 53) (it means google is 11 hops away, but we are not concerned about the exact number).
-- the tcp timestamp (28289791)
-- the tcp window size (win 5672). Note that OS defaults differ on this value (check dslreports.com speed tweaks faqs).



2. After the computer completes the handshake by sending an ACK and start the connection(PSH), here come the mysterious RSTs:
17:24:50.328010 IP (tos 0x0, ttl 51, id 57631, offset 0, flags [DF], proto TCP (6), length 52) py-in-f147.google.com.http > 192.168.12.3.36519: R, cksum 0x255e (correct), 1211513157:1211513157(0) win 10240 <nop,nop,timestamp 28290791 0>
17:24:50.328113 IP (tos 0x0, ttl 51, id 57632, offset 0, flags [DF], proto TCP (6), length 52) py-in-f147.google.com.http > 192.168.12.3.36519: R, cksum 0xf486 (correct), 1211525660:1211525660(0) win 10240 <nop,nop,timestamp 28290791 0>
 
Here we have a ttl 51 which indicates that these packet are not from the original source (Google). Another clue is the tcp window size (win 10240), which can not be that high without scaling after a connection is established. It is clear that this a different host sitting in the middle. Also note that the 2 RST packets were sent 0.0017 seconds in between! This is very unusual.
And tcp timestamp 28290791, although optional, but decreased!?


3.
The computer upon recieving the RST, replies with an RST and closes its side of the connection:
17:24:50.332401 IP (tos 0x20, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40) 192.168.12.3.36519 > py-in-
f147.google.com.http: R, cksum 0xf8e4 (correct), 4243224049:4243224049(0) win 0
 



4. And finally, ffw, Google upon recieving RSTs from the host in the middle, closes its side of the connection. Note here we confirm it is google: (ttl 53), a consistent tcp timestamp (28289791), and a low (win 6432). No need to go into window scaling discussion.
17:24:50.381396 IP (tos 0x20, ttl 53, id 57629, offset 0, flags [none], proto TCP (6), length 52) py-in-f147.google.com.http > 192.168.12.3.36519: R, cksum 0x44c9 (correct), 1:1(0) ack 257 win 6432 <nop,nop,timestamp 28289799 2587335018>
 



Conclusion: there is something out there resetting tcp connection to that ip address.



Now, the question is: how come it is possible to use the ip address directly to connect to Google?
My answer: most likely a DNS issue. It could be that at certain times of the day, Google.com DNS entries change to ip addresses that "the host in the middle" above does not like. Keep in mind that, as mentioned in this thread, that Google's DNS entries have very short TTLs:
C:\>nslookup
> set debug
> www.google.com
Server:  UnKnown
Address:  192.168.10.1
 
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 6, rcode = NXDOMAIN
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 0,  authority records = 0,  additional = 0
 
    QUESTIONS:
        www.google.com.lan, type = A, class = IN
 
------------
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 7, rcode = NOERROR
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 3,  authority records = 0,  additional = 0
 
    QUESTIONS:
        www.google.com, type = A, class = IN
    ANSWERS:
    ->  www.google.com
        canonical name = google.navigation.opendns.com
        ttl = 8 (8 secs)
    ->  google.navigation.opendns.com
        internet address = 208.67.216.231
        ttl = 8 (8 secs)
    ->  google.navigation.opendns.com
        internet address = 208.67.216.230
        ttl = 8 (8 secs)
 
------------
Non-authoritative answer:
Name:    google.navigation.opendns.com
Addresses:  208.67.216.231, 208.67.216.230
Aliases:  www.google.com
 
>
 

SuperNet
Go Ninja,Go Ninja Go..
Premium Member
join:2002-10-08
Hoffman Estates, IL

SuperNet to FromSnowLand

Premium Member

to FromSnowLand
said by FromSnowLand :

I have been having the same problem for over a week. Here is what I have found:
1) it only happens between 7 pm and midnight
2) only happens if I have been downloading for over an hour (news-reader - no impact on downloads)
3) only impacts iGoogle, Google search & Google Reader (gmail & google news work fine)

Almost the same thing but my speeds drop down to 2megs
clayjar
join:2007-10-30
Plainfield, IL

clayjar to Nubiatech

Member

to Nubiatech
Thanks, Nubiatech. That was the best detailed explanation/interpretation offered so far. Someone from Comcast needs to look into this and fix it.

SpaethCo
Digital Plumber
MVM
join:2001-04-21
Minneapolis, MN

SpaethCo

MVM

said by clayjar:

Thanks, Nubiatech. That was the best detailed explanation/interpretation offered so far. Someone from Comcast needs to look into this and fix it.
Actually that analysis would tend to suggest it's something on Google's side. The resets are coming from too far out to be Comcast.

Nubiatech
soy capitan
join:2007-09-02
Chicago, IL

Nubiatech

Member

said by SpaethCo:

said by clayjar:

Thanks, Nubiatech. That was the best detailed explanation/interpretation offered so far. Someone from Comcast needs to look into this and fix it.
Actually that analysis would tend to suggest it's something on Google's side. The resets are coming from too far out to be Comcast.
Although the ttl's suggest the resets are coming from further up (ttl 51 = 13 hops), it is not necessarily so. TTLs could easily be manipulated or setup to suggest just that.

It gets even more interesting,
Can't get to www.google.com? then try:
»google.navigation.opendns.com

This opens a new can of worms: is Comcast also trying to redirect and cache Google traffic just like OpenDNS is doing? (Ref: »forums.opendns.com/comme ··· 1#Item_0)

SpaethCo
Digital Plumber
MVM
join:2001-04-21
Minneapolis, MN

SpaethCo

MVM

said by Nubiatech:

Although the ttl's suggest the resets are coming from further up (ttl 51 = 13 hops), it is not necessarily so. TTLs could easily be manipulated or setup to suggest just that.
Agreed. All things being equal though, Comcast isn't the only ISP having this problem, which would also tend to suggest something closer to Google's network.
scottwed0
join:2005-06-23
Lehi, UT

scottwed0

Member

Just curious, does anyone have a list of other ISPs with the same issue?

I'm also perplexed as to how Hamachi is also having the same issue (the packet traces are identical in regards to the odd resets).

For the record, I haven't had the issue for 3 days now, but I've also scheduled any significant traffic to occur after midnight.

SpaethCo
Digital Plumber
MVM
join:2001-04-21
Minneapolis, MN

SpaethCo

MVM

ATT and Sonic.net have been reported here.

»Can't access Google domains.
»Can't access google.com?
Expand your moderator at work