espaeth
Digital Plumber
Premium,MVM
join:2001-04-21
Minneapolis, MN | reply to scottwed
Re: Problems loading google (DNS issues possibly?)
ATT and Sonic.net have been reported here.
»Can't access Google domains.
»Can't access google.com? |
|
scottwed
join:2005-06-23
Lehi, UT
| reply to espaeth
Just curious, does anyone have a list of other ISPs with the same issue?
I'm also perplexed as to how Hamachi is also having the same issue (the packet traces are identical in regards to the odd resets).
For the record, I haven't had the issue for 3 days now, but I've also scheduled any significant traffic to occur after midnight. |
|
espaeth
Digital Plumber
Premium,MVM
join:2001-04-21
Minneapolis, MN
 ·voip.ms
·Vitelity VOIP
·Callcentric
·VoiceStick
·ViaTalk
 ·Comcast
 ·Embarq
| reply to Nubiatech
said by Nubiatech  :Although the ttl's suggest the resets are coming from further up (ttl 51 = 13 hops), it is not necessarily so. TTLs could easily be manipulated or setup to suggest just that. Agreed. All things being equal though, Comcast isn't the only ISP having this problem, which would also tend to suggest something closer to Google's network. |
|
Nubiatech
soy capitan
join:2007-09-02
Illinois
| reply to espaeth
said by espaeth :said by clayjar :Thanks, Nubiatech. That was the best detailed explanation/interpretation offered so far. Someone from Comcast needs to look into this and fix it. Actually that analysis would tend to suggest it's something on Google's side. The resets are coming from too far out to be Comcast. Although the ttl's suggest the resets are coming from further up (ttl 51 = 13 hops), it is not necessarily so. TTLs could easily be manipulated or setup to suggest just that.
It gets even more interesting,
Can't get to www.google.com? then try:
»google.navigation.opendns.com
This opens a new can of worms: is Comcast also trying to redirect and cache Google traffic just like OpenDNS is doing? (Ref: »forums.opendns.com/comments.php?···1#Item_0) |
|
espaeth
Digital Plumber
Premium,MVM
join:2001-04-21
Minneapolis, MN
·voip.ms
·Vitelity VOIP
·Callcentric
·VoiceStick
·ViaTalk
·Comcast
·Embarq
| reply to clayjar
said by clayjar :Thanks, Nubiatech. That was the best detailed explanation/interpretation offered so far. Someone from Comcast needs to look into this and fix it. Actually that analysis would tend to suggest it's something on Google's side. The resets are coming from too far out to be Comcast. |
|
clayjar
join:2007-10-30
Plainfield, IL | reply to Nubiatech
Thanks, Nubiatech. That was the best detailed explanation/interpretation offered so far. Someone from Comcast needs to look into this and fix it. |
|
SuperNet33
Go Ninja,Go Ninja Go..
Premium
join:2002-10-08
Harwood Heights, IL
clubs:   
·AT&T CallVantage
| reply to FromSnowLand
said by FromSnowLand :
I have been having the same problem for over a week. Here is what I have found:
1) it only happens between 7 pm and midnight
2) only happens if I have been downloading for over an hour (news-reader - no impact on downloads)
3) only impacts iGoogle, Google search & Google Reader (gmail & google news work fine)
Almost the same thing but my speeds drop down to 2megs
--
Comcast HSI 8.0-768 / ATT CV - VoIP / »www.TalkAboutVoIP.COM - VoIP news & help / »www.ChicagoMaximaClub.net - My Website ! |
|
Nubiatech
soy capitan
join:2007-09-02
Illinois
| reply to xfezz2
Thanks clayjar and svideo for providing packet captures.
Here are my observations:
1. After google recieves an SYN from clayjar 's computer, it replies with SYN/ACK
Pay attention to the following:
-- time to live (ttl 53) (it means google is 11 hops away, but we are not concerned about the exact number).
-- the tcp timestamp (28289791)
-- the tcp window size (win 5672). Note that OS defaults differ on this value (check dslreports.com speed tweaks faqs).
2. After the computer completes the handshake by sending an ACK and start the connection(PSH), here come the mysterious RSTs:
Here we have a ttl 51 which indicates that these packet are not from the original source (Google). Another clue is the tcp window size (win 10240), which can not be that high without scaling after a connection is established. It is clear that this a different host sitting in the middle. Also note that the 2 RST packets were sent 0.0017 seconds in between! This is very unusual.
And tcp timestamp 28290791, although optional, but decreased!?
3.
The computer upon recieving the RST, replies with an RST and closes its side of the connection:
4. And finally, ffw, Google upon recieving RSTs from the host in the middle, closes its side of the connection. Note here we confirm it is google: (ttl 53), a consistent tcp timestamp (28289791), and a low (win 6432). No need to go into window scaling discussion.
Conclusion: there is something out there resetting tcp connection to that ip address.
Now, the question is: how come it is possible to use the ip address directly to connect to Google?
My answer: most likely a DNS issue. It could be that at certain times of the day, Google.com DNS entries change to ip addresses that "the host in the middle" above does not like. Keep in mind that, as mentioned in this thread, that Google's DNS entries have very short TTLs:
|
|
FromSnowLand
@comcast.net
| reply to xfezz2
I have been having the same problem for over a week. Here is what I have found:
1) it only happens between 7 pm and midnight
2) only happens if I have been downloading for over an hour (news-reader - no impact on downloads)
3) only impacts iGoogle, Google search & Google Reader (gmail & google news work fine)
I did some testing and if I go through a HTTP Proxy service, I can access the 3 google services just fine, even though at the same time I get the connection resets directly.
I have heard the Comcast is using the Sandvine service to manage traffic, that these devices are internal to the Comcast network (not at the edge) and are responsible for the resets. I suspect that Sandvine has determined certain Google services are p2p and is issuing resets to slow them down, but since they are really web applications, they die when reset (conjecture on my part).
|
|
clayjar
join:2007-10-30
Plainfield, IL
| reply to nosprings
 dump.txt 7,628 bytes
tcpdump from port 80 |
said by nosprings :Can anyone post traces showing the resets? I want to see if I was having the same problem Here's a dump of data on port 80 on my box while trying to connect to www.google.com. Hope that helps. |
|
nosprings
join:2007-10-30
Englewood, CO | reply to nosprings
Can anyone post traces showing the resets? I want to see if I was having the same problem |
|
nosprings
join:2007-10-30
Englewood, CO | reply to artlogic
I would like to see the captures |
|
Comspastic
join:2007-10-22
Seattle, WA | reply to telcolackey
Google hasn't changed anything. He's saying it's related to root and caching servers, and it doesn't have to do with google at all. Sorry i don't know that much about DNS. |
|
greeneg
@comcast.net
| reply to artlogic
Not sure if this violates posting policy, but for those that are trying to debug this can you please report your findings to »www.google.com/support/bin/reque···x=answer so the members of traffic team inside Google can either pressure Comcast to fix this issue or double check the settings on Google's infrastructure to assure that this issue is not being caused by any unintentional changes. Thanks. |
|
artlogic
@comcast.net
| reply to xfezz2
I live in a small town, South of Lansing, Michigan, and this exact issue was happening to me about a week ago. I lost connection to certain Google sites for 3 hours or so. I started investigating it, because I could access it from other locations. Here's what I found:
* Other folks with Comcast in Lansing proper were not having this issue.
* WireShark captures (which I saved) show what appear to be TCP RST packets being injected
* Whatever was doing this seemed to be looking for a HTTP header, specifically: "Host: google.com" or some variant - I could connect by IP - in fact I just used the IP I got from pinging google.com which worked.
IMHO, it appears as if Comcast is filtering HTTP traffic specifically - and looking for google.com in the host header. I'm not sure if someone if maliciously screwing with their bittorrent filtering software (which officially doesn't exist) or if Comcast is doing limited testing of web filtering technology, but I can say that the issue I experience was NOT a DNS issue. If anyone would like to see my packet captures, let me know. |
|
svideo
join:2007-10-30
| reply to xfezz2
I'm seeing the same thing on Comcast in Grand Rapids, MI. DNS does not appear to be the problem. I have packet captures and a detailed description posted up here. The short version is that I'll see TCP RSTs against google but nowhere else, across 2 different IPs (my TTL had expired while testing). All other traffic works as expected, including pings to the google hosts. |
|
MrDaBomb
@comcast.net
| reply to xfezz2
I'm sure it isn't a DNS issue. Google's web servers respond to straight IP requests. For instance, right now www.google.com resolves as 72.14.253.103 for me (its different depending on your geographical location). If you go to »72.14.253.103/ you'll see the google start page. When Comcast is blocking Google, I get HTTP reset errors from that site, but when I goto another location (SSH Proxy tunnel through a server thats connected through Global Crossing) I can bring up that IP's website fine.
I totally agree its something with Comcast's packet filter. Whether it is intentional or not, it shouldn't be happening, and Comcast should be a LOT faster about fixing this kind of problem. |
|
scottwed
join:2005-06-23
Lehi, UT
| reply to atreznik
Re: Google & Hamachi VPN timeouts
Thanks Atreznik, it's good to know that my symptoms are reproducible at other locations. From my location, the routes to Google and Hamachi servers are drastically different once they leave Comcast (Level3 vs ATT). I did not see the issue at all this Sunday.
So I had another thought today, this one doesn't depend upon any paranoia about private company battles. What if the packet shaper simply has a total daily traffic limit per domain? Everyone gets home from work, and starts surfing, the limit gets hit on many, but not all nights.
I'm still trying to identify when the problem starts each night, so if there is a consistent start period, then I'm going right back to 100% paranoia 
One partial argument against the sloppy Sandvine configuration theory; I've noticed that Yahoo is still available throughout the downtime. It also doesn't explain why Hamachi is affected.
If you want full-blown paranoia, one side or the other is trying to force the issue of a "net-neutrality" bill, something that I think is well intentioned but dangerous for the Internet. |
|
