macfreddy
join:2008-05-05
 ·Comcast Digital Vo..
 ·Comcast
| [DNS] Comcast DNS is broken.
...but most users don't notice.
When I connect, the DNS IPs that I receive from their DHCP server in the
lease data are 68.87.85.98 and 68.87.69.146.
Frequently, if I send a DNS request for name resolution to 68.87.69.146
for example, the reply comes back from 68.87.69.147 or 68.87.69.148 or 149
or 150. The same sort of off-by-1 or off-by-2, et.al., occurs using the other
IP address 68.87.85.98 as well. (I see 68.87.85.99 or 68.87.85.100, et.al. )
This occurs over 100X per week going back over a year or more.. not sure
when it started as I can't access the firewall logfiles earlier than that.
Now, most clients won't know the difference and just accept the traffic
coming back on port 53 and use it. Bring up a network tool like -tcpdump-
and start logging DNS data and you will see what I'm talking about.
But, my network has a firewall fronting it which does stateful packet
filtering using connection tracking. What that means is that if I
initiate a connection with 68.87.69.146, then any traffic coming back
from that IP is accepted by the firewall box as "ESTABLISTED,RELATED".
But NOT from any of 68.87.69.147, 148, 149, nor 150. That off-by-one or
off-by-two traffic is simply logged as "INVALID-NEW" and dropped by the firewall.
For example (edited for anonymity):
May 3 12:18:32 host8 kernel: INVALID-NEW: IN=eth1 OUT= MAC=00:mm:mm:mm:mm:mm:00:nn:nn:nn:nn:nn:nn:00 SRC=68.87.69.149 DST=6X.XXX.XX.XX LEN=74 TOS=0x00 PREC=0x00 TTL=52 ID=ZZZZZ DF PROTO=UDP SPT=53 DPT=PPPPP LEN=54
This results in delays or simply failure for the name lookup request.
Eventually, if the one browsing the web, persists, re-clicking upon the
desired hyperlink, the reply will randomly come back from the correct
targeted DNS IP address 68.87.69.146 and be accepted.
However, the problem comes about when the application sending the DNS
lookup request is NOT a browser with a human retrying the webpage
manually. Those DNS requests time out and lead to the application
issuing a failure code and ending. This makes life difficult as things
cannot be automated.
Comcast apparently must have some sort of DNS virtual cluster which handles
their millions of users. But, the way that the "virtual" address SHOULD
work is not correct: that cluster should have one IP address no matter how
many other internal computers, processors, exist inside that cluster are doing
the distributed work. What appears to be occurring is that each processor
has its own unique external public address which is not being NATed back to
the originating DNS server's IP when it returns the lookup response. Thus, I and
many others, are getting DNS replies returned from an IP address that they
didn't initiate connection to.
That is broken behavior. Most "real" firewall boxes (not SOHO ones like
your common black and blue ones) will react the same way to this broken
behavior. It violates established RFCs for DNS. |
|
NormanS
Premium,MVM
join:2001-02-14
San Jose, CA
 ·Pacific Bell - SBC
| said by macfreddy  :That is broken behavior. Most "real" firewall boxes (not SOHO ones like your common black and blue ones) will react the same way to this broken behavior. It violates established RFCs for DNS. By trace route, they don't appear to be Anycast DNS servers. What do you get for 4.2.2.1 and 4.2.2.2? I know those to be Anycast DNS servers. Just curious.
--
Norman
~Oh Lord, why have you come
~To Konnyu, with the Lion and the Drum |
|
Cabal
Premium
join:2007-01-21
Boston, MA
| reply to macfreddy
I use Comcast DNS servers with a stateful (pf) firewall and don't see this behavior. Perhaps something is misconfigured at your local/nearby Comcast DNS servers. Additionally, the vast majority of consumer routers made in the past 5 years (Linksys WRT54G* series, for sure) are stateful, so this problem would probably affect a few people.
--
Interested in open source engine management for your Subaru? |
|
macfreddy
join:2008-05-05
·Comcast Digital Vo..
·Comcast
| Cabal,
Yes, I too have used -pf- and unless you LOG and drop, you will be unaware of such rejects. Do you?
The behavior is seen with both DNS servers: 68.87.85.98 and 68.87.69.146, one is local and one elsewhere (Oregon, I believe). So, I would think a misconfiguration would not be likely in two disparate locations.
And, wrt54G routers may be stateful but same applies: do they log the drops/rejects? and can/do you see/retrieve these logs? Retries eventually get thru, so the behavior if not logged, would tend to look like slow DNS, or network traffic. Logging would nail it tho. |
|
espaeth
Misanthrope
Premium
join:2001-04-21
Minneapolis, MN
·voip.ms
·Callcentric
·VoiceStick
·ViaTalk
·Comcast
·Embarq
| reply to macfreddy
said by macfreddy :That is broken behavior. Most "real" firewall boxes (not SOHO ones like your common black and blue ones) will react the same way to this broken behavior. It violates established RFCs for DNS. I agree this is broken behavior, and is likely not what you think it is.
First off, this would be broken for *EVERYTHING* that uses port address translation (including the little black and blue SOHO boxes). If you don't initiate a connection to the outside, there's no way for an outside device to talk back in to you. (there wouldn't be a mapping present in the NAT table to get the return packet back to your box)
Each DNS request has a transaction ID; it's the first 2 bytes of the playload after the UDP header. I would be willing to bet that the responses you are seeing have a transaction ID that doesn't match any of your requests.
The next thing I would check is to issue a request against your DNS server, and check the TTL value of the response. Compare the TTL value of the "one off" DNS responses and see if it matches.
My guess would be someone is trying to run a DNS poisoning attack on your network segment with spoofed source UDP packets. |
|
Cabal
Premium
join:2007-01-21
Boston, MA
| reply to macfreddy
said by macfreddy :Cabal, Yes, I too have used -pf- and unless you LOG and drop, you will be unaware of such rejects. Do you? Yes, all day long. My local Comcast DNS servers are 68.87.71.226 and 68.87.73.242, and any requests sent to them are answered immediately and from the correct source. Anything else would be immediately apparent to a lot of people in the way of multi-second wait times.
--
Interested in open source engine management for your Subaru? |
|
Kearnstd
Elf Wizard
join:2002-01-22
Mullica Hill, NJ | reply to macfreddy
my Linksys router never has issues and my DNS requests go through fine.
--
[65 Arcanist]Filan(High Elf) Zone: Broadband Reports |
|
