dslreports logo
 story category
Earthlink Joins ISPs Snooping User Search Traffic
Use of 'Middle Man' Search Sniffing Technology Spreading

For years ISPs have been using DNS redirection, or redirecting users who visit misspelled or nonexistent domains to ISP run ad-laden search portals. The technology is a significant money maker for ISPs, estimated to bring them at least an additional $5 per user, per month. Recently however, DNS redirection companies like Paxfire have been taking things one step further, injecting themselves in between the traditional user and search engine relationship. Paxfire was recently sued for this practice, using search proxies to sniff user search requests and net search referral fees for both the hardware vendor and ISPs.

Click for full size
The folks behind the network tool Netalyzr at Berkeley have been at the forefront of investigating this new ISP technology for some time, and first described it in detail several months ago. Users in our Earthlink forum and the NANOG mailing list note that Earthlink is one of the most recent ISPs to employ this technology.

"Noticed past day or so that I started getting Captcha request for google searches saying that to many requests are originating from an IP other than my own," says a forum user. "Thought I was a victim of malware, but after troubleshooting noticed that 207.69.188.185 ns1.mindspring.com was returning 64.27.117.179 for www.google.com, www.bing.com & www.ask.com," they note. "It seems that 64.27.117.179 is a squid server which is configured to only proxy requests for major search engines."

Click for full size
While Paxfire insists they don't track this search data, there's no saying what search data ISPs are storing -- since none of them are willing to talk about this new technology. While DNS redirection technology can often be opted out of, ISPs aren't even admitting they're using this new search traffic hijack technology, much less giving users a way out of it.

In addition to privacy concerns, there's also a major annoyance for Earthlink users: in some instances they aren't getting the "real" Google and have to enter CAPTCHA passwords to actually see Google search results. We've confirmed with researchers at Berkeley that switching to alternative DNS providers like Google Public DNS and OpenDNS will stop the search engine traffic interception behavior.

"OpenDNS stops the search engine interception, but it doesn't stop the "helpful" results page when you have an incorrect hostname in a URL," says ICSI's Nicholas Weaver.

ISPs who currently use this technology or have in the past include Cavalier, Cincinnati Bell, Cogent, Frontier, Hughes, IBBS, Insight Broadband, Megapath, Paetec, RCN, Wide Open West, XO Communications, Charter and Iowa Telecom. While muted at the moment, political and legal inquiries about whether these technologies violate privacy and wiretap laws are expected to grow over the next year, with the lawsuit against Paxfire and RCN (and Paxfire's countersuit) playing center stage.

We've contacted Earthlink for comment and will update this post with any additional data received.
view:
topics flat nest 
nweaver
join:2010-01-13
Napa, CA

1 edit

nweaver

Member

Note, Google Public DNS will stop this behavior

Using Google Public DNS or other third party DNS services will stop this behavior as far as we know.

cowboyro
Premium Member
join:2000-10-11
CT

cowboyro

Premium Member

Re: Note, Google Public DNS will stop this behavior

...because obviously a party that involves in DNS redirection wouldn't dare to hijack legitimate DNS queries or simply redirect all third-party DNS queries to its own servers...

FFH5
Premium Member
join:2002-03-03
Tavistock NJ

4 edits

FFH5 to nweaver

Premium Member

to nweaver
said by nweaver:

Using Google Public DNS or other third party DNS services will stop this behavior as far as we know.

And if you use firefox as your browser, there is an extension that can turn on HTTPS for about 1000 web sites, including some major search engines, that would prevent Paxfire like search redirection by encrypting all the search terms.
»www.newscientist.com/art ··· ies.html
»www.eff.org/https-everywhere

Link to the firefox install extension:
»www.eff.org/files/https- ··· test.xpi

Is your ISP messing with your searches? Check using this tool: »netalyzr.icsi.berkeley.edu/

This is the line you are looking for in the netalyzer test that says your ISP is not doing the offending redirection behavior:
quote:
DNS-level redirection of specific sites (?): OK
Your ISP does not appear to be using DNS to redirect traffic for specific websites.
nweaver
join:2010-01-13
Napa, CA

nweaver

Member

Slight correction:

Switching to OpenDNS will stop THIS behavior. But it won't stop the other behavior that Paxfire uses, the "helpful" search results when you type a DNS error, since OpenDNS does the same thing.

FFH5
Premium Member
join:2002-03-03
Tavistock NJ

FFH5

Premium Member

Re: Slight correction.

said by nweaver:

Slight correction:

Switching to OpenDNS will stop THIS behavior. But it won't stop the other behavior that Paxfire uses, the "helpful" search results when you type a DNS error, since OpenDNS does the same thing.

If you register with OpenDNS(no pay is needed), you can make OpenDNS not redirect by selecting customer options. It will then not redirect you when you type a bad domain name. It will just give an error.

whfsdude
Premium Member
join:2003-04-05
Washington, DC

whfsdude

Premium Member

Switching DNS

Switching NS servers seems to work for me but of course Earthlink's setup might be different than Paxfire on RCN.

RCN controls their network, Earthlink does not. For example, Earthlink's LPV ADSL2+ is run by COVAD. Customers get put on COVADs IP network. All Earthlink can do is to have COVAD push Earthink's nameservers.

It's unlikely that Earthlink can do a transparent proxy on say COVAD/Megapath's IP network. This is likely why they're using this nasty technique of altering the A records to point to a proxy.

pnh102
Reptiles Are Cuddly And Pretty
Premium Member
join:2002-05-02
Mount Airy, MD

pnh102

Premium Member

One Aside...

Was I the only person who when reading this though, "Earthlink is still in business?"

sk1939
Premium Member
join:2010-10-23
Frederick, MD

sk1939

Premium Member

Re: One Aside...

I was slightly surprised too, but I hear Netzero is still around as well. You can actually still sign up for dial-up access for $10/month.

jchambers28
Premium Member
join:2007-05-12
Peculiar, MO

jchambers28

Premium Member

Still areound ?

I thought Earthlink was dead of near death because lack off access to next gen networks.
Wilsdom
join:2009-08-06

Wilsdom

Member

Re: Still areound ?

They've partnered with several cable companies and usually are cheaper than the host company (probably made possible by this hijacking).

cork1958
Cork
Premium Member
join:2000-02-26

cork1958

Premium Member

Re: Still areound ?

said by Wilsdom:

They've partnered with several cable companies and usually are cheaper than the host company (probably made possible by this hijacking).

I was thinking of something some what similar to this. If they are making an extra $5 a person, why can't any of these ISP's lower the cost to customers by $4 even, and still make a buck?!
cramer
Premium Member
join:2007-04-10
Raleigh, NC

cramer to jchambers28

Premium Member

to jchambers28
Earthlink bought a telco/isp/whatever. (ITC^Deltacom as I recall.)

Sodoshi
join:2009-04-27
Canton, OH

Sodoshi

Member

Epiphany...

Not certain, but this would explain why suddenly whenever I search from the Firefox search bar for Google, I get a "possible automated queries" message (but no CAPTCHA) instead of search results...

Am on Time Warner Cable for an ISP, which I notice was not listed in the article. However this makes more sense than anything else I could think of to explain why I started seeing that message this weekend.
Expand your moderator at work

whfsdude
Premium Member
join:2003-04-05
Washington, DC

whfsdude

Premium Member

CAPTCHA

Click for full size
Screenshot of the CAPCHA I get when I use Earthlink's DNS servers.

mix
join:2002-03-19
Romeo, MI
GL.iNet GL-B1300
Netgear CM500

1 recommendation

mix

Member

Re: CAPTCHA

So basically to google's servers, it looks like every search is coming from the same server since everyone is being routed through a proxy right? It must look like millions of searches are coming from the same ip (or small ip block)... Google should just ban this ip range, then all the calls to earthlink tech. support from angry customers will end this practice pretty quick.

JHW
@unc.edu

JHW

Anon

Domain Wildcarding

So how or why is open DNS wildcarding serch results whereas Google is not? I know what domain wildcarding is but I am still confused by this. At home I've used Open DNS for a long time, before Google had their public DNS 8.8.8.8
nweaver
join:2010-01-13
Napa, CA

nweaver

Member

Re: Domain Wildcarding

It has to do with business models.

OpenDNS's free service is supported by the advertisements users receive on the domain wildcards.

Google Public DNS's free service is supported by

1) Google's ability to perform aggregate analysis on search requests in privacy safe ways (Google Public DNS's excellent privacy policy)

2) That web browsers which encounter NXDOMAIN errors will generally launch a Google or Bing search. So its in Google's overall interest that errors remain errors: a wildcard won't result in more traffic to Google, as they'd be likely to get that traffic anyway.

OpenDNS is also a more costly service to operate than Google Public DNS, because even the free service offers additional filtering features, thus it requires a bigger revenue stream to justify.

JHW
@unc.edu

JHW

Anon

Re: Domain Wildcarding

Thanks for explaining nweaver. That makes more sense. I knew they had to make money somehow, just like the ISPs. I usually have *.wildcards on my personal domains but I've been using the University's DNS servers on my domain joined machine and didn't notice because I get the error page like I am supposed to. I tried a personal subdomain that isn't wild-carded and doesn't really exist on Open DNS. It does pull you to their search page. On Google's 8.8.8.8 I just get the server not found message, like the University's DNS servers when I change my DNS lookup settings.

Now here's the question, what do you all still use? Google or Open DNS? BTW, I really need to join this site. Too much good stuff to miss.
nweaver
join:2010-01-13
Napa, CA

nweaver

Member

Re: Domain Wildcarding

At work I use the work resolvers, and at home I use comcast's non-wildcarding resolvers, having done the opt-out on Comcast for my home account.

If I need to recommend a third-party DNS resolver service, I recommend Google Public DNS, since it is a clean DNS: no wildcarding or other special features, but just a public resolver service with no significant strings attached.

OpenDNS has some very powerful features, but also is aggressive in its wildcard operation, including some behavior that I've reported as a bug.

If a user understands what OpenDNS provides, its an excellent service. EG, a savvy user can use the OpenDNS dashboard to turn off the wildcarding (but this may disable other features).

A business account using OpenDNS Enterprise makes plenty of sense, as their non-free service for businesses allows tight control over various features.
n4bkn
join:2003-12-11
Memphis, TN

n4bkn

Member

Google SSL beta

You might wish to connect to: »encrypted.google.com/ to have a SSL connection (watching for any certificate mismatch messages!).

anon343w45
@anonymouse.org

anon343w45

Anon

Earthlink and Scientology

Earthlink is a company with strong connections to the "Church" of Scientology. Founded by Sky Dayton and Reed Slatkin, both hard-core members of this cult, Earthlink was also staffed by fellow Scientologists. Reed Slatkin later went to jail for defrauding investors, a practice which is, apparantly, exactly the kind of thing that Scientology encourages its business owners to do.

The Scientology cult is also notorious for spying on people, targeting it's own members as well as critics and government officials, employing both legal (though ethically questionable) and illegal methods.

So considering Earthlink's association with an organization that's always displayed utter contempt for people's privacy, maybe we shouldn't be surprised at this latest revelation of Earthlink digging into its user's browsing habits.

NormanS
I gave her time to steal my mind away
MVM
join:2001-02-14
San Jose, CA
TP-Link TD-8616
Asus RT-AC66U B1
Netgear FR114P

NormanS

MVM

Re: Earthlink and Scientology

said by anon343w45 :

Reed Slatkin later went to jail for defrauding investors, a practice which is, apparantly, exactly the kind of thing that Scientology encourages its business owners to do.

Apparently unrelated to Earthlink. They released the following statement:
quote:
"The legal proceedings concerning ex-Board member Reed Slatkin do not involve or impact EarthLink or EarthLink funds. The proceedings involve Mr. Slatkin and his personal clients."

So considering Earthlink's association with an organization that's always displayed utter contempt for people's privacy, maybe we shouldn't be surprised at this latest revelation of Earthlink digging into its user's browsing habits.

Earthlink's use of DNS redirection, and customer tracking is consistent with many smaller ISPs which don't have a demonstrated connection with Scientology. It is probably informed by corporatism, and MBAs; which afflict even the largest corporations (i.e., AT&T and Comcast).

aefstoggaflm
Open Source Fan
Premium Member
join:2002-03-04
Bethlehem, PA
Linksys E4200
ARRIS SB6141

aefstoggaflm

Premium Member

Verizon too

quote:
ISPs who currently use this technology or have in the past include Cavalier, Cincinnati Bell, Cogent, Frontier, Hughes, IBBS, Insight Broadband, Megapath, Paetec, RCN, Wide Open West, XO Communications, Charter and Iowa Telecom

But you guys/gals, forgot about Verizon..

sk1939
Premium Member
join:2010-10-23
Frederick, MD

sk1939

Premium Member

Re: Verizon too

Verizon doesn't use Paxfire technology, and they haven't (in my experience), ever hijacked search engine requests.
nweaver
join:2010-01-13
Napa, CA

1 recommendation

nweaver to aefstoggaflm

Member

to aefstoggaflm
Verizon has used (and may still use) Paxfire to implement their NXDOMAIN wildcarding but HAS NOT used Paxfire to intercept user search requests.

aefstoggaflm
Open Source Fan
Premium Member
join:2002-03-04
Bethlehem, PA
Linksys E4200
ARRIS SB6141

aefstoggaflm

Premium Member

Re: Verizon too

said by nweaver:

Verizon has used (and may still use) Paxfire to implement their NXDOMAIN wildcarding but HAS NOT used Paxfire to intercept user search requests.

It seems I read the story between the line(s), sorry.

Smith6612
MVM
join:2008-02-01
North Tonawanda, NY

Smith6612

MVM

Re: Verizon too

It happens...
Emiya
join:2006-03-30
Southington, OH

Emiya

Member

No need for Public DNS.

It's pretty simple to install some distro of Linux and run BIND as a private DNS resolver. No need to worry and you can enable DNSSEC.
jfmezei
Premium Member
join:2007-01-03
Pointe-Claire, QC

jfmezei

Premium Member

DNS tricks are so 1990s... DPI is the way to go

I am susprised that the ISPs would be playing DNS tricks, especially when they eventually become visible and cause a lot of bad publicity for that ISP.

With DPI equipment such as what Bell Canada installed, they can snoop on the traffic and extract search requests as they pass through and sell those to ad firms. The searches proceed normally and neither the user nor the search engine know that DPI equipent made a copy of the request.

whfsdude
Premium Member
join:2003-04-05
Washington, DC

whfsdude

Premium Member

Re: DNS tricks are so 1990s... DPI is the way to go

said by jfmezei:

I am susprised that the ISPs would be playing DNS tricks, especially when they eventually become visible and cause a lot of bad publicity for that ISP.

..... The searches proceed normally and neither the user nor the search engine know that DPI equipent made a copy of the request.

Earthlink doesn't use their own IP network for the majority of residential networks. It's really resold service (Cable and DSL) so they don't have a place where they could put such DPI equipment.
radambe
join:2011-07-14
San Francisco, CA

radambe

Member

wtf

I pay my ISP for an IP connection to the internet. Period. I don't want anything but that.

EarthlinkLaw
@24.42.91.x

EarthlinkLaw

Anon

Contacted Class Action firm and Earthlink

I contacted Earthlink on this via online support and they refused to acknowledge this.

Contacted earthlink via email and no response back. Also contacted class action firm on this matter.

Oh Earthlink is getting sued on this no doubt about it and everyone knows it now.

Earthlink added this technology and intercepted traffic without any notification to the consumer.
nweaver
join:2010-01-13
Napa, CA

nweaver

Member

Earthlink may have stopped...

An anonymous user in the forum thread just ran Netalyzr and did not see the search-engine interception.

I just did queries to ns1.mindspring.com and ns2.mindspring.com and did not see the search-engine interception either.

So it at least appears that they have stopped intercepting search requests.

sugarqb
@sbcglobal.net

sugarqb

Anon

Re: Earthlink may have stopped...

Nope. No matter the earlier results - earthlink is still intercepting all of my search requests.
EQ
sugarqb

sugarqb to nweaver

Anon

to nweaver
Just ran Netalyzer and confirmed that Earthlink is wildcarding my DNS - Results are as follows:

Major abnormalities:
Your DNS resolver returns IP addresses for names that do not exist

It does not validate DNSSEC. It wildcards NXDOMAIN errors. Instead of an error it returns the following IP address(es): 92.242.140.1.

Your ISP's DNS resolver does not randomize its local port number. This means your ISP's DNS resolver is probably vulnerable to DNS cache poisoning, which enables an attacker to intercept and modify effectively all communications of anyone using your ISP.
We suggest that, if possible, you immediately contact your network provider, as this represents a serious vulnerability.
The following graph shows DNS requests on the x-axis and the detected source ports on the y-axis.

Your ISP's DNS server returns IP addresses even for domain names which should not resolve. Instead of an error, the DNS server returns an address of 92.242.140.1, which resolves to unallocated.barefruit.co.uk. You can inspect the resulting HTML content here.
There are several possible explanations for this behavior. The most likely cause is that the ISP is attempting to profit from customer's typos by presenting advertisements in response to bad requests, but it could also be due to an error or misconfiguration in the DNS server.
The big problem with this behavior is that it can potentially break any network application which relies on DNS properly returning an error when a name does not exist.
The following lists your DNS server's behavior in more detail.
www.{random}.com is mapped to 92.242.140.1.
www.{random}.org is mapped to 92.242.140.1.
fubar.{random}.com is mapped to 92.242.140.1.
www.yahoo.cmo [sic] is mapped to 92.242.140.1.
nxdomain.{random}.netalyzr.icsi.berkeley.edu is mapped to 92.242.140.1.