dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
18737

TSI Gabe
Router of Packets
Premium Member
join:2007-01-03
Gatineau, QC

2 recommendations

TSI Gabe

Premium Member

Google DNS versus ours

Click for full size
Click for full size
There's been a lot of talking going around lately about our DNS being slower/worst than Google/OpenDNS. We're just in the middle of moving in some new improved recursive servers and I think I'm happy with the results.

Here's a graph showing response times comparing one of our DNS servers to Google's.

Not only are the response times almost 3 folds faster, you also get the benefit of using local DNS servers which means that services such as Akamai will deliver local content to you.

Some of these have already been rolled in production, so we are running a mix of old and new. We should have them all working within a week.

derekm
join:2008-02-26

derekm

Member

Curious: where in the network are these measurements taking place?

(i.e. will we see 9ms from home, or would it be more like 16ms (assuming ~7ms RTT to first hop) )

TSI Gabe
Router of Packets
Premium Member
join:2007-01-03
Gatineau, QC

TSI Gabe

Premium Member

That was from my house. (like 10ms away)
mlord
join:2006-11-05
Kanata, ON

mlord to derekm

Member

to derekm
said by derekm:

Curious: where in the network are these measurements taking place?

That's a very good question, though I suppose it shouldn't make any difference -- all of our home traffic passes through TSI on its way elsewhere, so at that point it should still be quicker to use the "local" TSI DNS.

But lots of other more capable ISPs have tried and failed to get this right. Let's see how it works here once rolled out to us masses.
Bugblndr
join:2010-03-02
Burlington, ON

Bugblndr to TSI Gabe

Member

to TSI Gabe
My issue with TSI's DNS Servers hasn't been the speed of them, it's the accuracy. Too many times over the past 3 years I haven't been able to resolve certain websites that worked fine using GoogleDNS.

mlord
join:2006-11-05
Kanata, ON

mlord

Member

Same here.

HiVolt
Premium Member
join:2000-12-28
Toronto, ON

HiVolt

Premium Member

said by mlord:

Same here.

Same thing I've experienced...

TSI Gabe
Router of Packets
Premium Member
join:2007-01-03
Gatineau, QC

TSI Gabe

Premium Member

k, well we are changing these. This is still surprising though since we were using bind...which is pretty much de facto standard DNS server out there.
mlord
join:2006-11-05
Kanata, ON

mlord

Member

Yes, very surprising to us when it happens, too.
If there was a good way to let TSI know about it (specific site lookup failing), we'd probably do it. But it's just easier to run my own DNS (bind) than to monkey around with rebooting my router, changing cables, disconnecting/reconnecting the modem etc.. just to report a DNS issue.

Cheers
mlord

mlord

Member

namebench gives me 25-26msec average lookups for the same server as in post #1 above, versus 45-46msec for 8.8.8.8.

Cheers
MaynardKrebs
We did it. We heaved Steve. Yipee.
Premium Member
join:2009-06-17

MaynardKrebs to TSI Gabe

Premium Member

to TSI Gabe
said by TSI Gabe:

There's been a lot of talking going around lately about our DNS being slower/worst than Google/OpenDNS. We're just in the middle of moving in some new improved recursive servers and I think I'm happy with the results.

Here's a graph showing response times comparing one of our DNS servers to Google's.

In the interest of completeness, since you raised it, where is the graph showing TSI DNS vs OpenDNS?

Just sayin'.

derekm
join:2008-02-26

derekm to mlord

Member

to mlord
said by mlord:

said by derekm:

Curious: where in the network are these measurements taking place?

That's a very good question, though I suppose it shouldn't make any difference -- all of our home traffic passes through TSI on its way elsewhere, so at that point it should still be quicker to use the "local" TSI DNS.

Yeah, it's relative, assuming the DNS server is before the xconnect to Google, which would make sense.

I was more asking if we're looking at 10ms + first hop RTT (done at NOC), or say 2ms + first hop RTT (done at home).

TSI Gabe
Router of Packets
Premium Member
join:2007-01-03
Gatineau, QC

TSI Gabe to MaynardKrebs

Premium Member

to MaynardKrebs
Click for full size
Click for full size
said by MaynardKrebs:

said by TSI Gabe:

There's been a lot of talking going around lately about our DNS being slower/worst than Google/OpenDNS. We're just in the middle of moving in some new improved recursive servers and I think I'm happy with the results.

Here's a graph showing response times comparing one of our DNS servers to Google's.

In the interest of completeness, since you raised it, where is the graph showing TSI DNS vs OpenDNS?

Just sayin'.

Here you go. OpenDNS performs better than Google, but ours is still better

TSI Marc
Premium Member
join:2006-06-23
Chatham, ON

1 edit

TSI Marc to Bugblndr

Premium Member

to Bugblndr
said by Bugblndr:

My issue with TSI's DNS Servers hasn't been the speed of them, it's the accuracy. Too many times over the past 3 years I haven't been able to resolve certain websites that worked fine using GoogleDNS.

Can you point to a thread on here where that was proven to be our server.

Each time I've seen this it turned out that it was the site at the other end that was in fact down or had their dns records messed up and only showed up on google or opendns because those had not yet updated to the latest records or visa-versa..

derekm
join:2008-02-26

derekm to TSI Gabe

Member

to TSI Gabe
These, in the vast majority of cases relate to who is caching what.

I looked at one of these complaints earlier, and it turns out, google had a later version of a DNS record.

A breakdown is like this:
Time 0:
 
A record is www.example.org 10.1.1.1  TTL 3600 (3600, one hour, usually much higher (like a day) for crappy dns providers)
 
TSI DNS: no copy
Google DNS: no copy
 
Time 1:
 
A Google user checks out the site
 
TSI DNS: no copy
Google DNS: 10.1.1.1, 3600  (will be cached for 3600 more seconds)
 
Time 1800:
 
A TekSavvy user checks out the site
 
TSI DNS: 10.1.1.1, 3600
Google DNS: 10.1.1.1, 1800
 
Time 3500:
 
Site decides they wish to update their A record to 10.1.1.2
 
TSI DNS: 10.1.1.1, 1900
Google DNS: 10.1.1.1, 100
 
Time 3600:
 
Googles cache expires
 
TSI DNS: 10.1.1.1, 1800
Google DNS: no copy
 
Time 3601:
 
Google user requests the site again
 
TSI DNS: 10.1.1.1, 1700
Google DNS: 10.1.1.2, 3600
 
For the next 1700 seconds (~29 mins), TSI will still be pointing at the old IP.

You can use dig to troubleshoot this:
$ dig teksavvy.com @8.8.8.8 && dig teksavvy.com @206.248.142.222
 
; <<>> DiG 9.8.1-P1 <<>> teksavvy.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27201
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
 
;; QUESTION SECTION:
;teksavvy.com. IN A
 
;; ANSWER SECTION:
teksavvy.com. 30 IN A 206.248.155.70
 
;; Query time: 35 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Oct 11 10:25:57 2012
;; MSG SIZE  rcvd: 46
 
; <<>> DiG 9.8.1-P1 <<>> teksavvy.com @206.248.142.222
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39145
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
 
;; QUESTION SECTION:
;teksavvy.com. IN A
 
;; ANSWER SECTION:
teksavvy.com. 1800 IN A 206.248.155.70
 
;; Query time: 10 msec
;; SERVER: 206.248.142.222#53(206.248.142.222)
;; WHEN: Thu Oct 11 10:25:57 2012
;; MSG SIZE  rcvd: 46
 

Here you can see the entry on 8.8.8.8 will live for another 30s (teksavvy.com. 30 IN A 206.248.155.70), while on 206.248.142.222, it will live for another 1800s. If the owner changes the record between 30 and 1800s google will get the right answer, TSI will not.

This is how DNS should work, just you are getting lucky sometimes, as opposed to TSI's servers being problematic.

Try this the next time you run into problems and see what you get.

EDIT: this was directed to people having problems with TSI servers, not to TSI
derekm

1 edit

derekm

Member

Might be interesting to automate a cache purge if requested from MyTools.

You could have the client submit problematic DNS record from the website, compare your DNS server's copy to google & opendns. If your serial is less than theirs, flush the cache entry.

You could report that the entry is fine, or out of date and flushed.

EDIT: all the lookups, and comparisons are done on the webserver, when authenticated, so it wouldn't affect actual DNS performance, and shouldn't raise security issues. Just the cache purge for that record happens on the name server see: rndc flushname

TSI Marc
Premium Member
join:2006-06-23
Chatham, ON

TSI Marc to derekm

Premium Member

to derekm
In the breakdown you show.. the times listed are how much longer it will remain cached based on when it was last refreshed.. in other words, in your example, it's not because google updates more often then ours, that's generally set by the DNS record holder.. it's like you say.. depends on when it was last updated..

Like I mentioned, each time I hear somebody say our dns this or that.. I kind of scratch my head because I haven't seen a case where it was clearly our dns server at fault...

this new upgrade though is pretty sweet.. nice to see that it's significantly faster.

derekm
join:2008-02-26

derekm

Member

Agreed - it's how DNS should work.

You could build the facility to report out of date zones - and have it automatically QC'd. Would it ever get used? That's a different question!

And yes, great job on the new servers!
bbiab
join:2004-05-26

bbiab to TSI Gabe

Member

to TSI Gabe
said by TSI Gabe:

Some of these have already been rolled in production, so we are running a mix of old and new. We should have them all working within a week.

Please update once you have completed this.
Bugblndr
join:2010-03-02
Burlington, ON

Bugblndr to TSI Marc

Member

to TSI Marc

said by TSI Marc See Profile
Can you point to a thread on here where that was proven to be our server.

Each time I've seen this it turned out that it was the site at the other end that was in fact down or had their dns records messed up and only showed up on google or opendns because those had not yet updated to the latest records or visa-versa..

I know when I've experienced the issue, it was not caused because the website was down. I don't know for sure it was a TSI DNS server issue, I just know that switching the DNS server I used solved the issue instantly.

Could it be related to caching as derekm suggests? Perhaps in the odd instance.

Either way, good work on speeding things up.

derekm
join:2008-02-26

derekm

Member

@TSIMarc:
An even easier solution, set the max-cache-ttl and max-ncache-ttl :

»www.zytrax.com/books/dns ··· ache-ttl

You can auto-expire your cache settings every 10 minutes - 1 hour.

TSI Marc
Premium Member
join:2006-06-23
Chatham, ON

1 edit

TSI Marc

Premium Member

yeah but that doesn't lead to better performance.. you want records that are accurate to stay cached so their response is fast.. having to go query the auth dns server where the records are hosted is the part that takes the longest...

sometimes this is why larger dns servers may perform better since they have more queries cached by virtue of simply having more people use it... in that way they dont need to delay while it fetches the record, somebody else has already done that in some cases so to you it appears quicker. typically that's where people might argue that its better.

at our size now though I'd say that's fairly moot but to me that's the only real legitimate thing one could say.. the more users use our servers.. the less of an issue that is too..
The Mongoose
join:2010-01-05
Toronto, ON

The Mongoose

Member

I caught this thread and decided to re-run namebench...a while back I was having some trouble and switched everything over to Google DNS. Today's test results:

Fastest: TekSavvy
2nd: UltraDNS (31% slower than TSI)
3rd: Primus (35% slower)

Other notables...OpenDNS was 37% slower, Google DNS was 48% slower.

I've switched back to TekSavvy's servers as a result, so far working fine.

HiVolt
Premium Member
join:2000-12-28
Toronto, ON

HiVolt to TSI Marc

Premium Member

to TSI Marc
I'll switch back to the TSI servers and see how they perform now...
voxframe
join:2010-08-02

voxframe to TSI Gabe

Member

to TSI Gabe
Bench tested TSI servers today and they blow the list away. Wow!

TSI Gabe
Router of Packets
Premium Member
join:2007-01-03
Gatineau, QC

TSI Gabe

Premium Member

We still have one server left to port over, so it's still technically not at it's best yet.
The Mongoose
join:2010-01-05
Toronto, ON

The Mongoose

Member

Do we get to find out which ones have been switched over so we can use the new ones? Or does it not matter (I can never fully wrap my head around DNS architecture)?

fluffybunny
@teksavvy.com

fluffybunny to TSI Marc

Anon

to TSI Marc
hmm..i get much worse performance with teksavvy DNS than with google :
using GRC nameserver benchmark :

206.248.142.222 | Min | Avg | Max |Std.Dev|Reliab%|
----------------+-------+-------+-------+-------+-------+
- Cached Name | 0.000 | 0.016 | 0.081 | 0.021 | 100.0 |
- Uncached Name | 0.055 | 0.196 | 0.514 | 0.111 | 100.0 |
- DotCom Lookup | 0.077 | 0.163 | 0.272 | 0.052 | 98.0 |
------+-------+-------+-------+-------+-------+

vs

8.8.8.8 | Min | Avg | Max |Std.Dev|Reliab%|
----------------+-------+-------+-------+-------+-------+
+ Cached Name | 0.000 | 0.019 | 0.082 | 0.023 | 100.0 |
+ Uncached Name | 0.047 | 0.165 | 0.440 | 0.101 | 100.0 |
+ DotCom Lookup | 0.048 | 0.166 | 0.283 | 0.060 | 100.0 |
------+-------+-------+-------+-------+-------+

derekm
join:2008-02-26

derekm to TSI Marc

Member

to TSI Marc
said by TSI Marc:

yeah but that doesn't lead to better performance.. you want records that are accurate to stay cached so their response is fast.. having to go query the auth dns server where the records are hosted is the part that takes the longest...

Understood. It wasn't to suggest it would improve performance. It would improve accuracy though. It's a trade off.

If it's 100-500ms (500 would be terrible) every 10-60 minutes that get hit with the latency, I don't think it would even be noticeable, if you look at the average response time.

Also, the end-users computer would respect the TTL in its DNS cache, so it would only be new requests.

You could override this TTL cap for the top 1% of the queries, fairly easily, which would give you better performance for most sites, and better accuracy for the 'long tail'.

Just throwing some (unrequested) ideas out there.

TSI Marc
Premium Member
join:2006-06-23
Chatham, ON

TSI Marc

Premium Member

yep. that's sort of how this is.. you kind of have to find a good balance based on the load that's on our servers.. there's no one size fits all kind of solution.

thanks for the input.