O Canada! > Canadian > TekSavvy > Google DNS versus ours

reply to TSI Gabe

Re: Google DNS versus ours

reply to TSI Marc

well.. there is one good reason.. and well, it's that it would take a bit of lifting to do it and make sure it's done right and for what? not much value..

we'd have to move everything else off that class C...
--
Marc - CEO/TekSavvy

reply to nbinont

Good news, time to change things up on my setup a bit and see for myself.

reply to nbinont

reply to TSI Gabe
I'm not a Teksavvy customer, but I have a question -- one which after scouring the Internet (assuming what you're using is namebench) nobody has asked.

What on earth does the vertical axis represent? It just says "%". Percentage of what? Packet loss? DNS query rejection (NXDOMAIN, or other error)?

Basically that graph doesn't mean anything unless it's explained 1) where the data is coming from, 2) the exact test being used, and 3) what each axis represents.

For example, namebench.py can send 250 queries to a DNS server. That's nice -- is that 250 concurrent lookups per second? Is that 250 queries total and then it graphs the response time? If the latter, then shouldn't the X axis be query number and the Y axis be response time (in milliseconds), with the visual results being a "scatter graph" followed by a line drawn which indicates the average median?

Surely I can't be the only one questioning what on earth that thing is actually showing.

Otherwise, if I take it to mean "percentage of queries and how long they took to be answered", it looks to me like TSI's servers are taking between 60-200ms about 70% of the time, and 10-20ms the remaining 30% of the time. While comparatively, Google's ervers are taking between 60-200ms about 80% of the time, and 20-30ms the remaining 20% of the time. And to me, that isn't impressive (if anything the results should be the opposite -- first-time query should be slow, but subsequent queries for the same NS/A/PTR/etc. should return almost instantly due to record caching, assuming all recursive records involved don't have stupid TTLs like 1 second. )

Let me show you what actual kernel developers working on UDP stacks tend to graph when it comes to nameserver performance:

»people.freebsd.org/~kris/scaling/bind-pt.png
»people.freebsd.org/~kris/scaling···gige.png
»people.freebsd.org/~kris/scaling···pt-2.png
»people.freebsd.org/~kris/scaling···-nsd.png

Welcome to why just blindly dumping "pretty pictures" isn't helpful without concise (and precise) documentation alongside.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.

I'm sure Gabe will chime in but I think it's pretty straight forward what the graph says...

85-90% of queries take 10ms to return a request and all requests always take less then 200ms...

your graphs show queries per second and load.. we're highlighting how quickly a query is returned not how many it can return which is also an important stat no doubt but given we have 4 servers.. load is less of an issue for us.
--
Marc - CEO/TekSavvy

I don't find this graph straight-forward in any way shape or form.

"85-90% of all queries take 10ms to get a result". Okay, that's because you look at the graph and see that the point where the graph "shoots off horizontally" starts at 85%, with the vertical axis being at 10ms, correct? That's the only way I can see how you reached that conclusion.

Except if you apply the same logic to the data shown on the rights side of the graph, you could safely say that 97% of all queries took 200ms to get a result...

The following graph (X axis = duration, Y axis = nameserver IP) makes perfect sense but doesn't really provide any hard data, though as I said, that one does make sense. It's the first graph that doesn't.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.

reply to NytOwl

reply to koitsu
it's a simple graph...

x axis = time in ms
y axis = % of querries..

if 100 querries were sent, order the results by shortest amount of time and put a dot along the y axis and how much time it took and that's the distribution you would get.
--
Marc - CEO/TekSavvy

reply to koitsu
and no axis of evil
--
Marc - CEO/TekSavvy

reply to TSI Marc
Marc, politely: I've had two other senior systems engineers (like myself) look at the graph. Both of them are equally as perplexed, and in the same way I am.

I'll let Gabe respond from here on out, but I'll explain more verbosely:

What you've described *makes sense* (as in conceptually what you want is doable), but what your first graph actually shows doesn't jibe with what you claim the results are -- and it's because of the type of graph being used + how the data is being graphed.

(Readers should note I ABSOLUTELY believe Teksavvy's claims that their nameservers take ~9ms on average vs. Google's 20-30ms. And the reason for that is quite honestly network round trip time between TekSavvy customer and Google's DNS servers, also taking into consideration authoritative nameservers on the Internet who do not work with large EDNS packets (this adds time to the response)).

I believe the data you have is confusing because you're using a line graph rather than a scatter graph or scatter plot.

Honestly what should be happening under the hood:

Loop iteration #1:

1. Issue 100 DNS queries and keeps track of the response time of each query. Query types will vary (different zones, TLDs, A vs. NS vs. PTR etc.), and response times will vary (some will be cached results, some won't be -- those which aren't should be much higher in response time)

2. Get an average response time: add up all 100 query response times, divide by 100. Result: average response time of 100 queries.

3. Graph result on Y axis, with Y axis label "average response time (in ms) of 100 DNS queries". X axis should be incremental based on time, or simply an incrementing variable ($loopcount++).

Loop iteration #2: repeat step 1/2/3, except in step 3, the X axis location should be further to the right than before, and that you can draw a line from iteration plot data point #1 to iteration plot data point #2.

The resulting graph would look roughly something like this.

The first loop iteration -- assuming all the nameservers its querying have *no cached records* -- should be very slow (high response times due to recursive, non-cached lookups). The 2nd loop iteration should be much faster (cached results), the 3rd as well, etc. etc...

The 2nd to Nth results should be "roughly" all within the same amount of time -- however, this greatly depends on the data set being measured (more specifically: what the per-record TTL is of something being resolved, or the SOA TTL associated with that record's zone).

If you were to take all the graphed averages (how many depends on how many loop iterations you let things run for -- it matters! If just one loop, then the results are worthless!) and put them in their own data set. You could then graph those using a bar graph or bar chart, where each bar would represent response time sections, e.g. 0-10ms, 11-20ms, 21-30ms, etc. and let people see what the "general average" response time is for everything. This is akin (mostly) to the 2nd graph you listed in your post (the blue horizontal bars), except with more granularity.

And trust me, I am quite familiar with data/metrics graphing -- I wrote all of what you see there, sans the dygraphs library, and have had to write an entire code base (all perl + dealing with the mess that is RRDTool) to graph VirtualHost bandwidth usage on Apache (using no third-party modules). Not trying to troll or give you a headache, mate!
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.

The graph is being generated by a tool called namedbench, I believe it's Google themselves that released it. This isn't something I created.

I understand what you are saying though, there are more details the namedbench report spews out that is missing here and I didn't necessarily want to publish it for fear of releasing internal network info.

Understood. And yeah, in my original/first reply, I linked to the namebench site -- their graphs are identical in layout (see "Response Distribution Chart"), meaning the use of a line plot model.

I had a 4th colleague of mine (better educated than myself, especially in mathematics) look at the graphs as well, and he agrees the presentation model is incorrect for what kind of data is trying to be plotted (not that the data itself is wrong!). There are better presentation/layout models (scatter, etc.) that would present the information in a way that makes more sense, but that's not your fault -- it's the fault of namebench. Although since it uses the Google Chart API, the HTTP arguments could be changed to refer to a different model.

The part that shocks me the most is that namebench was written by a pair of Google employees. I'm surprised that someone would write such a useful tool then completely botch the visual representation part. "It's open source, so go fix it, koitsu!" Yeah, and it's Python; I'd rather swallow hot coals.

Anyway, thanks for chiming in and clarifying a bit, TSI Gabe , very much appreciated!
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.

reply to TSI Gabe
Congrats Gabe/Marc et al. Internet access seems much more snappy here on new page loads.

Cheers!
--
If my online experience is enhanced, why are my speeds throttled?? BHell... A Public Futility.

reply to TSI Gabe
Question... On the "NAMEBENCH" tool...

Some of the DNS servers I tested are coming back with the message "Unable to get uncached results for: namebench2802998020.wordpress.com. ...".

They are then excluded from the results rankings, although some raw response times are still posted by namebench.

What exactly does that message mean, and what is its significance in regards to those servers?
edit1: And are there any steps to take to eliminate this situation? Flushing DNS cache, etc.?

edit2: NEVER MIND!!! Duh!

reply to TSI Gabe
Gabe,

You might want to invest in a copy of this bible
»www.edwardtufte.com/tufte/books_vdqi