Last week someone asked me if the DITL 2009 data could shed any light on the amount of queries sent to TLDs with wildcards. While we do have data from a few TLD operators, it wouldn't really help to answer this question. However, I think we can get a "first-order approximation" by looking at the queries to root nameservers. Note that by looking at queries to the roots, we have no knowledge of client queries that are cache hits and those that are sent to the TLD nameservers due to cached referrals. We could perhaps turn to a passive DNS collection, such as SIE for a second opinion.
The long, tall graph below shows the query rate to each TLD seen in the DITL 2009 data. Those TLDs known to have wildcards are shown in blue. Note, however, that some TLDs (such as CN and KR) implement wildcards only for IDN names.
Submitted by wessels@ on Mon, 2009-11-30 18:34.
A couple months ago I posted some data from the OARC reply size test service. Recently some folks have been wondering if the situation is getting better or staying the same. Today I created a graph that shows the monthly trend:
The data probably does not contain enough samples to ascertain any trends. The number of samples in each month is shown at the top of the bars. Also keep in mind that this data has a self-selecting bias. We're only measuring resolvers of people that choose to use this service.
Submitted by wessels@ on Tue, 2009-11-10 19:18.
Here at the RIPE 59 meeting in Lisbon, Joe Abley from ICANN and Matt Larson from VeriSign announced a plan and schedule for signing the Root Zone. A number of interesting tidbits:
The RIPE presentation contains additional details such as key sizes, algorithms and rolling intervals.
Submitted by wessels@ on Wed, 2009-10-07 11:17.
Earlier this year, ICANN contracted with DNS-OARC to study the impacts of potential changes facing the DNS root zone. These changes include: (1) a significant increase in the number of gTLDs, (2) signing the zone with DNSSEC, and (3) continued increase in IPv6 glue. In our study we explore how these changes affect:
This excerpt of the executive summary is taken from the full report:
Our analysis of zone size focuses on memory usage. As expected, we find that memory requirements increase linearly with zone size. We also find that, for a given number of TLDs, signing the zone increases the memory requirement by a factor of 1.5–2. Additionally, we find that 32 GB of memory is insufficient for serving a very large root zone (e.g., a signed zone with 10 million TLDs), particularly when using NSD.
The response latency measurements find negligible increases (typically less than one millisecond) with NSD. For BIND (9.6.0-P1), however, we find some response time degradation with a large signed root zone (e.g., greater than 100,000 TLDs). With a 100,000 TLD signed zone, BIND drops nearly 30% of all queries sent at a rate of 5000 queries per second. With a one million TLD signed zone, BIND drops over 80%. NSD also begins to show some signs of stress with a very large (4.5 million TLD) zone where over 40% of queries are dropped.
The reload and restart times measurements are relatively straightforward and contain no real surprises. Loading and reloading times are generally proportional to zone size. Loading a 1 million TLD signed zone takes 190 seconds with BIND and 227 seconds with NSD.
To measure inter-nameserver bandwidth we performed a number of zone transfers between master and slave nameservers. We tested both standard (AXFR) and incremental (IXFR) zone transfer mechanisms. One interesting result of the AXFR test is that an NSD master utilizes 20–30% less bandwidth than a BIND master to send a given zone. To assess the duration of a zone transfer under wide-area network conditions, we introduced simulated packet loss and delays. A zone transfer experiencing 1% packet loss takes more than 2.5 times longer than with no packet loss for any given tested latency.
To explore increased TCP at root servers, we replayed real query 1streams to servers with signed zones. We found that between 0.3% and 0.8% of responses to UDP queries would be truncated, likely causing most these clients to fall back to TCP. This means that root servers can expect to see at least an order of magnitude increase (e.g., from 5 to 50 per second) in queries over TCP when the root zone is signed. Additionally, we found that a large (e.g., one million TLD) signed root zone will likely result in a slightly higher proportion of TCP queries than a signed version of the current one. Finally, we examined data for the .org TLD from before and after DNSSEC was deployed and found evidence suggesting that the actual increase in TCP-based queries could be significantly higher than can be forecast by evaluating current DNS traffic patterns.
Submitted by admin on Thu, 2009-09-17 18:51.
I was recently asked if OARC had any data on the percentage of DNS queries with bad or disabled UDP checksums. After a few days of crunching through the DITL 2009 data, I have the following results:
Obviously, its interesting that most of the traces show 99% matching checksums, but a few have close to 25% or 50% with mismatches. I'm likely to suspect some kind of "middle boxes" (load balancers?) at play here, but have not yet investigated further.
Mauricio from NIC Chile reports that most of their bad UDP checksums are from replies they send out. They have some Dell hardware running Linux. The Linux installation doesn't support certain NIC hardware features, such as checksum calculations. Hardware checksumming can be disabled with this command:
# ethtool --offload ethXX tx off
Submitted by wessels@ on Wed, 2009-08-19 05:19.
Over on the IETF namedroppers mailing list there is a discussion about DNSSEC and UDP fragmentation. See this thread, for example. Since the OARC Reply Size Test has been going for a couple of months now, I thought maybe it would have enough data for a decent analysis. Here's what I found:
The sample isn't quite as large as I'd like. Only about 1650 unique client addresses tested so far.
The bar plots show histograms of the number of clients (on y-axis) receiving certain maximum reply sizes (x-axis). Each plot is for a different advertised EDNS receive buffer size. Note that the 512 data also includes clients that didn't send any EDNS information.
Normally the Reply Size Test utility won't return responses larger than the advertised buffer size. However, in the 512 data you can see a few counts around 4000. This can happen for one of two reasons: first, a clever user can "trick" the utility by sending queries with certain values in the query name. Second, there is a special mode for Nominum CNS, which won't send EDNS information unless it first receives a truncated response.
The most interesting data is in the 4096-bytes plot. Most of the clients can receive 4K responses. However, about 20% are limited to 2K or less. The bar just left of 1500 on the x-axis represents clients that cannot receive fragmented DNS responses.
Submitted by wessels@ on Tue, 2009-08-18 23:34.
Recent increases in DNSSEC deployment are exposing problems with DNS resolvers that cannot receive large responses.
The maximim reply size between a DNS server and resolver may be limited by a number of factors:
The BIND resolver, since version 9.5.0, includes a feature to decrease its advertised EDNS receive buffer size (down to 512) when its queries time out. We've seen this lead to significant increases in TCP for DNSSEC-signed zones.
DNS-OARC built the DNS Reply Size Test Server to help users identify resolvers that cannot receive large DNS replies.
How To Use
To use the DNS Reply Size Test Server, simply use dig command line tool to issue a TXT query for the name rs.dns-oarc.net:
$ dig +short rs.dns-oarc.net txt
You can test a specific DNS resolver by using the @server feature of dig.
The output should look something like this:
rst.x4001.rs.dns-oarc.net. rst.x3985.x4001.rs.dns-oarc.net. rst.x4023.x3985.x4001.rs.dns-oarc.net. "192.168.1.1 sent EDNS buffer size 4096" "192.168.1.1 DNS reply size limit is at least 4023 bytes"
The first three lines of the output are CNAME records in the response. The "x" numbers represent the sizes of successfully received responses at each step in the test. The final two lines are TXT records that provide information on the test results. Here we can see that the resolver advertised a receive buffer size of 4096 and the server was able to send a response of 4023 bytes.
If your test results in a reply size limit of less than about 4,000 you may want to investigate further for one of the following problems:
The following result comes from a DSL router that does not support EDNS:
rst.x486.rs.dns-oarc.net. rst.x454.x486.rs.dns-oarc.net. rst.x384.x454.x486.rs.dns-oarc.net. "X.X.X.X DNS reply size limit is at least 486 bytes" "X.X.X.X lacks EDNS, defaults to 512"
IP Fragments Filtered
If you're behind a firewall that filters IP fragments, you can expect to see a reply size limit slightly less than 1400 bytes:
rst.x1014.rs.dns-oarc.net. rst.x1202.x1014.rs.dns-oarc.net. rst.x1382.x1202.x1014.rs.dns-oarc.net. "X.X.X.X sent EDNS buffer size 4096" "X.X.X.X DNS reply size limit is at least 1382 bytes"
The test may provide inconsistent results if any of the DNS reply packets are dropped or reordered. The records in the reply have a TTL of 60 seconds or less, so feel free to rerun your test again after waiting 60 seconds.
Truncated, retrying in TCP mode
Some resolvers (e.g. BIND-9.6) send the bloated authority section back to dig. Since dig doesn't set an EDNS receive buffer size by default, the reply may be truncated. You can avoid this problem by telling dig to advertise a large receive buffer. For example:
$ dig +bufsize=1024 rs.dns-oarc.net txt
Note that the EDNS receive buffer size is a "hop-by-hop" parameter, which means that setting it here does not affect the test between your resolver and the OARC server.
Nominum's CNS resolver is designed to utilize EDNS only after first receiving a truncated response. To use this test with a CNS resolver, issue the following query:
$ dig tcf.rs.dns-oarc.net txt
The special name "tcf" instructs the server to set the TC bit in responses if the query doesn't have an EDNS pseudo-record. This should cause CNS to re-query with EDNS.
How It Works
For the technically inclined, here's how the DNS Reply Size Test Server works:
Submitted by admin on Tue, 2009-07-07 21:54.
A couple weeks ago I gave a lightning talk at NANOG46 titled DNSSEC, EDNS and TCP using data from before and after the .ORG zone became signed. Afilias and PIR have been gracious enough to share data from this event with DNS-OARC.
As we recently speculated, the .ORG data clearly shows that queries with DO=1 and a small EDNS buffer size leads to an increase in DNS queries over TCP:
Prior to publishing the signed zone, this server recieved very little TCP traffic (maybe once per 5 seconds). After the signed zone was fully published at 4PM, the TCP query rate is closer to
Next I looked at what types of queries are in the TCP traffic. The following plot shows it is mostly A, MX, and AAAA:
Prior to signing, the TCP traffic is just noise and should be ignored. After signing we see about 70% A, 15% MX, and 10% AAAA. This is more-or-less normal. Upon comparison to the UDP query types, we see that TCP has a higher percentage of A queries, lower percentage of MX, and a higher percentage of AAAA.
I also looked at the Rcodes in the TCP traffic:
It shows 82% NOERROR and 17% NXDOMAIN for TCP, whereas it is 71% and 29% respectively for UDP. In other words, the TCP queries are not due a particular query type or response code.
I was asked for some more specific numbers since it can be difficult to determine values from the logarithmically scaled graphs. The following table shows hourly mean values for the data in the first plot above. All numbers are queries per second:
Submitted by wessels@ on Thu, 2009-07-02 00:20.
Recently, the U.S. Department of Commerece, ICANN, and Verisign announced their cooperation to get the DNS Root zone signed by the end of 2009.
Anyone who has had the pleasure of signing a DNS zone knows that the DNSSEC keys and signatures are much larger than most DNS resource records (and not particularly pretty, either). There is some concern among DNS operators that a signed root zone will lead to truncated responses and queries arriving over TCP, rather than UDP. RFC 4035 has something to say about DNSSEC and message sizes:
DNS-OARC looked at EDNS data last year and found about 65% of queries used EDNS and a message size of 4096, a little less than 35% did not use EDNS at all, and a very small percentage used EDNS with an advertized message size of 512 bytes. More recently we've been asked to look at EDNS message sizes again and see how often a root server receives queries with the DO bit set and a message size of 512 bytes. The results of our analysis of the DITL 2009 data are below:
The first graph (above) shows distributions of queries received from three classes of clients: No EDNS, EDNS with the DNSSEC OK bit clear, and EDNS with the DNSSEC OK bit set. For example, in this plot we can see that "DO=1" queries were received from about one million different clients, the busiest of that group sent more than 10 million queries in this 72 hour period. The "No EDNS" group sent fewer queries overall, but there are more clients in that group.
The second graph shows, again, that about 65% of queries arrive with DO=1 and a buffer size of 4096. These DNS clients meet the requirements of RFC 4035 should be fine with respect to large DNSSEC responses. The blue area represents the approximately 31% of queries that lack EDNS. These should be fine also, since they are not requesting to be sent any DNSSEC records. However, the green area beneath the blue shows that about 2% of queries arrive with DO=1 and a message size of 512 bytes. If these clients switch to TCP, it could present a significant load to the root server system.
The final graph shows a distribution, similar to the first, for the clients in this "DO=1 512 bytes" category. In this 72 hour period, the root server received such queries from about 75,000 unique sources.
Submitted by admin on Tue, 2009-06-09 21:10. categories [ ]
Like .GOV, .ORG is also using the NSEC3 algorithm, which means that versions of BIND prior to 9.6.0 will have problems securely resolving names under .ORG.
As Ram Mohan of Afilias noted in his message to the dnssec-deployment list, there are still significant hurdles in getting the actual registrations signed. Not many registrars accept DS records from customers yet.
Submitted by wessels@ on Tue, 2009-06-02 21:25.
ARI Registry Services
Public Interest Registry
University of Maryland