Mr. DNS Lives!

Great news everyone! Matt Larson and Cricket Liu are resurrecting their Ask Mr. DNS advice column as a podcast:

Sadly, after Acme’s acquisition by VeriSign in 2000, Mr. DNS began the downward spiral into dissolution and iniquity so familiar to those in the public eye. When Matt and Cricket tracked Mr. DNS down in late 2008, he had been living under the raised floor in the data centers of various charitable hostmasters, answering the occasional technical question on USENET newsgroups and fencing obsolescent Cisco gear to feed his sundry nasty habits.

After months of care and patience, Matt and Cricket have managed to rehabilitate Mr. DNS. He looks forward to the opportunity to earn his redemption by answering your DNS questions. He eagerly awaits your email...

Submitted by on Mon, 2009-01-12 21:56

PowerDNS Server Selection Algorithm

A number of years ago I did some simulations and measurements to show how different DNS resolvers distribute their queries to a set of authoritative servers (see PAM 2004 paper and NANOG29 presentation). At that time I tested BIND, Windows, DJBDNS, and CNS.

Recently someone asked if I know the behavior for PowerDNS. I ran a quick test against PowerDNS 3.1.7 that shows it having a strong affinity to a small number of authoritative servers. It seems to favor the nameservers with the lowest latency, which is not too surprising. In this case, it sent most of its queries to I- and F-roots, both of which are less than 1 ms away from the resolver host.

(click for larger image)

Submitted by on Wed, 2008-12-31 19:45

dig +bufsiz=2048 XN--9T4B11YI5A RRSIG

While working on the TLDmon plugins a couple of weeks ago, I noticed that a certain query to was consistenly failing:

$ dig +bufsiz=2048 XN--9T4B11YI5A RRSIG

; <<>> DiG 9.3.5-P2 <<>> +bufsiz=2048 XN--9T4B11YI5A RRSIG
; (1 server found)
;; global options:  printcmd
;; connection timed out; no servers could be reached

It was strange because queries to the same server for all of the other TLD hosted there work just fine:

$ dig +short +bufsiz=2048 XN--KGBECHTV RRSIG | wc
       6      78    1794
$ dig +short +bufsiz=2048 XN--HGBK6AJ7F53BBA RRSIG | wc
       6      78    1831

It was also strange because the problematic TLD works fine from hosts outside of ISC's network (which is where the OARC servers are located), and it works if the query is sent to or

A tcpdump shows that the DNS reply message is fragmented and we only get the first fragment.

That this problem happens only (?) when querying from ISC's network seems to imply it is caused by something on ISC's network. But then why does it work when querying Why would the second fragment from c arrive, but not the fragment from b? Here's a tcpdump showing the correctly received second fragment from c.

I think it safe to assume that the fragment leaving b is the same, except with different values some TCP header fields (ip_sum, ip_id, ip_ttl, ip_src). Here's another tcpdump showing the fragment from b received outside ISC's network.

Note that both of these fragments are smaller (ip_len=31) than the minimum Ethernet payload size so they are padded out to 46 bytes.

Submitted by on Tue, 2008-12-09 21:22 categories [ ]

TechCrunch Article: Who Protects The Internet?

In his Who Protects The Internet? article, Matt Rutherford mentions something near and dear to us as an example of hackers, er, citizens protecting the Internet:

Just look at Dan Kaminsky, a computer consultant who discovered a fundamental flaw in DNS, allowing him control over any website online. This flaw was astounding in what it gave access to – yet Dan Kaminsky didn’t turn to a government agency or organization, or abuse the hack himself. Instead he made a phone call to Paul Vixie, one of the creators of the BIND9 DNS routing software, and they assembled a team of civilians and private companies to resolve this apocalyptic vulnerability.

Submitted by on Thu, 2008-12-04 17:24

NTIA publishes comments from DNSSEC Notice of Inquiry

On October 9, 2008, the U.S. National Telecommunications and Information Administration solicited public comments on the deployment of DNSSEC. Yesterday was the deadline for submission and today all of the comments have been published. Looks like slightly over 53 comments (of varying coherence) were received.

Submitted by on Wed, 2008-11-26 00:29

Kaminsky Vulnerabilty Leads to Increased DNSSEC Adoption

Eric Osterweil gave an interesting talk at the NANOG44 DNSSEC BOF. His data (shown below) from SecSpider indicates a significant increase in signed zones right around the time that CERT VU #800113 was released:

(click for larger)

The blue line shows a large increase in the number of signed zones automatically discovered by the SecSpider crawlers in August 2008. A message to the dnssec-deployment list confirms this as well.

Submitted by on Mon, 2008-10-13 22:02

DSC now reports EDNS buffer sizes

Last week David Conrad of ICANN asked if I could make DSC show the EDNS buffer sizes advertized by clients. This is now available in DSC versions dated after 2008-08-22. Furthermore, the new version is running on the F-root collector nodes.

The breakdown of buffer sizes looked different than I remembered for recent DITL data, so I generated some graphs for the 2006 through 2008 DITL data (F-root nodes) using the same size ranges.

January 2006

January 2007

March 2008

August 2008

The trend is good news. Back in Jan 2007, 50% of queries did not indicate any EDNS support. 20% had bufsiz=2048, and 30% had bufsiz=4096. Now we have about 65% of queries with bufsiz=4096, while 35% still don't support EDNS.


Submitted by wessels on Mon, 2008-08-25 20:17

ISC SIE cache poisoning attempt detection tool

Date: Mon, 4 Aug 2008 18:22:46 -0400
From: Robert Edmonds
To: dns-operations
Subject: [dns-operations] release of ISC SIE cache poisoning attempt detection tool


ISC SIE has developed a tool for detecting cache poisoning attempts. it consists of two parts: ncaptool, the part which performs packet gathering, reassembly, and dns filtering; and mod_urstate, a message processing module which attempts to statefully detect unsolicited responses that may be indicative of cache poisoning attempts. specifically, the tool is designed to listen at the network layer of a recursive dns server, auditing the query-response stream between recursive and authoritative dns servers. when a potential cache poisoning attempt is detected, both the offending and original dns responses will be emitted into the output stream.

ncaptool and mod_urstate may be obtained via ftp:

the defaults are tuned for a dedicated IDS-style setup; e.g. fairly fast machines with >= 1 GB of memory and aggregated taps of dns traffic between recursive and authoritative nameservers. it ought to be possible to run it directly on a machine running a recursive nameserver, however.

we would like people to use it, and if possible provide feed back or contribute the data it generates to SIE.

mod_urstate is an ncaptool dns message parsing plugin that attempts to detect unsolicited dns responses that may be indicative of cache poisoning attempts. it does this by statefully tracking the application layer state of the dns transactions between recursive and authoritative dns servers. it gracefully handles query retransmissions due to client timeouts and byte identical responses from dns authorities.

two data collections are employed by mod_urstate:

  1. a fifo cache of the query / response payloads generated by dns transactions. a hard limit on the number of entries in this cache will be enforced to bound memory consumption. the default limit of 1048576 entries tends to consume around 700 MB of memory. the larger this cache is, the more accurate the output will be. as a rule of thumb, the cache should be large enough to hold at least 5 to 30 seconds or so of the most recent dns transactions.
  2. an associative array for locating entries in the cache based on a tuple of fields from the packet headers and payload.

when a query for a previously unknown tuple arrives, the query payload is cached so that subsequent byte identical queries may be discarded. when the first response for this query arrives, its payload is also cached. the vast majority of dns queries will only elicit a single response, so most cache entries will be quietly expired in fifo order.

however, in the usual case that a domain name's authoritative nameservers are reachable and functioning, a potentially successful cache poisoning attempt has these properties:

  • the malicious response payload differs in some way from legitimate response payloads.
  • the malicious response matches the tuple for an outstanding query.
  • the malicious response arrives at approximately the same time as a legitimate response.

(it should be noted that certain types of benign dns responses unfortunately also match these criteria. it will be possible to filter these out in the post-processing / analysis phase.)

the mod_urstate module will output dns responses matching the above criteria. this output can then be analyzed to help answer the following questions:

  • was an attempt made to introduce malicious data into my cache?
  • did the attempt succeed? i.e., did the malicious response arrive prior to the legitimate response?
  • what are the contents of the malicious dns response? e.g., in the most common scenario, where is the attacker's doppelgaenger located?

it should be noted that an out of band method of distinguishing malicious from legitimate responses will be needed by the analyst.

Robert Edmonds

Submitted by wessels on Fri, 2008-08-08 22:30

Information about CERT VU #800113

This page provides information relating to CERT VU #800113. Please write to Admin if you have corrections or additions.

Timeline of Events

?, 2008

Dan Kaminsky stumbles upon a serious problem in the DNS protocol that makes poisoning easier than most everyone previously thought.

March 31, 2008

DNS Summit at Microsoft's offices to discuss the problem and solutions.

July 8, 2008

CERT VU #800113 published in conjunction with patches/updates from almost all DNS vendors. Attack details are to be kept secret until August 6th.

July 9, 2008

Kaminsky gives details to skeptical security colleages in confidence.

July 21, 2008

A so-called security expert publicly speculates about the flaw on his blog and more-or-less gets it right. The speculation is confirmed when Matasano publishes a blog-post-in-waiting, which is later retracted. At this point the flaw is effectively leaked.

July 23, 2008

Working exploit code is released.

July 24, 2008

Another metasploit script is released, as well as a python implementation.

July 30, 2008

PCWorld reports that an AT&T nameserver is poisoned, which ends up affecting the author who published the first exploit code.

August 7,2008

Dan Kaminsky gives his talk (ppt file) at the Blackhat conference.

Tools for Testing and Monitoring

OARC's own porttest tool is a DNS server that will tell you if the queries coming from your resolver appear to be random. Use it with command-line tools like dig and nslookup.

Very similar to porttest, except that this one reports on transaction IDs. Most DNS resolver implementations were already using random transaction IDs well before this vulnerability came out, but now you can double-check it with this service.

OARC's Web-based entropy test

Similar to porttest and txidtest, except that the test provides more information and is done with your web browser, rather than command line tools. Another benefit to the web-based test is that you can see scatter plots of the received ports and IDs to visualize the data.

DNS Checker at

Dan's DNS checker also reports on source ports and transaction IDs, although it uses fewer queries than OARC's, and requires javascript.

Niels Provos home page

When you visit Neils' page, an image at the top will tell you if your DNS resolver is using random ports or not.

ONZRA's CacheAudit

CacheAudit is a BSD-licensed tool that allows recursive providers to detect cache poisoning events using cache dumps from
their DNS servers.

ISC's SIE cache poisoning attempt detection tool

ISC SIE has developed a tool for detecting cache poisoning attempts. It consists of two parts: ncaptool, the part which performs packet gathering, reassembly, and dns filtering; and mod_urstate, a message processing module which attempts to statefully detect unsolicited responses that may be indicative of cache poisoning attempts.

Data, Documentation, Papers, and Presentations

Dan's Blackhat talk

Straight from the horse's mouth.

Sid's DNS Ephemeral Port Measurement

Sid Faber is generates data showing levels of patching from SIE data.

Report from reports on patch rates based on the queries they see at the .at authoritative nameservers.

Kaminsky Viz

A flashy visualization of patch adoption from Dan's data.

Steve Friedl's Illustrated Guide to the Kaminsky DNS Vulnerability

A very nice explanation of how DNS works and of the poisoning vulnerability.

Submitted by wessels on Thu, 2008-07-31 22:36 -- Check your resolver's transaction ID behavior

A number of people have been asking for a way to check transaction ID randomness, in addition to source port randomness. OARC's porttest tool has now been expanded to also report on transaction IDs. To use it, issue a TXT query for the name For example, with dig:

$ dig +short TXT
" is GREAT: 26 queries in 2.7 seconds from 26 txids with std dev 20574.11"

Also note that in conjunction with this enhancement, the scoring critera for porttest and txidtest have been changed to match the web-based port test. The scoring is as follows:

Submitted by wessels on Mon, 2008-07-28 17:24 categories [ ]