Upward Referrals Considered Harmful

Note: you can skip ahead to BIND fixes.

Recently the hosting company ISPrime became the victim of a DNS-based DDoS attack using spoofed source addresses. Some are calling it an amplification attack because the query ". IN NS" is quite small (47 octets) while an upward referral response is a bit larger (256 octets). For some additional information on this attack, see ISC SANS story #5713. OARC members can track this as incident #10780.

One interesting aspect of this attack is that the queries are apparently sent to authoritative nameservers only. In the past we have seen DNS-based attacks bounce queries through open resolvers. Its not clear why this attack is using authoritative nameservers. Perhaps because it is easy to get a list of nameservers if you already have a list of domain names and you can be reasonably sure that an authoritative nameserver will give some kind of answer.

The attack brings an old debate back into the light: What is an authoriative nameserver's appropriate response to a query that cannot be answered?

The configuration and/or implementation of some authoritative nameservers causes them to return an upward referral to the root zone. We recommend against this behavior for a number of reasons:

  1. Upward referrals are generally useless. The resolver that is iterating through the space already knows where to start.
  2. A proper iterative resolver should consider the upward referral "out of bailiwick" and ignore the data anyway.
  3. The authoritative nameserver's root zone "hints" may become stale over time if not properly maintained, causing delivery of queries to decommissioned root server addresses.
  4. Upward referrals are known to cause "referral loops" that result in hundreds of useless queries.

We feel that a REFUSED response code is better than an upward referral.

To find out if your authoritative nameservers return an upward referral, send them a query with the dig program:

$ dig @ns.example.net . NS

If you see a long list of ROOT-SERVERS.NET names and addresses, you have an upward referral. Click here to see an example.

Eliminating Upward Referrals


  • Disable recursion on authoritative nameservers with this global option:
    recursion no;
  • If your BIND nameserver is a master for some zones, it needs the root hints to correctly send NOTIFY messages to the slave nameservers. To prevent upward referral responses, you can add this line to the global options:
    additional-from-cache no;

    Then, a query such as ". IN NS" should result in a REFUSED response.

    Alternatively, you can use access controls to accomplish the same thing by denying all queries globally and then allowing queries for each zone. Click here to see an example.

  • If your nameserver is authoritative and acting only as a slave, it doesn't really need the root hints because it won't send NOTIFY messages. However, simply removing the root hints from your configuration does not solve this problem because BIND comes with the root hints built into the source code. Therefore, you should still use one of the above techniques.
  • If your nameserver is both authoritative and caching, you really should separate the two functions. Caching nameservers are susceptible to poisoning and other types of attacks. You don't necessarily need separate hardware for each. You might need to use separate IP addresses, or possibly configure the authoritative nameserver to use an external address while the caching nameserver uses an internal address.
  • If you have a caching-only nameserver, make sure that it has proper access controls in place to prevent its abuse by external users.
Submitted by admin on Thu, 2009-01-22 00:06 categories [ ]

Mr. DNS Lives!

Great news everyone! Matt Larson and Cricket Liu are resurrecting their Ask Mr. DNS advice column as a podcast:

Sadly, after Acme’s acquisition by VeriSign in 2000, Mr. DNS began the downward spiral into dissolution and iniquity so familiar to those in the public eye. When Matt and Cricket tracked Mr. DNS down in late 2008, he had been living under the raised floor in the data centers of various charitable hostmasters, answering the occasional technical question on USENET newsgroups and fencing obsolescent Cisco gear to feed his sundry nasty habits.

After months of care and patience, Matt and Cricket have managed to rehabilitate Mr. DNS. He looks forward to the opportunity to earn his redemption by answering your DNS questions. He eagerly awaits your email...

Submitted by wessels@dns-oarc.net on Mon, 2009-01-12 21:56

PowerDNS Server Selection Algorithm

A number of years ago I did some simulations and measurements to show how different DNS resolvers distribute their queries to a set of authoritative servers (see PAM 2004 paper and NANOG29 presentation). At that time I tested BIND, Windows, DJBDNS, and CNS.

Recently someone asked if I know the behavior for PowerDNS. I ran a quick test against PowerDNS 3.1.7 that shows it having a strong affinity to a small number of authoritative servers. It seems to favor the nameservers with the lowest latency, which is not too surprising. In this case, it sent most of its queries to I- and F-roots, both of which are less than 1 ms away from the resolver host.

(click for larger image)

Submitted by wessels@dns-oarc.net on Wed, 2008-12-31 19:45

dig +bufsiz=2048 @b.iana-servers.net XN--9T4B11YI5A RRSIG

While working on the TLDmon plugins a couple of weeks ago, I noticed that a certain query to b.iana-servers.net was consistenly failing:

$ dig +bufsiz=2048 @b.iana-servers.net XN--9T4B11YI5A RRSIG

; <<>> DiG 9.3.5-P2 <<>> +bufsiz=2048 @b.iana-servers.net XN--9T4B11YI5A RRSIG
; (1 server found)
;; global options:  printcmd
;; connection timed out; no servers could be reached

It was strange because queries to the same server for all of the other TLD hosted there work just fine:

$ dig +short +bufsiz=2048 @b.iana-servers.net XN--KGBECHTV RRSIG | wc
       6      78    1794
$ dig +short +bufsiz=2048 @b.iana-servers.net XN--HGBK6AJ7F53BBA RRSIG | wc
       6      78    1831

It was also strange because the problematic TLD works fine from hosts outside of ISC's network (which is where the OARC servers are located), and it works if the query is sent to a.iana-servers.net or c.iana-servers.net.

A tcpdump shows that the DNS reply message is fragmented and we only get the first fragment.

That this problem happens only (?) when querying from ISC's network seems to imply it is caused by something on ISC's network. But then why does it work when querying c.iana-servers.net? Why would the second fragment from c arrive, but not the fragment from b? Here's a tcpdump showing the correctly received second fragment from c.

I think it safe to assume that the fragment leaving b is the same, except with different values some TCP header fields (ip_sum, ip_id, ip_ttl, ip_src). Here's another tcpdump showing the fragment from b received outside ISC's network.

Note that both of these fragments are smaller (ip_len=31) than the minimum Ethernet payload size so they are padded out to 46 bytes.

Submitted by wessels@dns-oarc.net on Tue, 2008-12-09 21:22 categories [ ]

TechCrunch Article: Who Protects The Internet?

In his Who Protects The Internet? article, Matt Rutherford mentions something near and dear to us as an example of hackers, er, citizens protecting the Internet:

Just look at Dan Kaminsky, a computer consultant who discovered a fundamental flaw in DNS, allowing him control over any website online. This flaw was astounding in what it gave access to – yet Dan Kaminsky didn’t turn to a government agency or organization, or abuse the hack himself. Instead he made a phone call to Paul Vixie, one of the creators of the BIND9 DNS routing software, and they assembled a team of civilians and private companies to resolve this apocalyptic vulnerability.

Submitted by wessels@dns-oarc.net on Thu, 2008-12-04 17:24

NTIA publishes comments from DNSSEC Notice of Inquiry

On October 9, 2008, the U.S. National Telecommunications and Information Administration solicited public comments on the deployment of DNSSEC. Yesterday was the deadline for submission and today all of the comments have been published. Looks like slightly over 53 comments (of varying coherence) were received.

Submitted by wessels@dns-oarc.net on Wed, 2008-11-26 00:29

Kaminsky Vulnerabilty Leads to Increased DNSSEC Adoption

Eric Osterweil gave an interesting talk at the NANOG44 DNSSEC BOF. His data (shown below) from SecSpider indicates a significant increase in signed zones right around the time that CERT VU #800113 was released:

(click for larger)

The blue line shows a large increase in the number of signed zones automatically discovered by the SecSpider crawlers in August 2008. A message to the dnssec-deployment list confirms this as well.

Submitted by wessels@dns-oarc.net on Mon, 2008-10-13 22:02

DSC now reports EDNS buffer sizes

Last week David Conrad of ICANN asked if I could make DSC show the EDNS buffer sizes advertized by clients. This is now available in DSC versions dated after 2008-08-22. Furthermore, the new version is running on the F-root collector nodes.

The breakdown of buffer sizes looked different than I remembered for recent DITL data, so I generated some graphs for the 2006 through 2008 DITL data (F-root nodes) using the same size ranges.

January 2006

January 2007

March 2008

August 2008

The trend is good news. Back in Jan 2007, 50% of queries did not indicate any EDNS support. 20% had bufsiz=2048, and 30% had bufsiz=4096. Now we have about 65% of queries with bufsiz=4096, while 35% still don't support EDNS.


Submitted by wessels on Mon, 2008-08-25 20:17

ISC SIE cache poisoning attempt detection tool

Date: Mon, 4 Aug 2008 18:22:46 -0400
From: Robert Edmonds
To: dns-operations
Subject: [dns-operations] release of ISC SIE cache poisoning attempt detection tool


ISC SIE has developed a tool for detecting cache poisoning attempts. it consists of two parts: ncaptool, the part which performs packet gathering, reassembly, and dns filtering; and mod_urstate, a message processing module which attempts to statefully detect unsolicited responses that may be indicative of cache poisoning attempts. specifically, the tool is designed to listen at the network layer of a recursive dns server, auditing the query-response stream between recursive and authoritative dns servers. when a potential cache poisoning attempt is detected, both the offending and original dns responses will be emitted into the output stream.

ncaptool and mod_urstate may be obtained via ftp:


the defaults are tuned for a dedicated IDS-style setup; e.g. fairly fast machines with >= 1 GB of memory and aggregated taps of dns traffic between recursive and authoritative nameservers. it ought to be possible to run it directly on a machine running a recursive nameserver, however.

we would like people to use it, and if possible provide feed back or contribute the data it generates to SIE.

mod_urstate is an ncaptool dns message parsing plugin that attempts to detect unsolicited dns responses that may be indicative of cache poisoning attempts. it does this by statefully tracking the application layer state of the dns transactions between recursive and authoritative dns servers. it gracefully handles query retransmissions due to client timeouts and byte identical responses from dns authorities.

two data collections are employed by mod_urstate:

  1. a fifo cache of the query / response payloads generated by dns transactions. a hard limit on the number of entries in this cache will be enforced to bound memory consumption. the default limit of 1048576 entries tends to consume around 700 MB of memory. the larger this cache is, the more accurate the output will be. as a rule of thumb, the cache should be large enough to hold at least 5 to 30 seconds or so of the most recent dns transactions.
  2. an associative array for locating entries in the cache based on a tuple of fields from the packet headers and payload.

when a query for a previously unknown tuple arrives, the query payload is cached so that subsequent byte identical queries may be discarded. when the first response for this query arrives, its payload is also cached. the vast majority of dns queries will only elicit a single response, so most cache entries will be quietly expired in fifo order.

however, in the usual case that a domain name's authoritative nameservers are reachable and functioning, a potentially successful cache poisoning attempt has these properties:

  • the malicious response payload differs in some way from legitimate response payloads.
  • the malicious response matches the tuple for an outstanding query.
  • the malicious response arrives at approximately the same time as a legitimate response.

(it should be noted that certain types of benign dns responses unfortunately also match these criteria. it will be possible to filter these out in the post-processing / analysis phase.)

the mod_urstate module will output dns responses matching the above criteria. this output can then be analyzed to help answer the following questions:

  • was an attempt made to introduce malicious data into my cache?
  • did the attempt succeed? i.e., did the malicious response arrive prior to the legitimate response?
  • what are the contents of the malicious dns response? e.g., in the most common scenario, where is the attacker's doppelgaenger located?

it should be noted that an out of band method of distinguishing malicious from legitimate responses will be needed by the analyst.

Robert Edmonds

Submitted by wessels on Fri, 2008-08-08 22:30

Information about CERT VU #800113

This page provides information relating to CERT VU #800113. Please write to Admin if you have corrections or additions.

Timeline of Events

?, 2008

Dan Kaminsky stumbles upon a serious problem in the DNS protocol that makes poisoning easier than most everyone previously thought.

March 31, 2008

DNS Summit at Microsoft's offices to discuss the problem and solutions.

July 8, 2008

CERT VU #800113 published in conjunction with patches/updates from almost all DNS vendors. Attack details are to be kept secret until August 6th.

July 9, 2008

Kaminsky gives details to skeptical security colleages in confidence.

July 21, 2008

A so-called security expert publicly speculates about the flaw on his blog and more-or-less gets it right. The speculation is confirmed when Matasano publishes a blog-post-in-waiting, which is later retracted. At this point the flaw is effectively leaked.

July 23, 2008

Working exploit code is released.

July 24, 2008

Another metasploit script is released, as well as a python implementation.

July 30, 2008

PCWorld reports that an AT&T nameserver is poisoned, which ends up affecting the author who published the first exploit code.

August 7,2008

Dan Kaminsky gives his talk (ppt file) at the Blackhat conference.

Tools for Testing and Monitoring


OARC's own porttest tool is a DNS server that will tell you if the queries coming from your resolver appear to be random. Use it with command-line tools like dig and nslookup.


Very similar to porttest, except that this one reports on transaction IDs. Most DNS resolver implementations were already using random transaction IDs well before this vulnerability came out, but now you can double-check it with this service.

OARC's Web-based entropy test

Similar to porttest and txidtest, except that the test provides more information and is done with your web browser, rather than command line tools. Another benefit to the web-based test is that you can see scatter plots of the received ports and IDs to visualize the data.

DNS Checker at doxpara.com

Dan's DNS checker also reports on source ports and transaction IDs, although it uses fewer queries than OARC's, and requires javascript.

Niels Provos home page

When you visit Neils' page, an image at the top will tell you if your DNS resolver is using random ports or not.

ONZRA's CacheAudit

CacheAudit is a BSD-licensed tool that allows recursive providers to detect cache poisoning events using cache dumps from
their DNS servers.

ISC's SIE cache poisoning attempt detection tool

ISC SIE has developed a tool for detecting cache poisoning attempts. It consists of two parts: ncaptool, the part which performs packet gathering, reassembly, and dns filtering; and mod_urstate, a message processing module which attempts to statefully detect unsolicited responses that may be indicative of cache poisoning attempts.

Data, Documentation, Papers, and Presentations

Dan's Blackhat talk

Straight from the horse's mouth.

Sid's DNS Ephemeral Port Measurement

Sid Faber is generates data showing levels of patching from SIE data.

Report from CERT.at

CERT.at reports on patch rates based on the queries they see at the .at authoritative nameservers.

Kaminsky Viz

A flashy visualization of patch adoption from Dan's data.

Steve Friedl's Illustrated Guide to the Kaminsky DNS Vulnerability

A very nice explanation of how DNS works and of the poisoning vulnerability.

Submitted by wessels on Thu, 2008-07-31 22:36