* * * Important notice to ODVR users * * *
On September 3, 2010, ICANN decommissioned the IANA DNSSEC testbed (announcement here). As an interim measure, we continue to resolve DNS queries sent to the IP addresses that that were used for access to the testbed, namely: 188.8.131.52 and 2001:4f8:3:2bc:1::64:22. However, we will discontinue this service on October 1, 2010. We will continue to offer the BIND and Unbound open DNSSEC-validating resolvers on the following IP addresses:
Submitted by email@example.com on Wed, 2010-09-01 09:35. categories [ ]
The Domain Name System Operations Analysis and Research Center (DNS-OARC) is a non-profit, membership organization that seeks to improve the security, stability, and understanding of the Internet's DNS infrastructure.
DNS-OARC's mission is: to build relationships among its community of members and facilitate an environment where information can be shared confidentially; to enable knowledge transfer by organizing workshops; to promote research with operational relevance through data collection and analysis; to increase awareness of the DNS's significance; and to offer useful, publicly available tools and services.
Provide technical, outreach, and governance leadership to ensure the smooth operation of DNS-OARC as a center of co-ordination, innovation and excellence in the Internet Domain Name System operator community. Manage engineering and research projects required to maintain and develop DNS-OARC's infrastructure and data collection. The setting and raising of high standards of availability and support for DNS-OARC activities. Play a key role in building DNS-OARC as a growing, autonomous, neutral organization.
- All aspects of Management of DNS-OARC, including but not limited to:
- Liaison and outreach to existing and new DNS-OARC Members.
- Implementation of the policies and instructions of the OARC Inc. Board.
- Administration of twice-yearly DNS-OARC public/member meetings.
- Presentation and promotion of DNS-OARC projects, activities and analysis to international meetings.
- Development and evolution of DNS-OARC as a neutral membership organization.
- Creation and management of DNS-OARC documents, website content, and mailing lists.
- Liaison with ISC DNS-OARC Secretariat for administration of OARC Inc. as an autonomous nonprofit legal entity.
- Generation of regular monthly and quarterly reports to DNS-OARC's Board and Members.
- Management, support and development of DNS-OARC's infrastructure, including but not limited to:
- Ongoing system administration of DNS-OARC's multi-server collaboration and data gathering platform.
- Ensure availability and connectivity of DNS-OARC network infrastructure to Internet via ISC's backbone.
- Primary point of contact and assistance for DNS-OARC members and other operators/researchers requiring support.
- Liaison with ISC Operations for hosting and remote-hands services for DNS-OARC infrastructure.
- Enable and ensure effective communication and information sharing within DNS-OARC community.
- Contribute to research on topics relevant to DNS-OARC, including but not limited to:
- Development of DNS-OARC toolset for supporting member community.
- Large-scale gathering of data from DNS servers for projects such as DITL and Root Zone scaling.
- Development of the Domain Statistics Collector and other software.
- Characterization of traffic to key name servers, including the root and top-level domains.
- Analysis of abuse and anomalies in DNS traffic.
- Other operational collaboration and research projects as they emerge.
- Other duties as required from time to time as directed by the OARC Inc. Board in support of DNS-OARC activities and projects.
Minimum job requirements:
- Education: Bachelors in Computer Science or related engineering discipline.
- Experience: At least 5 years experience of Internet Engineering and Unix System Administration in an operational environment.
- Specific Skills:
- Proactive in making decisions, takes initiative and has autonomy.
- Ability to work unsupervised as part of a geographically-distributed team in an international environment.
- Good communications skills both in person and via electronic media, including excellent written and spoken English abilities.
- Presentation skills to provide updates on work to member and industry meetings.
- Unix system adminstration, preferably including FreeBSD, Linux and Solaris.
- Server installation, management, and content generation, preferably including Apache, Postfix, Mailman, Drupal, Jabber.
- Experience of Internet protocol software development projects.
- TCP/IP routing and applications. IP router and ethernet switch configuration and management desirable.
- Specialized Knowledge:
- Understanding of Internet Domain Name System governance, architecture, protocols and current development issues. Familiarity with current best practice and issues in ensuring security and integrity of systems connected to the public Internet in a "hostile" environment. Clue. Candidates already recognized by and active within the Internet technical community are preferred.
Reports to: OARC Inc. Board
Supervises: DNS-OARC Engineering staff
DNS-OARC does not currently have physical offices, so this role will be performed remotely, preferably from a suitably equipped office within easy reach of an international airport.
Travel at least once per quarter to DNS-OARC and other industry meetings in the US and internationally, and to visit DNS-OARC's infrastructure in the ISC data center in Redwood City, California, will be required.
DNS-OARC has assigned a budget of US$80-100k for the fulfilment of the above role.
Candidates should submit a Resume/CV and any supporting documents by-email to firstname.lastname@example.org
by 23:59 UTC on 8th Jan 2010.
Prepared by: Keith Mitchell
Last updated: 18-Dec-09 v1.0
Submitted by admin on Fri, 2009-12-18 17:27. categories [ ]
Recently, the U.S. Department of Commerece
, and Verisign announced their cooperation to get the DNS Root zone signed by the end of 2009.
Anyone who has had the pleasure of signing a DNS zone knows that the DNSSEC keys and signatures are much larger than most DNS resource records (and not particularly pretty, either). There is some concern among DNS operators that a signed root zone will lead to truncated responses and queries arriving over TCP, rather than UDP. RFC 4035 has something to say about DNSSEC and message sizes:
A security-aware name server MUST support the EDNS0 ([RFC2671]) message size extension, MUST support a message size of at least 1220 octets, and SHOULD support a message size of 4000 octets.
DNS-OARC looked at EDNS data last year and found about 65% of queries used EDNS and a message size of 4096, a little less than 35% did not use EDNS at all, and a very small percentage used EDNS with an advertized message size of 512 bytes. More recently we've been asked to look at EDNS message sizes again and see how often a root server receives queries with the DO bit set and a message size of 512 bytes. The results of our analysis of the DITL 2009 data are below:
The first graph (above) shows distributions of queries received from three classes of clients: No EDNS, EDNS with the DNSSEC OK bit clear, and EDNS with the DNSSEC OK bit set. For example, in this plot we can see that "DO=1" queries were received from about one million different clients, the busiest of that group sent more than 10 million queries in this 72 hour period. The "No EDNS" group sent fewer queries overall, but there are more clients in that group.
The second graph shows, again, that about 65% of queries arrive with DO=1 and a buffer size of 4096. These DNS clients meet the requirements of RFC 4035 should be fine with respect to large DNSSEC responses. The blue area represents the approximately 31% of queries that lack EDNS. These should be fine also, since they are not requesting to be sent any DNSSEC records. However, the green area beneath the blue shows that about 2% of queries arrive with DO=1 and a message size of 512 bytes. If these clients switch to TCP, it could present a significant load to the root server system.
The final graph shows a distribution, similar to the first, for the clients in this "DO=1 512 bytes" category. In this 72 hour period, the root server received such queries from about 75,000 unique sources.
Submitted by admin on Tue, 2009-06-09 21:10. categories [ ]
OARC, Inc. is a nonprofit corporation formed under the laws of the State of Delaware. The corporation was created June 30, 2008 (file number 4569769).
Submitted by admin on Wed, 2009-05-13 22:44. categories [ ]
Note: you can skip ahead to BIND fixes.
Recently the hosting company ISPrime became the victim of a DNS-based DDoS attack using spoofed source addresses. Some are calling it an amplification attack because the query ". IN NS" is quite small (47 octets) while an upward referral response is a bit larger (256 octets). For some additional information on this attack, see ISC SANS story #5713. OARC members can track this as incident #10780.
One interesting aspect of this attack is that the queries are apparently sent to authoritative nameservers only. In the past we have seen DNS-based attacks bounce queries through open resolvers. Its not clear why this attack is using authoritative nameservers. Perhaps because it is easy to get a list of nameservers if you already have a list of domain names and you can be reasonably sure that an authoritative nameserver will give some kind of answer.
The attack brings an old debate back into the light: What is an authoriative nameserver's appropriate response to a query that cannot be answered?
The configuration and/or implementation of some authoritative nameservers causes them to return an upward referral to the root zone. We recommend against this behavior for a number of reasons:
- Upward referrals are generally useless. The resolver that is iterating through the space already knows where to start.
- A proper iterative resolver should consider the upward referral "out of bailiwick" and ignore the data anyway.
- The authoritative nameserver's root zone "hints" may become stale over time if not properly maintained, causing delivery of queries to decommissioned root server addresses.
- Upward referrals are known to cause "referral loops" that result in hundreds of useless queries.
We feel that a REFUSED response code is better than an upward referral.
To find out if your authoritative nameservers return an upward referral, send them a query with the dig program:
$ dig @ns.example.net . NS
If you see a long list of ROOT-SERVERS.NET names and addresses, you have an upward referral. Click here to see an example.
Eliminating Upward Referrals
- Disable recursion on authoritative nameservers with this global option:
- If your BIND nameserver is a master for some zones, it needs the root hints to correctly send NOTIFY messages to the slave nameservers. To prevent upward referral responses, you can add this line to the global options:
Then, a query such as ". IN NS" should result in a REFUSED response.
Alternatively, you can use access controls to accomplish the same thing by denying all queries globally and then allowing queries for each zone. Click here to see an example.
- If your nameserver is authoritative and acting only as a slave, it doesn't really need the root hints because it won't send NOTIFY messages. However, simply removing the root hints from your configuration does not solve this problem because BIND comes with the root hints built into the source code. Therefore, you should still use one of the above techniques.
- If your nameserver is both authoritative and caching, you really should separate the two functions. Caching nameservers are susceptible to poisoning and other types of attacks. You don't necessarily need separate hardware for each. You might need to use separate IP addresses, or possibly configure the authoritative nameserver to use an external address while the caching nameserver uses an internal address.
- If you have a caching-only nameserver, make sure that it has proper access controls in place to prevent its abuse by external users.
Submitted by admin on Thu, 2009-01-22 00:06. categories [ ]
The Georgia Tech Information Security Center (GTISC) and ICANN are holding a two-day Global DNS Security, Stability and Resiliency Symposium
in early February 2009. OARC has been invited to coordinate a short technical meeting on February 2nd, the day before the symposium. Participation in the GTISC/ICANN Symposium is by invitation only. Because of limited space and time for the Symposium, the GTISC/ICANN organizers are hoping to direct technical discussion to the Technical Operations day.
||February 2 (Mon), 2009
||Georgia Institute of Technology Campus
||266 Ferst Drive, Atlanta, GA 30332-0765
||Klaus Advanced Computing Building (KACB)
Room 1116 West
Call For Participation
OARC is seeking technical and operational contributions from its members and others to be presented to the symposium audience. Participation in the Technical Operations Meeting is open to:
- All DNS-OARC members
- Anyone willing to present relevant material
- Others by invitation only
The following topics are within the scope of this meeting:
- Case studies of attacks on DNS servers
- Fast-flux domains
- Techniques for quickly identifying attack/abuse conditions
- Evidence of DNS cache poisoning
- Case studies of DNSSEC deployments
- Is overprovisioning the best way to survive an attack? What are the minimums and maximums?
If you would like to make a presentation during the Technical Operations day, please contact the OARC Admin before January 16, 2009
Submitted by admin on Thu, 2008-12-18 22:26. categories [ ]
While working on the TLDmon plugins a couple of weeks ago, I noticed that a certain query to b.iana-servers.net was consistenly failing:
$ dig +bufsiz=2048 @b.iana-servers.net XN--9T4B11YI5A RRSIG
; <<>> DiG 9.3.5-P2 <<>> +bufsiz=2048 @b.iana-servers.net XN--9T4B11YI5A RRSIG
; (1 server found)
;; global options: printcmd
;; connection timed out; no servers could be reached
It was strange because queries to the same server for all of the other TLD hosted there work just fine:
$ dig +short +bufsiz=2048 @b.iana-servers.net XN--KGBECHTV RRSIG | wc
6 78 1794
$ dig +short +bufsiz=2048 @b.iana-servers.net XN--HGBK6AJ7F53BBA RRSIG | wc
6 78 1831
It was also strange because the problematic TLD works fine from hosts outside of ISC's network (which is where the OARC servers are located), and it works if the query is sent to a.iana-servers.net or c.iana-servers.net.
A tcpdump shows that the DNS reply message is fragmented and we only get the first fragment.
That this problem happens only (?) when querying from ISC's network seems to imply it is caused by something on ISC's network. But then why does it work when querying c.iana-servers.net? Why would the second fragment from c arrive, but not the fragment from b? Here's a tcpdump showing the correctly received second fragment from c.
I think it safe to assume that the fragment leaving b is the same, except with different values some TCP header fields (ip_sum, ip_id, ip_ttl, ip_src). Here's another tcpdump showing the fragment from b received outside ISC's network.
Note that both of these fragments are smaller (ip_len=31) than the minimum Ethernet payload size so they are padded out to 46 bytes.
Submitted by email@example.com on Tue, 2008-12-09 21:22. categories [ ]
to monitor operational characteristics of authoritative nameservers for the Root Zone and all Top Level Domains. TLDmon checks for authoritative answers, EDNS support, lame delegations, consistent NS RR sets, open resolvers, expired RRSIGs, matching serial numbers, and TCP support. As the Domain Name System continues its evolution, it becomes increasingly important that these critical nameservers are configured correctly.
TLDmon is available to the public. OARC members can receive notifications (via email) about zone problems directly from Nagios. Members can also request monitoring of additional (non-TLD) zones. Please contact the OARC Admin for more information.
Nagios is really designed to monitor hosts and services that run on those hosts. We've configured Nagios such that each DNS zone is a "host" each and characteristic to be monitored is a "service." TLDmon checks the following operational characteristics of each zone:
The AA service checks that all nameservers set the Authoritative Answer (AA) bit in responses to SOA queries for the zone. When all nameservers set the AA bit, the AA service is marked OK and shown in green. If one or more do not set the AA bit, the service is marked WARNING and shown in yellow. A nameserver that does not set the AA bit may be configured as a caching resolver, rather than an authoritative server. Caching resolvers are susceptible to DNS cache poisoning.
The CLKSKEW service checks that all the clocks of all nameservers are reasonbly within sync to the current time. The check uses a TSIG query to learn the server's time. When all nameservers are within 15 seconds of the correct time, the service is marked OK and shown in green. If any nameserver's clock skew is greater than 15 seconds, the service is marked WARNING and shown in yellow. In this case the nameserver with the largest skew will be shown in the status information field.
The EDNS service checks that all nameservers support EDNS0 extensions. When all nameservers support EDNS0, the EDNS service is marked OK and shown in green. If one or more do not, the service is marked WARNING and shown in yellow. The EDNS0 protocol extension (written in 1999) is necessary for the transmission of UDP DNS messages larger than 512 octets. It is also used to request DNSSEC validation.
The IPV6 service checks that the zone's IPv6-enabled nameservers are working. Only those zones with at least one IPv6-enabled nameserver are checked. When all IPv6-enabled nameservers are working, the service is marked OK and shown in green. If one or more do not, the service is marked WARNING and shown in yellow. IPv6 is increasingly important as the IPv4 free pool shrinks in size.
The LAME service checks that no nameservers are lame for the zone. A nameserver is considered lame when the response to an SOA query for the zone contains no records in the answer section. When no nameservers are lame, the LAME service is marked OK and shown in green. If one or more are lame, the service is marked WARNING and shown in yellow. A lame nameserver results in wasted queries and additional latency for end users.
The NSSET service checks that all nameservers report the same set of NS records for the zone and that they match the delegations from the parent zone. Note that lame nameservers are excluded from this check. If all (non-lame) nameservers report the same NS set, the service is marked OK and shown in green. If there is at least one inconsistency, the service is marked WARNING and shown in yellow.
A nameserver that is known by different names appears as an inconsistency when the delegation name does not match the name listed in the zone. Some people may not consider this a real problem. The NSSET service only checks the nameserver names, not their IP addresses.
The OPENRES service checks that none of the nameservers are open resolvers (i.e., providing recursive resolution to any client). If no nameservers are open resolvers, the OPENRES service is marked OK and shown in green. If one or more are open, the service is marked WARNING and shown in yellow. It is usually a bad idea to mix recursive and authoritative DNS services in a single process, and especially so for top-level zones in the DNS infrastructure. Open resolvers have an increased vulnerability to cache poisoning and denial of service.
The RCODE service checks that all nameservers return response code zero ("NOERROR") in response to an SOA query for the zone. If all nameservers return NOERROR, the RCODE server is marked OK and shown in green. If one or more return an error code (such as SERVFAIL, REFUSED) or no response at all, the service is marked WARNING and shown in yellow. Nameservers that return errors or cause timeouts lead to wasted queries and increased latency for end users.
The RRSIG service checks the expiration time of DNSSEC RRSIG records for zone itself. Zones that do not implement DNSSEC are excluded from this check. If the expiration time for RRSIG records is greater than 3 days, the service is marked OK and shown in green. If one or more RRSIG records is already expired, the service is marked CRITICAL and shown in red. If records expire in less than 3 days, the service is marked WARNING and shown in yellow.
The SERIAL service checks that all nameservers report the same serial number for the zone. When all nameservers report the same serial number, the SERIAL service is marked OK and shown in green. If one or more nameservers has a different serial number, the service is marked WARNING and shown in yellow.
Serial number checking is prone to false alarms due to latencies involved in master/slave synchronization, and in the time that it takes to query multiple nameservers. To reduce false alarms, we tolerate two exceptions to the requirement that serial numbers must match:
- We always tolerate off-by-one differences, such as 12345 and 12346.
- We tolerate a difference of up to 3600 if the serial number appears to be a Unix epoch time value and the maximum serial differs from the current time by less than 3600 seconds.
The TCP service checks that all nameservers respond to queries over TCP, rather than UDP. When all nameservers support TCP, the TCP service is marked OK and shown in green. If one or more do not support TCP, the service is marked WARNING and shown in yellow. TCP support is becoming increasingly important with the deployment of DNSSEC and IPv6.
Long Term Trends
The Trends Graph page shows the number of zones with services in OK, WARNING, and CRITICAL states since the project began. These graphs will help us understand whether things are getting better, worse, or not changing over time.
Please visit the TLDmon code page if you'd like to see the Nagios plugins that we use.
Submitted by admin on Wed, 2008-11-12 18:23. categories [ ]
7 June 2011 UPDATE: The .de zone is now fully signed and the corresponding DS Resource Record has been added to the root zone, so the testbed redirection has been removed from both resolvers.
4 October 2010 UPDATE: We have now added the .de DNSSEC Testbed to both resolvers.
How To Use ODVR
OARC is pleased to offer open DNSSEC-validating resolvers ("ODVR") that anyone can use to experiment with DNSSEC. The IP addresses for ODVR nameservers are:
You might like to manually query the ODVR nameservers with a tool such as dig. Be sure to add the +dnssec option:
$ dig +dnssec @184.108.40.206 iis.se | less
The AD bit in the response flags tells you that the reply data has been validated:
;; flags: qr rd ra ad; ...
Another way to use ODVR is to place the following lines in your Unix /etc/resolv.conf file:
Windows users can manually set DNS servers in the Internet Protocol Properties dialog of a network connection.
ODVR has been configured with the following list of trust anchors:
|Zone||Key Verified||PGP Sig Verified|
ODVR also validates against ISC's DLV registry.
OARC collects data from the ODVR nameservers and makes this data available to our members for research purposes.
These graphs, updated nightly, show the number of queries received with and without the "DO" bit set, and the number of responses sent with and without the "AD" bit set.
Frequently Anticipated Questions
Q: Does it mean all my DNS lookups are secure if I use OARC's validating resolvers?
A: No, probably not, for the following reasons:
- Most zones are not yet signed. Chances are that, for most of your DNS queries, there will not be any DNSSEC signatures. However, we expect this to improve over time as more and more zones take advantage of DNSSEC.
- Most end-user applications (think Web browser) and stub resolvers (a part your computer's operating system) do not yet perform DNSSEC validation. This means that the channel between you and the OARC nameserver is still vulnerable to attack. In other words, security of the DNSSEC transaction is only guaranteed up to the point where the validation has been performed.
Q: Then why are you doing this?
A: A few reasons:
- So that you can play with DNSSEC without changing the configuration of your own nameserver.
- To convince you that a DNSSEC-validating resolver works almost exactly like a non-validating resolver and that you should go ahead and enable DNSSEC on your own resolvers.
- To collect and publish data on adoption of DNSSEC over time.
Q: Can I use ODVR nameservers provide protection from Kaminsky-style spoofing attacks?
A: The answer is complicated and depends on a number of other factors. Generally, this should not be your motivation for using ODVR. If you are stuck using a DNS resolver with poor source port randomization then ODVR may make you more secure. However, a determined attacker could probably spoof answers that appear to come from the ODVR nameservers and give you bad answers.
Q: I thought open resolvers were a bad thing?
A: It's true that open resolvers are usually considered to be a problem and have been used — in combination with source address spoofing — to conduct large-scale DDoS attacks. Such attacks are made possible because (1) there are hundreds of thousands, if not millions, of open resolvers, and (2) their owners/operators are unaware of the openness. The ODVR nameservers are rate-limited and closely monitored. If we have reason to suspect abuse of the ODVR nameservers, we will act quickly to stop it. Please contact the OARC Admin if you have abuse concerns.
Q: Can DNS-OARC members have non-rate-limited access?
A: Absolutely. Write to the OARC Admin to find out how.
Submitted by admin on Tue, 2008-10-28 21:13. categories [ ]
This page catalogs OARC's services to members and the public, its projects and other activities.
Submitted by wessels on Mon, 2008-08-11 21:19. categories [ ]