AWS Engineer here. I haven't worked directly on Route 53 in a few years, but I wrote most of the DNS answering logic back when we built Route 53. Route 53 is a so-called "child centric" DNS server, in that it considers the child zone to be fully-authoritative for its own NS records. That's why the name servers for a parent zone reply with a referral, rather than in the answer, because from their perspective, they don't really "know" the answer. The proper thing a resolver should do is follow the referral and ask the child name servers.
I don't think that's in violation of what the DNS RFCs specify (though the older DNS RFCs are not great at being specific), and I remember making the decision based on what would work best, and be least likely to poison resolvers, in the event of domains being moved between customers, providers, or across accounts for some reason. Setting up the child name servers to say "have moved here ..." can speed those processes up.
I'm sure the Route 53 team will look at this and make whatever change is best, but I also strongly suspect that QNAME minimization may rest on under-specified or un-specified ideas about what authoritative name servers should do. I'd have it follow referrals if it doesn't already.
In any event, I think what's going in this case is more subtle but simpler, basically "ws.fdmg.org." isn't occluding "prod.ws.fdmg.org" when it should, but that takes a very convoluted edge-case set up involving two over-lapping delegations from the same parent zone.
> Route 53 is a so-called "child centric" DNS server, in that it considers the child zone to be fully-authoritative for its own NS records. That's why the name servers for a parent zone reply with a referral, rather than in the answer, because from their perspective, they don't really "know" the answer. The proper thing a resolver should do is follow the referral and ask the child name servers.
> I don't think that's in violation of what the DNS RFCs specify
Agreed, and in fact it's pretty easy to see that the root and .org name servers behave exactly like Route 53 in this regard.
> In any event, I think what's going in this case is more subtle but simpler, basically "ws.fdmg.org." isn't occluding "prod.ws.fdmg.org" when it should, but that takes a very convoluted edge-case set up involving two over-lapping delegations from the same parent zone.
Yes, that appears to be what is happening, and it breaks regular DNS resolution, not just QNAME minimization. If your DNS resolver has the NS records for ws.fdmg.org cached because you previously looked up a ws.fdmg.org hostname, you won't be able to resolve any prod.ws.fdmg.org hostnames, because your resolver will send the queries to the name servers for ws.fdmg.org. I don't know if this is a Route 53 bug or if the customer misconfigured their zones, but if the latter it would be nice if Route 53 could stop people from shooting themselves in the foot this way. To reiterate, this breaks regular DNS resolution, not just QNAME minimization.
That said, Route 53 improperly returns NXDOMAIN for empty non-terminals, which does break QNAME minimization (and violates RFC8020). For a proper example, see opsmate0._domainkey.sslmate.com and _domainkey.sslmate.com.
RFC8020 is an example of the DNS WG making backwards incompatible changes. The "NXDOMAIN" for an empty non-terminal has long been the behavior in DJB's TinyDNS.
That isn't what Route 53 is using, but I bring it up because there have been mailing list threads (or flames?) where folks complained about what DJB's software is doing, and he repeatedly asked "Show me where in the standards this is is specified or forbidden?" and no-one would come up with the goods, but folks still basically wanted "Just do what bind does".
Here's DJB's own mail about it, I had to forward it because his mails don't get through to the mailing list for some reason (that might be the fault of his own stupid qsecretary moderation system): https://www.ietf.org/mail-archive/web/dnsext/current/msg1108...
I generally sympathize with DJB in this case, because DNS really needs some robust and competing implementations. The standards should be tolerant of differences in behavior (as others are), rather than dominated by the proponents of one implementation. It doesn't help that the WG is toxic, at least in my opinion as someone who has been reading it (and mostly just reading) for over a decade. Anyway, I'm not trying to start an argument, just explain. In this case, the bind proponents did dominate, and we get RFC8020, which for some software is a backwards incompatible change.
This isn't the first backwards-incompatible change either. The invention of DNAME introduced a new kind of backwards-incompatible occlusion, and name servers "before" and "after" DNAME respond in different ways, for the same zone file.
Those kinds of changes place providers in a pickle: one of our golden rules is never to make backward incompatible changes or break anything that was working. A promise is a promise. So it means we have to lock things down (the previous paragraph is the reason Route 53 does not accept arbitrary literally-encoded RRs) and when changes come; either not make those changes, or exhaustively audit and determine how many zones, queries it might impact and whether that would change anything. At scale, you'd be amazed how small and subtle behaviors can end up being embedded as assumptions in customer scripts and so on.
I'm with you. I read the complaint twice, and the RFC twice, and I don't quite think it's a bug. It's a weird thing to reason out, but I don't think Amazon's behavior here is wrong, though it might be unexpected...
> The proper thing a resolver should do is follow the referral and ask the child name servers.
You're right, I was clearly not paying attention when writing that part down (fixed now in the forum post).
> [...] but I also strongly suspect that QNAME minimization may rest on under-specified or un-specified ideas about what authoritative name servers should do. I'd have it follow referrals if it doesn't already.
AFAIK, that is not the case, QNAME minimisation is well-specified, and the resolver will follow referrals if it gets them. But in this case the resolver is getting an NXDOMAIN where it should be getting either a NODATA response (if there is no zone at ws.fdmg.org), or it should be getting a referral, if there is a zone at ws.fdmg.org, which should then contain a delegation for prod.ws.fdmg.org.
Since it's getting neither, the resolver will stop resolution when it gets the NXDOMAIN. Now it may be the case that this is either broken empty non-terminal behaviour, or there is actually a zone at ws.fdmg.org, but that zone is missing a delegation for prod.ws.fdmg.org. It's hard to tell from the outside, but if the latter is the case it should be easy to fix.
There is a zone cut at ws.fdmg.org, so I think it's the latter. The zone is misconfigured, that's pretty simple to fix for whoever owns the zone, but the general question of whether Route 53 should be more strict and prevent it from being imported or not is a tough call.
On the one hand it's good to prevent these edge cases and keep things deterministic, but on the other it can cause availability issues for customers if Route 53 were to refuse to import a zone that probably loaded and "worked" just fine with the previous or secondary set up. There's all sorts of ways it's possible to write a mis-configured but "valid" zone file, where records and subtrees of records become unreachable. Most customers rely on test queries and traffic monitoring to observe if there are problems.
In general, at AWS we try to avoid being paternalistic; we believe in giving customers powerful tools and then making those tools intuitive and easy, but I'll admit this can be very challenging with DNS, where the standards and obscure corner cases were never really intended to be understood by an inexpert user-base.
I resisted making this point last night because I didn't want to be seen to deflect. But on reflection, the privacy advocate in me really needs to get this out: QNAME minimization is a really really bad idea and doesn't do anything meaningful for privacy IMO.
The most privacy sensitive domains are still leaked. DNS queries are still in the clear, including the ultimate specific queries for the domain you want to resolve, observable to anyone who can look at your traffic, maybe including your wifi buddies at the local café.
Then along the way the sensitive names are still leaked to parties you probably shouldn't trust in the privacy-protecting model. For example: Verisign can still tell that you are querying for "oddlyspecificpornfetish.com". Dropping the "www", if there even is one, isn't much protection. In return for essentially zero meaningful privacy protection, you have to deal with all sorts of name resolution problems due to making NS queries. That's not a smart trade-off. I'm not saying the goal isn't worthy though, it is.
Agreed, it is a tough call on whether or not Route 53 should try to prevent this type of misconfiguration. I would err on the side of saying you probably shouldn't because before you know it you're going down the rabbit hole of fixing all sorts of weird misconfigurations users make in the DNS.
And yes. DNS seems so simple on the outside until you let a non-expert near it. And even experts make mistakes analysing corner cases (mea culpa).
As to whether or not QNAME minimisation offers privacy, this is debatable, but here I would err on the side of saying: it's a building block that at least prevents some leakage (to e.g. the root).
A more important takeaway for me here is that QNAME minimisation makes misconfigurations such as missing delegations actually cause problems, whereas this would have been masked if QNAME minimisation is not enabled. I'm planning to do a measurement of the frequency at which this occurs.
Finally, as agwa pointed out, Route 53 still has broken empty non-terminal responses, I've added a separate reply to the AWS forum thread with the example agwa used.
> On the one hand it's good to prevent these edge cases and keep things deterministic, but on the other it can cause availability issues for customers if Route 53 were to refuse to import a zone that probably loaded and "worked" just fine with the previous or secondary set up. There's all sorts of ways it's possible to write a mis-configured but "valid" zone file, where records and subtrees of records become unreachable. Most customers rely on test queries and traffic monitoring to observe if there are problems.
Since this problem only materializes if you prime your resolver cache correctly, it's really easy for testing/monitoring to miss it.
Route 53 could issue a warning instead of rejecting the zone outright. For example, Google Cloud DNS issues these warnings if you try to add an SPF TXT record without quotes:
"Warning: A record for this domain has whitespace but is not a "quoted string" and therefore is split into separate strings at whitespace. SPF,
DKIM, and DMARC join those strings without spaces, which can cause "problems, especially for Sender Policy Framework records. Warning: A record for this domain starts with "v=spf1" but lacks a quoted space following the '1'. This may be a badly formatted Sender Policy Framework record that will be ignored by mail software."
That saved someone I know from publishing a bad SPF record.
Heh, this is a fairly easy mistake to make when implementing a DNS server. I’m the person that had to fix this very same bug (and possibly introduced it) for a certain class of domains in the Cloudflare DNS server. “Empty non-terminal” still has an emotional reaction attached to it. :)
Just to clarify, the qname minimization is a privacy technique used during recursion to only disclose what's absolutely necessary.
Traditionally if you're querying www.google.com, you'll:
1. ask root name servers for www.google.com and get list of com name servers
2. ask com name servers for www.google.com and get list of google name servers
3. ask google name servers for www.google.com and get the response
With qname minimization you would:
1. ask root name servers for com TLD and get list of com name servers
2. ask com name servers for google.com and get list of google name servers
3. ask google name servers for www.google.com and get the response
The AWS DNS servers don't strictly follow rfc and don't provide correct responses in specific cases
well I'm not sure if it's sane to implement this rfc.
it's experimental since march 2016. and I do not think that privacy should be a concern when resolving...
I mean what is private about the dns system itself?!
Try counting number of DNS queries for all lookups every day. The software I wrote for myself does this and I can attest that sites using AWS nameservers in almost all cases require more queries than others, including those using similar managed DNS services. The number, which does not include queries to the root can be quite high, e.g., eight. No matter how fast the response times, that delay adds up.
root.zone data is reasonably static. Having a local copy of this data, resolution can be done in two queries: 1. tld ns 2. auth ns, which is the case for many, many www domainnames (the majority, perhaps), or even one query (I think some .mil domains used to be a well-known example). As a user, I view every added query as an additional cost.
It is possible to minimize queries via other means that are under user control and do not rely on third parties.
Running a root locally. Custom stub resolvers with tld ns IPs compiled in. Storing RRs for common nameservers in local zone. Storing DNS data in bulk for important sites that will be revisited, copying to HOSTS and updating the data periodically. I have been using techniques like these for many years. And, it is getting easier every year as more bulk DNS data is available, e.g., internet scans of DNS, TCP DNS services that accept pipelined queries, etc. (If I am not mistaken, the author of the "qname minimisation" draft also was the one with the "DNS looking glass" idea.)
Despite usual mailing list/forum comment/blog memes and mantras, if one surveys all websites (not just popular ones), DNS data is mostly static. That provides great opportunities for speedups by avoiding querying ICANN DNS for the same data 24/7/365. Cue Schopenhauer quote about truth. Someday maybe we will see an internet draft about "DNS minimization" for users.
Did AWS forums start requiring an account for read access?
> Did AWS forums start requiring an account for read access?
No but if you timed out from a session with your account through the console, you might see something about not being signed in. Clicking the "log out" button (or hosing cookies) should rectify it.
AWS DNS is all kinds of fucked. They have implemented a range of undocumented rate limits both to their own DNS server and to external (!) DNS, presumably to reduce amplification attacks. These things have hobbled our services many times, which rely on making thousands of DNS queries per second.
Can you describe a bit more about what you’re resolving? And by rate-limiting external, are you saying they’re throttling DNS traffic to say 8.8.8.8 as well, or do you mean external as opposed to “internal” in a split horizon sense?
I don't think that's in violation of what the DNS RFCs specify (though the older DNS RFCs are not great at being specific), and I remember making the decision based on what would work best, and be least likely to poison resolvers, in the event of domains being moved between customers, providers, or across accounts for some reason. Setting up the child name servers to say "have moved here ..." can speed those processes up.
I'm sure the Route 53 team will look at this and make whatever change is best, but I also strongly suspect that QNAME minimization may rest on under-specified or un-specified ideas about what authoritative name servers should do. I'd have it follow referrals if it doesn't already.
In any event, I think what's going in this case is more subtle but simpler, basically "ws.fdmg.org." isn't occluding "prod.ws.fdmg.org" when it should, but that takes a very convoluted edge-case set up involving two over-lapping delegations from the same parent zone.