A Young Goku (from Dragon Ball) wearing his Crane School uniform
Kunall Banerjee

Learning how to dig made me a better programmer

27th April, 2023

This week at work we had a massive DNS resolution failure with one of our datacenters. The issue was quickly resolved, but in the process, they assigned our services new IPs which took forever to propagate. While it is known to most people that DNS propagation can take up to 72 hours, this meant total downtime for our customers in the meantime. Also, because our team could not do anything about it, I simply updated all my tickets, and then took the time to finally sit down and understand how dig works.

Digging with no shovel

A typical dig query looks like dig @server name type. But if you simply dig and do nothing else, you end up getting the 13 root nameservers. They play a fundamental role in the operation of DNS infrastructure.

Root nameservers

Limitations in the original architecture of DNS require there to be a maximum of 13 server addresses in the root zone. Today, due to Anycast routing, it is possible to distribute requests based on load and proximity. Right now, there are over 600 different DNS root servers distributed across every populated continent on earth.

The root nameservers do not directly handle queries for specific domain names but rather provide information about the authoritative nameservers for each top-level domain. These authoritative nameservers, in turn, handle queries for their respective domains.

dig; <<>> DiG 9.10.6 <<>>;; global options: +cmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11663;; flags: qr rd ra; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 0;; QUESTION SECTION:;.				IN	NS;; ANSWER SECTION:.			1800	IN	NS	d.root-servers.net..			1800	IN	NS	h.root-servers.net..			1800	IN	NS	j.root-servers.net..			1800	IN	NS	a.root-servers.net..			1800	IN	NS	k.root-servers.net..			1800	IN	NS	i.root-servers.net..			1800	IN	NS	e.root-servers.net..			1800	IN	NS	g.root-servers.net..			1800	IN	NS	l.root-servers.net..			1800	IN	NS	m.root-servers.net..			1800	IN	NS	f.root-servers.net..			1800	IN	NS	b.root-servers.net..			1800	IN	NS	c.root-servers.net.;; Query time: 127 msec;; SERVER: 100.64.0.2#53(100.64.0.2);; WHEN: Sun Jun 25 18:21:17 IST 2023;; MSG SIZE  rcvd: 228

That… is a lot of noise. Perhaps that’s because I don’t fully know what every line means. Today, I’m going to change that.

The first thing that caught my attention was that dig was using (at the time) an unknown IP for resolving DNS queries. I know I set my primary DNS resolver to Cloudflare’s 1.1.1.1, so where was this IP coming from? man dig to the rescue:

If no server argument is provided, dig consults /etc/resolv.conf; if an address is found there, it queries the nameserver at that address.

Let’s check what my /etc/resolve.conf looks like:

## macOS Notice## This file is not consulted for DNS hostname resolution, address# resolution, or the DNS query routing mechanism used by most# processes on this system.## To view the DNS configuration used by this system, use:#   scutil --dns## SEE ALSO#   dns-sd(1), scutil(8)## This file is automatically generated.#nameserver 100.64.0.2

Ah, macOS, you never fail to amaze me piss me off. Let me check what scutil says:

scutil --dns | grep 'nameserver\[[0-9]*\]' | sort | uniqnameserver[0] : 1.1.1.1nameserver[0] : 100.64.0.2nameserver[0] : 127.0.0.1nameserver[1] : 1.0.0.1nameserver[2] : 2606:4700:4700::1111nameserver[3] : 2606:4700:4700::1001

My machine is using 1.1.1.1 for scoped queries, and 100.64.0.2 is used as fallback.

;; SERVER: 100.64.0.2#53(100.64.0.2)

Now, what is the #53 above? That’s the port number the Internet Assigned Numbers Authority (IANA) assigned to be used for DNS.

Going back to the original query, because I passed no arguments/flags/options to dig, it performed a nameserver (NS) query to the root ., which is at the top of the DNS hierarchy. Basically, if you did a tree of the DNS hierarchy, this is what it would look like:

. (root)├── com (TLDs)│   ├── example│   │   ├── www (SLDs)│   │   └── mail├── ca (CC-TLDs)│   ├── example

The DNS hierarchy is a lot more dense, and I have intentionally left out those bits. You’ll also notice that each domain in the ANSWER section ends with a .. That’s (a.root-servers.net.) what you call a Fully Qualified Domain Name (FQDN). I won’t go any further into this topic, but you should know that that is not an error.

.			1800	IN	NS	a.root-servers.net.

The 1800 above is the time-to-live (TTL) for the entry, or how long the DNS resolver will cache the particular entry to optimize subsequent queries.

.			1800	IN	NS	a.root-servers.net.

The IN stands for the Internet class. Other classes include CH (Chaos net), HS (Hesoid), and NONE for placeholder records. I’ll use CH in an upcoming section for debugging purposes. The chaos class is a special class used for querying server-related information.

Personally, I would have liked it if Chaosnet took off instead of the Internet. I like the name more.

Digging past the surface-level

Being able to query the root nameservers is great, but that’s rarely why you would use dig in day-to-day life. You’re reading this post on kimchiii.space; but what nameservers does my domain use?

dig +short @1.1.1.1 NS kimchiii.space.ns1.vercel-dns.com.ns2.vercel-dns.com.

I’m sure Vercel informs its users what DNS service they use (it’s NS1, but I think in the past they use AWS’ Route 53), but it’s cool that you can find out this information from the comfort of your command-line.

Nice. It looks like ns1.vercel-dns.com. is the primary DNS server for my domain, whereas ns2.vercel-dns.com. is the secondary DNS server.

I want to take it a step further. ns1.vercel-dns.com. and ns2.vercel-dns.com. are authoritative nameservers for Vercel, but what IPs do they resolve to?

dig +short @1.1.1.1 A ns1.vercel-dns.com.198.51.44.13

Who does this IP address belong to? Well if I haven’t fallen victim to DNS spoofing, then whois 198.51.44.13 should return info on NS1.

whois 198.51.44.13...NetRange:       198.51.44.0 - 198.51.45.255CIDR:           198.51.44.0/23NetName:        NSONE-DNSNetHandle:      NET-198-51-44-0-1Parent:         NET198 (NET-198-0-0-0-0)NetType:        Direct AllocationOriginAS:       AS62597Organization:   NSONE Inc (NSONE)RegDate:        2013-08-07Updated:        2021-12-14Comment:        http://nsone.netRef:            https://rdap.arin.net/registry/ip/198.51.44.0...

Phew. Not seeing NSONE Inc as the Organization would have me tripping right now. Looks like I can move on with my experiment.

This shit gettin’ deeper and deeper, I dig it

I’m having way too much fun. I don’t want to stop now. I have already confirmed that they have moved away from AWS for their DNS needs, but what about their hosting? I want to find out more about the IPs used to resolve my domain kimchiii.space.

dig +short @1.1.1.1 A kimchiii.space.76.76.21.2276.76.21.241

Who do these IPs belong to? As of me writing this post, I think Vercel still uses AWS. Let’s confirm that:

...OrgName:        Vercel, IncOrgId:          ZEITIAddress:        340 S LEMON AVE #4133City:           WalnutStateProv:      CAPostalCode:     91789Country:        USRegDate:        2020-03-26Updated:        2020-06-05Comment:        https://vercel.comRef:            https://rdap.arin.net/registry/entity/ZEITI...

Hmm… this is not alarming, but it is also not what I expected. Does this mean Vercel uses its own infrastructure for hosting? I’m not so certain, because ipinfo.io tells me that the ASN (AS16509) for the IP 76.76.21.22 is Amazon. It also confirms that it is allocated for hosting purposes. If you further drill down information on the allocated IP address ranges, you will find that 76.76.21.0/24 is assigned to Vercel, which to me, confirms that Vercel still uses AWS for hosting.

If you were to run the same query, you’d most likely get a different set of IPs each time. This is most likely because Vercel (or rather, NS1) is implementing DNS load balancing or DNS round-robin techniques. This is a good thing.

Surfacing back to the root

Going back to the tree structure of the DNS hierarchy, I should be able to trace the path to my domain’s nameserver from the root nameserver by iteratively querying the authoritative nameservers for each level of the DNS hierarchy. Let’s try that:

dig +short @1.1.1.1 NS .a.root-servers.net.b.root-servers.net.c.root-servers.net.d.root-servers.net.e.root-servers.net.f.root-servers.net.g.root-servers.net.h.root-servers.net.i.root-servers.net.j.root-servers.net.k.root-servers.net.l.root-servers.net.m.root-servers.net.

Now, replacing 1.1.1.1 with either one of the 13 root nameservers:

dig @a.root-servers.net. NS space....;; AUTHORITY SECTION:space.			172800	IN	NS	b.nic.space.space.			172800	IN	NS	e.nic.space.space.			172800	IN	NS	f.nic.space.space.			172800	IN	NS	a.nic.space....

Then, replacing one of the root nameservers with the authoritative nameservers above:

dig @a.nic.space. NS kimchiii.space....;; AUTHORITY SECTION:kimchiii.space.		3600	IN	NS	ns2.vercel-dns.com.kimchiii.space.		3600	IN	NS	ns1.vercel-dns.com....

Bingo. We have arrived at the same nameservers from the previous section. FWIW, you can find out this information using dig +all +trace @server <fqdn>, albeit the output is a bit too verbose for this use-case.

What if I want to find out which nameserver node responds to my queries using dig? As I’m writing this post, I’m connected to a NordVPN server hosted somewhere in London. Because my site is hosted at the edge, I should be hitting an edge datacenter hosted in London, or closest to it. Let’s confirm that:

dig +norec +short @ns1.vercel-dns.com. CH TXT hostname.bind"ns1dns-lhr04-11184-5310"

No recursion here The +norec option here needs to be set so that the direct node is contacted, and nothing in between. hostname.bind is a well-known query name used to obtain information about the hostname of the DNS server being queried

FeelsGoodMan to be right. lhr04 gives it away. LHR is the airport code for Heathrow Airport. Now I’m going to connect to a server in Canada (Montréal):

dig +norec +short @ns1.vercel-dns.com. CH TXT hostname.bind"ns1dns-yyz03-11199-5323"

Interesting. Looks like NS1 does not have any datacenters in Montréal because YYZ is the airport code for Toronto Pearson International Airport. Or I just don’t happen to be hitting it. This opens up an avenue for another experiment. 😏

Other records

There are several DNS record types that I need to check from time-to-time, so I thought I’d mention them here.

ANY record

This is slowly being phased out, but a lot of public DNS servers still respond to this record with results. Cloudflare’s 1.1.1.1 already returns nothing:

dig +short @1.1.1.1 ANY kimchiii.space.

AdGuard and Quad9 DNS return an HINFO in the answer section set to “RFC8482” "".

dig @9.9.9.9 ANY kimchiii.space....;; ANSWER SECTION:kimchiii.space.		60	IN	HINFO	"RFC8482" ""...dig @94.140.14.14 ANY kimchiii.space....;; ANSWER SECTION:kimchiii.space.		60	IN	HINFO	"RFC8482" ""...

You can try other DNS servers. I’ll update this post once I find one that still returns results. But you should probably stop using this record type.

MX record

An MX record directs email to a mail server. In my case, I use ImprovMX for email forwarding. One can confirm this by doing:

dig +short @1.1.1.1 MX kimchiii.space.10 mx1.improvmx.com.20 mx2.improvmx.com.

The highlighted parts above tell you the priority (or preference) with which a mail server, or rather, the Message Transfer Agent (MTA) will try to send emails. Lower number is higher priority. But one can also set the same value for the priority to enable load-balancing.

Good to know MX records have to point directly to a server’s A record or AAAA record. Pointing to a CNAME is forbidden according to RFC 2181

PTR record

DNS PTR records serve the opposite purpose of A records, which provide the IP address associated with a domain name. PTR records are used in reverse DNS lookups. The most common use-case I can think of is a mail server using the reverse lookup to confirm that an email came from the source it claims to have come from.

dig +short @1.1.1.1 NS kimchiii.space.ns1.vercel-dns.com.ns2.vercel-dns.com.dig +short @1.1.1.1 ns1.vercel-dns.com.198.51.44.13dig -x 198.51.44.13...;; QUESTION SECTION:;13.44.51.198.in-addr.arpa.	IN	PTR;; ANSWER SECTION:13.44.51.198.in-addr.arpa. 1800	IN	PTR	dns1.p13.nsone.net....

This is all well and good, but this is the crucial part: looking up the A record of dns1.p13.nsone.net. should return the IP address 198.51.44.13:

dig +short @1.1.1.1 A dns1.p13.nsone.net.198.51.44.13

SOA record

No, not Points of Authority, although that be a way cooler name, IMO. The Start of Authority (SOA) record must exist for every DNS zone.

dig +all +multiline @1.1.1.1 SOA kimchiii.space....;; ANSWER SECTION:kimchiii.space.		3600 IN	SOA ns1.vercel-dns.com. hostmaster.nsone.net. (1655751660 ; serial43200      ; refresh (12 hours)7200       ; retry (2 hours)1209600    ; expire (2 weeks)60         ; minimum (1 minute))...

You can also use the SOA query to find out the primary nameserver for a domain. Here’s another example:

dig +all +multiline @1.1.1.1 SOA example.com....;; ANSWER SECTION:example.com.		3600 IN	SOA ns.icann.org. noc.dns.icann.org. (2022091303 ; serial7200       ; refresh (2 hours)3600       ; retry (1 hour)1209600    ; expire (2 weeks)3600       ; minimum (1 hour))...

The FQDN that follows the highlighted ones above are not actually domains, but rather an email address. For example, in the case of NS1, you would contact hostmaster@nsone.net for any concerns. This email is different from the email address used to contact for abuse. That can be found using whois instead.

SPF record

The Sender Policy Framework (SPF) record type doesn’t actually exist any more, but it can be set using the TXT record. It exists because the Simple Mail Transfer Protocol (SMTP), by default, performs no authentication on the “From” address in an email. Here’s what the SPF record for my domain looks like:

dig +short @1.1.1.1 TXT kimchiii.space."v=spf1 include:spf.improvmx.com ~all"

Because there is no dedicated SPF record, your TXT record must begin with v=spf1, or the server querying your domain won’t know of its existence. You cannot have more than one SPF record per domain, either.

I use an include tag, which tells the server what third-party organizations are authorized to send emails on behalf of the domain. The domain must be a valid one, and in my case, it is set to ImprovMX’s domain.

I also made it so that unlisted emails will be marked as insecure or spam but still accepted, using the ~all option. Other options are -all (reject any and all emails not listed in the SPF record) and +all (any server can send emails on your behalf). You probably don’t want to set the last option. If you don’t set either of the all options, then you must set a redirect: tag which tells the MTA that the SPF record is hosted by another domain.

Final thoughts

I had a lot of fun with this experiment. I started working on this post at ~0100h EST, and it is now ~0800h EST. Now, I’m going to go make me some coffee.

I left out any and all record types pertaining to DNSSEC because otherwise this post would get too long. I think that DNSSEC deserves its own post, so I’m working towards it as you read this.

Another thing I’d like to mention is that I used dig that came pre-installed with macOS (9.10.6), which has no support for DoH or DoT. If you have that requirement, you may want to upgrade dig to 9.16 or higher. You may also wish to use dnscrypt-proxy instead.