Working with public/private/hybrid DNS in Oracle OCI

What you should know: I am currently working for Orbit Cloud Solutions as Cloud Advisor, but any posts on this blog reflect my own views and opinions only.

Interestingly, one of my posts on this blog getting most attention is covering the use of dnsmasq for using hybrid DNS on OCI. Since there is quite a lot of interesting things around DNS on OCI, i decided to prepare another post, giving some overview and then show approaches for using private and hybrid DNS.

DNS in Oracle OCI – Concepts

By default, DNS is pretty simple in OCI.

If you try to reach an internal host using a name following the schema of host.subnet.vcn.oraclevcn.com in your current VCN, a VCN resolver will return an answer containing a private IP address.

If you try to reach an external host like blog.maxjahn.at an internet resolver will pick up the request and return a public IP address.

OCI DNS Servers

There is a default OCI DNS server that will be used by default for the internet and VCN resolver. It is actually a magic, non-routable IP address that is used for instance metadata queries as well: 169.254.169.254.

Clients

So how does a client instance know which DNS servers to query? For Linux there is /etc/resolv.conf that contains IP addresses of nameservers to use. It will look like this for your typical Oracle Linux instance in OCI:

search woods.oraclevcn.com sett.woods.oraclevcn.com
nameserver 169.254.169.254

And how is that information added to /etc/resolv.conf? If you feel couragous, you of course can edit resolv.conf by hand. But most current environments simply will use DHCP to not only get information about IP address assignments but also the DNS servers to use.

With DHCP there are several configurations and settings that can be broadcasted with the DHCP message by the DHCP server. These are generally called DHCP options.

So if we wanted to use a different set of nameservers, e.g. Googles 8.8.8.8 and 8.8.8.4, all we have to do is to modify the DHCP options that will be broadcasted by the DHCP server. This then will break the VCN resolver, so you will no longer be able to look up fox.den.woods.oraclevcn.com.

DNS names

Another thing we need to discuss briefly is how DNS servers actually manage to resolve the millions of domains and hostnames that exist on public internet today.

If a new host is added in a domain, e.g. if i decide i need badger.maxjahn.at to host some badger content, the associated IP address is not written by me directly to one of the root DNS servers. Instead there are several DNS servers (e.g. ns1.easyname.eu and ns2.easyname.eu) that i will be able to add badger.maxjahn.at. These servers are the authorative nameservers for my domain maxjahn.at, whatever is on their records counts.

So other nameservers that need to resolve video.maxjahn.at will generally query these authorative nameservers and then cache the response for some time. This is called DNS recursion and will only work if other nameservers actualy know which authorative nameservers to query for a specific domain.

Another strategy is iterative DNS lookup where the client will query the authorative nameservers directly to get the IP address of the host. This will be slower since no caching will be available and makes the client having to track down the authorative nameservers themself.

To illustrate this, we can look at two simplified examples for recursive and iterative lookup.

Recursive DNS Lookup

In the first example our host badger needs to resolve blog.maxjahn.at using a recursive strategy. It asks dns.google.com for the ip address and receives the answer. In the background dns.google.com might have asked the authorative DNS servers for .at domains at dns.nic.at. As dns.nic.at does not have the answer itself it will ask ns1.easyname.eu. As this is the authorative nameserver for maxjahn.at they have to relevant records and can return the answer 77.244.243.43 for blog.maxjahn.at. Good thing is that the client did not notice what was happening in the background here.

Iterative DNS Lookup

In the second example our host badger needs to resolve build.slashdot.org using an iterative strategy. This time it asks dns.google.com for an answer (non-recursive) and only receives an error message. In the next step the host badger asks the authorative nameserver for .org and too receives an error. Finally it asks the authorative nameserver for slashdot.org at ns0.dnsmadeeasy.com and gets the ip address 216.105.38.15 for build.slashdot.org. Again only this authorative nameserver had the relevant records for this request.

So this simplified overview now should have covered the concepts needed for discussing how to use public, private and hybrid DNS in Oracle Cloud.

Using Public DNS in OCI

When OCI DNS Zones were introduced, i guess they were thinking of people needing to manage their public DNS records with OCI tooling. So this is the case that is documented pretty well in the official documentation.

You first create a DNS Zone for the domain you want to manage either with the web console, oci cli or terraform. But as discussed above, for managing the DNS records you will need access to the authorative nameservers. Therefore you need to issue a request with your DNS registrar to delegate your domain.

And in case you do not already have your own domain: You will not be able to purchase a new domain from within OCI as you could with Azure or AWS.

To use the public DNS you manage with OCI DNS Zones, there will be no need for any changes in your networking or compute instance configuration. As long as the subnet you are using is using the Internet and VCN resolver, you should be able to resolve the public IP addresses of your hosts.

Using Private DNS in OCI

The second scenario of using OCI DNS Zones to manage private DNS records now seems to require some kind of workaround.

Adding a private DNS Zone

The first step is pretty similar to setting up a public DNS zone: You create a DNS Zone for the domain you want to manage either with the web console, oci cli or terraform. The catch this time is to use a domain that you only want to use internally to resolve private IP addresses of your instances on OCI. Things to consider when choosing a private domain:

• As the authorative nameservers will be used by all OCI tenancies alike, your favourite domain name might already be used by somebody else. For example the domain foo.bar is not available anymore for your internal use at the moment i write this post. I guess that some of the obvious choices will not be around for a long time either.
• While it is not an issue if you only use private DNS, you should try to avoid domains that could be resolved by public DNS. First of all, it would make it harder to switch to hybrid DNS later on. And then it is a constant source of confusions and errors to have ambiguous use of domains in your environment.

Creating DHCP Options

From the discussion above we know that now we need to use the authorative nameservers for that DNS Zones instead of the default nameserver 169.254.169.254.

So first we need to get the ips of these authorative nameservers. Just grab the NS records from the DNS Zone and use dig (or nslookup) to get their IP addresses. You can do this by hand or just use the CLI:

oci dns zone get --compartment-id YOUR_COMPARTMENT_OCID --zone-name-or-id YOUR_PRIVATE_DNS_DOMAIN | jq -r '.data.nameservers[] | .hostname' | xargs dig +short

These are the nameservers that we need to be in DHCP options for server instances to use. These DHCP options will use a custom resolver and optionally your private DNS zone name as search domain. Next comes the decision of the scope the DHCP options will be used:

Private DNS for the whole VCN

If you want all subnets in your VCN to use your private DNS, then you simply can change the settings for the default DHCP options of your VCN. First you need to get the OCID of the default DHCP options for that VCN, e.g. using CLI:

oci network dhcp-options list --compartment-id YOUR_COMPARTMENT_OCID --vcn-id YOUR_VCN_OCID --query 'data[?starts_with("display-name",`Default DHCP Options`)].id|[0]' 

Then you can update these default DHCP options and set up to three custom DNS servers and your private search domain.

oci network dhcp-options update --dhcp-id YOUR_DHCP_OPTIONS_OCID --options '[{"type": "DomainNameServer", "customDnsServers": ["162.88.34.6", "169.254.169.254"], "serverType": "CustomDnsServer"}, {"type": "SearchDomain", "searchDomainNames": ["YOUR_PRIVATE_DNS_DOMAIN"]}]'

Now this configuration will be used in all the subnets of that VCN.

Private DNS for only some subnets in the VCN

If you want some subnets to use the internet and VCN resolver and only some others to use private DNS, you want to create a new set of DHCP options for this purpose and assign them to only the latter subnets. First you need to create the new DHCP options for the VCN, e.g. via CLI:

oci network dhcp-options create --compartment-id YOUR_COMPARTMENT_OCID --vcn-id YOUR_VCN_OCID --options '[{"type": "DomainNameServer", "customDnsServers": ["162.88.34.6", "169.254.169.254"], "serverType": "CustomDnsServer"}, {"type": "SearchDomain", "searchDomainNames": ["YOUR_PRIVATE_DNS_DOMAIN"]}]' --display-name "Private DNS DHCP Options"

Make sure to note down the OCID of the newly created DHCP options. Then find the subnet you want to update and update it to use the new DHCP options.

oci network subnet list --compartment-id YOUR_COMPARTMENT_OCID --vcn-id YOUR_VCN_OCID --query 'data[*].["display-name","id","cidr-block"]'

oci network subnet update --subnet-id YOUR_SUBNET_ID --dhcp-options-id YOUR_DHCP_OPTIONS_OCID

Repeat for all subnets you want the DHCP options to use.

Testing Private DNS

So let us take a look at an environment that only uses one of the authorative nameservers for the private OCI DNS zone maxian.oci.

As you can see, you can resolve records from the private DNS zone, but resolving public DNS records fails.

Using Hybrid DNS in OCI

But what if you need to resolve some hosts on the public internet, e.g. for downloading your CentOS updates from https://mirror.easyname.at/centos/ ? This is where the next scenario comes into play.

Adding the Default DNS Server Again

At first glance the solution to add resolution of public DNS is pretty straightforward: Just add the default OCI DNS server 169.254.169.254 again at the end of list of nameserves in DHCP Options.

This works since the authorative nameservers for the OCI DNS zones will not recurse, but the defaul OCI nameserver does. So the client should first try to resolve a domain with those non-recursion authorative OCI DNS zone nameservers and if that fails try the default OCI nameserver.

Again, let us take a look at an environment using this setup:

OK, what was happening here? We cannot trust the default resolving strategy of nslookup, as it will give an answer of a nameserver using recursion precedence over a nameserver not using recursion. If we force it not to rely on recursion with the -norecurse flag, it suddenly works again.

To get a better idea what resolving looks to an application running in the OS, we use a different tooling: getent. This will use the Name Service Switch libraries to look up various informations, one of them being the hosts database. So getent ahosts will return similar results as an application like ping or ssh will see.

So do not fall into the trap of trusting only nslookup when testing DNS resultion on your machines.

Spots on the Shiny New DNS Solution

Great, so this is a simple solution for having hybrid DNS in your VCN. But wait, there still is a catch here:

So there is something bad happening here: With this solution only DNS A records will be resolved. Other record types such as CNAME will not work any longer…

A big caveat here: Do not use this approach if you cannot rule out that the hosts you will be resolving might use CNAME records or similar today or in the future.

Plan B: dnsmasq

What to do if you still need to use hybrid DNS but do not want to take any risks here? There still is the more cumbersome (and costly) approach to set up some DNS forwarders that will route requests for your private domain to the OCI DNS zone authorative nameservers and all others to the default OCI DNS server. I described this approach in one of my earlier posts on private/hybrid DNS in OCI.

Mixing public, private and hybrid DNS in a single VCN

As we have discussed above, the key element to the way domains are resolved in a subnet are the DHCP options. You can mix these approaches in a single VCN by attaching different DHCP options to the subnets.

And since the authorative nameservers for OCI DNS zones are not tied to a specific compartment or VCN, you can use a single private domain for your whole tenancy. As long as the IP addresses are routable, this should work perfectly well.

What next?

One major drawback of the solutions discussed in this post is that you need to manually update DNS records for each server instance that is created or that has its private IP changed. And then you probably want to clean up your DNS records once a VM gets terminated.

One approach to remedy this is using cloud events to trigger updates to the DNS records using serverless functions whenever a VM is changed. I prepared an example implementation for this and describe it in a follow-up blog post.