DNS Failover for the Home Lab and Why You Need It

We have all heard the very common expression, “it is always DNS”. We joke about this often, but very often DNS is at the heart of issues with connectivity. Incorrect DNS records, reverse lookups, or just overall DNS connectivity issues play a role in major outages. This can also affect the home lab environment. You may not think about it this way at first, but having DNS failover and redundancy is extremely important to have a resilient environment for your learning and home network. Having something go wrong in your home lab network with DNS can cause a major outage. Let’s look closely at DNS failover and a few different designs to think about.

What Is DNS Failover in the context DNS resolvers?

When thinking about DNS resolution and DNS failover, it refers to having more than one DNS server or resolver that is listed in your client’s network configuration. If the primary DNS server fails or isn’t reachable any longer by clients, the system will automatically fall back to the secondary server for DNS resolution.

With this setup, it makes sure that devices and services in your home lab can still resolve your hostnames to IP addresses. This can happen even if your main DNS server is down due to hardware failure, power outage, or some type of misconfiguration.

Why DNS Failover is important with a Home Lab

In most enterprise networks, having redundant DNS servers is pretty standard. However, it may be something that is skipped in the home lab, or just overlooked.

However, this is an aspect of your home lab infrastructure that you shouldn’t skimp on since most likely the following are things you have implemented or are experimenting with:

• Internal DNS zones for your local services (like plex.home, nas.home, vm1.lab)
• Custom DHCP and DNS setups (maybe using pfSense, Pi-hole, dnsmasq, unbound)
• Split-horizon DNS (internal and external domains resolving differently)

If you only have one DNS server, if this DNS server goes offline, your entire lab environment can become unusable. Even if the VMs, containers, or pods are running just fine, if DNS goes down, clients or other internal components won’t know how to reach the resources it needs.

How DNS failover works

Multiple DNS addresses can be configured in most operating systems. When you configure multiple DNS servers, these can be configured by DHCP for clients or manually when setting an IP address on a specific subnet.

Clients will try these in order:

1. The primary DNS server is queried for records first
2. If it responds, resolution is successful
3. If it times out or is unreachable, then the secondary DNS server is queried for name resolution
4. If both fail, resolution will just fail

Different OSes and devices handle failover slightly differently, but the principle is the same: DNS failover makes sure you have name resolution by switching to the next available server.

Important Note:

DNS failover at the resolver level does not load balance. It’s strictly failover. Most clients won’t switch to the secondary server unless the first one fails or is slow to respond.

Real-World Home Lab benefits for DNS failover

Note the following scenarios and benefits to having DNS failover capabilities with your DNS servers.

1. Primary DNS Server Reboot or Maintenance

If you’re running Windows Server DNS, Unbound, BIND, or Pi-hole on a Raspberry Pi or VM and need to reboot for updates or other maintenance, you don’t want to have to take down your entire lab stack. A secondary DNS server keeps everything working during downtime of the primary host.

2. DNS on the Edge Router Only

Many home lab setups use pfSense or OPNsense as both firewall and DNS forwarder. If the router reboots, you lose DNS resolution. By configuring and having a secondary DNS server running on another device, like a Linux VM, it keeps from having this single point of failure.

3. Dynamic Home Networks with DHCP

Most clients, including PCs, VMs, mobile devices, get their DNS info from DHCP. Making sure your DHCP server has two or more DNS servers configured gives your clients multiple hosts to forward DNS queries to. This helps in situations where you need to perform maintenance as we have already mentioned or have a failure.

4. Cloud resolvers also need secondary addresses

Even if you’re using public DNS (like Google DNS, Cloudflare, or Quad9), it’s a good idea to configure more than one address for your resolvers. If your ISP blocks, throttles, or tries to misroute traffic to one address, clients and your network will still be able to use the secondary address. Many also like to configure secondary addresses from a different public DNS provider so that if one provider has issues, your second resolver should still be working.

Recommended DNS Failover Configurations

Let’s now look at a few DNS failover configurations for having multiple resolvers in your home lab environment and which configurations you may choose.

Option 1: Two local DNS servers

This is probably the most basic and it is to have two DNS servers inside your home lab. Also, run these on separate virtual hosts, if you are running in VMs or separate physical hosts if you are running on baremetal.

Example configuration that provides DNS failover between different types of hardware:

• Primary: dns1.lab.local on 192.168.1.10 (Unbound on Ubuntu)
• Secondary: dns2.lab.local on 192.168.1.11 (Pi-hole on Raspberry Pi)

You would then update your DHCP server configuration to hand out both addresses to your clients.

DNS Servers: 192.168.1.10, 192.168.1.11

Below is an example in my environment of two DNS servers (Pi-Hole servers) configured for a client DHCP scope:

Option 2: One local DNS server, One public DNS server

If you don’t have the hardware or want to have two separate physical servers or virtual hosts to run multiple DNS servers, a great workaround is to have one local DNS server and one public DNS resolver.

Example:

• Primary: 192.168.1.10 (Pi-hole or Unbound)
• Secondary: 9.9.9.9 (Quad9) or 1.1.1.1 (Cloudflare)

Below is an example of a Windows host in this configuration, with the internal resolver as the primary and the secondary resolver configured to a public DNS server:

This setup allows you to have local hostname resolution(via the local server) but falls back to external DNS to allow internet DNS resolution when your local service is down. Keep in mind your resolution of local resources would still fail in this configuration if you have an outage of your local DNS resolver. However, this is a good mixed configuration that at least keeps the family happy if Internet resolution still works.

Option 3: Public DNS Only with Redundancy

If you don’t self-host any DNS servers in your home network or any self-hosted resources that rely on DNS internally, you can still configure redundant public DNS resolvers. Note the following as examples of multiple resolvers.

1.1.1.1 (Cloudflare)
1.0.0.1 (Cloudflare backup)
8.8.8.8 (Google)
9.9.9.9 (Quad9)

Some routers and DHCP servers let you list 2-4 DNS servers.

Monitoring your setup and failing over

Let’s look at a few considerations for monitoring and testing your configuration.

1. Configure multiple DNS servers in your DHCP configuration

There is no wrong device or solution to serve out DHCP. However, just make sure you configure multiple DNS servers in your DHCP scope for resolution.

2. Monitor your DNS server health

There are many enterprise solutions and open-source solutions that allow you to monitor DNS name resolution and query health. Just a few to mention:

• Nagios or check_dig: To alert if a DNS server becomes unresponsive
• Netdata: Monitors DNS queries per second and failure rates
• Grafana + Prometheus: This is a great open source solution for visualizing your DNS query health

3. Testing Failover

You can test DNS failover using tools like dig or nslookup:

dig @192.168.1.10 google.com
dig @192.168.1.11 google.com

To test failover, you can shutdown or disconnect the network on your primary DNS server and then see if your clients are able to failover to using the secondary address. As a note, I have found that different client OS’es are better than others at doing this. Windows is notorious for hanging on to primary addresses for longer than it should before it fails over to using the secondary addresses configured.

Best Practices when configuring and using DNS failover

• Use Static IPs – DNS servers should always be configured with static network addresses
• Keep DNS services on separate physical or virtual hosts – having your DNS infrastructure segregated from other hosts or virtualization environments is a great way to separate your fault domains, especially in a volatile lab environment where you are constantly taking things down or making changes
• Use lightweight DNS solutions – You can run Unbound, Pi-hole, or BIND even inside a Docker container
• Segment DNS from critical firewall functions when possible
• Test DNS failover regularly – simulate DNS failure and make sure your failback works like you would expect it to
• Log your DNS queries for troubleshooting and testing

Authoritative servers

Likely if you are running in a home lab environment, you will be running an authoritative server for an internal domain like lab.local or internal.mydomain.com. For this, you can use BIND or something like PiHole to replicate zones across multiple servers and configure primary/secondary roles. I am doing this on my LAN home network side of things for the family.

• I have two PiHole servers configured and gravity sync working between them. As a note, gravity sync is now retired and doesn’t work for PiHole 6
• If you run Windows DNS servers, you can use Windows DNS Server to have an Active Directory integrated zone which replicates between all your DCs in the environment. This is an extremely popular option in the enterprise and SMB environment. So replicating this for your home lab is a good way to get familiar with what many enterprise environments are running.
• You can run things like BIND or Unbound on two linux servers and use things like Keepalived to have a virtual IP address and have one of those servers as the primary for your Keepalived configuration and the other as a backup. This is a great way to do this as you will only have one IP address configured for your clients for your internal servers. However, the redundancy comes in at the application layer with Keepalived as if a server goes down, the other server becomes the owner of the virtual IP address.

Video

Check out my video on what I think is one of the best DNS server configurations for home lab and self-hosting:

Wrapping up

DNS is a foundational technology in any modern computer network, including even our home labs. It is extremely important to think about DNS failover if you want to have services still reachable if you have a failure in hardware or softtare as part of your solution. It helps to make sure that name resolution can continue, even in the event of an outage. Using a combination of internal DNS resolvers and public cloud resolvers is also a great way to have DNS failover so that at the very least, your Internet resolution can continue, even if you have an outage for your internal authoritative domains. Let me know in the comments what you are using in the home lab for DNS failover.