Skip to content

Systemd.md

Latest commit

 

History

History
330 lines (239 loc) · 15.2 KB

Systemd.md

File metadata and controls

330 lines (239 loc) · 15.2 KB

systemd & DNS Resolver Integration

How docker-dns integrates with the DNS resolver stack on Ubuntu and Debian, and why it works the way it does.

The Problem

Docker's built-in DNS (127.0.0.11) only works container-to-container on user-defined networks. From the host machine: where developers actually work, run browsers, and use curl: there is no way to resolve a container by name without manual intervention (docker inspect, editing /etc/hosts, or publishing ports with -p).

docker-dns solves this by running a lightweight DNS server on the host that resolves containername.docker to the container's IP. The hard part is not the DNS server itself: it's making the host's DNS stack actually use it.

Ubuntu and Debian handle DNS resolution in fundamentally different ways, and the details change across versions. This document explains what each system does, what docker-dns does to integrate, and why.

How DNS Resolution Works on Each System

Ubuntu (20.04, 22.04, 24.04)

All supported Ubuntu LTS versions use systemd-resolved as the default DNS resolver. The setup looks like this:

Version systemd resolv.conf target resolvectl DNSStubListenerExtra
20.04 Focal 245 stub-resolv.conf ✓ ✗
22.04 Jammy 249 stub-resolv.conf ✓ ✓
24.04 Noble 255 stub-resolv.conf ✓ ✓

systemd-resolved provides DNS to applications through three paths:

  1. NSS module (nss-resolve): glibc's getaddrinfo() talks to resolved via D-Bus. Most native Linux applications use this.
  2. Stub listener at 127.0.0.53:53: For software that reads /etc/resolv.conf directly and issues its own DNS queries. This includes web browsers (Chrome, Firefox), Go programs, and curl.
  3. D-Bus API: Direct programmatic access.

/etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf, which contains nameserver 127.0.0.53. Applications query 127.0.0.53, and systemd-resolved decides which upstream server handles each query.

This means you cannot simply add nameserver 127.0.0.153 to /etc/resolv.conf: it is a symlink to a generated file that gets overwritten.

Ubuntu 24.04 note: systemd-resolved was split into a separate package. On minimal or cloud images it may not be installed. The postinst detects this and falls back to the plain resolv.conf path.

Debian (11, 12, 13)

Debian does not install or enable systemd-resolved by default on any version. DNS is handled by a plain /etc/resolv.conf file, managed by NetworkManager (desktop) or dhclient/ifupdown (server).

Version systemd Default resolver resolv.conf
11 Bullseye 247 None Regular file
12 Bookworm 252 None Regular file
13 Trixie 257 None Regular file

Applications read /etc/resolv.conf directly and query whichever nameservers are listed. Adding a nameserver to this file works, but it may be overwritten by DHCP renewal or NetworkManager reconnection.

How docker-dns Integrates

On systems with systemd-resolved (Ubuntu default)

docker-dns uses routing domains: a systemd-resolved feature (available since systemd 229) that routes queries for specific domain suffixes to specific DNS servers.

The postinst creates a drop-in configuration:

# /etc/systemd/resolved.conf.d/docker-dns.conf
[Resolve]
DNS=127.0.0.153
Domains=~docker

The ~ prefix on docker makes it a routing domain, not a search domain. This tells systemd-resolved:

  • Queries for *.docker -> forward to 127.0.0.153 (docker-dns)
  • All other queries -> unchanged, forwarded to whatever DNS the network provides

This is the same mechanism used by VPNs, LXD, and corporate split-DNS setups.

On systems with NetworkManager (Debian Desktop)

On Debian Desktop, NetworkManager is active but systemd-resolved is not installed. NM runs with dns=default, meaning it writes /etc/resolv.conf directly on every connection event, DHCP renewal, or service restart. A one-shot prepend to /etc/resolv.conf would be overwritten the next time NM touches the file.

The postinst detects this (NM active, resolved not active) and installs a dispatcher script:

# /etc/NetworkManager/dispatcher.d/docker-dns

The dispatcher re-prepends nameserver 127.0.0.153 to /etc/resolv.conf after every NM connection event (up, down, dhcp4-change, dhcp6-change). The postinst also does an immediate one-shot prepend so DNS works between install and the next NM event.

The postrm removes the dispatcher script and cleans up /etc/resolv.conf on uninstall.

On systems without systemd-resolved or NetworkManager (Debian Server)

On minimal or server installs where neither systemd-resolved nor NetworkManager is active, the postinst prepends docker-dns as a nameserver to /etc/resolv.conf:

# Generated by docker-dns
nameserver 127.0.0.153

Since docker-dns handles .docker queries itself and forwards everything else to its configured upstream resolvers (default: 8.8.8.8, 1.1.1.1, 8.8.4.4), this works for both container resolution and normal DNS.

The postrm removes this entry on uninstall using the # Generated by docker-dns comment as a marker.

Custom DNS Resolvers (Manual Configuration)

docker-dns's automatic integration covers the default DNS stack on all supported Ubuntu and Debian versions (server and desktop). If you run a custom local resolver (dnsmasq, unbound, Pi-hole, or NM with dns=dnsmasq), you already know your DNS stack: add the forwarding rule manually.

dnsmasq (standalone or Pi-hole):

# /etc/dnsmasq.d/docker-dns.conf
server=/docker/127.0.0.153

Then: sudo systemctl restart dnsmasq

unbound:

# /etc/unbound/unbound.conf.d/docker-dns.conf
stub-zone:
    name: "docker"
    stub-addr: 127.0.0.153

Then: sudo systemctl restart unbound

NetworkManager with dns=dnsmasq:

# /etc/NetworkManager/dnsmasq.d/docker-dns.conf
server=/docker/127.0.0.153

Then: sudo systemctl restart NetworkManager

Note: the postrm does not clean up manually-created configs. If you uninstall docker-dns, remove these files yourself.

Why Routing Domains Instead of Global DNS

An earlier version of the postinst set DNS=127.0.0.153 globally without Domains=~docker. This made docker-dns the upstream for all queries: google.com, apt repositories, everything. Problems with that approach:

  • If docker-dns is down, all DNS breaks
  • DHCP-provided DNS servers get deprioritized or ignored
  • VPN split-horizon DNS breaks
  • docker-dns becomes responsible for the entire internet

Routing domains avoid all of these. systemd-resolved only sends .docker queries to docker-dns. The rest of the DNS stack is untouched.

The Browser Problem

How browsers resolve DNS

Browsers do not use glibc's getaddrinfo(). They read /etc/resolv.conf and issue DNS queries directly. This means:

  • On Ubuntu (resolv.conf -> 127.0.0.53): browser queries go through systemd-resolved, routing domains work, .docker queries reach docker-dns. ✓
  • On Debian (resolv.conf is a plain file): the browser queries whatever nameservers are listed. If docker-dns is listed, it works. ✓

DNS-over-HTTPS (DoH)

Modern browsers can bypass system DNS entirely:

  • Firefox: DoH is enabled by default for US users (Cloudflare). All queries go to https://mozilla.cloudflare-dns.com/dns-query, completely bypassing system DNS and docker-dns.
  • Chrome/Chromium: "Use secure DNS" upgrades to DoH if the configured server supports it. Since 127.0.0.53 has no known DoH endpoint, Chrome falls back to plain DNS. This usually works.

If Firefox DoH is on, .docker domains will not resolve. Users need to either disable DoH in Firefox settings ( Privacy & Security -> DNS over HTTPS -> Off) or use a Firefox enterprise policy:

{
  "policies": {
    "DNSOverHTTPS": {
      "Enabled": false
    }
  }
}

Chrome HTTPS records (Type 65)

Chrome queries for HTTPS DNS records (type 65) for every domain. docker-dns returns NOTIMP for unsupported query types on .docker domains. If this causes issues in Chrome (delays, errors), the handler could be changed to return an empty NOERROR instead.

Uninstall Cleanup (postrm)

The postrm reverses all integration changes:

# systemd-resolved: remove drop-in, restart resolved
rm -f /etc/systemd/resolved.conf.d/docker-dns.conf
systemctl restart systemd-resolved 2>/dev/null || true

# dnsmasq
rm -f /etc/dnsmasq.d/docker-dns.conf
systemctl restart dnsmasq 2>/dev/null || true

# unbound
rm -f /etc/unbound/unbound.conf.d/docker-dns.conf
systemctl restart unbound 2>/dev/null || true

# NetworkManager dispatcher: removed before NM restart so a restart event
# cannot fire the dispatcher and re-add the nameserver to resolv.conf
rm -f /etc/NetworkManager/dispatcher.d/docker-dns

# NetworkManager dnsmasq
rm -f /etc/NetworkManager/dnsmasq.d/docker-dns.conf
systemctl restart NetworkManager 2>/dev/null || true

# Plain resolv.conf: remove the prepended entry
awk '/# Generated by docker-dns/{skip=1; next} skip{skip=0; next} 1' \
    /etc/resolv.conf > /tmp/resolv.tmp \
    && cat /tmp/resolv.tmp > /etc/resolv.conf && rm -f /tmp/resolv.tmp

Each systemctl restart is guarded with 2>/dev/null || true so it silently skips services that aren't installed.

Tested Configurations

The systemd integration tests (make test-systemd OS=<codename>) verify the full install -> resolve -> uninstall lifecycle on each target. Both the direct path (dig @127.0.0.153) and the system-level path (dig without @, going through whatever the OS uses for DNS) are tested.

OS Version Variant DNS Method Assertions
Noble Ubuntu 24.04 Server systemd-resolved + routing domain 13
Noble Ubuntu 24.04 Desktop systemd-resolved + NM (dns=systemd-resolved) 14
Jammy Ubuntu 22.04 Server systemd-resolved + routing domain 13
Jammy Ubuntu 22.04 Desktop systemd-resolved + NM (dns=systemd-resolved) 14
Focal Ubuntu 20.04 Server systemd-resolved + routing domain 13
Focal Ubuntu 20.04 Desktop systemd-resolved + NM (dns=systemd-resolved) 14
Trixie Debian 13 Server plain resolv.conf 12
Trixie Debian 13 Desktop NM dispatcher + resolv.conf 15
Bookworm Debian 12 Server plain resolv.conf 12
Bookworm Debian 12 Desktop NM dispatcher + resolv.conf 15
Bullseye Debian 11 Server plain resolv.conf 12
Bullseye Debian 11 Desktop NM dispatcher + resolv.conf 15

Desktop variants additionally test that docker-dns resolution survives a full systemctl restart NetworkManager cycle.

Each test runs inside a privileged Docker container with systemd as PID 1, Docker-in-Docker for container resolution, and full DNS chain verification including post-uninstall cleanup.

systemd-resolved Feature Matrix

For reference, the systemd-resolved features available on each Ubuntu version (Debian does not ship resolved by default):

Feature systemd version Focal 20.04 Jammy 22.04 Noble 24.04
Routing domains (~docker) 229+ ✓ ✓ ✓
resolvectl command ~239+ ✓ ✓ ✓
DNSStubListenerExtra 247+ ✗ ✓ ✓
127.0.0.54 proxy stub 247+ ✗ ✓ ✓
Drop-in config directory All ✓ ✓ ✓
Separate systemd-resolved package 255+ ✗ ✗ ✓

Alternative Approach: Dummy Interface with resolvectl

The current implementation uses a global drop-in config. A more robust alternative (used by LXD) creates a dummy network interface and configures per-link DNS on it:

ip link add docker-dns0 type dummy
ip link set docker-dns0 up
resolvectl dns docker-dns0 127.0.0.153
resolvectl domain docker-dns0 '~docker'

Per-link routing domains are the primary mechanism systemd-resolved was designed for: the global config is secondary. This approach avoids edge cases where per-link DNS configured by NetworkManager or netplan might interact unexpectedly with global settings.

This is documented here for future consideration but is not currently implemented because the drop-in approach has proven reliable across all tested Ubuntu versions.

References