From abdc45b47ba5449192df49e0e1b70d78f2245867 Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Mon, 3 Aug 2026 10:55:49 +0000 Subject: [PATCH] fix(resolved): the fallback resolver is the bug, not the safety net MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 00-moshpit.conf assumed systemd-resolved "uses the first server for every query, rotating only on failure", so listing DigitalOcean's resolvers after the bridge was harmless insurance. It rotates away from a failing server and does not rotate back. One restart of moshpit-dns moves every query to the fallback permanently, and the fallback answers NXDOMAIN for every Moshpit name — so DNS looks healthy while the whole namespace is dead. That is the worst shape a failure can take: nothing is obviously broken to look at. Seen three times in one afternoon before it was understood. `dig @127.0.0.1 -p 5354 seo.rank` answered, `curl seo.rank` did not, and resolvectl showed the bridge listed and 67.207.67.3 current. The fallback also buys nothing. The bridge forwards what it cannot answer upstream itself (mode=clearnet), so a second entry adds no capability — only that failure. If the bridge stops, restart it: the unit already does, in two seconds. Two seconds of no DNS beats an afternoon of DNS that lies. Documents that this must be the only drop-in setting DNS=, and how to remove the others. --- systemd/00-moshpit.conf | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/systemd/00-moshpit.conf b/systemd/00-moshpit.conf index 1f52229..5084c59 100644 --- a/systemd/00-moshpit.conf +++ b/systemd/00-moshpit.conf @@ -1,9 +1,30 @@ # The Moshpit resolver, as this machine's primary DNS. # -# Named 00- so it is read before DigitalOcean.conf: systemd-resolved appends -# DNS= across drop-ins in filename order, and it uses the first server for -# every query, rotating only on failure. So the bridge answers, and DO's -# resolvers stay listed as a fallback for if it ever stops. +# This must be the ONLY drop-in that sets DNS=. Delete any other — a cloud +# image's DigitalOcean.conf, or the moshpit.conf that `moshcode dns enable` +# used to write: +# +# rm /etc/systemd/resolved.conf.d/DigitalOcean.conf +# systemctl restart systemd-resolved +# resolvectl status | grep -A1 'DNS Servers' # only 127.0.0.1:5354 +# +# A second drop-in appends its servers to the same list, and that list is a +# trap rather than a safety net. systemd-resolved rotates away from a server +# that fails and DOES NOT ROTATE BACK, so one restart of the bridge moves every +# query to the fallback permanently. The fallback answers NXDOMAIN for every +# Moshpit name, which means DNS looks healthy while the entire namespace is +# dead — the worst shape a failure can take, because nothing is obviously +# broken to look at. +# +# Observed three times in one afternoon before it was understood. `dig +# @127.0.0.1 -p 5354 seo.rank` answered, `curl seo.rank` did not, and +# `resolvectl status` showed the bridge listed and 67.207.67.3 current. +# +# A fallback also buys nothing. The bridge forwards everything it cannot answer +# to the upstreams itself (mode=clearnet), so a second entry adds no capability +# — only the failure above. If the bridge stops, the answer is to restart it, +# which moshpit-dns.service already does in two seconds (Restart=always, +# RestartSec=2). Two seconds of no DNS beats an afternoon of DNS that lies. # # There is deliberately no Domains= line, and that is the whole point. #