local connect: add opt-in --local-dns for resolving cluster names via a local DNS resolver

[scott-cotton]

What

Adds opt-in local-DNS resolution to signadot local connect. When enabled, the CLI resolves Kubernetes service names through a local DNS resolver wired into your OS resolver, instead of the default mechanism that edits /etc/hosts.

Two ways to opt in:

• Per connection (encouraged) — set resolver: LocalDNS on the connection in your config. resolver: EtcHosts (the default when the field is omitted) keeps the /etc/hosts behavior:

  local:
    connections:
      - cluster: <cluster>
        resolver: LocalDNS

• Per invocation — the --local-dns flag overrides the config field for a single connect. --local-dns=false forces /etc/hosts even when the config selects LocalDNS:

  signadot local connect --local-dns --cluster <cluster>
  

Default behavior is unchanged — omitted config field and no flag = current /etc/hosts behavior, byte-for-byte. This is additive and experimental.

Why

The default path enumerates a fixed set of hostnames per service and rewrites the shared /etc/hosts file on every cluster change. Local DNS instead runs a real resolver that is authoritative for the cluster zones: it answers cluster names directly, forwards everything else to your existing upstream resolvers, and restores your DNS configuration on disconnect. If a session exits uncleanly, DNS can be left pointed at the stopped resolver; the next signadot local connect repairs it on startup regardless of mode — recovery is no longer gated on --local-dns, so a plain reconnect (the natural move when your DNS just broke) restores working resolution. Net effect for users: cluster name resolution that behaves more like in-cluster DNS, without touching /etc/hosts.

What resolves

With local DNS active, all the usual service name forms resolve on both macOS and Linux:

• my-svc.my-ns.svc.cluster.local (FQDN)
• my-svc.my-ns.svc
• my-svc.my-ns

Non-cluster names (e.g. github.com) continue to resolve normally — they're forwarded to the resolvers you were already using.

local hosts

signadot local hosts lists the cluster hostnames currently resolvable from your machine and the address each maps to. It reads the shared address map, so it works in both resolution modes (/etc/hosts and local DNS):

signadot local hosts            # aligned NAME / IP table
signadot local hosts -o json    # [{ "name": ..., "ip": ... }, ...]
signadot local hosts -o yaml

Each host reports a single virtual address (IPv4-preferred); the addresses are the synthetic ones the tunnel intercepts, deterministic and name-sorted.

Requires root

Local DNS configures the system DNS resolver (and on Linux binds 127.0.0.54:53), so it needs the same root privileges local connect already escalates for. The privilege-rationale prompt now mentions DNS-resolver configuration when active. It is not supported with --unprivileged — combining them fails fast with a clear error rather than silently ignoring the setting (there's no root-manager in that mode to run the resolver). This applies however local DNS was selected: the same conflict check fires when the mode comes from the resolver: config field, and the error names the source.

local status

When local DNS is active, signadot local status shows the resolver in place of the /etc/hosts line — resolvable record count, the resolver's bind address, and a prominent warning if a system DNS manager (e.g. systemd-resolved owning /etc/resolv.conf) might revert the configuration. The /etc/hosts section is omitted in this mode rather than reported as a failure. Reflected in both the table and JSON/--details output.

Compatibility & scope

• Opt-in: omitted config field and unset flag = current /etc/hosts behavior, byte-for-byte.
• Platforms: macOS and Linux. (Other OSes reject resolver management.)
• Intentional difference vs /etc/hosts: the resolver mirrors what a real cluster serves, so it will not resolve non-standard truncations the old /etc/hosts path happened to emit (e.g. my-svc.my-ns.svc.cluster) — those never resolve in-cluster either. Surfacing this as a deliberate behavior change since it's opt-in.
• IPv6 — groundwork only: the underlying libconnect work includes first steps toward IPv6 (a dual-stack address-allocation model and an AAAA-capable resolver), but it is not yet employed by either the cluster or the CLI/client side — AAAA queries return NODATA for now. local hosts accordingly reports one address per host; the resolver's []ip backing (for future A/AAAA answers) is unaffected.
• Namespace-qualified bare names: my-svc → current namespace, not yet.

Try it

signadot local connect --local-dns --cluster <cluster>
curl my-svc.my-ns:<port>/...        # short name
curl my-svc.my-ns.svc.cluster.local:<port>/...
signadot local hosts                # resolvable cluster names and their addresses
signadot local status               # shows the local DNS resolver
signadot local disconnect           # restores your original DNS config

⚠️ Merge order

This branch pins libconnect to an unmerged branch pseudo-version (feat/localdns-1f, signadot/libconnect#163) that carries the resolver, the standalone crash-recovery entry point, and the per-connection resolver config type. Do not merge until that lands; then repoint go.mod to the merged libconnect commit/tag. Kept as a draft for this reason.

[cursor]

Stale comment

Risk: medium. Cursor Bugbot did not run on this PR and there are no human reviews yet. Deferring to the already-assigned reviewers for this opt-in system DNS resolver change.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}

[joehorsnell]

Just adding a comment here @scott-cotton as per our call earlier - from a usability perspective, using the current /etc/hosts approach lets you easily see what all the host entries are. Using the local DNS resolver, you lose this, so maybe having a new command, or flag on an existing command, to enumerate the hosts would be useful?

[scott-cotton]

Just adding a comment here @scott-cotton as per our call earlier - from a usability perspective, using the current /etc/hosts approach lets you easily see what all the host entries are. Using the local DNS resolver, you lose this, so maybe having a new command, or flag on an existing command, to enumerate the hosts would be useful?

Good call-out, and I had forgotten about that.

In 0e6c121
I added

26-07-06 scott@air cli % ./signadot local -h
Connect local machine with cluster

Usage:
  signadot local [command]

Available Commands:
  connect     Connect local machine to cluster
  disconnect  Disconnect local machine from cluster
  hosts       List the cluster hosts resolvable from the local machine and their IP addresses
  override    Override sandbox HTTP traffic using a local service
  proxy       Proxy connections based on the specified mappings
  status      Show status of the local machine's connection with cluster

Flags:
  -h, --help   help for local

Global Flags:
      --config string   config file (default is $HOME/.signadot/config.yaml)
      --debug           enable debug output
  -o, --output string   output format (json|yaml)

Use "signadot local [command] --help" for more information about a command.
26-07-06 scott@air cli % ./signadot local hosts | head -10
agent-metrics.signadot.svc.cluster.local 10.22.0.26
apiserver-metrics.signadot-control.svc.cluster.local 10.22.0.46
apiserver.signadot-control.svc.cluster.local 10.22.0.5
argo-rollouts-metrics.argo-rollouts.svc.cluster.local 10.22.0.22
artifactsserver-internal.signadot-control.svc.cluster.local 10.22.0.16
artifactsserver.signadot-control.svc.cluster.local 10.22.0.18
basic-istio-e2e-bcfa99-t-8c86-0.basic-istio-e2e-bcfa99-t-8c86.signadot.svc.cluster.local 10.22.0.39
basic-istio-e2e-bcfa99-t-8c86.signadot.svc.cluster.local 10.22.0.27
frontend.hotrod-istio-gwa.svc.cluster.local 10.22.0.25
httpbin.default.svc.cluster.local 10.22.0.11
26-07-06 scott@air cli % ./signadot local hosts -o json | head -10
[
  {
    "name": "agent-metrics.signadot.svc.cluster.local",
    "ip": "10.22.0.26"
  },
  {
    "name": "apiserver-metrics.signadot-control.svc.cluster.local",
    "ip": "10.22.0.46"
  },
  {
26-07-06 scott@air cli % ./signadot local hosts -o yaml | head -10
- ip: 10.22.0.26
  name: agent-metrics.signadot.svc.cluster.local
- ip: 10.22.0.46
  name: apiserver-metrics.signadot-control.svc.cluster.local
- ip: 10.22.0.5
  name: apiserver.signadot-control.svc.cluster.local
- ip: 10.22.0.22
  name: argo-rollouts-metrics.argo-rollouts.svc.cluster.local
- ip: 10.22.0.16
  name: artifactsserver-internal.signadot-control.svc.cluster.local
26-07-06 scott@air cli %

ptal

[cursor]

Stale comment

Risk: medium. Cursor Bugbot is not configured on this PR, so I am deferring to human review. Reviewers daniel-de-vera and foxish are already assigned for this opt-in system DNS resolver change.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}

[joehorsnell]

ptal

TIL; PTAL!

LGTM 😂

Tested locally with the new commit on this branch it looks good (same output from signadot local hosts using the current default /etc/hosts and the new --local-dns mode).

[foxish]

Reviewed and tested this locally: all packages build, go vet is clean, the full test suite passes, and I smoke-tested the new CLI surface (--local-dns + --unprivileged fails fast as intended, help text, local hosts output paths). Nice work overall: the runNameService/stopNameService split is clean, teardown ordering (resolver restored before the tunnel drops) is well thought out, and localdns.New failing synchronously on bind conflicts while refreshing records asynchronously gives the right failure semantics.

Inline comments cover the code-level items. Three of them I'd consider blocking: the tpMonitor restart loop in --local-dns mode, the stray tp_monitor.go.head file, and the orphaned sandboxmanager on the new startup error path.

Cross-cutting items that don't anchor to a single line:

1. Crash recovery only runs if the next connect also passes --local-dns. The repair logic lives in libconnect's localdns installer and only executes inside localdns.New, which the CLI only calls when EnableLocalDNS is set. On Linux, a --local-dns session that dies uncleanly leaves /etc/resolv.conf pointing at a dead 127.0.0.54:53, i.e. all DNS on the machine is broken; if the user then runs a plain signadot local connect (the default, and the likely move for someone whose DNS just broke), recovery never runs and system DNS stays broken. That contradicts the description's claim that "the next signadot local connect detects the leftover configuration and repairs it on startup". Suggest exporting a standalone repair entry point from libconnect (feedback for signadot/libconnect#163 while it's still open) and calling it unconditionally at root-manager startup in both modes; at minimum the PR description should be corrected.

2. signadot local hosts is a new user-facing command but isn't mentioned in the PR description. It's independently useful (works in both resolution modes since it reads the shared ipmap); it should be described here and get docs.

3. No tests accompany the change. pickIP, checkLocalDNSStatus, and the status-printer gating are pure functions that are cheap to unit test, and a mode-matrix test on the tpMonitor health check would have caught the restart loop.

Also: we'd like the config file, not the flag, to be the encouraged way to opt in; see the inline comment on the flag definition for a concrete field proposal (resolver: EtcHosts | LocalDNS per connection).

[foxish]

Blocking: restart loop in --local-dns mode. checkRootServer (L206) still gates health on the /etc/hosts service: if r.etcHostsSVC == nil || !r.etcHostsSVC.Status().Healthy. With --local-dns, etcHostsSVC is always nil (localDNSSVC is set instead), and this monitor runs for exactly the PortForward and ControlPlaneProxy link types. So once the startup grace period (connectTimeout, default 10s) expires, every check returns restart=true, and localnet + localdns get torn down and restarted every ~10s indefinitely, uninstalling and reinstalling the OS resolver config each cycle. It also means localdns health is never actually monitored.

checkRootServer needs to branch on ciConfig.EnableLocalDNS and check localDNSSVC health instead, ideally via the mutex-guarded accessors (the raw field reads here are a pre-existing race that now extends to the new fields).

[foxish]

Confirmed empirically: built this branch, connected to a real cluster with --local-dns over a PortForward link. The daemon restarts localnet + localdns every ~10s, continuously (observed for over an hour), binding a new ephemeral port each cycle and rewriting all 12 /etc/resolver files + HUPing mDNSResponder each time:

15:50:41 INFO msg="restarting localnet and name-resolution services"
15:50:41 INFO msg="localdns listening" addr=127.0.0.1:55286
15:50:51 INFO msg="restarting localnet and name-resolution services"
15:50:51 INFO msg="localdns listening" addr=127.0.0.1:63746
15:51:01 INFO msg="restarting localnet and name-resolution services"
15:51:02 INFO msg="localdns listening" addr=127.0.0.1:53087
...

Easy to miss because local status reports healthy between restarts and resolution genuinely works (FQDN/short forms resolve, end-to-end curl through localnet returns 200).

[foxish]

Blocking (trivial): this looks like a leftover merge-conflict artifact, an old copy of tp_monitor.go (it still has the pre-connectTimeout maxStartingTime logic). The extension keeps it out of the build, but it should be deleted.

[foxish]

Blocking: orphaned sandboxmanager on this error path. This early return skips the cleanup block at the end of Run(), notably m.sbmMonitor.stop(). The sandbox manager was already spawned above (go m.sbmMonitor.run()) as a separately daemonized child via sudo -u <user>, and stop() is the only thing that kills it. So a localdns startup failure here (realistic case: something else already bound 127.0.0.54:53 on Linux) exits the root manager but leaves an orphaned sandboxmanager running with its pidfile held. Before this PR this branch had no error path, so this is new. Suggest routing the error through the same cleanup used by the normal shutdown path below.

[foxish]

We'd like the config file to be the encouraged way to opt in, with this flag as a per-invocation override: forgetting the flag on the next connect is the natural failure mode, and a --local-dns session followed by a plain reconnect is also exactly the case where crash recovery doesn't run (see review body).

Concrete proposal: a per-connection field in libconnect's ConnectionConfig, e.g.

local:
  connections:
    - cluster: demo
      resolver: LocalDNS   # or EtcHosts; omitted = EtcHosts

Enum rather than bool (leaves room for Auto later), PascalCase values to match type:, zero value EtcHosts so existing configs are untouched. runConnect already has the ConnectionConfig in hand before building the ciConfig, so the CLI plumbing is small; the --unprivileged conflict check should then also fire when the mode comes from config, with an error naming the config field. Fine as a fast-follow alongside libconnect#163, but worth agreeing on the field now so docs can lead with it.

[joehorsnell]

This is a good idea @foxish.

One question here @scott-cotton - do we still check at runtime, or record at the actual point of connecting, which resolver method was used, rather than relying on the config file/command-line switches to tell us?

I'm just thinking of the case where the value in the config file changes while the CLI is connected, eg someone connects using the default /etc/hosts method, then while connected modifies the config file to use the LocalDNS setting.

[scott-cotton]

The CLI is designed to have race-free local connect config, it records the config at invocation time and sends that to the locald process, thus making it impossible to unexpectedly change the config or be unaware of what the config is due to it changing while running.

this is actually necessary because the config includes, for example, the auth (and apiserver) and the possibility of having parse errors, of cli invocation overriding the config, etc. All these things make the expected flow quite complex for a live config file.

So, if someone changes anything in the config, it has no effect and they should restart, which IMO is by far the least friction approach.

[foxish]

Schema forward-compat: this bakes a single ip into the -o json/-o yaml output, while the proto already carries repeated ips and this PR lays IPv6 dual-stack groundwork. When AAAA support lands, either ip gets a breaking rename to ips for script consumers, or it stays and silently under-reports dual-stack hosts. Emitting "ips": [...] from day one avoids the breaking change; the human-readable view can keep showing one address.

[scott-cotton]

I will change the proto to be singleton ip. This doesn't affect ipv6/dual-stack forward compatibility and is simpler, following the table below:

ipv4	dual-stack	ipv6
use ipv4	use ipv4 (both virtual addrs reach the service; v4 by preference)	use ipv6

the ipMap representation retains []ip as it is the backing for the resolver which needs to answer AAAA and A record requests.

[foxish]

Default output should use internal/sdtab like every other list command (cluster, routegroup, resourceplugin, ...), so humans get an aligned NAME/IP table with a header, and scripts use -o json. The bare "%s %s" lines are also not hosts-file format (that's <ip> <name>, reversed), so there's no copy-paste argument for keeping them. Following the pattern from cluster/printers.go:

type hostRow struct {
	Name string `sdtab:"NAME"`
	IP   string `sdtab:"IP"`
}

func printHosts(out io.Writer, hosts []printableHost) error {
	t := sdtab.New[hostRow](out)
	t.AddHeader()
	for _, h := range hosts {
		t.AddRow(hostRow{Name: h.Name, IP: h.IP})
	}
	return t.Flush()
}

[scott-cotton]

My reasoning for the standard output was that this command is different because it is likely to get many many more outputs than another list using sdtab. So, the headers basically are likely to be invisible, off screen in any case -- why force the acrobatics and extra machinery of json or header filtering on consumers?

More generally, removing the headers is a basic UX concern regarding standard unix pipelining.

But, since it seems this reasoning only subtly justifies deviating from the unusable-in-pipelines UX elsewhere, I'll add the sdtab.

[foxish]

gofmt: this insertion breaks the struct-tag alignment for the rest of the field block; gofmt -l flags this file on the branch (it's clean on main).

[foxish]

The inverse gating is missing on getRawHosts: in --local-dns mode the root manager reports Hosts == nil, so the non-details JSON/YAML status output still emits "hosts": {"healthy": false, "numHosts": 0}, which reads as a failure even though /etc/hosts management is intentionally not running. Suggest getRawHosts return nil when ciConfig.EnableLocalDNS, mirroring how this function returns nil when local DNS is disabled. (The human-readable view is already handled by the printSuccess branch.)

[foxish]

Tested against a running daemon from a previous CLI release: this surfaces as tunnel-api unsupported method: consider upgrading the operator, because processGRPCError maps codes.Unimplemented to operator guidance. For GetHosts the call is CLI-to-local-daemon, and Unimplemented means the running locald predates this CLI, which will happen to everyone who upgrades the CLI while connected. Worth catching Unimplemented here with a targeted message, e.g. "the running local daemon predates 'local hosts'; run signadot local disconnect && signadot local connect".

[scott-cotton]

oops. good catch!

[scott-cotton]

Thanks @foxish for the thorough review.

All items addressed, some in-line comments have specific responses. Everything else is straightforward implementation of your comments. Last 6 commits attempt to break down the changes into independently reviewable chunks.

[cursor]

Stale comment

Risk: medium. Cursor Bugbot is not configured on this PR. Human review is still required: foxish left unresolved blocking feedback and new commits landed since that review. daniel-de-vera remains assigned as reviewer.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}

[daniel-de-vera]

@scott-cotton, check this:

$ signadot local connect --local-dns
signadot local connect needs root privileges for:
	- configuring the system DNS resolver for cluster service names
	- configuring networking to direct local traffic to the cluster

signadot local connect has been started ✓
* runtime config: cluster signadot-staging, running with root-daemon
✗ Local connection not healthy!
* failed to configure local DNS resolver
disconnected.
Error: connect failed and is no longer running

time=2026-07-10T10:03:38.406-03:00 level=ERROR msg="error starting name resolution; will retry" locald-component=root-manager error="starting localdns: cannot bind 127.0.0.54:53: address already in use — another signadot/libconnect instance is likely already running (one per host in this phase): binding udp 127.0.0.54:53: listen udp 127.0.0.54:53: bind: address already in use"

$ cat /etc/resolv.conf 
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search tail473aa.ts.net lan


$ sudo netstat -lnp | grep ":53 "
tcp        0      0 127.0.0.54:53           0.0.0.0:*               LISTEN      920/systemd-resolve 
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      920/systemd-resolve 
udp        0      0 127.0.0.54:53           0.0.0.0:*                           920/systemd-resolve 
udp        0      0 127.0.0.53:53           0.0.0.0:*                           920/systemd-resolve 

Note that systemd-resolved is the default DNS resolver on Ubuntu (since version 18.04), but not sure about other distributions, but this seems like a common problem to me.

[scott-cotton]

@scott-cotton, check this:

@daniel-de-vera thanks for the test and report. Fixed in libconnect
https://github.com/signadot/libconnect/pull/163/commits/e1e03fd7f0cd279f5ed7afb6010507b8c3d451ef

and that fix linked here in last commit.

[cursor]

Stale comment

Risk: medium. Cursor Bugbot is not configured on this PR. Human review is still required: foxish left unresolved blocking feedback on the latest changes. daniel-de-vera and foxish remain assigned as reviewers.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}

[daniel-de-vera]

@daniel-de-vera thanks for the test and report. Fixed in libconnect signadot/libconnect@e1e03fd

and that fix linked here in last commit.

Thanks Scott, it works now, but I noticed this behaviour, which I think it'd be worth flagging:

# Using --local-dns
#
$ time curl location.hotrod-devmesh.svc.cluster.local:8081/locations
[{"id":1,"name":"My Home","coordinates":"231,773"},{"id":123,"name":"Rachel's Floral Designs","coordinates":"115,277"},{"id":392,"name":"Trom Chocolatier","coordinates":"577,322"},{"id":567,"name":"Amazing Coffee Roasters","coordinates":"211,653"},{"id":731,"name":"Japanese Desserts","coordinates":"728,326"}]
real	0m0,549s
user	0m0,002s
sys	0m0,007s

# Note this one was veeery slow (almost 7sec, compared to the 500ms before):
$ time curl location.hotrod-devmesh:8081/locations
[{"id":1,"name":"My Home","coordinates":"231,773"},{"id":123,"name":"Rachel's Floral Designs","coordinates":"115,277"},{"id":392,"name":"Trom Chocolatier","coordinates":"577,322"},{"id":567,"name":"Amazing Coffee Roasters","coordinates":"211,653"},{"id":731,"name":"Japanese Desserts","coordinates":"728,326"}]
real	0m6,743s
user	0m0,007s
sys	0m0,011s


# Using /etc/hosts
#

$ time curl location.hotrod-devmesh.svc.cluster.local:8081/locations
[{"id":1,"name":"My Home","coordinates":"231,773"},{"id":123,"name":"Rachel's Floral Designs","coordinates":"115,277"},{"id":392,"name":"Trom Chocolatier","coordinates":"577,322"},{"id":567,"name":"Amazing Coffee Roasters","coordinates":"211,653"},{"id":731,"name":"Japanese Desserts","coordinates":"728,326"}]
real	0m0,709s
user	0m0,005s
sys	0m0,007s

$ time curl location.hotrod-devmesh:8081/locations
[{"id":1,"name":"My Home","coordinates":"231,773"},{"id":123,"name":"Rachel's Floral Designs","coordinates":"115,277"},{"id":392,"name":"Trom Chocolatier","coordinates":"577,322"},{"id":567,"name":"Amazing Coffee Roasters","coordinates":"211,653"},{"id":731,"name":"Japanese Desserts","coordinates":"728,326"}]
real	0m0,999s
user	0m0,008s
sys	0m0,008s

Note sure if there's anything we can do about the above, but could have a clear impact in performance for some users.

Also, when using --local-dns I could notice an almost never ending waterfall of these messages:

time=2026-07-10T12:05:19.711-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:37403->127.0.0.53:53: i/o timeout"
time=2026-07-10T12:05:19.786-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:50686->127.0.0.53:53: i/o timeout"
time=2026-07-10T12:05:19.889-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:36216->127.0.0.53:53: i/o timeout"
time=2026-07-10T12:05:19.960-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:55973->127.0.0.53:53: i/o timeout"
time=2026-07-10T12:05:20.065-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:40432->127.0.0.53:53: i/o timeout"
time=2026-07-10T12:05:20.135-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:52578->127.0.0.53:53: i/o timeout"
time=2026-07-10T12:05:20.209-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:38024->127.0.0.53:53: i/o timeout"
time=2026-07-10T12:05:20.282-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:33313->127.0.0.53:53: i/o timeout"
time=2026-07-10T12:05:20.351-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:42970->127.0.0.53:53: i/o timeout"
time=2026-07-10T12:05:20.421-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:43099->127.0.0.53:53: i/o timeout"
time=2026-07-10T12:05:20.493-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:43462->127.0.0.53:53: i/o timeout"
time=2026-07-10T12:05:20.564-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:39891->127.0.0.53:53: i/o timeout"
time=2026-07-10T12:05:20.635-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:51456->127.0.0.53:53: i/o timeout"
time=2026-07-10T12:05:20.704-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:52100->127.0.0.53:53: i/o timeout"
time=2026-07-10T12:05:20.822-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:34815->127.0.0.53:53: i/o timeout"
time=2026-07-10T12:05:20.902-03:00 level=DEBUG msg="localdns forward failed" locald-component=root-manager name=haproxy-high-qps.delivery-1.us-east1.eqtv.io. error="read udp 127.0.0.1:49149->127.0.0.53:53: i/o timeout"

[cursor]

Stale comment

Risk: medium. Cursor Bugbot is not configured on this PR. Human review is still required: foxish left unresolved blocking feedback on the latest changes. daniel-de-vera and foxish remain assigned as reviewers.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}

[scott-cotton]

@daniel-de-vera thanks for the test and report. Fixed in libconnect signadot/libconnect@e1e03fd
and that fix linked here in last commit.

Thanks Scott, it works now, but I noticed this behaviour, which I think it'd be worth flagging:

Using --local-dns

Thanks again, please see last commit and ndots option for fixing. Could you verify if that fixes the problem?

[daniel-de-vera]

Thanks again, please see last commit and ndots option for fixing. Could you verify if that fixes the problem?

Now the problem got inverted:

$ time curl location.hotrod-devmesh.svc.cluster.local:8081/locations
[{"id":1,"name":"My Home","coordinates":"231,773"},{"id":123,"name":"Rachel's Floral Designs","coordinates":"115,277"},{"id":392,"name":"Trom Chocolatier","coordinates":"577,322"},{"id":567,"name":"Amazing Coffee Roasters","coordinates":"211,653"},{"id":731,"name":"Japanese Desserts","coordinates":"728,326"}]
real	0m9,067s
user	0m0,003s
sys	0m0,006s

$ time curl location.hotrod-devmesh:8081/locations
[{"id":1,"name":"My Home","coordinates":"231,773"},{"id":123,"name":"Rachel's Floral Designs","coordinates":"115,277"},{"id":392,"name":"Trom Chocolatier","coordinates":"577,322"},{"id":567,"name":"Amazing Coffee Roasters","coordinates":"211,653"},{"id":731,"name":"Japanese Desserts","coordinates":"728,326"}]
real	0m0,633s
user	0m0,003s
sys	0m0,006s

[cursor]

Stale comment

Risk: medium. Cursor Bugbot is not configured on this PR. Human review is still required: foxish left unresolved blocking feedback on the latest changes. daniel-de-vera and foxish remain assigned as reviewers.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}

[scott-cotton]

Thanks again, please see last commit and ndots option for fixing. Could you verify if that fixes the problem?

Now the problem got inverted:

sigh.

Last commits in libconnect and here remove the dependency on ndots and embed the name expansion in the resolver lookup, can you try again?

[daniel-de-vera]

Thanks again, please see last commit and ndots option for fixing. Could you verify if that fixes the problem?

Now the problem got inverted:

sigh.

Last commits in libconnect and here remove the dependency on ndots and embed the name expansion in the resolver lookup, can you try again?

Yes! 🙂

$ time curl location.hotrod-devmesh.svc.cluster.local:8081/locations
[{"id":1,"name":"My Home","coordinates":"231,773"},{"id":123,"name":"Rachel's Floral Designs","coordinates":"115,277"},{"id":392,"name":"Trom Chocolatier","coordinates":"577,322"},{"id":567,"name":"Amazing Coffee Roasters","coordinates":"211,653"},{"id":731,"name":"Japanese Desserts","coordinates":"728,326"}]
real	0m0,601s
user	0m0,005s
sys	0m0,014s

$ time curl location.hotrod-devmesh:8081/locations
[{"id":1,"name":"My Home","coordinates":"231,773"},{"id":123,"name":"Rachel's Floral Designs","coordinates":"115,277"},{"id":392,"name":"Trom Chocolatier","coordinates":"577,322"},{"id":567,"name":"Amazing Coffee Roasters","coordinates":"211,653"},{"id":731,"name":"Japanese Desserts","coordinates":"728,326"}]
real	0m0,606s
user	0m0,007s
sys	0m0,009s

[foxish]

Re-reviewed the latest state (f2ebe46, libconnect pinned at c142183 which matches the current feat/localdns-1f head) and ran it live against the demo cluster on macOS: PortForward link, --local-dns, 15+ minute session.

Everything from the previous round is verifiably fixed, in code and live:

• No restart loop: exactly one "restarting localnet and name-resolution services" at startup (the expected initial start when the port-forward address first materializes), then stable for the whole session on one bind address, resolver-file mtimes frozen at daemon start.
• All three name forms resolve through the real macOS resolver path (dscacheutil, i.e. getaddrinfo): FQDN, <svc>.<ns>.svc, <svc>.<ns>. External names unaffected.
• Correct DNS semantics probed directly: A answers authoritatively; AAAA is authoritative NODATA (the documented IPv6 groundwork behavior); a miss under cluster.local is an authoritative NXDOMAIN; a miss under a borrowed namespace zone is forwarded upstream (verified against captured Tailscale MagicDNS upstreams), so the short-name zones don't shadow real domains. TCP works.
• Hygiene held: /etc/hosts has zero signadot entries in this mode, all managed files carry the sentinel, and a pre-existing foreign /etc/resolver/test file on my machine was correctly left untouched. scutil --dns shows all scoped resolvers registered.
• local hosts (table/JSON), local status (table/JSON/--details) all gate correctly per mode; clean disconnect left nothing behind.

Approving, modulo the two bugs below being addressed. Both live in the pinned libconnect branch rather than this repo, so practically: fix in signadot/libconnect#163 and bump the pin here, which the merge-order note requires before undrafting anyway.

1. Linux: a stale sentinel resolv.conf without its backup produces a self-forwarding loop

The chain: a crashed --local-dns session leaves /etc/resolv.conf signadot-managed (sentinel + nameserver 127.0.0.54), and /var/lib/signadot/resolv.conf.bak is missing (wiped dir, failed backup write). On the next connect:

1. localdns.Recover no-ops, because Linux recovery is keyed on the backup file existing (installer_linux.go recover()).
2. localdns.New binds 127.0.0.54 (free, the old process is dead).
3. captureUpstreams reads the stale resolv.conf and captures our own address as the upstream. Every non-cluster query now self-forwards: each forwarded query arrives as a new inbound query, so this is a query loop with amplification, not just a timeout.
4. Worse, linuxInstaller.capture() then records the stale sentinel file as the "original", so disconnect restores broken DNS.

Two cheap defenses, both worth doing:

• Filter the server's own bind IP out of captured upstreams. buildResolvConf already does exactly this for secondary nameservers (if ns == bindIP { continue }), so the forwarder is asymmetric today.
• Make capture() refuse to treat a sentinel-bearing file as the original (treat it as "no usable original" and log loudly).

There's also a milder in-session variant: on a tunnel-proxy restart where the outgoing service's resolv.conf restore failed, New runs captureUpstreams before newInstaller's recovery, so the forwarder can capture the previous (dead) bind address as first upstream and eat the 2s per-server timeout on every forwarded query for the rest of the session. The self-filter above covers most of it.

2. macOS: "writing more than 10 resolver files" warns every refresh cycle, forever

darwinInstaller.install() emits the len(desired) > 10 warning unconditionally, and the refresh loop calls install() every 15s. Observed live with 12 zones on the demo cluster: ~4 warnings/minute into root-manager.log for the lifetime of the session (thousands of lines/day). Fix: gate the warning on the desired set changing (there's already a changed flag in that function) or on first crossing the threshold.

---

Not a bug, but noticed live and worth a wording tweak: local status reports "147 cluster names resolvable via local DNS" while local hosts lists 49 rows. Both are correct (147 is exactly 49 x 3: each FQDN plus its two synthesized short forms), but a user comparing them will wonder where 98 names went. Something like "147 names (49 hosts)" in the status line, or reporting the host count there, would preempt the confusion.

[cursor]

Stale comment

Risk: medium. Cursor Bugbot is not configured on this PR. Human re-review is still needed: foxish's conditional approval predates two commits that address the noted libconnect fixes and status wording. daniel-de-vera remains assigned; foxish requested for re-verification.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}

[cursor]

Stale comment

Risk: medium. Cursor Bugbot is not configured on this PR. Human re-review is still needed: foxish's conditional approval predates three commits that address the noted libconnect fixes and status wording, and review threads remain open. daniel-de-vera and foxish remain assigned as reviewers.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}

[cursor]

Stale comment

Risk: medium. Cursor Bugbot is not configured on this PR. Human re-review is still needed: foxish's conditional approval predates four commits that address the noted libconnect fixes and status wording, and nine review threads remain unresolved. daniel-de-vera and foxish remain assigned as reviewers.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}

[cursor]

Stale comment

Risk: medium. Cursor Bugbot is not configured on this PR. Not approving: foxish's conditional approval predates five commits that address the noted libconnect fixes and status wording, and nine review threads remain unresolved. daniel-de-vera and foxish are already assigned as reviewers.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}

[cursor]

Stale comment

Risk: medium. Cursor Bugbot is not configured on this PR. Not approving: foxish's conditional approval predates five commits that address the noted libconnect fixes, and nine review threads remain unresolved. daniel-de-vera and foxish are already assigned as reviewers.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}

[cursor]

Risk: medium. Cursor Bugbot is not configured on this PR. Not approving: foxish's approval predates seven commits and nine review threads remain unresolved. daniel-de-vera and foxish are already assigned as reviewers.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}

[scott-cotton]

Thanks — all of the 07-20 review is addressed, plus the hardening that came out of the discussion. Both blocking bugs are fixed in libconnect (signadot/libconnect#163) and the pin here is bumped to include them.

Bug 1 — Linux self-forward loop — fixed structurally, not just patched:

• The bind probe now refuses any address already present as a nameserver in /etc/resolv.conf (libconnect 45a1c0f), on top of the existing in-use skip. So our bind can never be an address the resolver forwards to — the loop is impossible by construction. This is exactly what catches the crashed-session leftover (nameserver 127.0.0.54, no listener) that the in-use check alone missed.
• Belt-and-suspenders: newForwarder also drops our own bind address from captured upstreams (5dc7dd1).
• Crash-recovery hardening (from the "a competing DNS manager can strip our comment" thread): a leftover is now identified by two independent signals — an in-file @@@SIGNADOT-BIND-<ip>@@@ marker (substring-matched, survives comment reflow) and an uncontested on-disk bind record (survives a wiped file) — then neutralized with a faithful backup restore or a surgical strip that preserves a competing manager's own entries. Every action is logged causally, to kill the "why is DNS periodically slow?" support ghost (c5d81ec).

Bug 2 — macOS >10 resolver files warning — removed entirely (0788891), not just gated. It was a holdover from the earlier installer, which wrote roughly one file per cluster suffix (~2), where >10 plausibly signalled something odd; per-namespace short-name routing later made >10 the normal count. It was never meant to fire in the common path, so throttling its frequency was treating the symptom — removing it is the actual fix.

Host count (the 147-vs-49 note) — went with the explicit count: added HostCount to LocalDNSStatus (5a7b7ce); local status now shows 147 names (49 hosts) … (and in JSON). New-CLI/old-daemon skew handled too — the (N hosts) suffix is omitted when a pre-host_count daemon is queried, with tests (5a5e17c).

Also de-raced a pre-existing flake I hit testing on Linux: TestServiceCloseIdempotent / TestListenerDeathRollsBackInstaller raced serve() vs shutdown() ("server not started") — ~36/40 iterations under Docker at the pre-review commit. serve() now blocks until the listeners are actually serving; 0/40 after (008c84a). Full libconnect suite verified under -race on Linux (Docker) and macOS.

Merge order unchanged: land libconnect#163 first, then I'll bump the pin here to the merged commit and mark ready.