feat: etcd-backed store registries behind a StoreRegistry trait, with a mirrored migration path

[beinan]

Design: docs/src/design/registry-etcd.md (in this PR).

Problem

The store registries (_registry.rollout.lance, .generic, .datagen) are a name -> uri directory that every worker consults on an LRU miss (get_or_open_*) and writes on every create/delete. As a Lance table:

• every read is a checkout_latest = one manifest fetch from object storage (0.7 s on ADLS today, 1.5 s+ under throttling), on the hot path of every API call that misses the worker cache;
• every write is delete + append = two commits contended by 20 workers on one commit point; POST /rollouts averaged 47 s and 500'd 79 times in 15 h when ADLS throttled mid-way;
• unmaintained it grows quadratically (fix(master): compact and version-prune the store registries alongside _stats #274 fixes that; this PR is the next step from "linear and slow" to "fast").

The master already runs etcd for its task queue and locks, and the cluster exposes it to the namespace. A directory of ~11k {uri, created_at} entries is exactly what etcd is for.

Change

core

• StoreRegistry trait: contains / get / list / upsert / remove / insert_missing, all &self. Today's RwLock<RolloutRegistry> existed only because Lance handles must checkout_latest; dropping it removes a serialisation point every worker request went through.
• LanceRegistry (wraps RolloutRegistry; keeps fix(master): compact and version-prune the store registries alongside _stats #274's maintenance via lance_table()), EtcdRegistry (one key per store under <ETCD_PREFIX>/registry/<kind>/<name>, upsert is a plain put that preserves created_at, insert_missing is per-entry put-if-absent txns), MirroredRegistry (read primary, write both, mirror best-effort with a warning).
• backfill_registry(from, to) and diff_registries(a, b).
• EtcdConfig (the ETCD_* flags, moved out of the master so the server can share them) and RegistryConfig (REGISTRY_BACKEND=lance|etcd, REGISTRY_MIRROR=lance|etcd) with an open_registry() factory. Defaults are Lance with no mirror, so this PR is behaviour-neutral until the flags are set.

server: the three registries become Arc<dyn StoreRegistry>; create_*, get_or_open_*, unregister_*, list_* are unchanged in shape. Config gains the two flag groups.

master: same swap; TaskStore connects through EtcdConfig (prefix unchanged, keys unchanged); seeds the mirror from the primary at startup and heals it every maintenance round; GET /api/v1/registry/diff?kind= and POST /api/v1/registry/backfill?kind= for verifying a migration step; #274's Lance maintenance is a no-op on an etcd backend.

Migration (env-only, each step revertable)

1. REGISTRY_BACKEND=lance REGISTRY_MIRROR=etcd: reads unchanged, writes mirrored, master backfills. Check diff is empty.
2. REGISTRY_BACKEND=etcd REGISTRY_MIRROR=lance: reads from etcd; Lance kept current for rollback.
3. REGISTRY_BACKEND=etcd: done. Lance tables stay on disk as cold backup; discovery.rs can rebuild etcd from the data directory regardless.

Verification

• registry_etcd tests (etcd-backed): upsert/get/contains/remove incl. created_at preservation on retried create; list + insert_missing with duplicates; prefix names (a vs ab) do not collide.
• mirrored_registry_keeps_both_backends_in_step: the step-1 configuration end to end — pre-existing Lance rows, diff shows them, backfill copies them, mirrored writes and removes land in both, diff is empty.
• Existing suites unchanged: core 242, master 84 (+29 etcd-gated), server 43 (+5/8/1 integration). Clippy -D warnings, fmt clean.

Not in scope: _stats (needs range queries; stays in Lance), MemWAL shard manifests.

🤖 Generated with Claude Code

Rebase and review follow-up (2026-10-02)

Rebased onto upstream b85db02, including #292 and #284. Resolved the registry/merge configuration overlap without duplicate etcd CLI arguments, preserved lazy merge-only connection behavior, and retained the recovery coordinator, repair/rescan routes, and serial fan-out. Updated registry call sites added since the original branch.

Local validation on the rebased code: 88 master tests, 92 worker tests, and 14 core registry tests passed, with the etcd-backed cases enabled against isolated local etcd. Clippy passed for core/master/server/merge with all targets and warnings denied; formatting and diff checks passed. CI has restarted on the rebased head.

Migration review remains unresolved: master backfill/diff does not cover datagen even though worker backend selection switches it; insert-only mirror healing cannot repair missed deletions; and the name-only diff can report agreement despite different URIs. Two local diagnostic probes reproduced the latter two failures. These probes are not part of the committed suite. Do not treat the existing migration/rollback description or passing happy-path tests as evidence that these cases are safe. This rebase does not enable or deploy the registry backend.