Skip to content

chore(scripts): let seeding establish its own port-forwards - #643

Merged
ericfitz merged 1 commit into
mainfrom
chore/ensure-port-forwards
Jul 31, 2026
Merged

chore(scripts): let seeding establish its own port-forwards#643
ericfitz merged 1 commit into
mainfrom
chore/ensure-port-forwards

Conversation

@ericfitz

Copy link
Copy Markdown
Owner

Seeding needs localhost:8080 and localhost:5432, but nothing established either — they
were a documented manual prerequisite.

Why the manual approach could never work

This isn't a lapse in developer diligence. stop_port_forward()'s legacy reaper matches any
-n tmi-platform port-forward svc/ command, so a hand-started forward is killed as an
"orphan"
by the next dev-up/dev-restart/dev-down. It then resurfaces as a confusing
failure inside the seed rather than as a missing prerequisite.

The change

ensure_port_forward(name), modelled on tmi_common.ensure_oauth_stub: probe → return
quietly if healthy → otherwise start and wait. Generic over the three named dev forwards;
only seeding is wired to it for now, per the agreed scope.

A forward whose recorded kube context is no longer the current one counts as unhealthy and
is replaced rather than reused — a live supervisor pointing localhost at another cluster is
the #580 failure mode.

scripts/run-dbtool.py is the single wiring point, which covers every way seeding is invoked:
make cats-seed, make e2e-seed, and the cats plugin's seed hook (hence make cats-fuzz).
Skipped for --oci (external ADB, no Service to forward) and when --server isn't loopback.

Postgres gets a first-class supervised forward, pidfile-tracked so stop_port_forward() tears
it down deliberately. It is not started by dev-up — 5432 collides with a locally
installed PostgreSQL, and a developer who never runs CATS shouldn't have to care. On a
collision _preflight_port already hard-fails naming the squatter; the message now adds that
a local PostgreSQL commonly holds that port, and why seeding must not silently use it.

Two bugs found while doing this

  1. start_redis_port_forward() called the blanket stop_port_forward(), clearing every
    pidfile. That forced the server forward to start afterwards to survive — an ordering
    constraint that would silently extend to any third forward — and destroyed the postgres
    forward on every dev-up/dev-restart. Each starter now stops only its own pidfile.
  2. ensure returned ~3s before the port was bound, so the contract was "started" rather
    than "usable" and an immediate connect could race it. It now waits for the bind, as
    ensure_oauth_stub does.

Test change, called out explicitly

test_server_port_forward_is_k3s_only is widened for the one new call site — the by-name
dispatch inside ensure_port_forward, whose callers know the server URL is loopback rather
than knowing cluster_target. Any other unguarded call site still fails. The campaign still
hits the NodePort directly; this path only ever carries low-volume seeding traffic, which is
what #463/#578 actually protect.

Verification

make test-dev-scripts — 164 tests, OK (10 new). make lint — 0 issues. Only Python changed,
so the Go gates don't apply.

Against the live k3s cluster:

check result
make cats-seed with 5432 down starts the forward, waits for bind, seeds successfully
second ensure call Port-forward already running — no churn
forward survives make dev-restart survived (previously killed as an orphan)
stop_port_forward() tears down all three deliberately, by pidfile
connect immediately after ensure returns connects — race gone

🤖 Generated with Claude Code

https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc

Seeding needs localhost:8080 and localhost:5432, but nothing established
either -- they were a documented manual prerequisite. That was never
workable, for a reason that is structural rather than a lapse in
diligence: stop_port_forward()'s legacy reaper matches any
"-n tmi-platform port-forward svc/" command, so a hand-started forward is
killed as an "orphan" by the next dev-up/dev-restart/dev-down. It then
resurfaces as a confusing failure inside the seed rather than as a missing
prerequisite.

Adds ensure_port_forward(name), modelled on tmi_common.ensure_oauth_stub:
probe, return quietly if already healthy, otherwise start and wait. It is
generic over the three named dev forwards; only seeding is wired to it for
now. A forward whose recorded kube context is no longer the current one
counts as unhealthy and is replaced rather than reused -- a live supervisor
pointing localhost at another cluster is the #580 failure mode.

scripts/run-dbtool.py is the single wiring point, which covers every way
seeding is invoked: make cats-seed, make e2e-seed, and the cats plugin's
seed hook (hence make cats-fuzz). Skipped for --oci (external ADB, no
Service to forward) and when --server is not loopback.

Postgres gets a first-class supervised forward, pidfile-tracked like the
others so stop_port_forward() tears it down deliberately. It is deliberately
NOT started by dev-up: 5432 collides with a locally installed PostgreSQL,
and a developer who never runs CATS should not have to care. On a collision
_preflight_port already hard-fails naming the squatter; the message now adds
that a local PostgreSQL commonly holds that port, and why seeding must not
silently use it.

Two fixes found while doing this:

  - start_redis_port_forward() called the blanket stop_port_forward(), which
    cleared every pidfile. That forced the server forward to be started
    afterwards to survive -- an ordering constraint that would silently
    extend to any third forward -- and destroyed the postgres forward on
    every dev-up/dev-restart. Each starter now stops only its own pidfile.
    Squatters are still handled by the existing marker-scoped orphan reaping
    and _preflight_port.

  - ensure returned ~3s before the port was bound, so the contract was
    "started" rather than "usable" and an immediate connect could race it.
    It now waits for the bind, as ensure_oauth_stub does.

test_server_port_forward_is_k3s_only is widened for the one new call site,
the by-name dispatch inside ensure_port_forward, whose callers know the
server URL is loopback rather than knowing cluster_target. Any other
unguarded call site still fails. The campaign still hits the NodePort
directly -- this path only ever carries low-volume seeding traffic, which is
what #463/#578 actually protect.

Verified against the live k3s cluster: cats-seed starts the forward and
seeds from a cold start; a second call is a no-op; the forward now survives
dev-restart; stop_port_forward() still tears all three down deliberately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc
@ericfitz
ericfitz merged commit 2b49b1b into main Jul 31, 2026
12 checks passed
@ericfitz
ericfitz deleted the chore/ensure-port-forwards branch July 31, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant