Skip to content

fix(serve): keep a public endpoint authenticated across respawns - #151

Open
volen-silo wants to merge 2 commits into
mainfrom
fix/public-endpoint-restart-auth
Open

fix(serve): keep a public endpoint authenticated across respawns#151
volen-silo wants to merge 2 commits into
mainfrom
fix/public-endpoint-restart-auth

Conversation

@volen-silo

Copy link
Copy Markdown
Collaborator
  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No EAI-7409 rows exist; endpoint auth has never had e2e coverage.)

Problem

Public-endpoint authentication (EAI-7409, added in #124) held for the initial rocm serve, but not for any path that respawns a recorded service. Those paths treated "a key file exists" as the source of truth. The real invariant is a property of the recorded host: a non-loopback bind must always be authenticated.

Reachable through first-class commands:

rocm serve --host 0.0.0.0 --allow-public-bind   # key minted, stored 0600, printed
rocm services stop <id>                         # key file deleted
rocm services restart <id>                      # comes back with NO key

restart_internal_managed_service captured None for the preserved key, stored nothing, and spawned the engine child with no ROCM_SERVE_API_KEY_FILE, so the service returned on 0.0.0.0 completely unauthenticated. rocm services restart documents its argument as an id from rocm services list --all, which includes stopped records, so this is the documented flow rather than an edge case. rocmd's recovery supervisor had the same gap.

Approach

Enforce the invariant at each spawn site and fail closed, following the existing ensure_public_bind_engine_supported precedent. Fail-closed rather than mint-and-print a replacement key: restart_server is also an automation/sandbox tool whose JSON result can land in agent transcripts, so regenerating would add a secret-disclosure surface.

  • rocm_engine_protocol::is_public_bind_host is now the single classification, so rocm and rocmd cannot disagree about a recorded host. Both crates' is_loopback_host delegate to it.
  • Refuse a keyless public respawn in restart_internal_managed_service (before the stop, so a refused restart leaves a running service running), in spawn_managed_engine_child, and in rocmd's supervise_service (before the manifest write, so a refusal does not clobber restart_count/timestamps).
  • The guards check key validity, not file existence — the engine adapters resolve the key with endpoint_api_key_from_file and enforce nothing when it yields None, so an empty file would otherwise pass the guard and still serve anonymously.
  • A stop drops the key only once it has confirmed termination. An unconfirmed stop may have left the engine alive and still enforcing the key, and discarding the only copy locked the CLI's own probes, chat, and discovery out of a healthy service. The deferred cleanup lands on the liveness refresh that later observes the process dead, so no plaintext secret is stranded.
  • Daemon recovery refuses at the recovery boundary with a recorded restart_managed_service_refused event, rather than letting a permanent failure propagate out of evaluate_watchers and take every watcher down on each 30s tick.

Loopback behavior is unchanged: it stays credential-free.

Behavior change

A public-host service whose key is gone (including records predating #124) is no longer restartable in place; relaunch it with rocm serve --host <host> --allow-public-bind to issue a new key. The refusal message says so. This is the intended cost of fail-closed.

Known gap

rocmd's own stop_managed_service still clears the key unconditionally: its terminate_process reports only that a signal was accepted, so it has no confirmed-termination notion to gate on. Giving it one is a larger change than this fix.

Testing

cargo fmt --all -- --check and cargo clippy --workspace --all-targets -- -D warnings clean. cargo test --workspace --no-fail-fast passes except two pre-existing rocm-core proc_lifecycle::tests::tree_* failures, which fail on WSL2 for process-tree signalling reasons unrelated to this change (untouched file).

Not run locally: GPU acceptance (scripts/vllm_therock_gpu_test.py) and the GPU e2e lanes — no GPU on this host. The end-to-end assertion that a restarted public server actually rejects an unauthenticated request needs a live engine; endpoint auth has no e2e coverage today, so these paths are unit-tested only. Filters are listed in docs/testing.md.

Closes EAI-7409.

A public (non-loopback) `rocm serve` bind mints an endpoint API key and
persists it in a 0600 file, but every path that *respawns* a recorded
service treated the presence of that file as the source of truth. The real
invariant is a property of the host, not the file: a non-loopback bind must
always be authenticated.

Reachable through ordinary commands:

  rocm serve --host 0.0.0.0 --allow-public-bind   # key minted and stored
  rocm services stop <id>                         # key file deleted
  rocm services restart <id>                      # respawns with NO key

`restart_internal_managed_service` captured `None` for the preserved key,
stored nothing, and spawned the engine child without ROCM_SERVE_API_KEY_FILE,
so the service came back on 0.0.0.0 completely unauthenticated. `rocm services
restart` documents its argument as an id from `rocm services list --all`,
which includes stopped records, so this is the documented flow rather than an
edge case. `rocmd`'s recovery supervisor had the same gap.

Enforce the invariant at each spawn site and fail closed, mirroring the
existing `ensure_public_bind_engine_supported` precedent:

- Add `rocm_engine_protocol::is_public_bind_host` so `rocm` and `rocmd`
  classify a recorded host identically; `is_loopback_host` now delegates to it.
- Refuse to respawn a public service with no key in
  `restart_internal_managed_service` (before the stop, so a refused restart
  leaves a running service running), in `spawn_managed_engine_child`, and in
  `rocmd`'s `supervise_service`.
- Stop only drops the key file once termination is confirmed. An unconfirmed
  stop may have left the engine alive and still enforcing the key, and
  discarding the only copy locked the CLI's own probes, chat, and service
  discovery out of a healthy service.

Loopback behavior is unchanged: it stays credential-free.

Closes EAI-7409.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
- Check key *validity*, not file existence. `endpoint_key_file_if_present` is
  documented as an existence check only, and the engine adapters resolve the key
  with `endpoint_api_key_from_file` and enforce nothing when it yields `None`.
  An empty or malformed key file therefore satisfied the guard at two of its
  three sites and still produced an unauthenticated public listener — the exact
  failure the guard exists to prevent.

- Do not let a refused recovery kill the daemon. `supervise_service`'s bail
  propagated through `restart_managed_service` and `evaluate_watchers` to
  `run_daemon`, which `?`-propagates on both the startup and ticker paths. Since
  a missing key is a permanent condition, that took down every watcher on each
  30s recovery tick. Refuse at the recovery boundary instead, recording a
  `restart_managed_service_refused` event, mirroring the containment the webhook
  branch already uses.

- Do not orphan the key file. Gating the stop-time clear on confirmed
  termination left the 0600 secret on disk forever: the liveness refresh that
  later observes the process dead sets "stopped" but never cleared the key, and
  no sweep exists. Clear it there, so the key's lifetime tracks confirmed death
  rather than the stop command.

- Refuse before the manifest write in `supervise_service`, so a refused respawn
  leaves restart_count and timestamps intact.

- Delegate rocmd's copied `is_loopback_host` to the shared predicate — a second
  hand-maintained classification is the divergence hazard this change closes.

- Cover the empty-key-file case, record why the ordering assertion holds, and
  update docs/testing.md with the new invariant and test filters.

Known gap, not addressed here: `rocmd`'s own `stop_managed_service` still clears
the key unconditionally, because `terminate_process` there reports only that a
signal was accepted, so it has no confirmed-termination notion to gate on.
Giving it one is a larger change than this fix.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
@volen-silo
volen-silo requested a review from a team as a code owner July 30, 2026 12:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant