fix(serve): keep a public endpoint authenticated across respawns - #151
Open
volen-silo wants to merge 2 commits into
Open
fix(serve): keep a public endpoint authenticated across respawns#151volen-silo wants to merge 2 commits into
volen-silo wants to merge 2 commits into
Conversation
A public (non-loopback) `rocm serve` bind mints an endpoint API key and persists it in a 0600 file, but every path that *respawns* a recorded service treated the presence of that file as the source of truth. The real invariant is a property of the host, not the file: a non-loopback bind must always be authenticated. Reachable through ordinary commands: rocm serve --host 0.0.0.0 --allow-public-bind # key minted and stored rocm services stop <id> # key file deleted rocm services restart <id> # respawns with NO key `restart_internal_managed_service` captured `None` for the preserved key, stored nothing, and spawned the engine child without ROCM_SERVE_API_KEY_FILE, so the service came back on 0.0.0.0 completely unauthenticated. `rocm services restart` documents its argument as an id from `rocm services list --all`, which includes stopped records, so this is the documented flow rather than an edge case. `rocmd`'s recovery supervisor had the same gap. Enforce the invariant at each spawn site and fail closed, mirroring the existing `ensure_public_bind_engine_supported` precedent: - Add `rocm_engine_protocol::is_public_bind_host` so `rocm` and `rocmd` classify a recorded host identically; `is_loopback_host` now delegates to it. - Refuse to respawn a public service with no key in `restart_internal_managed_service` (before the stop, so a refused restart leaves a running service running), in `spawn_managed_engine_child`, and in `rocmd`'s `supervise_service`. - Stop only drops the key file once termination is confirmed. An unconfirmed stop may have left the engine alive and still enforcing the key, and discarding the only copy locked the CLI's own probes, chat, and service discovery out of a healthy service. Loopback behavior is unchanged: it stays credential-free. Closes EAI-7409. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
- Check key *validity*, not file existence. `endpoint_key_file_if_present` is documented as an existence check only, and the engine adapters resolve the key with `endpoint_api_key_from_file` and enforce nothing when it yields `None`. An empty or malformed key file therefore satisfied the guard at two of its three sites and still produced an unauthenticated public listener — the exact failure the guard exists to prevent. - Do not let a refused recovery kill the daemon. `supervise_service`'s bail propagated through `restart_managed_service` and `evaluate_watchers` to `run_daemon`, which `?`-propagates on both the startup and ticker paths. Since a missing key is a permanent condition, that took down every watcher on each 30s recovery tick. Refuse at the recovery boundary instead, recording a `restart_managed_service_refused` event, mirroring the containment the webhook branch already uses. - Do not orphan the key file. Gating the stop-time clear on confirmed termination left the 0600 secret on disk forever: the liveness refresh that later observes the process dead sets "stopped" but never cleared the key, and no sweep exists. Clear it there, so the key's lifetime tracks confirmed death rather than the stop command. - Refuse before the manifest write in `supervise_service`, so a refused respawn leaves restart_count and timestamps intact. - Delegate rocmd's copied `is_loopback_host` to the shared predicate — a second hand-maintained classification is the divergence hazard this change closes. - Cover the empty-key-file case, record why the ordering assertion holds, and update docs/testing.md with the new invariant and test filters. Known gap, not addressed here: `rocmd`'s own `stop_managed_service` still clears the key unconditionally, because `terminate_process` there reports only that a signal was accepted, so it has no confirmed-termination notion to gate on. Giving it one is a larger change than this fix. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No EAI-7409 rows exist; endpoint auth has never had e2e coverage.)Problem
Public-endpoint authentication (EAI-7409, added in #124) held for the initial
rocm serve, but not for any path that respawns a recorded service. Those paths treated "a key file exists" as the source of truth. The real invariant is a property of the recorded host: a non-loopback bind must always be authenticated.Reachable through first-class commands:
restart_internal_managed_servicecapturedNonefor the preserved key, stored nothing, and spawned the engine child with noROCM_SERVE_API_KEY_FILE, so the service returned on0.0.0.0completely unauthenticated.rocm services restartdocuments its argument as an id fromrocm services list --all, which includes stopped records, so this is the documented flow rather than an edge case.rocmd's recovery supervisor had the same gap.Approach
Enforce the invariant at each spawn site and fail closed, following the existing
ensure_public_bind_engine_supportedprecedent. Fail-closed rather than mint-and-print a replacement key:restart_serveris also an automation/sandbox tool whose JSON result can land in agent transcripts, so regenerating would add a secret-disclosure surface.rocm_engine_protocol::is_public_bind_hostis now the single classification, sorocmandrocmdcannot disagree about a recorded host. Both crates'is_loopback_hostdelegate to it.restart_internal_managed_service(before the stop, so a refused restart leaves a running service running), inspawn_managed_engine_child, and inrocmd'ssupervise_service(before the manifest write, so a refusal does not clobber restart_count/timestamps).endpoint_api_key_from_fileand enforce nothing when it yieldsNone, so an empty file would otherwise pass the guard and still serve anonymously.restart_managed_service_refusedevent, rather than letting a permanent failure propagate out ofevaluate_watchersand take every watcher down on each 30s tick.Loopback behavior is unchanged: it stays credential-free.
Behavior change
A public-host service whose key is gone (including records predating #124) is no longer restartable in place; relaunch it with
rocm serve --host <host> --allow-public-bindto issue a new key. The refusal message says so. This is the intended cost of fail-closed.Known gap
rocmd's ownstop_managed_servicestill clears the key unconditionally: itsterminate_processreports only that a signal was accepted, so it has no confirmed-termination notion to gate on. Giving it one is a larger change than this fix.Testing
cargo fmt --all -- --checkandcargo clippy --workspace --all-targets -- -D warningsclean.cargo test --workspace --no-fail-fastpasses except two pre-existingrocm-coreproc_lifecycle::tests::tree_*failures, which fail on WSL2 for process-tree signalling reasons unrelated to this change (untouched file).Not run locally: GPU acceptance (
scripts/vllm_therock_gpu_test.py) and the GPU e2e lanes — no GPU on this host. The end-to-end assertion that a restarted public server actually rejects an unauthenticated request needs a live engine; endpoint auth has no e2e coverage today, so these paths are unit-tested only. Filters are listed indocs/testing.md.Closes EAI-7409.