Resolve and deliver tong sidecar secrets through host provider CLIs#10
Merged
Conversation
Tong definitions reference secrets as ${secret:<provider>:<ref>}. Resolve
them on the host by shelling out to a provider CLI declared once in the
user layer (~/.swarmforge/secret-providers.yaml), the
docker-credential-helper pattern, so Swarmforge knows nothing about any
individual secret manager. Loading the provider table and building the
argv are pure (in tongs.py); the subprocess that runs the CLI lives in the
launcher's resolver closure, which surfaces a clear error -- never the
secret -- on an unknown provider, an unrunnable CLI, or a non-zero exit.
Only the CLI's stdout is captured; its stderr passes through to the user's
terminal, so interactive unlocks (op signin, biometrics) stay visible.
Deliver resolved secrets to a tong through an in-memory tmpfs file rather
than a docker -e env var: anything holding the docker socket could read an
env value back via docker inspect. Each secret-bearing env var becomes a
tmpfs file plus a <NAME>_FILE pointer the tong reads; plain env keeps
flowing through -e. The resolved value only ever appears in the file plan,
never in the env the launcher passes as -e, and the launcher never writes
a secret to host disk. An env name that would escape the tmpfs dir as a
path component is refused, and a declared env var that collides with a
secret's <NAME>_FILE pointer fails the launch (and validation) unless it
already names the generated path, so a tong is never started unable to
find its secret.
These are host-side helpers exercised only when a tong carries a secret
reference; the empty-discovery launch path is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sidecar (tong) definitions reference secrets as
${secret:<provider>:<ref>}.This adds host-side resolution of those references and a leak-resistant path for
delivering the resolved values into a sidecar.
Secrets are resolved on the host by shelling out to a provider CLI declared once
in the user layer (
~/.swarmforge/secret-providers.yaml), e.g.op: ["op", "read", "{ref}"]. This is the docker-credential-helper pattern, soSwarmforge knows nothing about any individual secret manager and 1Password, pass,
AWS, Doppler, etc. work without bespoke support. Interactive unlocks (
op signin,biometrics) work because the launcher runs in the user's terminal.
Resolved secrets reach a sidecar through an in-memory tmpfs file plus a
<NAME>_FILEenv pointer, never a docker-eenv var — anything holding thedocker socket could otherwise read an env value back via
docker inspect.These are host-side helpers exercised only when a definition carries a secret
reference; a launch for a repo with no sidecar definitions is unchanged.
Changes
scripts/tongs.py:load_secret_providersparses the user-layer provider table, returning{}for a missing file and raising on a malformed one so a typo surfaces at load
time;
secret_provider_commandsubstitutes the literal{ref}token inevery argv element.
partition_secret_env,secret_delivery_plan, andplan_tong_secretsroutesecret-bearing env to tmpfs files with
<NAME>_FILEpointers while leavingplain env as
-e. The resolved value only ever appears in the file plan,never in the
-eenv. An env name that would escape the tmpfs dir as a pathcomponent is refused, and a file pointer that collides with a declared env
var keeps the declared value with a warning.
make_secret_resolverinscripts/run_anvil.py: the impure resolverclosure that shells out to the provider CLI and returns the secret, raising a
clear error — naming the provider/reference, never the secret — on an unknown
provider, an unrunnable CLI, or a non-zero exit. A single trailing newline is
stripped; other whitespace is preserved.
scripts/test_tongs.pyandscripts/test_run_anvil.pywith coveragefor provider-table loading and validation,
{ref}substitution, the execresolver (success, newline trimming, unknown provider, non-zero exit,
unrunnable CLI), the secret-never-in-env invariant end to end, the
traversal-name rejection, and the pointer-collision warning. The existing
passthrough-invariant tests still pass.