Skip to content

fix(launchers): Keep resolved input values out of error messages - #423

Open
yuechao-qin wants to merge 1 commit into
masterfrom
ycq/do-not-print-resolved-input-argument-values
Open

yuechao-qin wants to merge 1 commit into
masterfrom
ycq/do-not-print-resolved-input-argument-values

Conversation

@yuechao-qin

Copy link
Copy Markdown
Collaborator

Problem

A secret consumed as an input value is held in InputArgument.value. Launcher error messages interpolate the whole argument — f"... {input_name=}, {input_argument=}" — and a dataclass prints every field in the repr it generates, so the message carries the plaintext.

The orchestrator then copies that one string into four places: the application log (_logger.exception), the error reporter, ExecutionNode.extra_data, and the execution-log API response (system_error_exception_full). A failed launch therefore publishes the value to readers who were never granted the secret, and two of those copies cannot be withdrawn.

There are 19 such messages across the Kubernetes, Docker and HuggingFace launchers. Editing them one by one is the same fix 19 times, and the 20th would leak again.

Before / after

Surface Before After
a launcher error message input_argument=InputArgument(total_size=44, is_dir=False, value='<plaintext>', uri=None, staging_uri='gs://...') input_argument=InputArgument(total_size=44, is_dir=False, uri=None, staging_uri='gs://...')
the stored exception, and the log the same string, with the value the same string, without it
input_argument.value in launcher code the value unchanged — the container still receives it

Fix

# cloud_pipelines_backend/launchers/interfaces.py
value: str | None = dataclasses.field(default=None, repr=False)

repr=False omits the field from the generated __repr__, which str(), print(), %s, .format() and f"{arg=}" all route through. The field is unchanged in every other respect: still present, still assigned, still read by the launchers that need to pass the value to a container.

Notes

  • Not retroactive: messages already written to logs, to the error reporter, and to ExecutionNode.extra_data keep their copy. The stored ones can be cleaned by a one-off pass; the external ones cannot, so anything already exposed needs rotating.
  • dataclasses.asdict ignores repr=False. No launcher uses it on an input argument, and a test pins that.

Tests

tests/launchers/test_input_argument_repr.py, 7 tests:

  • every way of stringifying an argument — repr, str, f"{a}", f"{a!r}", f"{a=}", .format, %s — omits the value
  • the other fields still print, since the messages exist to identify an input
  • the value is still readable and still a field
  • a traceback built the way the launchers build one is clean
  • no launcher interpolates .value into a string, and none calls asdict on an argument
  • asdict would have leaked it, so the test above is not asserting a bare convention

Suite: 581 passed. Reverting the one-line change turns two of the new tests red.

A dataclass prints every field in its generated repr, so interpolating an InputArgument into a launcher error message published the resolved secret to the log, the error reporter, the database and the execution-log API alike. repr=False on the value field removes it from all 19 of those messages at once, while leaving the field itself unchanged.
@yuechao-qin
yuechao-qin requested a review from a team November 2, 2026 16:58
@yuechao-qin
yuechao-qin requested a review from Ark-kun as a code owner November 2, 2026 16:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant