Repository navigation
fix(launchers): Keep resolved input values out of error messages - #423
Open
yuechao-qin wants to merge 1 commit into
Open
yuechao-qin wants to merge 1 commit into
yuechao-qin wants to merge 1 commit into
Conversation
A dataclass prints every field in its generated repr, so interpolating an InputArgument into a launcher error message published the resolved secret to the log, the error reporter, the database and the execution-log API alike. repr=False on the value field removes it from all 19 of those messages at once, while leaving the field itself unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A secret consumed as an input value is held in
InputArgument.value. Launcher error messages interpolate the whole argument —f"... {input_name=}, {input_argument=}"— and a dataclass prints every field in the repr it generates, so the message carries the plaintext.The orchestrator then copies that one string into four places: the application log (
_logger.exception), the error reporter,ExecutionNode.extra_data, and the execution-log API response (system_error_exception_full). A failed launch therefore publishes the value to readers who were never granted the secret, and two of those copies cannot be withdrawn.There are 19 such messages across the Kubernetes, Docker and HuggingFace launchers. Editing them one by one is the same fix 19 times, and the 20th would leak again.
Before / after
input_argument=InputArgument(total_size=44, is_dir=False, value='<plaintext>', uri=None, staging_uri='gs://...')input_argument=InputArgument(total_size=44, is_dir=False, uri=None, staging_uri='gs://...')input_argument.valuein launcher codeFix
repr=Falseomits the field from the generated__repr__, whichstr(),print(),%s,.format()andf"{arg=}"all route through. The field is unchanged in every other respect: still present, still assigned, still read by the launchers that need to pass the value to a container.Notes
ExecutionNode.extra_datakeep their copy. The stored ones can be cleaned by a one-off pass; the external ones cannot, so anything already exposed needs rotating.dataclasses.asdictignoresrepr=False. No launcher uses it on an input argument, and a test pins that.Tests
tests/launchers/test_input_argument_repr.py, 7 tests:repr,str,f"{a}",f"{a!r}",f"{a=}",.format,%s— omits the value.valueinto a string, and none callsasdicton an argumentasdictwould have leaked it, so the test above is not asserting a bare conventionSuite: 581 passed. Reverting the one-line change turns two of the new tests red.