Repository navigation
feat(qsv): run qsv through qsv-client - #342
Merged
Merged
Conversation
QSVCommand now runs qsv through the qsv-client package. Its public
surface is unchanged: the same methods, a subprocess.CompletedProcess on
success, JobError on failure (or the {"stdout", "stderr"} dict with
uses_stdio), and the result itself with check=False.
What changes:
- A timeout kills qsv's whole process group, so children qsv started
cannot outlive it or hold its pipes open and hang the worker.
- qsv is asked for structured errors (QSV_ERROR_FORMAT=json); with qsv
builds that support them, JobError carries the error kind and message.
Older builds print text, which is used as before.
- The version check reads `qsv --capabilities`, falling back to
`--version` on older builds.
- `env=` is merged into the worker's environment instead of replacing it
(subprocess.run's env= dropped PATH and everything else).
- describegpt's api_key goes in QSV_LLM_APIKEY instead of --api-key:
describegpt copies its command line into the attribution it writes, and
argv is visible in ps. The "NONE" sentinel, which qsv honors only on the
flag, stays on the command line.
Any non-zero exit is still a failure, including 255, which qsv-client
itself counts as success.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The old JobError text came from CalledProcessError, which included the whole argv. Keep that: operators reading a failed job's log need to see which file and flags were used, not just the subcommand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
QSVCommandnow runs qsv throughqsv-client(qsv-client>=0.1.0, no dependencies). This is the shared Python client from the dathere/qsv#1248 follow-up. Its public surface is unchanged: the same methods, asubprocess.CompletedProcesson success,JobErroron failure (or the{"stdout", "stderr"}dict withuses_stdio), and the result itself withcheck=False. No caller changes.What changes
subprocess.run(timeout=...)killed only qsv. A child qsv had started could outlive it and keep the pipes open, so the worker kept waiting even after the timeout fired. Now the group gets SIGTERM, then SIGKILL.QSV_ERROR_FORMAT=json(feat: QSV_ERROR_FORMAT=json and --capabilities for subprocess callers qsv#4765). Builds that support it report the error kind and message, andJobErrorcarries them, e.g.qsv command failed: qsv validate failed (csv, exit 1): …. Builds that don't (all current releases, including 20.1.0) print text, which is used as before. Theqsv command failed:prefix thattests/log_analyzer.pymatches is kept.qsv --capabilities, falling back to--version(and--list) on older builds.MINIMUM_QSV_VERSIONand theQSV_VERSIONthe worker image installs are unchanged.env=is merged into the worker's environment instead of replacing it, assubprocess.run(env=...)did, which droppedPATHand everything else. No current caller passesenv.api_keygoes inQSV_LLM_APIKEY, not--api-key. describegpt copies its command line into the attribution of the dictionary, description and tags it generates, and argv shows up inps. (fix(describegpt): redact --api-key and --ckan-token from the attribution qsv#4766 also redacts it from the attribution on the qsv side.) The"NONE"sentinel stays on the command line, because qsv honors it only on the flag. No current caller passesapi_key.capture_output=Falseis now ignored (output is always captured; no caller sets it), and qsv's stdin is/dev/null(no DP+ command reads stdin).Any non-zero exit is still a failure, including 255, which qsv-client itself counts as success.
Tests
tests/test_qsv_command_client.pyrunsQSVCommandend to end against a fakeqsvdp. It covers the version check (passing and too old), success, structured and text failures, the empty-stderr case, exit 255,uses_stdio,check=False,text=False, env merging, and a timeout that must kill qsv's child process.tests/test_ai_suggestions.py: the describegpt tests mockedsubprocess.run. They now mock qsv-client's process layer, which has the same call shape. There's a new test for the key going in the environment, andapi_key=""now goes there too.uses_stdiodict, putting the key back on argv, replacing instead of mergingenv, and not raising on failure.dpp-testcontainer (321 passed). I also smoke-tested against its real qsvdp 20.1.0: version via the--versionfallback,count,headers,sortcheckreturning theuses_stdiodict on an unsorted file, avalidatefailure raisingJobError, andsniff.Found while reviewing, not changed here
With
ckanext.datapusher_plus.pii_quick_screen = true(default false), a file with no PII fails the job.qsv searchset --quickexits 1 when nothing matches, andscreen_for_piitreats that asJobError("Cannot quickly search CSV for PII"). I confirmed this with a real binary. The fix is passing--not-onein the quick call. It's left for a separate PR so this one changes no pipeline behavior.🤖 Generated with Claude Code