Skip to content

feat(qsv): run qsv through qsv-client - #342

Merged
jqnatividad merged 2 commits into
mainfrom
qsv-client-adoption
Oct 8, 2026
Merged

jqnatividad merged 2 commits into
mainfrom
qsv-client-adoption

Conversation

@jqnatividad

Copy link
Copy Markdown
Collaborator

QSVCommand now runs qsv through qsv-client (qsv-client>=0.1.0, no dependencies). This is the shared Python client from the dathere/qsv#1248 follow-up. Its public surface is unchanged: the same methods, a subprocess.CompletedProcess on success, JobError on failure (or the {"stdout", "stderr"} dict with uses_stdio), and the result itself with check=False. No caller changes.

What changes

  • Timeouts kill the whole process group. Before, subprocess.run(timeout=...) killed only qsv. A child qsv had started could outlive it and keep the pipes open, so the worker kept waiting even after the timeout fired. Now the group gets SIGTERM, then SIGKILL.
  • Structured errors. qsv is asked for QSV_ERROR_FORMAT=json (feat: QSV_ERROR_FORMAT=json and --capabilities for subprocess callers qsv#4765). Builds that support it report the error kind and message, and JobError carries them, e.g. qsv command failed: qsv validate failed (csv, exit 1): …. Builds that don't (all current releases, including 20.1.0) print text, which is used as before. The qsv command failed: prefix that tests/log_analyzer.py matches is kept.
  • Version check reads qsv --capabilities, falling back to --version (and --list) on older builds. MINIMUM_QSV_VERSION and the QSV_VERSION the worker image installs are unchanged.
  • env= is merged into the worker's environment instead of replacing it, as subprocess.run(env=...) did, which dropped PATH and everything else. No current caller passes env.
  • describegpt's api_key goes in QSV_LLM_APIKEY, not --api-key. describegpt copies its command line into the attribution of the dictionary, description and tags it generates, and argv shows up in ps. (fix(describegpt): redact --api-key and --ckan-token from the attribution qsv#4766 also redacts it from the attribution on the qsv side.) The "NONE" sentinel stays on the command line, because qsv honors it only on the flag. No current caller passes api_key.
  • capture_output=False is now ignored (output is always captured; no caller sets it), and qsv's stdin is /dev/null (no DP+ command reads stdin).

Any non-zero exit is still a failure, including 255, which qsv-client itself counts as success.

Tests

  • New tests/test_qsv_command_client.py runs QSVCommand end to end against a fake qsvdp. It covers the version check (passing and too old), success, structured and text failures, the empty-stderr case, exit 255, uses_stdio, check=False, text=False, env merging, and a timeout that must kill qsv's child process.
  • tests/test_ai_suggestions.py: the describegpt tests mocked subprocess.run. They now mock qsv-client's process layer, which has the same call shape. There's a new test for the key going in the environment, and api_key="" now goes there too.
  • I mutation-checked five mutations; each made the new tests fail: treating 255 as success, dropping the uses_stdio dict, putting the key back on argv, replacing instead of merging env, and not raising on failure.
  • The unit suite passes in the dpp-test container (321 passed). I also smoke-tested against its real qsvdp 20.1.0: version via the --version fallback, count, headers, sortcheck returning the uses_stdio dict on an unsorted file, a validate failure raising JobError, and sniff.

Found while reviewing, not changed here

With ckanext.datapusher_plus.pii_quick_screen = true (default false), a file with no PII fails the job. qsv searchset --quick exits 1 when nothing matches, and screen_for_pii treats that as JobError("Cannot quickly search CSV for PII"). I confirmed this with a real binary. The fix is passing --not-one in the quick call. It's left for a separate PR so this one changes no pipeline behavior.

🤖 Generated with Claude Code

jqnatividad and others added 2 commits October 8, 2026 15:39
QSVCommand now runs qsv through the qsv-client package. Its public
surface is unchanged: the same methods, a subprocess.CompletedProcess on
success, JobError on failure (or the {"stdout", "stderr"} dict with
uses_stdio), and the result itself with check=False.

What changes:
- A timeout kills qsv's whole process group, so children qsv started
  cannot outlive it or hold its pipes open and hang the worker.
- qsv is asked for structured errors (QSV_ERROR_FORMAT=json); with qsv
  builds that support them, JobError carries the error kind and message.
  Older builds print text, which is used as before.
- The version check reads `qsv --capabilities`, falling back to
  `--version` on older builds.
- `env=` is merged into the worker's environment instead of replacing it
  (subprocess.run's env= dropped PATH and everything else).
- describegpt's api_key goes in QSV_LLM_APIKEY instead of --api-key:
  describegpt copies its command line into the attribution it writes, and
  argv is visible in ps. The "NONE" sentinel, which qsv honors only on the
  flag, stays on the command line.

Any non-zero exit is still a failure, including 255, which qsv-client
itself counts as success.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The old JobError text came from CalledProcessError, which included the
whole argv. Keep that: operators reading a failed job's log need to see
which file and flags were used, not just the subcommand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jqnatividad
jqnatividad merged commit 88c8a54 into main Oct 8, 2026
1 of 2 checks passed
@jqnatividad
jqnatividad deleted the qsv-client-adoption branch October 8, 2026 20:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant