From 7f36fc46dc5c7818d78d9a770e600ef14b6dffdd Mon Sep 17 00:00:00 2001 From: Joel Natividad <1980690+jqnatividad@users.noreply.github.com> Date: Fri, 9 Oct 2026 13:36:40 -0400 Subject: [PATCH] build(deps): require qsv-client>=0.2.0 qsv-client 0.2.0 bounds a timeout even when a process qsv started (e.g. one spawned by describegpt) leaves the process group and keeps the output pipes open; with >=0.1.0, an environment that already has 0.1.0 installed keeps it on `pip install -r requirements.txt`. No DP+ code changes are needed: QSVCommand only uses Qsv(binary=, timeout=), run(check=False, text=) and error_from_run, all unchanged. test_a_failed_search_raises_job_error patched pathlib.Path.absolute process-wide to point the PII regex file at a missing path. qsv-client 0.2.0 resolves the qsv binary with Path.absolute(), so the patch also turned the binary into the missing regex file (FileNotFoundError instead of JobError). It now moves pii_screening.__file__ into tmp_path instead, which makes only the default regex path missing. Passes on 0.1.0 and 0.2.0. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + requirements.txt | 2 +- tests/test_pii_screening_search.py | 9 ++++++--- 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d896353..c1d1f72 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 * New `ckanext.datapusher_plus.prefect_enabled` setting (default `true`). Setting it to `false` turns Prefect off entirely: `datapusher_submit` enqueues the job on CKAN's own background-job queue and the new `ckanext/datapusher_plus/jobs/local_runner.py` executes the same nine ingestion stages in-process, so no Prefect server, worker, work pool — or `import prefect` — is involved. Operators run `ckan jobs worker` instead. This is the fix for deployments where importing Prefect *itself* fails, e.g. a CKAN process running with `HOME=/root` but no write access there, which surfaces as `Error submitting job to DataPusher: [Errno 13] Permission denied: '/root/.prefect/profiles.toml'` (the alternative fix, if you want to keep Prefect, is to point `PREFECT_HOME` at a writable directory). The local runner keeps the `Jobs`/`Logs` tables, the `datapusher_hook` callbacks, the complete-with-skip contract, the `flow_timeout` deadline, and the post-database-failure datastore cleanup; it does *not* provide per-stage retries, result caching, the Prefect run graph/artifacts/events, or human-in-the-loop PII review — with Prefect off, a job crossing `pii_review_threshold` aborts before any datastore write rather than suspending for approval. The task_status row records the RQ job id as `rq_job_id` (not `flow_run_id`), so nothing builds a Prefect-UI deep link out of it. See [Running without Prefect](README.md#running-without-prefect). ### Changed +* Require `qsv-client>=0.2.0` (was `>=0.1.0`), so existing environments pick up its fixes on the next `pip install -r requirements.txt`. The one that matters for DP+: a timeout is bounded again even when a process qsv started — e.g. one spawned by `describegpt` in the AI suggestions stage — leaves the process group and keeps the output pipes open; previously the job waited for that process to exit. No DP+ code changes are needed. ([qsv-client 0.2.0 release notes](https://github.com/dathere/qsv-client/releases/tag/0.2.0)) * The Prefect-free half of the pipeline moved out of `jobs/prefect_flow.py` into the new `jobs/pipeline_core.py` (the CKAN status callback, input validation, `RuntimeContext` construction, the stage invoker and its `StageAbort` signal, and the datastore rollback body), shared verbatim by the Prefect flow and the local runner. `prefect_flow` keeps its historical private names as aliases, so custom flows composed from its `@task` primitives are unaffected. Tests that patched `prefect_flow.dsu` / `.QSVCommand` / `.Path` now patch `pipeline_core`. * Auto-indexing now uses a closed cardinality range `[auto_index_min_threshold, auto_index_threshold]` instead of a single upper bound (issue #142). Defaults bumped: `auto_index_threshold` `3` → `10` and the new `auto_index_min_threshold` defaults to `3`. The new lower bound skips the useless-single-value-column case @EricSoroos flagged in #142 (a 1-value text column previously produced a 10–40MB B-tree the Postgres planner would never choose). The upper bump from 3 → 10 widens the DataTables-SearchBuilder filtering sweet spot to cover typical enum-shaped columns (status = 3–10 values, common enums under ~10). **Operator-facing impact at default settings:** columns with cardinality 1–2 lose their auto-index (intentional — the indexes were dead weight); columns with cardinality 4–10 gain an auto-index. No migration is required; the change re-applies on the next resubmit. To restore pre-#142 behavior (no lower floor) set `ckanext.datapusher_plus.auto_index_min_threshold = 0`. The setting `auto_index_threshold = -1` ("index every column") still works but now hits the `min_threshold = 3` floor unless paired with `auto_index_min_threshold = 0`. * **BREAKING** Bumped `MINIMUM_QSV_VERSION` from `4.0.0` to `20.0.0`. Operators must upgrade their `qsv` binary at the path configured by `ckanext.datapusher_plus.qsv_bin` before deploying this version — DP+ will refuse to start otherwise. See [qsv 20.0.0 release notes](https://github.com/dathere/qsv/releases/tag/20.0.0) and the migration notes below. ([README install snippet](README.md#option-2-install-prebuilt-qsv-binaries) updated accordingly.) diff --git a/requirements.txt b/requirements.txt index 1b1354c..9fd4f2c 100644 --- a/requirements.txt +++ b/requirements.txt @@ -32,4 +32,4 @@ babel>=2.9 # qsv-client: runs qsv for QSVCommand. Kills the whole process group on # timeout (qsv and anything it started) and reads qsv's structured errors # (QSV_ERROR_FORMAT=json) where the binary supports them. No dependencies. -qsv-client>=0.1.0 +qsv-client>=0.2.0 diff --git a/tests/test_pii_screening_search.py b/tests/test_pii_screening_search.py index 2975949..94a192f 100644 --- a/tests/test_pii_screening_search.py +++ b/tests/test_pii_screening_search.py @@ -106,9 +106,12 @@ def test_full_screen_aborts_on_pii_with_the_counts(screen): def test_a_failed_search_raises_job_error(screen, quick, message, tmp_path): from ckanext.datapusher_plus import pii_screening, utils - # a regex file qsv cannot read makes searchset itself fail - missing = tmp_path / "missing-regexes.txt" - with mock.patch.object(pii_screening.Path, "absolute", return_value=missing), \ + # a regex file qsv cannot read makes searchset itself fail. Moving the module's + # __file__ points the default regex path (Path(__file__).with_name(...)) at a + # missing file; patching pathlib.Path itself would also hijack qsv-client, + # which resolves the qsv binary with Path.absolute(). + assert not (tmp_path / "default-pii-regexes.txt").exists() + with mock.patch.object(pii_screening, "__file__", str(tmp_path / "pii_screening.py")), \ pytest.raises(utils.JobError, match=message): screen(CLEAN, quick)