Skip to content

fix(pii): make PII screening work with its default regexes - #343

Merged
jqnatividad merged 5 commits into
mainfrom
fix-pii-quick-screen-no-match
Oct 8, 2026
Merged

jqnatividad merged 5 commits into
mainfrom
fix-pii-quick-screen-no-match

Conversation

@jqnatividad

@jqnatividad jqnatividad commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

This started as the quick-screen bug noted in #342: with pii_quick_screen on, a file with no PII failed the job. While fixing it, I found that PII screening with the default regex set doesn't work in either mode. All the bugs below are confirmed against qsvdp 20.1.0 in the dpp-test container. Each fix is its own commit.

  1. The default regex file wasn't where the code looks for it. In 2023, default-pii-regexes.txt moved to the repo root (2f204cf), but screen_for_pii resolves it next to pii_screening.py. Quick mode therefore always failed with io error: No such file or directory. Full mode passed the bare file name, which only resolves when the worker's current directory is the repo root. With a custom regex resource, full mode passed the URL's basename rather than the file it had just downloaded.
    Fix: move the file into ckanext/datapusher_plus/, add it to MANIFEST.in (I checked a built wheel includes it), and have both modes search the same absolute path. The README link is updated to match.

  2. The quick-screen no-match bug. qsv searchset --quick exits 1 when nothing matches, and the quick screen treated that like any error.
    Fix: run it with check=False (a new check parameter on QSVCommand.searchset) and read the exit code:

    • 0 means a match, with its row on stderr;
    • 1 with no error message means no PII. So does a structured no_match error, which is what qsv returns once feat(qsv): run qsv through qsv-client #342 asks for QSV_ERROR_FORMAT=json.
    • Anything else is still an error.

    --not-one doesn't work here: with it, qsv's sequential search prints the row count to stderr when nothing matched, which reads as a match. That's a qsv bug, fixed in fix(search, searchset): --quick reports no row when nothing matched qsv#4767; the indexed path doesn't do it.

  3. Errors raised TypeError instead of JobError. Seven JobError("… %s", e) calls passed logging-style arguments, but JobError takes a single message. That includes the "PII candidate found, job aborted" check in both modes. They're now f-strings.

  4. A missing custom regex resource crashed with UnboundLocalError. If ckanext.datapusher_plus.pii_regex_resource_id_or_alias names a resource that isn't in the DataStore, the regex path was never set.
    Fix: raise a JobError that names the resource and the setting, and says to unset it to use the default regexes. The existence check itself is unchanged: it asks whether the resource is in the DataStore, though the code then downloads the resource's URL.

Tests: the new tests/test_pii_screening_search.py runs screen_for_pii against the real binary in both modes, from a temp directory as a worker would. It covers:

  • a file without PII;
  • aborting on PII (quick mode reports the row, full mode the counts);
  • a failed search raising JobError;
  • a missing custom regex resource raising a clear JobError;
  • table-driven no-match detection.

I mutation-checked five mutations, and each made these tests fail: a relative regex path in full mode, check=True for the quick search, any exit 1 counting as no match, and the printf-style JobError in both the abort and the search-failure paths. With the missing-resource check removed, its test fails with the original UnboundLocalError. The unit suite passes in dpp-test (324 passed).

This doesn't conflict with #342. With #342 merged, _run_command(check=False) still returns the CompletedProcess, and its structured no_match is handled.

🤖 Generated with Claude Code

jqnatividad and others added 4 commits October 8, 2026 15:49
… by absolute path

The default regex file was moved to the repo root in 2023, but the
screening code looks for it next to pii_screening.py, so quick screening
always failed with "No such file or directory". Full screening passed
the bare file name instead, which only resolved when the worker's
current directory happened to be the repo root, and with a custom regex
resource it passed the URL's basename rather than the downloaded file.

Move the file into ckanext/datapusher_plus/, include it in the package
data, and have both modes search the same absolute path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e job

`qsv searchset --quick` exits 1 when nothing matches, and the quick
screen treated that like any failure, so with pii_quick_screen enabled
every file without PII failed with "Cannot quickly search CSV for PII".

Run it with check=False and read the exit code: 0 is a match (its row
number is on stderr); 1 with no error message (or, with
QSV_ERROR_FORMAT=json, a `no_match` error) is no PII; anything else is
still an error. `--not-one` is no help: with it, qsv's sequential search
prints the row count to stderr when nothing matched, which reads as a
match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seven JobError calls in pii_screening.py passed logging-style arguments
("... %s", e), but JobError takes a single message, so each of those
paths raised TypeError instead: a failed search, the "PII candidate
found, job aborted" check in both modes, and the PII preview's stats,
COPY and connection errors. Format the messages with f-strings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs both modes from a directory that isn't the repo root, covering a
file without PII, aborting on PII (quick: the row; full: the counts), a
failed search raising JobError, and the no-match detection, including
the structured `no_match` error qsv-client will request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When ckanext.datapusher_plus.pii_regex_resource_id_or_alias named a
resource that isn't in the DataStore, the regex path was never set and
screen_for_pii crashed with UnboundLocalError. Raise a JobError that
names the resource and the setting instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jqnatividad
jqnatividad merged commit f43f525 into main Oct 8, 2026
1 of 2 checks passed
@jqnatividad
jqnatividad deleted the fix-pii-quick-screen-no-match branch October 8, 2026 20:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant