Repository navigation
fix(pii): make PII screening work with its default regexes - #343
Merged
Merged
Conversation
… by absolute path The default regex file was moved to the repo root in 2023, but the screening code looks for it next to pii_screening.py, so quick screening always failed with "No such file or directory". Full screening passed the bare file name instead, which only resolved when the worker's current directory happened to be the repo root, and with a custom regex resource it passed the URL's basename rather than the downloaded file. Move the file into ckanext/datapusher_plus/, include it in the package data, and have both modes search the same absolute path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e job `qsv searchset --quick` exits 1 when nothing matches, and the quick screen treated that like any failure, so with pii_quick_screen enabled every file without PII failed with "Cannot quickly search CSV for PII". Run it with check=False and read the exit code: 0 is a match (its row number is on stderr); 1 with no error message (or, with QSV_ERROR_FORMAT=json, a `no_match` error) is no PII; anything else is still an error. `--not-one` is no help: with it, qsv's sequential search prints the row count to stderr when nothing matched, which reads as a match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seven JobError calls in pii_screening.py passed logging-style arguments
("... %s", e), but JobError takes a single message, so each of those
paths raised TypeError instead: a failed search, the "PII candidate
found, job aborted" check in both modes, and the PII preview's stats,
COPY and connection errors. Format the messages with f-strings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs both modes from a directory that isn't the repo root, covering a file without PII, aborting on PII (quick: the row; full: the counts), a failed search raising JobError, and the no-match detection, including the structured `no_match` error qsv-client will request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When ckanext.datapusher_plus.pii_regex_resource_id_or_alias named a resource that isn't in the DataStore, the regex path was never set and screen_for_pii crashed with UnboundLocalError. Raise a JobError that names the resource and the setting instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This started as the quick-screen bug noted in #342: with
pii_quick_screenon, a file with no PII failed the job. While fixing it, I found that PII screening with the default regex set doesn't work in either mode. All the bugs below are confirmed against qsvdp 20.1.0 in thedpp-testcontainer. Each fix is its own commit.The default regex file wasn't where the code looks for it. In 2023,
default-pii-regexes.txtmoved to the repo root (2f204cf), butscreen_for_piiresolves it next topii_screening.py. Quick mode therefore always failed withio error: No such file or directory. Full mode passed the bare file name, which only resolves when the worker's current directory is the repo root. With a custom regex resource, full mode passed the URL's basename rather than the file it had just downloaded.Fix: move the file into
ckanext/datapusher_plus/, add it toMANIFEST.in(I checked a built wheel includes it), and have both modes search the same absolute path. The README link is updated to match.The quick-screen no-match bug.
qsv searchset --quickexits 1 when nothing matches, and the quick screen treated that like any error.Fix: run it with
check=False(a newcheckparameter onQSVCommand.searchset) and read the exit code:no_matcherror, which is what qsv returns once feat(qsv): run qsv through qsv-client #342 asks forQSV_ERROR_FORMAT=json.--not-onedoesn't work here: with it, qsv's sequential search prints the row count to stderr when nothing matched, which reads as a match. That's a qsv bug, fixed in fix(search, searchset): --quick reports no row when nothing matched qsv#4767; the indexed path doesn't do it.Errors raised
TypeErrorinstead ofJobError. SevenJobError("… %s", e)calls passed logging-style arguments, butJobErrortakes a single message. That includes the "PII candidate found, job aborted" check in both modes. They're now f-strings.A missing custom regex resource crashed with
UnboundLocalError. Ifckanext.datapusher_plus.pii_regex_resource_id_or_aliasnames a resource that isn't in the DataStore, the regex path was never set.Fix: raise a
JobErrorthat names the resource and the setting, and says to unset it to use the default regexes. The existence check itself is unchanged: it asks whether the resource is in the DataStore, though the code then downloads the resource's URL.Tests: the new
tests/test_pii_screening_search.pyrunsscreen_for_piiagainst the real binary in both modes, from a temp directory as a worker would. It covers:JobError;JobError;I mutation-checked five mutations, and each made these tests fail: a relative regex path in full mode,
check=Truefor the quick search, any exit 1 counting as no match, and the printf-styleJobErrorin both the abort and the search-failure paths. With the missing-resource check removed, its test fails with the originalUnboundLocalError. The unit suite passes indpp-test(324 passed).This doesn't conflict with #342. With #342 merged,
_run_command(check=False)still returns theCompletedProcess, and its structuredno_matchis handled.🤖 Generated with Claude Code