fix(store,uploads): close the three preconditions for a strict-ciphertext read (BACKLOG #1169) - #877
Open
wshallwshall wants to merge 3 commits into
Open
fix(store,uploads): close the three preconditions for a strict-ciphertext read (BACKLOG #1169)#877wshallwshall wants to merge 3 commits into
wshallwshall wants to merge 3 commits into
Conversation
added 3 commits
September 4, 2026 18:30
… (BACKLOG #1169) `attachment_chunk` (#149, ADR 0105) shipped on all three backends with a rotation pass but, on Postgres and SQL Server, no ON-OPEN migration pass. SQLite had one. So a keyless-to-keyed transition sealed every other cipher column and left detached-document chunks as plaintext at rest on the two server backends until someone happened to run rotate-key. Adds the missing `_encrypt_existing_rows` pass to both, binding the same cell AAD their read paths use. Batch is 16, not the sibling passes' 500: one row is a whole DETACH_CHUNK_BYTES (1 MiB) slice, so 500 would hold ~650 MiB resident in a single transaction while the store is still opening. Adds a reconciliation guard so the next covered table cannot ship half-added. The existing SQL Server guard checks that a swept table exists; it cannot see the opposite drift, a covered table no sweep names, because an omission has no literal to inspect. The new check reads each backend's own `cell_aad` literals and asserts both sweeps reach every one whose table that backend INSERTs into. The INSERT scoping is load-bearing, not incidental: without it the check falsely accuses both server backends over `shared_body`, which they declare for schema parity and never write. Both mutations were measured red before the fix and green after. The first version of the guard matched raw source text and reported the real Postgres omission as CLEAN, because a comment above the missing call said the word `attachment_chunk`. It now matches only string constants the code executes, with docstrings excluded and comments absent from the AST.
… failure (BACKLOG #1169) The uploaded-file store rides the same cipher as the message store but had no migration or rotation pass of any kind, and `rotate-key` called only `store.reencrypt_to_active()`. So a rotation re-sealed every database cell and left every uploaded file under the retired key. The command's own docstring then tells the operator to drop that key, at which point every upload written before the rotation stops decrypting. Measured: after a rotation the on-disk blob was byte-identical, and dropping the prior key raised CipherError. Adds `UploadStore.reseal_to_active()` and wires it into `rotate-key` on the store's own live cipher instance, so the AES-GCM invocation bound (ASVS 11.3.4) charges to the new key. Same contract as the store's pass: rewrites plaintext or retired-key values, skips values already under the active key, and lets a CipherError propagate before any write. Skipped files are counted and reported on stderr, because a skipped file is still under the prior key and "OK" alone would invite the operator to retire a key that is still load-bearing. Only the ROTATION transition is closed automatically. The store seals legacy plaintext at every keyed open; this pass has one caller, so a first key-enable is closed only when an operator runs the command. Wiring a whole-directory crypto sweep into API startup is unbounded boot-time work and a separate decision, so the docstring names the asymmetry instead of implying parity. Also splits `_scan_metas_sync`'s blanket `except Exception`, which folded a cipher refusal, a damaged file and a malformed sidecar into one identical warning. That is the handler a strict-ciphertext read would raise through, on the surface where planting a file is easiest, so a refusal would have been indistinguishable from a routine post-rotation skip. Each class is now caught and logged separately (CLAUDE.md section 6). The malformed branch logs the exception TYPE only, because a ValueError from coercing a metadata field can echo that field's value. The refusal itself is NOT built: it needs an owner ruling. An unmarked sidecar is still accepted, and this pass launders it into a genuine AAD-bound ciphertext exactly as the store's rotation does. Both are pinned by tests, the second so the builder who ships the refusal converts them rather than meeting them in CI.
…nd false premise Progress note only. The status banner is untouched: #1169 closes by a vault scorecard re-score, and the refusal it researches still needs the re-put owner ruling. Records what was re-measured by execution before any code was written, since every line number in the 2026-08-20 research had moved. The core claim survives verbatim, with both controls firing in the same run. Corrects a SECOND precondition, the item's own this time. `attachment_chunk` was called "unmigrated on all three backends" on the strength of its three INSERT sites, but those answer where it is written, not where it is sealed. SQLite had an on-open pass all along and rotation covered all three; the real gap was one pass on two backends. The `shared_body` correction holds and needed no new evidence: tests/test_phi_at_rest_inventory.py already pinned it, with the same instrument and the same reasoning, before this item was filed. The record contained its own refutation. Names what was measured and deliberately not done, including the ~1,000-line declarative refactor that would retire the new guard but lands in the one code path no CI leg executes.
Collaborator
Author
|
Manager note. I am deliberately NOT applying a What this pull request should be read for, named rather than left to be found:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes nothing. BACKLOG #1169 stays open — it closes by a vault scorecard re-score, and the strict-ciphertext read it researches still needs the re-put owner ruling. This lands only the three preconditions the item says must close first, or the refusal fires on legitimate data. The status banner is untouched.
Re-measured first, and two reported premises were wrong
Every line number in the 2026-08-20 research had moved. Measured against
a2eef0f37, with a positive control in each run.The core claim survives verbatim.
crypto.py:756-757returned a forged plaintext unchanged from a keyed cipher, while both controls fired — a mutated ciphertext and a wrong-cell AAD each raisedCipherError. On the uploads surface a hand-written plaintext sidecar was accepted by a KEYED store and returned filename, uploader and uploader id all attacker-chosen; a relocated SEALED sidecar raised, which is the control separating a real passthrough from a broken test.The
shared_bodyprecondition is false, as briefed — and it needed no new evidence.INSERT INTO shared_bodyoccurs exactly once tree-wide, atstore/store.py:3640, the SQLite backend. Positive control in the same run: the bare token appears 55, 10 and 7 times across the three backend files. Buttests/test_phi_at_rest_inventory.pyalready pinned this, with the sameINSERT INTOinstrument and the same SQLite-only reasoning, before the item was filed. The record contained its own refutation. I did not restate it (SDS-3.5).A SECOND premise is false, and it is the item's own.
attachment_chunkwas called "unmigrated on all three backends" on the strength of its threeINSERT INTOsites. Those answer where it is written, not where it is sealed — an SDS-3.8 instrument mismatch.store.py(SQLite):2765)postgres.pysqlserver.pySo the real gap was one pass on two backends, not three.
What this closes
attachment_chunkon-open parity on Postgres and SQL Server, binding the same cell AAD their read paths use. Batch is 16, not the sibling passes' 500: one row is a whole 1 MiBDETACH_CHUNK_BYTESslice, so 500 would hold ~650 MiB resident in a single transaction while the store is still opening.The uploaded-file store's missing pass — worse than reported, and a standing data-loss defect rather than only a blocker. It had no migration pass and no rotation pass. Measured: after a rotation the on-disk blob was byte-identical, and once the prior key was dropped — the step
rotate-key's own docstring tells the operator to take — the upload raisedCipherErrorpermanently.UploadStore.reseal_to_active()now closes both transitions androtate-keycalls it on the store's live cipher instance (ASVS 11.3.4). Skipped files are counted and warned about on stderr, because a skipped file is still under the prior key.One asymmetry is deliberate and named in the docstring rather than papered over: the store seals legacy plaintext automatically at every keyed open, whereas this pass has one caller, so a first key-enable is closed only when an operator runs the command. A whole-directory crypto sweep at API startup is unbounded boot-time work and a separate decision.
The blanket swallow at
_scan_metas_sync. Measured: three unrelated causes produced three identical warnings. Each class is now caught and logged separately. The malformed branch logs the exception TYPE only, because aValueErrorfrom coercing a metadata field can echo that field's value.Plus a mechanical reconciliation (
tests/test_store_cipher_sweep_parity.py) so a future covered table cannot ship half-added: everycell_aadcell whose table a backend INSERTs into must be reached by BOTH sweeps. TheINSERT INTOscoping is load-bearing — without it the check falsely accuses both server backends overshared_body.The guard nearly shipped broken, and that is recorded
The first version matched raw source text and passed the real Postgres omission, because a comment above the missing call contained the word
attachment_chunk. Prose satisfied a check about code. It now matches only string constants the code executes, with docstrings excluded and comments absent from the AST. Both backend mutations were measured red before the fix and green after.Not built, deliberately
The refusal, pending the ruling.
test_a_planted_plaintext_sidecar_is_still_acceptedpins the standing behaviour as a TRIGGER — it fails when the refusal ships, which tells that builder to convert it rather than meet it in CI. It also asserts that the new reseal pass launders an unmarked value into a genuine AAD-bound ciphertext, exactly as the store's rotation does. That is why the pass is a precondition for a refusal and not a substitute for one.Also measured and left: each backend still hand-maintains its own composite-pass list (~26 sites), and SQL Server has no
_CIPHER_COLUMNSat all, sopostgres.py:856's "mirrorsMessageStore._CIPHER_COLUMNS" is untrue of one backend. Driving all passes off one declaration would retire this guard and ~1,300 lines of sibling AST scaffolding, but it is ~1,000 lines in the one path no CI leg executes — every SQL Server and Postgres leg is keyless and returns before the sweep body. That is a separate item, not a bounded change.Checks
Run locally and green:
ruff checkandruff format --checkon the whole tree,mypy messagefoundrystrict (267 files), andtest_uploads_reseal/test_store_cipher_sweep_parity/test_uploads/test_uploads_cross_process_quota/test_sqlserver_encrypt_pass_tables/test_tooling_partition/test_phi_at_rest_inventory(99 passed), plustest_cli/test_store_encryption/test_store_aad_binding/test_attachment_substrate/test_asvs_gcm_invocation_bound(231 passed).Not run, and these legs must be read on the runner. No full suite — the box is contended. Critically, the SQL Server and Postgres legs: the new on-open passes live behind
if not self._cipher.encrypts: return, and every SS and PG CI step is keyless, so no leg executes the new SQL at all. The source-reading guard is the only mechanism that can see it, which is why it exists. Neither new test file needs atests/tooling_manifest.txtentry (both import the engine); confirmed by runningtest_tooling_partition.