Skip to content

fix(store,uploads): close the three preconditions for a strict-ciphertext read (BACKLOG #1169) - #877

Open
wshallwshall wants to merge 3 commits into
mainfrom
claude/builder-1169-strict-read-preconditions
Open

fix(store,uploads): close the three preconditions for a strict-ciphertext read (BACKLOG #1169)#877
wshallwshall wants to merge 3 commits into
mainfrom
claude/builder-1169-strict-read-preconditions

Conversation

@wshallwshall

Copy link
Copy Markdown
Collaborator

Closes nothing. BACKLOG #1169 stays open — it closes by a vault scorecard re-score, and the strict-ciphertext read it researches still needs the re-put owner ruling. This lands only the three preconditions the item says must close first, or the refusal fires on legitimate data. The status banner is untouched.

Re-measured first, and two reported premises were wrong

Every line number in the 2026-08-20 research had moved. Measured against a2eef0f37, with a positive control in each run.

The core claim survives verbatim. crypto.py:756-757 returned a forged plaintext unchanged from a keyed cipher, while both controls fired — a mutated ciphertext and a wrong-cell AAD each raised CipherError. On the uploads surface a hand-written plaintext sidecar was accepted by a KEYED store and returned filename, uploader and uploader id all attacker-chosen; a relocated SEALED sidecar raised, which is the control separating a real passthrough from a broken test.

The shared_body precondition is false, as briefed — and it needed no new evidence. INSERT INTO shared_body occurs exactly once tree-wide, at store/store.py:3640, the SQLite backend. Positive control in the same run: the bare token appears 55, 10 and 7 times across the three backend files. But tests/test_phi_at_rest_inventory.py already pinned this, with the same INSERT INTO instrument and the same SQLite-only reasoning, before the item was filed. The record contained its own refutation. I did not restate it (SDS-3.5).

A SECOND premise is false, and it is the item's own. attachment_chunk was called "unmigrated on all three backends" on the strength of its three INSERT INTO sites. Those answer where it is written, not where it is sealed — an SDS-3.8 instrument mismatch.

on-open migration rotation
store.py (SQLite) present all along (:2765) present
postgres.py missing present
sqlserver.py missing present

So the real gap was one pass on two backends, not three.

What this closes

  1. attachment_chunk on-open parity on Postgres and SQL Server, binding the same cell AAD their read paths use. Batch is 16, not the sibling passes' 500: one row is a whole 1 MiB DETACH_CHUNK_BYTES slice, so 500 would hold ~650 MiB resident in a single transaction while the store is still opening.

  2. The uploaded-file store's missing pass — worse than reported, and a standing data-loss defect rather than only a blocker. It had no migration pass and no rotation pass. Measured: after a rotation the on-disk blob was byte-identical, and once the prior key was dropped — the step rotate-key's own docstring tells the operator to take — the upload raised CipherError permanently. UploadStore.reseal_to_active() now closes both transitions and rotate-key calls it on the store's live cipher instance (ASVS 11.3.4). Skipped files are counted and warned about on stderr, because a skipped file is still under the prior key.

    One asymmetry is deliberate and named in the docstring rather than papered over: the store seals legacy plaintext automatically at every keyed open, whereas this pass has one caller, so a first key-enable is closed only when an operator runs the command. A whole-directory crypto sweep at API startup is unbounded boot-time work and a separate decision.

  3. The blanket swallow at _scan_metas_sync. Measured: three unrelated causes produced three identical warnings. Each class is now caught and logged separately. The malformed branch logs the exception TYPE only, because a ValueError from coercing a metadata field can echo that field's value.

Plus a mechanical reconciliation (tests/test_store_cipher_sweep_parity.py) so a future covered table cannot ship half-added: every cell_aad cell whose table a backend INSERTs into must be reached by BOTH sweeps. The INSERT INTO scoping is load-bearing — without it the check falsely accuses both server backends over shared_body.

The guard nearly shipped broken, and that is recorded

The first version matched raw source text and passed the real Postgres omission, because a comment above the missing call contained the word attachment_chunk. Prose satisfied a check about code. It now matches only string constants the code executes, with docstrings excluded and comments absent from the AST. Both backend mutations were measured red before the fix and green after.

Not built, deliberately

The refusal, pending the ruling. test_a_planted_plaintext_sidecar_is_still_accepted pins the standing behaviour as a TRIGGER — it fails when the refusal ships, which tells that builder to convert it rather than meet it in CI. It also asserts that the new reseal pass launders an unmarked value into a genuine AAD-bound ciphertext, exactly as the store's rotation does. That is why the pass is a precondition for a refusal and not a substitute for one.

Also measured and left: each backend still hand-maintains its own composite-pass list (~26 sites), and SQL Server has no _CIPHER_COLUMNS at all, so postgres.py:856's "mirrors MessageStore._CIPHER_COLUMNS" is untrue of one backend. Driving all passes off one declaration would retire this guard and ~1,300 lines of sibling AST scaffolding, but it is ~1,000 lines in the one path no CI leg executes — every SQL Server and Postgres leg is keyless and returns before the sweep body. That is a separate item, not a bounded change.

Checks

Run locally and green: ruff check and ruff format --check on the whole tree, mypy messagefoundry strict (267 files), and test_uploads_reseal / test_store_cipher_sweep_parity / test_uploads / test_uploads_cross_process_quota / test_sqlserver_encrypt_pass_tables / test_tooling_partition / test_phi_at_rest_inventory (99 passed), plus test_cli / test_store_encryption / test_store_aad_binding / test_attachment_substrate / test_asvs_gcm_invocation_bound (231 passed).

Not run, and these legs must be read on the runner. No full suite — the box is contended. Critically, the SQL Server and Postgres legs: the new on-open passes live behind if not self._cipher.encrypts: return, and every SS and PG CI step is keyless, so no leg executes the new SQL at all. The source-reading guard is the only mechanism that can see it, which is why it exists. Neither new test file needs a tests/tooling_manifest.txt entry (both import the engine); confirmed by running test_tooling_partition.

wshallwshall added 3 commits September 4, 2026 18:30
… (BACKLOG #1169)

`attachment_chunk` (#149, ADR 0105) shipped on all three backends with a
rotation pass but, on Postgres and SQL Server, no ON-OPEN migration pass.
SQLite had one. So a keyless-to-keyed transition sealed every other cipher
column and left detached-document chunks as plaintext at rest on the two
server backends until someone happened to run rotate-key.

Adds the missing `_encrypt_existing_rows` pass to both, binding the same
cell AAD their read paths use. Batch is 16, not the sibling passes' 500:
one row is a whole DETACH_CHUNK_BYTES (1 MiB) slice, so 500 would hold
~650 MiB resident in a single transaction while the store is still opening.

Adds a reconciliation guard so the next covered table cannot ship half-added.
The existing SQL Server guard checks that a swept table exists; it cannot see
the opposite drift, a covered table no sweep names, because an omission has no
literal to inspect. The new check reads each backend's own `cell_aad` literals
and asserts both sweeps reach every one whose table that backend INSERTs into.

The INSERT scoping is load-bearing, not incidental: without it the check
falsely accuses both server backends over `shared_body`, which they declare
for schema parity and never write. Both mutations were measured red before
the fix and green after.

The first version of the guard matched raw source text and reported the real
Postgres omission as CLEAN, because a comment above the missing call said the
word `attachment_chunk`. It now matches only string constants the code
executes, with docstrings excluded and comments absent from the AST.
… failure (BACKLOG #1169)

The uploaded-file store rides the same cipher as the message store but had no
migration or rotation pass of any kind, and `rotate-key` called only
`store.reencrypt_to_active()`. So a rotation re-sealed every database cell and
left every uploaded file under the retired key. The command's own docstring
then tells the operator to drop that key, at which point every upload written
before the rotation stops decrypting. Measured: after a rotation the on-disk
blob was byte-identical, and dropping the prior key raised CipherError.

Adds `UploadStore.reseal_to_active()` and wires it into `rotate-key` on the
store's own live cipher instance, so the AES-GCM invocation bound (ASVS 11.3.4)
charges to the new key. Same contract as the store's pass: rewrites plaintext
or retired-key values, skips values already under the active key, and lets a
CipherError propagate before any write. Skipped files are counted and reported
on stderr, because a skipped file is still under the prior key and "OK" alone
would invite the operator to retire a key that is still load-bearing.

Only the ROTATION transition is closed automatically. The store seals legacy
plaintext at every keyed open; this pass has one caller, so a first key-enable
is closed only when an operator runs the command. Wiring a whole-directory
crypto sweep into API startup is unbounded boot-time work and a separate
decision, so the docstring names the asymmetry instead of implying parity.

Also splits `_scan_metas_sync`'s blanket `except Exception`, which folded a
cipher refusal, a damaged file and a malformed sidecar into one identical
warning. That is the handler a strict-ciphertext read would raise through, on
the surface where planting a file is easiest, so a refusal would have been
indistinguishable from a routine post-rotation skip. Each class is now caught
and logged separately (CLAUDE.md section 6). The malformed branch logs the
exception TYPE only, because a ValueError from coercing a metadata field can
echo that field's value.

The refusal itself is NOT built: it needs an owner ruling. An unmarked sidecar
is still accepted, and this pass launders it into a genuine AAD-bound
ciphertext exactly as the store's rotation does. Both are pinned by tests, the
second so the builder who ships the refusal converts them rather than meeting
them in CI.
…nd false premise

Progress note only. The status banner is untouched: #1169 closes by a vault
scorecard re-score, and the refusal it researches still needs the re-put owner
ruling.

Records what was re-measured by execution before any code was written, since
every line number in the 2026-08-20 research had moved. The core claim survives
verbatim, with both controls firing in the same run.

Corrects a SECOND precondition, the item's own this time. `attachment_chunk`
was called "unmigrated on all three backends" on the strength of its three
INSERT sites, but those answer where it is written, not where it is sealed.
SQLite had an on-open pass all along and rotation covered all three; the real
gap was one pass on two backends.

The `shared_body` correction holds and needed no new evidence:
tests/test_phi_at_rest_inventory.py already pinned it, with the same instrument
and the same reasoning, before this item was filed. The record contained its
own refutation.

Names what was measured and deliberately not done, including the ~1,000-line
declarative refactor that would retire the new guard but lands in the one code
path no CI leg executes.
@wshallwshall

Copy link
Copy Markdown
Collaborator Author

Manager note. I am deliberately NOT applying a reviewed label here, and the reason is a change that landed while this was being built: the owner retired the reviewer requirement, and a reviewer has read this was removed from branch protection at roughly 23:31Z. Verified by printing the full required set rather than grepping for a zero - 13 contexts, reviewer absent. So the label would clear nothing and would only assert a review that did not happen.

What this pull request should be read for, named rather than left to be found:

  • It is BEHIND main.
  • The SQL Server and Postgres legs cannot execute the new SQL, because every such leg is keyless. The source-reading guard is the only mechanism covering it, so read that guard rather than assuming a green database leg means anything here.
  • The full suite was not run on a contended box.
  • The strict refusal is deliberately NOT built - it needs an owner ruling, and it would fire on legitimate unmigrated data until these preconditions land.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant