Skip to content

ci: fail a PR that deletes an e2e attestation receipt - #1052

Merged
thekevinscott merged 4 commits into
mainfrom
claude/1043-receipt-deletion-guard
Sep 5, 2026
Merged

thekevinscott merged 4 commits into
mainfrom
claude/1043-receipt-deletion-guard

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Sep 5, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1043

The hole

e2e-verify is diff-based and asks only whether a branch adds or updates its own receipt. A diff that deletes other branches' receipts is invisible to it — #1036 merged exactly that, 28 receipts, fully green with 65 passing checks. Anything that deletes a receipt still slips through: a stale global testing-conventions invoked by hand, a bad git add -A, a botched rebase.

This adds a dirsql-checks attestation-guard check and its own workflow lane that fails a PR whose base...HEAD diff deletes any e2e attestation receipt.

This is a repo-side guard, not a tool-version workaround. It holds at any testing-conventions version: nothing should ever delete a receipt it does not own, and nothing can regenerate a merged branch's.

Design decisions

The rule is total, not ownership-scoped. The issue title says "a receipt it does not own"; the check fails on any receipt deletion. Reasons:

  • Receipts are append-only by convention (agents/reference/e2e-attestation.md) — a merged branch's receipt is the standing record that its suite ran.
  • A branch that re-attests modifies its receipt; it never deletes one. There is no legitimate deletion for ownership to carve out.
  • Slug-keyed ownership would have mis-handled the real case in Split the worker-call progress count by cache hits #1036, where a legitimately-merged child branch's receipt rode in under a different slug and looked foreign. "Own" is not a property the diff reliably carries.

Renames count as deletions. The diff runs with --no-renames, so a receipt moved to another path registers as a deletion of the original. A receipt's path is its branch slug, so renaming one loses the record as surely as removing it.

Escape hatch follows changelog-gate. An allow-receipt-deletion: <reason> line anywhere in any commit body bypasses the gate, scanned over raw %B bodies rather than git's trailer parser (same reason as skip-changelog: — a blank line must not split it off). The reason is mandatory, so a bare word cannot disarm the gate by accident.

Near-miss diagnostics. A bypass that was attempted but misspelled (wrong verb, wrong noun, pluralised, reason omitted) is quoted back with the exact accepted form rather than falling through to a generic failure.

Actionable output. Every deleted receipt gets its own ::error file= annotation, followed by the literal git checkout <base-sha> -- <paths…> that restores them all. Verified below: the printed command was pasted verbatim and turned the check green.

Wiring

Its own workflow, .github/workflows/attestation-guard.yml, filtered on '**/e2e-attestations/**' (plus the check's source and the workflow file).

  • Receipts live under packages/python/, packages/ts/, plugins/dirsql-plugin-embeddings/ and internals/checks/, so no single package lane sees them all.
  • changelog-check.yml explicitly filters e2e-attestations out of its paths, so it was not a candidate.
  • The ** filter is the point: a deletion can ride in any diff, and the receipt path itself is the only reliable trigger.
  • Dispatch verified on this PR — the workflow ran on the RED commit (run 33935055197), so it does not hang CI Gate waiting for a run that never appears.

The run: step is one invocation; all logic (receipt-path detection, bypass scanning, the restore instructions) lives in internals/checks/src/checks/attestation_guard/, split into a pure decide.py / report.py and a thin gate.py composition root, per AGENTS.md "CI Workflows".

Red/green

RED commit c620e58 pushed on its own against inert stubs. CI went red for the right reason:

Red run: https://github.com/thekevinscott/dirsql/actions/runs/33935055548 — 25 unit failures + 1 integration failure, every one of them a new attestation_guard test asserting the guarded behavior:

  • Unit-test coverage (python) — 25 failed, 433 passed
    • decide_test.py::describe_deleted_receipts::{keeps_only_paths_under_an_e2e_attestations_directory, spans_every_package_root_that_holds_receipts, sorts_the_result} — assert [] == ['packages/py...laude-a.json']
    • decide_test.py::describe_has_allow_line::{accepts_the_bypass_with_a_reason, accepts_it_regardless_of_case, accepts_it_from_any_line_not_only_a_formal_trailer} — assert False is True
    • decide_test.py::describe_near_miss_lines::{names_a_bypass_spelled_with_the_wrong_verb, …_wrong_noun, …_missing_its_reason, …_missing_its_colon, strips_surrounding_whitespace} — assert [] == ['skip-receip...ion: cleanup']
    • gate_test.py::describe_run::{fails_when_the_diff_deletes_a_receipt, names_the_restore_command_with_the_base_sha, fails_and_names_a_near_miss_bypass, reads_commit_messages_over_the_same_range} and describe_run::passes_when_a_bypass_line_is_present — assert 0 == 1
    • gate_test.py::describe_verdict::reports_every_deleted_receipt — assert 0 == 1
    • report_test.py::describe_restore_command::names_the_base_sha_and_every_path and 6 describe_report cases — assert '' == 'git checkout...a.json b.json'
  • internals/checks integration tests — 1 failed, 7 passed: tests/integration/attestation_guard_test.py::describe_run_against_a_real_repo::it_fails_when_the_branch_deletes_another_branchs_receipt — AssertionError: assert 0 == 1

GREEN commit 4e22dbe fills the stubs in; CI flips red → green.

f1179a1 adds one further assertion to kill a surviving mutant that unit mutation found: ReplaceBinaryOperator_Mul_Div on gate.py:30 turns the keyword-only * marker into the positional-only /, which every keyword-passing test tolerated. Killed with a new assertion (run("base", "head", <positional>) must raise TypeError), not by weakening anything.

Manual exercise

Against a real clone of main with real receipts, using the packaged dirsql-checks console script.

1. A branch that adds its own receipt and prunes three siblings — fails, names each file:

$ git diff --name-status main...HEAD
D  packages/python/e2e-attestations/claude-819-oneshot-timeout.json
D  packages/ts/e2e-attestations/cc-keen-knuth-f11gyq.json
D  packages/ts/e2e-attestations/claude-819-oneshot-timeout.json
A  packages/ts/e2e-attestations/demo-pruned-siblings.json

$ BASE_SHA=$(git rev-parse main) HEAD_SHA=$(git rev-parse HEAD) dirsql-checks attestation-guard
::error file=packages/python/e2e-attestations/claude-819-oneshot-timeout.json::packages/python/e2e-attestations/claude-819-oneshot-timeout.json is an e2e attestation receipt. Receipts are append-only; this PR deletes it.
::error file=packages/ts/e2e-attestations/cc-keen-knuth-f11gyq.json::… Receipts are append-only; this PR deletes it.
::error file=packages/ts/e2e-attestations/claude-819-oneshot-timeout.json::… Receipts are append-only; this PR deletes it.
::error::Restore 3 deleted receipt(s) with: git checkout 7bfdb1da7a21ac20a1666f4662716f4b17e96dbf -- packages/python/e2e-attestations/claude-819-oneshot-timeout.json packages/ts/e2e-attestations/cc-keen-knuth-f11gyq.json packages/ts/e2e-attestations/claude-819-oneshot-timeout.json
::error::A receipt records that another branch's e2e suite ran, and nothing can regenerate a merged branch's. Restore them and re-attest only your own branch. If the deletion is deliberate (retiring a package), add an 'allow-receipt-deletion: <reason>' line to any commit message.
EXIT=1

2. The printed restore command, pasted verbatim — legitimate diff, passes:

$ git checkout 7bfdb1da7a21ac20a1666f4662716f4b17e96dbf -- packages/python/…/claude-819-oneshot-timeout.json packages/ts/…/cc-keen-knuth-f11gyq.json packages/ts/…/claude-819-oneshot-timeout.json
$ git commit -am "restore pruned receipts"
$ git diff --name-status main...HEAD
A  packages/ts/e2e-attestations/demo-pruned-siblings.json

$ dirsql-checks attestation-guard
No e2e attestation receipts deleted.
EXIT=0

3. A misspelled bypass — named as a near miss, not honoured:

$ git commit -m "retire the ts receipt

skip-receipt-deletion: cleaning up"
$ dirsql-checks attestation-guard
::error file=packages/ts/e2e-attestations/cc-keen-knuth-f11gyq.json::… Receipts are append-only; this PR deletes it.
::error::'skip-receipt-deletion: cleaning up' is not the bypass line. The exact form is 'allow-receipt-deletion: <reason>', reason required.
::error::Restore 1 deleted receipt(s) with: git checkout 7bfdb1da… -- packages/ts/e2e-attestations/cc-keen-knuth-f11gyq.json
EXIT=1

4. The correct bypass — honoured:

$ git commit --amend -m "retire the ts receipt

allow-receipt-deletion: package retired"
$ dirsql-checks attestation-guard
allow-receipt-deletion line present; permitting receipt deletion.
EXIT=0

5. Dogfooded on this branch (base = origin/main, head = f1179a1): No e2e attestation receipts deleted. EXIT=0. Every e2e-attestations entry in this PR's file list is added.

Gates

just preflight --conventions .github/workflows/internals-checks-ci.yml — 0 failing pairs, 0 skipped: uv-sync, declared-deps, colocated-test, one-function-per-file, unit-lint, integration-lint, unit-coverage, mutation (34 mutants, no survivors), e2e-verify. No exemptions added.

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: N/A — no packages/python change
  • TypeScript SDK e2e: N/A — no packages/ts change
  • Rust core e2e (if applicable): N/A — no packages/rust change
  • packages/python/e2e-attestations/<branch>.json receipt written if packages/python changed — N/A
  • packages/ts/e2e-attestations/<branch>.json receipt written if packages/ts changed — N/A
  • Command(s) run: just e2e-attest-internals-checks (cd internals/checks && uvx testing-conventions e2e attest 'uv run python -m pytest tests/e2e -x -q')
  • Result summary: 28 passed; receipt committed at internals/checks/e2e-attestations/claude-1043-receipt-deletion-guard.json in b4f20fb, which git show --stat confirms is a single added file.

Changelog / Migrations

  • Not required. changelog_gate/decide.py scopes to _ROOTS = ("packages", "plugins"); internals/ is outside it, and changelog-check.yml's path filter does not match this diff. Verified, not assumed.
  • Migration fragment: not required — no public API, additive repo tooling.

PARITY.md: unchanged — no SDK public API touched.

Adds the failing unit, integration and e2e tests for a new
`dirsql-checks attestation-guard` check, against inert stubs, plus the
workflow that will invoke it.

The `e2e-verify` gate is diff-based and asks only whether a branch adds or
updates ITS OWN receipt, so a diff that deletes other branches' receipts
passes it green. Receipts are append-only and nothing can regenerate a
merged branch's, so the rule asserted here is total: any deletion under an
`e2e-attestations/` directory fails, with an `allow-receipt-deletion:
<reason>` commit-body bypass mirroring changelog-gate's `skip-changelog:`.

30 tests fail against the stubs; the implementation lands next.
Fills in the stubs the previous commit left inert: receipt-path detection,
the `allow-receipt-deletion: <reason>` bypass and its near-miss diagnostic,
and the failure report naming every deleted receipt plus the exact
`git checkout <base> -- <paths>` that restores them.

`--no-renames` on the diff so a receipt moved to another path counts as a
deletion of the original — a receipt's path is its branch slug, so renaming
one loses the record as surely as removing it.
Kills a surviving `*` -> `/` mutant on the signature: the mutant turns the
positional-only marker on, which every keyword-passing test still tolerates.
@thekevinscott
thekevinscott merged commit e07723e into main Sep 5, 2026
10 checks passed
@thekevinscott
thekevinscott deleted the claude/1043-receipt-deletion-guard branch September 5, 2026 01:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gate: fail a PR that deletes an e2e attestation receipt it does not own

2 participants