Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
eb0c346
feat(formal-verification): add the plugin skeleton, settings and tool…
kyle-sexton Nov 15, 2026
aaff5bc
feat(formal-verification): add the formal-area skill contract
kyle-sexton Nov 15, 2026
647187a
feat(formal-verification): add the chart skill, TLA+ with TLC
kyle-sexton Nov 15, 2026
2cb5436
feat(formal-verification): add the setup skill
kyle-sexton Nov 15, 2026
a24d8d6
feat(formal-verification): draft the claim-protocol model as formal-c…
kyle-sexton Nov 15, 2026
4446aac
docs(upstream): record the formal-verification adoption at both upstr…
kyle-sexton Nov 15, 2026
29460f2
test(formal-verification): keep the chart corpus out of markdown and …
kyle-sexton Nov 15, 2026
43fba2c
test(formal-verification): add eval cases for setup check and an unat…
kyle-sexton Nov 15, 2026
d851e21
fix(formal-verification): guard the pid a chart test kills
kyle-sexton Nov 15, 2026
86427e4
fix(formal-verification): accept default labels and keep region condi…
kyle-sexton Nov 15, 2026
d6a259c
fix(formal-verification): take download sources only from the shipped…
kyle-sexton Nov 15, 2026
146c03d
fix(formal-verification): stage the spec and bound each Java run
kyle-sexton Nov 15, 2026
b369b06
fix(formal-verification): fence and escape spec text in the report, d…
kyle-sexton Nov 15, 2026
93f6a07
docs(formal-verification): state that a view file is code and report …
kyle-sexton Nov 15, 2026
db59e0f
test(formal-verification): keep fixture rewrites out of the chart tes…
kyle-sexton Nov 15, 2026
e6323d4
test(formal-verification): use US spelling in a guard test name
kyle-sexton Nov 15, 2026
18e4771
fix(formal-verification): escape only spec text in report sentences
kyle-sexton Nov 15, 2026
9c17f9a
Merge commit 'e1491a4badcd2ebe063a306ee201d5a6933e5b0e' into feat/for…
kyle-sexton Nov 15, 2026
0e24d61
feat(formal-verification): regenerate chart corpus, record the securi…
kyle-sexton Nov 16, 2026
327bb34
refactor(formal-verification): name the fake-java variable prefix
kyle-sexton Nov 16, 2026
c644a99
Merge remote-tracking branch 'origin/main' into feat/formal-verificat…
kyle-sexton Nov 16, 2026
d662a24
test(formal-verification): expect the output-folder refusal the chart…
kyle-sexton Nov 16, 2026
f55ed11
feat(formal-verification): model the claim protocol with renew, relea…
kyle-sexton Nov 16, 2026
c1d472e
style(formal-verification): drop a comment the view file does not need
kyle-sexton Nov 16, 2026
67bce52
Merge remote-tracking branch 'origin/main' into feat/formal-verificat…
kyle-sexton Nov 16, 2026
0d2ced5
fix(formal-verification): declare the TLC jar as a prerequisite
kyle-sexton Nov 16, 2026
cad1e32
Merge remote-tracking branch 'origin/main' into feat/formal-verificat…
kyle-sexton Nov 16, 2026
57380c2
chore(formal-verification): take main's shared prerequisites copies
kyle-sexton Nov 16, 2026
405c35c
Merge remote-tracking branch 'origin/main' into feat/formal-verificat…
kyle-sexton Nov 16, 2026
bf5c619
feat(formal-verification): model the claim protocol at a548b3aaab8c w…
kyle-sexton Nov 16, 2026
e852414
fix(formal-verification): address the review round and two CI gates
kyle-sexton Nov 16, 2026
804cac9
Merge remote-tracking branch 'origin/main' into feat/formal-verificat…
kyle-sexton Nov 16, 2026
e28be56
ci: exclude the chart corpus graph files from the editorconfig check
kyle-sexton Nov 16, 2026
0af0d1a
Merge remote-tracking branch 'origin/main' into feat/formal-verificat…
kyle-sexton Nov 16, 2026
9eb40bc
fix(formal-verification): close the gaps the done-when check of #1564…
kyle-sexton Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
bump: minor
---

### Added

- The `formal-verification` plugin, off by default.
- `/formal-verification:chart` model-checks a TLA+ spec with TLC and writes a Mermaid chart of its states and a report with the shortest path to each broken rule. Its scripts take the jar, Java and bounds as arguments and install nothing.
- `/formal-verification:setup` checks the script runtime, Java, the model checker and the renderer, and on a typed `apply install-model-checker` or `apply install-renderer` downloads the pinned file into the plugin data directory, keeping it only when its checksum matches.
- `reference/formal-skill.md`, the contract for a consumer's `formal-<area>` skill: its sections, one `sync.yaml` entry per check, the view file the chart reads, and where model files and unattended writes may go.
- Version pins and default bounds as keys of the consumer's `docs/conventions/formal-verification.yaml`, and the jar, Java and script-runtime paths as plugin options.
15 changes: 15 additions & 0 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -611,6 +611,21 @@
"skill"
]
},
{
"name": "formal-verification",
"source": "./plugins/formal-verification",
"category": "testing",
"tags": [
"formal-verification",
"tla-plus",
"tlc",
"model-checking",
"statechart",
"mermaid",
"skill"
],
"defaultEnabled": false
},
Comment thread
kyle-sexton marked this conversation as resolved.
Resolved
{
"name": "code-tidying",
"source": "./plugins/code-tidying",
Expand Down
162 changes: 162 additions & 0 deletions .claude/skills/formal-claim-protocol/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
---
name: formal-claim-protocol
description: "TLA+ model of the work-item tracker's GitHub claim protocol: one issue claimed at once by a local session's default claim, the issue lane's lease-only claim and a triage claim, each able to renew once and release, with lease expiry, the 300 s skew margin, the completed-run rule and the assignee hand-back. Use when a change touches the GitHub adapter's claim.sh, renew-lease.sh, release.sh, common.sh, lib/lease.sh or the tracker contract's Lease protocol, or to ask whether two claimants can both work one item."
disable-model-invocation: true
---

# Formal check: the claim protocol

## Property

Each rule is an INVARIANT in `model/ClaimProtocol.tla`; the quoted line is the source it encodes.
Paths are under `plugins/work-items/tools/work-item-tracker/`.

| Rule | Plain sentence | Source |
|---|---|---|
| `OneHolder` | At most one claimant works the item while its own lease is live. | `CONTRACT.md` "Lease protocol" step 4: "If an earlier live lease exists that is not our own comment, the foreign lease wins → supersede own comment, exit `7`."; `adapters/github/claim.sh`: "# Earliest LIVE lease wins." |
| `LaneAssignsNoOne` | The issue lane's lease-only claim never assigns anyone. | CONTRACT.md: "**Lease-only (`--lease-only`):** the claim posts the lease comment and assigns no one." |
| `HolderAssigned` | A default claimant that holds the item is assigned to it. | CONTRACT.md: "Claim = **assignee (authenticated human user) + lease record**." |
| `NoStrandedAssignee` | Every assigned login belongs to a claim still in flight. | `adapters/github/claim.sh`: "any failure before a successful lease-comment write (step 3) would otherwise strand the issue assigned with no lease record" |
| `LoserSupersedesOwn` | A claimant that lost arbitration has superseded its own lease. | CONTRACT.md step 4: "the foreign lease wins → supersede own comment, exit `7`." |
| `CompletedRunHoldsNothing` | No claimant backs off from a lease whose run had completed. | CONTRACT.md step 4: "A run whose `status` is `completed`, whatever its conclusion, holds nothing, so its lease is superseded in place" |
| `ReleaseFreesItem` | A claimant that released no longer holds a live lease. | CONTRACT.md, Release: "a session that has finished with an item hands it back at once instead of blocking every later claim" |

How each rule reads the model's states:

- "Works the item" (`Holding`): the claim returned success, no release returned success, no renew
told the session it lost or is still re-checking after a late renew, and its own lease is live.
- `HolderAssigned` is checked in quiescent states only (every claimant holds, has finished, or has
not started): the hand-back removes and restores a same-login holder's assignee two calls apart.
- "In flight" (`NoStrandedAssignee`): a default claimant of that login has started and not
returned: it is claiming, holds the item, or is renewing, releasing or handing back.
- `LoserSupersedesOwn` covers a claimant that lost, failed or died; it assumes writes succeed,
since the supersede is best-effort.
- `CompletedRunHoldsNothing` records the moment a claimant's decision goes against the lane's
lease while the lane's run has completed.

`TypeOK` checks the variables' types. `EndStateReclaimable` is an extra rule, outside the seven the owner approved
(`adapters/github/claim.sh`: "reclaim ignores items with no lease"): once every claimant
has finished and no lease is live, any assignee left is the holder of the active lease, which
reclaim unassigns.

## Harm

Two sessions work the same issue at once and push conflicting changes, or an issue stays assigned
with nobody working it and drops off the frontier.

## Tool and why

TLA+ with TLC, read through `/formal-verification:chart`. The question is whether the GitHub calls
of three sessions can interleave into a bad order; a model checker tries every order of a small
model, and the races live between comment, assignee and clock reads that no unit test sequences.

## Bounds and assumptions

- One issue; three claimants: `local` (login `human`), the lane (holder `bot`, `--lease-only`,
`session_id` `run-<id>`) and `triage` (login `human` or `other`; both are explored). The cfg's
`Starters` names the claimants that start a claim.
- Each claimant claims once, renews at most once (`WithRenew`) and releases at most once; at most
three lease comments; one renewal generation per lease.
- Every GitHub read and write is its own step. A lease list retried once is one step.
- One clock for every claimant: entering the 300 s margin and expiring are global events per lease.
With `BoundedStall`, a body read more than 300 s before its expiry is judged before it expires,
and a renew that judged its lease outside the margin PATCHes before it expires.
- Lease comment ids grow in posting order.
- The lane's run can end at any point after its lease is posted, without releasing.
- Failures are off unless a cfg turns them on: `WritesMayFail` (each best-effort write may fail),
`ReadsMayFail` (the reads whose failure does not exit may fail), `MayDie` (a claimant may die
between any two steps; a checked call's failure exits the same way). `RenewDeathKeepsHolding`
is `FALSE`: a renew that dies after its PATCH ends the session.
- The cfg uses `VIEW View`, which drops script-local values a claimant never reads again. Under
it TLC's distinct-state count depends on exploration order; the chart command runs one worker,
so its count is stable.
- The model is finite, so no bound limits the first pass; the run commands pass
`--rerun-states` to cover it after a break.
- The seeded break in `sync.yaml` was run against this model: with the winner taken as the latest
live lease, `OneHolder` breaks in `LocalLane.cfg` after 1,415 distinct states (`caught: true`).

## Not covered

- Reclaim, the local-markdown and other adapters, the role-label flip, a second renew, and an
assignee a person adds by hand.
- A comment from an author the adapter does not count, a cross-issue or foreign handle, and
`gh api user` reads.
- Time passing between renew's PATCH and its clock read after it: the model takes both in one
step, so it never shows a renew that landed in time but reads its clock inside the margin or
past the old expiry (the code then re-lists or backs off).
- A PATCH that GitHub applies but reports as failed (renew's or release's), a lease
`renewed_at` more than 300 s in the future, a TTL over the cap, and the pause between writes.
- Three sessions on one login: the model has at most two. With three, a hand-back's last removal
can land after a third session's claim won, leaving that holder unassigned (issue 16505).
- `WithRenew = TRUE` with all three claimants: that space is over 100 million states. The shipped
checks run two claimants each. All three without renew (`Starters = {"local", "lane", "triage"}`,
`WithRenew = FALSE`) hold every rule at a548b3aaab8c: 202,568,284 states, 25 minutes on 20
workers, so it is a hand run and not a shipped check.
- The tracker contract's documented limits are reported as known limits, never as defects:
clock skew ("Liveness compares provider timestamps (`renewed_at` in the marker) to the local
clock"), ttl-0 ("A claim with `ttl_hours: 0` and `ttl_minutes: 0` is born expired"),
comment-id monotonicity ("adapters MUST emit ordered unique numeric `lease_comment_id`
handles"), the ambiguous lease post ("A lease comment POST that GitHub applies but reports as
failed (a timeout) returns no comment id"), and no compare-and-swap ("GitHub's comment PATCH
has no If-Match/CAS, so a concurrent writer can still interleave").

## Run

One command per cfg; each takes two to three minutes:

```text
/formal-verification:chart .claude/skills/formal-claim-protocol/model/ClaimProtocol.tla --config .claude/skills/formal-claim-protocol/model/LocalLane.cfg --view .claude/skills/formal-claim-protocol/model/view.mjs --rerun-states 2000000
/formal-verification:chart .claude/skills/formal-claim-protocol/model/ClaimProtocol.tla --config .claude/skills/formal-claim-protocol/model/LocalTriage.cfg --view .claude/skills/formal-claim-protocol/model/view.mjs --rerun-states 2000000
```

`LocalLane.cfg` runs the local session against the issue lane, `LocalTriage.cfg` the local session
against a triage claim; both with renew, every failure off and `BoundedStall`. `--rerun-states`
must cover the whole space (1,686,588 and 1,356,633 states): after a rule breaks, the command rechecks
the others under that bound, and a smaller bound leaves them not checked.

Pass a new `--out` folder each run.

## Reading a result

Replay each counterexample step by step against the scripts on a scratch issue, one call per
action. The actions are named for the calls they model: `Assign`, `CheckRead`, `ReAssign`,
`PostLease`, `ReadLeases`, `ArbJudge`, `ArbRunRead`, `ArbRunSup`, `LossSup`, `WinRead` and
`WinAdd` are `claim.sh`; `RenewStart`, `RenewPatch`, `Ren2Leases`, `Ren2Judge` and `RenSup` are
`renew-lease.sh`; `ReleaseStart`, `RelJudgeOwn` and `RelPatch` are `release.sh`; `GLeases` to
`GRm3` are the hand-back, `wit_hand_back_assignee` in `common.sh`; `TrapSup` is claim's abort
trap; `Die` is a death or an exiting failure; `RunCompletes`, `Approach` (300 s before expiry) and
`Expire` are the world. Each action's comment in the model cites its lines. If a bug does not
occur, think that the model is wrong until you show that it is not. A path that reproduces is a
bug: write it up with `/bugs:write`. A path that needs a documented limit to fail is reported as
that limit.

Known results at the sync commit (a run that shows only these has found nothing new):

- Both shipped cfgs hold every rule.
- With `WritesMayFail = TRUE`, `NoStrandedAssignee` and `EndStateReclaimable` break in both
starter sets: a release's PATCH fails, the release exits with no hand-back, and its login stays
assigned beside its live lease; once a later lease is the active one, reclaim no longer removes
that login. In `{"local","triage"}` `HolderAssigned` breaks: a same-login hand-back removed the
login and the winner's re-add fails. A known limit: these writes are best-effort or exit on
failure.
- With `ReadsMayFail = TRUE`, `OneHolder` breaks: renew's re-list after a PATCH inside the margin
fails twice and the renewed lease is kept while a newer claimant holds. By design: "a list it
still cannot read keeps the renewed lease (exit `0`, with a warning)". `CompletedRunHoldsNothing`
breaks in `{"local","lane"}`: "A run in any other status, or a read that fails, keeps the lease
winning." In `{"local","triage"}` `HolderAssigned`, `NoStrandedAssignee` and
`EndStateReclaimable` break: a hand-back whose lease list fails twice counts it as empty or
stops at the re-list, so it removes a same-login holder's login or leaves a login no lease
holds. By design: "a failed list counts as empty".
- With `MayDie = TRUE`, `NoStrandedAssignee` and `EndStateReclaimable` break (a claim killed
during its own rollback leaves its assignee with no lease, which reclaim ignores), and in
`{"local","triage"}` `HolderAssigned` breaks (a same-login hand-back killed between its removal
and its re-list). A known limit: a killed process cannot finish its exit trap.

## Updating

When a covered file in `sync.yaml` changes, read the change against the model: a call split in
two, a new branch, a retry, or a moved check means the model changes; a promise changed on purpose
means a rule changes, and only with the owner's yes. Never weaken a rule to make it hold. Move
`commit` forward only after confirming the model describes the code at the new commit, re-cite
the changed line numbers in the model's comments, rerun every shipped cfg and the seeded break,
and update the known results above.
Loading
Loading