Repository navigation
feat(formal-verification): add the chart and setup skills and the formal-area contract #16458
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
kyle-sexton
wants to merge
35
commits into
main
Choose a base branch
from
feat/formal-verification-chart-setup
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+25,185
−30
Open
Changes from all commits
Commits
Show all changes
35 commits
Select commit
Hold shift + click to select a range
eb0c346
feat(formal-verification): add the plugin skeleton, settings and tool…
kyle-sexton aaff5bc
feat(formal-verification): add the formal-area skill contract
kyle-sexton 647187a
feat(formal-verification): add the chart skill, TLA+ with TLC
kyle-sexton 2cb5436
feat(formal-verification): add the setup skill
kyle-sexton a24d8d6
feat(formal-verification): draft the claim-protocol model as formal-c…
kyle-sexton 4446aac
docs(upstream): record the formal-verification adoption at both upstr…
kyle-sexton 29460f2
test(formal-verification): keep the chart corpus out of markdown and …
kyle-sexton 43fba2c
test(formal-verification): add eval cases for setup check and an unat…
kyle-sexton d851e21
fix(formal-verification): guard the pid a chart test kills
kyle-sexton 86427e4
fix(formal-verification): accept default labels and keep region condi…
kyle-sexton d6a259c
fix(formal-verification): take download sources only from the shipped…
kyle-sexton 146c03d
fix(formal-verification): stage the spec and bound each Java run
kyle-sexton b369b06
fix(formal-verification): fence and escape spec text in the report, d…
kyle-sexton 93f6a07
docs(formal-verification): state that a view file is code and report …
kyle-sexton db59e0f
test(formal-verification): keep fixture rewrites out of the chart tes…
kyle-sexton e6323d4
test(formal-verification): use US spelling in a guard test name
kyle-sexton 18e4771
fix(formal-verification): escape only spec text in report sentences
kyle-sexton 9c17f9a
Merge commit 'e1491a4badcd2ebe063a306ee201d5a6933e5b0e' into feat/for…
kyle-sexton 0e24d61
feat(formal-verification): regenerate chart corpus, record the securi…
kyle-sexton 327bb34
refactor(formal-verification): name the fake-java variable prefix
kyle-sexton c644a99
Merge remote-tracking branch 'origin/main' into feat/formal-verificat…
kyle-sexton d662a24
test(formal-verification): expect the output-folder refusal the chart…
kyle-sexton f55ed11
feat(formal-verification): model the claim protocol with renew, relea…
kyle-sexton c1d472e
style(formal-verification): drop a comment the view file does not need
kyle-sexton 67bce52
Merge remote-tracking branch 'origin/main' into feat/formal-verificat…
kyle-sexton 0d2ced5
fix(formal-verification): declare the TLC jar as a prerequisite
kyle-sexton cad1e32
Merge remote-tracking branch 'origin/main' into feat/formal-verificat…
kyle-sexton 57380c2
chore(formal-verification): take main's shared prerequisites copies
kyle-sexton 405c35c
Merge remote-tracking branch 'origin/main' into feat/formal-verificat…
kyle-sexton bf5c619
feat(formal-verification): model the claim protocol at a548b3aaab8c w…
kyle-sexton e852414
fix(formal-verification): address the review round and two CI gates
kyle-sexton 804cac9
Merge remote-tracking branch 'origin/main' into feat/formal-verificat…
kyle-sexton e28be56
ci: exclude the chart corpus graph files from the editorconfig check
kyle-sexton 0af0d1a
Merge remote-tracking branch 'origin/main' into feat/formal-verificat…
kyle-sexton 9eb40bc
fix(formal-verification): close the gaps the done-when check of #1564…
kyle-sexton File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
There are no files selected for viewing
11 changes: 11 additions & 0 deletions
11
.changes/formal-verification/feat-formal-verification-chart-setup-2f19b369.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| bump: minor | ||
| --- | ||
|
|
||
| ### Added | ||
|
|
||
| - The `formal-verification` plugin, off by default. | ||
| - `/formal-verification:chart` model-checks a TLA+ spec with TLC and writes a Mermaid chart of its states and a report with the shortest path to each broken rule. Its scripts take the jar, Java and bounds as arguments and install nothing. | ||
| - `/formal-verification:setup` checks the script runtime, Java, the model checker and the renderer, and on a typed `apply install-model-checker` or `apply install-renderer` downloads the pinned file into the plugin data directory, keeping it only when its checksum matches. | ||
| - `reference/formal-skill.md`, the contract for a consumer's `formal-<area>` skill: its sections, one `sync.yaml` entry per check, the view file the chart reads, and where model files and unattended writes may go. | ||
| - Version pins and default bounds as keys of the consumer's `docs/conventions/formal-verification.yaml`, and the jar, Java and script-runtime paths as plugin options. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,162 @@ | ||
| --- | ||
| name: formal-claim-protocol | ||
| description: "TLA+ model of the work-item tracker's GitHub claim protocol: one issue claimed at once by a local session's default claim, the issue lane's lease-only claim and a triage claim, each able to renew once and release, with lease expiry, the 300 s skew margin, the completed-run rule and the assignee hand-back. Use when a change touches the GitHub adapter's claim.sh, renew-lease.sh, release.sh, common.sh, lib/lease.sh or the tracker contract's Lease protocol, or to ask whether two claimants can both work one item." | ||
| disable-model-invocation: true | ||
| --- | ||
|
|
||
| # Formal check: the claim protocol | ||
|
|
||
| ## Property | ||
|
|
||
| Each rule is an INVARIANT in `model/ClaimProtocol.tla`; the quoted line is the source it encodes. | ||
| Paths are under `plugins/work-items/tools/work-item-tracker/`. | ||
|
|
||
| | Rule | Plain sentence | Source | | ||
| |---|---|---| | ||
| | `OneHolder` | At most one claimant works the item while its own lease is live. | `CONTRACT.md` "Lease protocol" step 4: "If an earlier live lease exists that is not our own comment, the foreign lease wins → supersede own comment, exit `7`."; `adapters/github/claim.sh`: "# Earliest LIVE lease wins." | | ||
| | `LaneAssignsNoOne` | The issue lane's lease-only claim never assigns anyone. | CONTRACT.md: "**Lease-only (`--lease-only`):** the claim posts the lease comment and assigns no one." | | ||
| | `HolderAssigned` | A default claimant that holds the item is assigned to it. | CONTRACT.md: "Claim = **assignee (authenticated human user) + lease record**." | | ||
| | `NoStrandedAssignee` | Every assigned login belongs to a claim still in flight. | `adapters/github/claim.sh`: "any failure before a successful lease-comment write (step 3) would otherwise strand the issue assigned with no lease record" | | ||
| | `LoserSupersedesOwn` | A claimant that lost arbitration has superseded its own lease. | CONTRACT.md step 4: "the foreign lease wins → supersede own comment, exit `7`." | | ||
| | `CompletedRunHoldsNothing` | No claimant backs off from a lease whose run had completed. | CONTRACT.md step 4: "A run whose `status` is `completed`, whatever its conclusion, holds nothing, so its lease is superseded in place" | | ||
| | `ReleaseFreesItem` | A claimant that released no longer holds a live lease. | CONTRACT.md, Release: "a session that has finished with an item hands it back at once instead of blocking every later claim" | | ||
|
|
||
| How each rule reads the model's states: | ||
|
|
||
| - "Works the item" (`Holding`): the claim returned success, no release returned success, no renew | ||
| told the session it lost or is still re-checking after a late renew, and its own lease is live. | ||
| - `HolderAssigned` is checked in quiescent states only (every claimant holds, has finished, or has | ||
| not started): the hand-back removes and restores a same-login holder's assignee two calls apart. | ||
| - "In flight" (`NoStrandedAssignee`): a default claimant of that login has started and not | ||
| returned: it is claiming, holds the item, or is renewing, releasing or handing back. | ||
| - `LoserSupersedesOwn` covers a claimant that lost, failed or died; it assumes writes succeed, | ||
| since the supersede is best-effort. | ||
| - `CompletedRunHoldsNothing` records the moment a claimant's decision goes against the lane's | ||
| lease while the lane's run has completed. | ||
|
|
||
| `TypeOK` checks the variables' types. `EndStateReclaimable` is an extra rule, outside the seven the owner approved | ||
| (`adapters/github/claim.sh`: "reclaim ignores items with no lease"): once every claimant | ||
| has finished and no lease is live, any assignee left is the holder of the active lease, which | ||
| reclaim unassigns. | ||
|
|
||
| ## Harm | ||
|
|
||
| Two sessions work the same issue at once and push conflicting changes, or an issue stays assigned | ||
| with nobody working it and drops off the frontier. | ||
|
|
||
| ## Tool and why | ||
|
|
||
| TLA+ with TLC, read through `/formal-verification:chart`. The question is whether the GitHub calls | ||
| of three sessions can interleave into a bad order; a model checker tries every order of a small | ||
| model, and the races live between comment, assignee and clock reads that no unit test sequences. | ||
|
|
||
| ## Bounds and assumptions | ||
|
|
||
| - One issue; three claimants: `local` (login `human`), the lane (holder `bot`, `--lease-only`, | ||
| `session_id` `run-<id>`) and `triage` (login `human` or `other`; both are explored). The cfg's | ||
| `Starters` names the claimants that start a claim. | ||
| - Each claimant claims once, renews at most once (`WithRenew`) and releases at most once; at most | ||
| three lease comments; one renewal generation per lease. | ||
| - Every GitHub read and write is its own step. A lease list retried once is one step. | ||
| - One clock for every claimant: entering the 300 s margin and expiring are global events per lease. | ||
| With `BoundedStall`, a body read more than 300 s before its expiry is judged before it expires, | ||
| and a renew that judged its lease outside the margin PATCHes before it expires. | ||
| - Lease comment ids grow in posting order. | ||
| - The lane's run can end at any point after its lease is posted, without releasing. | ||
| - Failures are off unless a cfg turns them on: `WritesMayFail` (each best-effort write may fail), | ||
| `ReadsMayFail` (the reads whose failure does not exit may fail), `MayDie` (a claimant may die | ||
| between any two steps; a checked call's failure exits the same way). `RenewDeathKeepsHolding` | ||
| is `FALSE`: a renew that dies after its PATCH ends the session. | ||
| - The cfg uses `VIEW View`, which drops script-local values a claimant never reads again. Under | ||
| it TLC's distinct-state count depends on exploration order; the chart command runs one worker, | ||
| so its count is stable. | ||
| - The model is finite, so no bound limits the first pass; the run commands pass | ||
| `--rerun-states` to cover it after a break. | ||
| - The seeded break in `sync.yaml` was run against this model: with the winner taken as the latest | ||
| live lease, `OneHolder` breaks in `LocalLane.cfg` after 1,415 distinct states (`caught: true`). | ||
|
|
||
| ## Not covered | ||
|
|
||
| - Reclaim, the local-markdown and other adapters, the role-label flip, a second renew, and an | ||
| assignee a person adds by hand. | ||
| - A comment from an author the adapter does not count, a cross-issue or foreign handle, and | ||
| `gh api user` reads. | ||
| - Time passing between renew's PATCH and its clock read after it: the model takes both in one | ||
| step, so it never shows a renew that landed in time but reads its clock inside the margin or | ||
| past the old expiry (the code then re-lists or backs off). | ||
| - A PATCH that GitHub applies but reports as failed (renew's or release's), a lease | ||
| `renewed_at` more than 300 s in the future, a TTL over the cap, and the pause between writes. | ||
| - Three sessions on one login: the model has at most two. With three, a hand-back's last removal | ||
| can land after a third session's claim won, leaving that holder unassigned (issue 16505). | ||
| - `WithRenew = TRUE` with all three claimants: that space is over 100 million states. The shipped | ||
| checks run two claimants each. All three without renew (`Starters = {"local", "lane", "triage"}`, | ||
| `WithRenew = FALSE`) hold every rule at a548b3aaab8c: 202,568,284 states, 25 minutes on 20 | ||
| workers, so it is a hand run and not a shipped check. | ||
| - The tracker contract's documented limits are reported as known limits, never as defects: | ||
| clock skew ("Liveness compares provider timestamps (`renewed_at` in the marker) to the local | ||
| clock"), ttl-0 ("A claim with `ttl_hours: 0` and `ttl_minutes: 0` is born expired"), | ||
| comment-id monotonicity ("adapters MUST emit ordered unique numeric `lease_comment_id` | ||
| handles"), the ambiguous lease post ("A lease comment POST that GitHub applies but reports as | ||
| failed (a timeout) returns no comment id"), and no compare-and-swap ("GitHub's comment PATCH | ||
| has no If-Match/CAS, so a concurrent writer can still interleave"). | ||
|
|
||
| ## Run | ||
|
|
||
| One command per cfg; each takes two to three minutes: | ||
|
|
||
| ```text | ||
| /formal-verification:chart .claude/skills/formal-claim-protocol/model/ClaimProtocol.tla --config .claude/skills/formal-claim-protocol/model/LocalLane.cfg --view .claude/skills/formal-claim-protocol/model/view.mjs --rerun-states 2000000 | ||
| /formal-verification:chart .claude/skills/formal-claim-protocol/model/ClaimProtocol.tla --config .claude/skills/formal-claim-protocol/model/LocalTriage.cfg --view .claude/skills/formal-claim-protocol/model/view.mjs --rerun-states 2000000 | ||
| ``` | ||
|
|
||
| `LocalLane.cfg` runs the local session against the issue lane, `LocalTriage.cfg` the local session | ||
| against a triage claim; both with renew, every failure off and `BoundedStall`. `--rerun-states` | ||
| must cover the whole space (1,686,588 and 1,356,633 states): after a rule breaks, the command rechecks | ||
| the others under that bound, and a smaller bound leaves them not checked. | ||
|
|
||
| Pass a new `--out` folder each run. | ||
|
|
||
| ## Reading a result | ||
|
|
||
| Replay each counterexample step by step against the scripts on a scratch issue, one call per | ||
| action. The actions are named for the calls they model: `Assign`, `CheckRead`, `ReAssign`, | ||
| `PostLease`, `ReadLeases`, `ArbJudge`, `ArbRunRead`, `ArbRunSup`, `LossSup`, `WinRead` and | ||
| `WinAdd` are `claim.sh`; `RenewStart`, `RenewPatch`, `Ren2Leases`, `Ren2Judge` and `RenSup` are | ||
| `renew-lease.sh`; `ReleaseStart`, `RelJudgeOwn` and `RelPatch` are `release.sh`; `GLeases` to | ||
| `GRm3` are the hand-back, `wit_hand_back_assignee` in `common.sh`; `TrapSup` is claim's abort | ||
| trap; `Die` is a death or an exiting failure; `RunCompletes`, `Approach` (300 s before expiry) and | ||
| `Expire` are the world. Each action's comment in the model cites its lines. If a bug does not | ||
| occur, think that the model is wrong until you show that it is not. A path that reproduces is a | ||
| bug: write it up with `/bugs:write`. A path that needs a documented limit to fail is reported as | ||
| that limit. | ||
|
|
||
| Known results at the sync commit (a run that shows only these has found nothing new): | ||
|
|
||
| - Both shipped cfgs hold every rule. | ||
| - With `WritesMayFail = TRUE`, `NoStrandedAssignee` and `EndStateReclaimable` break in both | ||
| starter sets: a release's PATCH fails, the release exits with no hand-back, and its login stays | ||
| assigned beside its live lease; once a later lease is the active one, reclaim no longer removes | ||
| that login. In `{"local","triage"}` `HolderAssigned` breaks: a same-login hand-back removed the | ||
| login and the winner's re-add fails. A known limit: these writes are best-effort or exit on | ||
| failure. | ||
| - With `ReadsMayFail = TRUE`, `OneHolder` breaks: renew's re-list after a PATCH inside the margin | ||
| fails twice and the renewed lease is kept while a newer claimant holds. By design: "a list it | ||
| still cannot read keeps the renewed lease (exit `0`, with a warning)". `CompletedRunHoldsNothing` | ||
| breaks in `{"local","lane"}`: "A run in any other status, or a read that fails, keeps the lease | ||
| winning." In `{"local","triage"}` `HolderAssigned`, `NoStrandedAssignee` and | ||
| `EndStateReclaimable` break: a hand-back whose lease list fails twice counts it as empty or | ||
| stops at the re-list, so it removes a same-login holder's login or leaves a login no lease | ||
| holds. By design: "a failed list counts as empty". | ||
| - With `MayDie = TRUE`, `NoStrandedAssignee` and `EndStateReclaimable` break (a claim killed | ||
| during its own rollback leaves its assignee with no lease, which reclaim ignores), and in | ||
| `{"local","triage"}` `HolderAssigned` breaks (a same-login hand-back killed between its removal | ||
| and its re-list). A known limit: a killed process cannot finish its exit trap. | ||
|
|
||
| ## Updating | ||
|
|
||
| When a covered file in `sync.yaml` changes, read the change against the model: a call split in | ||
| two, a new branch, a retry, or a moved check means the model changes; a promise changed on purpose | ||
| means a rule changes, and only with the owner's yes. Never weaken a rule to make it hold. Move | ||
| `commit` forward only after confirming the model describes the code at the new commit, re-cite | ||
| the changed line numbers in the model's comments, rerun every shipped cfg and the seeded break, | ||
| and update the known results above. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.