Skip to content

Add the formal-verification plugin's chart skill, setup and consumer contract, proven on the loop-lane claim protocol #15648

Description

@kyle-sexton

Decision (owner, 2026-11-12, revised the same day)

Source: the talk "All Code Is Slop (Except Mine)" (Lauren Tan, AI Engineer Code Summit, 2026-11-11), slides 27 and 32 and the transcript at 1:12:08 to 1:12:39 and 1:14:05: formal verification with TLA+ and Lean, and statecharts reviewed alongside videos and screenshots. "We've already found quite a few bugs with formal verification. But it's certainly not a silver bullet."

We absorb her three formal-verification skills (create-formal-verification-skill, maintain-formal-verification-skill, statechart in cursor/plugins pstack/skills/, MIT) as one plugin, keeping as much of her text and scripts as possible, placed, renamed and packaged to this marketplace's philosophy. Owner: "we keep as much of her stuff as possible. We just place it, rename it, and package it differently." This replaces the earlier throwaway proof of concept and its "ship a skill only if it finds a real defect" gate, and revises the parked "no TLA+ skill, written in our own words" decision.

The shape mirrors app-control, which absorbed her runtime verification pair the same way: a generator skill with actions over a consumer-owned skill under a contract, plus a skill that operates it.

plugins/formal-verification/
├── .claude-plugin/plugin.json     options for the model checker and Java paths
├── README.md  + MIT notice for her scripts
├── prerequisites.json             script runtime, Java, model checker, renderer (chart only)
├── reference/formal-skill.md      contract for the consumer's generated skill
└── skills/
    ├── specify/                   bare = propose, writes nothing | create | check | refresh
    │   └── reference/  create.md  refresh.md  choose-a-tool.md
    ├── chart/                     model-check a TLA+ spec and chart its states
    │   ├── reference/  view-module.md  read-report.md  render.md
    │   └── scripts/               her scripts, install path removed
    └── setup/                     checks prerequisites; installs only on an interactive yes

<consumer repo>/.claude/skills/formal-<area>/
├── SKILL.md      what is verified, tool and why, bounds, the run command
├── sync.yaml     covers: <code>, commit: <last confirmed>
└── model/        spec, config, view file, harness

Names are decided: formal-verification, specify, chart, setup, formal-<area>, sync.yaml.

This item: slice 1 of 3

The chart skill, the setup skill and the consumer contract, proven on this repository's work-item claim protocol, defined in the work-item tracker contract's "Lease protocol" (one item claimed at once by a default claim, a lease-only claim and a triage-phase claim).

  • chart keeps her statechart skill's text and scripts: check a TLA+ model, draw its states as a Mermaid chart, give the shortest path to each broken rule, for a person to judge. "Never weaken a rule to make it hold." A bug path is replayed against the real code before it counts: "If a bug does not occur, think that the model is wrong until you show that it is not."
  • chart is the plugin's only TLA+-specific skill: its description and body declare "TLA+ with TLC", and it keeps her "Write the spec" rules.
  • Changed from hers: the scripts no longer install the model checker or Java while the skill runs; they take the jar, Java and bounds as arguments and never install. Each prerequisite is declared and setup checks it. Machine paths (jar, Java, script runtime) are userConfig options, not environment variables. Numbers and pinned versions the skill text restates point at the script or carry an as-of date and a recheck trigger.
  • setup installs the pinned, checksummed toolchain into the plugin data directory only on an interactive yes; unattended runs never install and report "blocked: toolchain missing". No CI job runs the model checker in this slice.
  • The contract (reference/formal-skill.md) names what a consumer's formal-<area> skill holds: frontmatter; the sections Property, Harm, Tool and why, Bounds and assumptions, Not covered, Run, Reading a result, Updating; one sync.yaml entry per check (id, covered paths, sync commit, command, recorded seeded break); and the view file the chart reads. Model files default to formal-<area>/model/; paths outside it are allowed only when listed in sync.yaml, and unattended writes are limited to the folder plus those paths. It is the only surface the two skills share; specify never copies chart's commands.
  • A human owns the invariants. Outside evidence is mixed: agents handle TLA+ mechanics but score under half on conformance and invariants unaided. The agent drafts them, each quoting the contract or code line it encodes; the owner approves the list before the model is checked.
  • Both upstream records are updated in this slice's pull request: the parked TLA+ record gains the revised decision and attribution rows at the new pin, and the pstack record's three rows change from pending adoption to adopted.

Design rules for all three slices

  • One responsibility per skill; specify's create, check and refresh are actions because they read and write the same consumer skill and contract.
  • specify names no tool. Her sentence "TLA+ is the default" is replaced, recorded as a change (ADR 0101).
  • The tool is chosen by the question and the consumer's repository. A second tool is recorded only in the consumer's formal-<area> skill ("Tool and why"); no specify/reference/tools/<tool>.md file is added for any tool. A state-transition checker other than TLC adds a front end in chart/scripts. No tool interface until a second tool exists.
  • Skills reach each other only by slash reference with an "if installed" fallback: /corrections:codify (the ladder's opt-in formal-verification rung, which Release enforcement ladder 3.0.0 with the verification tier and capability-named rungs added), /testing:write when no verifier exists for a language, /bugs:write for a bug, /verification:confirm for verdict evidence.
  • Nothing a consumer may need to change is hard-coded. Owner, 2026-11-12: "making things configurable and extensible, not shipping assumptions or hard-coded things that can't be changed by the downstream consumers. Every single plugin and skill must align with our plugin philosophy, our user config, and our extensibility and configurability requirements." Her scripts pin a model-checker version, a Java version, a script runtime, a renderer, and state and bound limits. Each becomes a setting with her value as the default, and a consumer can add a tool or a front end without editing shipped files. Machine paths (jar, Java, script runtime) are userConfig options; version pins and default bounds are keys in the consumer's docs/conventions/formal-verification.yaml, her values as defaults; a per-model bound is an argument in that model's run command. The image renderer is a setting (default: her Pele 1.9.0), used only when an image is asked for.
  • Each SKILL.md holds routing and the quality bar; everything else loads on demand from reference files.
  • An unattended run installs nothing and writes nothing outside the consumer's formal-<area> folder and the paths its sync.yaml lists.

Before build

The interview ran and the owner confirmed it on 2026-11-14 ("Go with all recommendations.", then "Confirmed."). Its answers:

  • States and actions the first model covers: one item claimed at once by a default claim, a lease-only claim and a triage-phase claim on the GitHub adapter, with lease expiry, release and the completed-run rule. Renew, reclaim and other adapters are left out. The race is not "two lanes": two issue-lane runs on one issue are already serialized.
  • Who writes the invariants: the agent drafts them, each quoting the contract or code line it encodes; the owner approves the list before the model is checked.
  • What counts as a real defect: a counterexample replayed step by step against the GitHub adapter's claim script. The tracker contract's three documented limits (clock skew, ttl-0, comment-id monotonicity) are reported as known limits. A defect is written up through /bugs:write.
  • First formal-<area> skill or one-off: the model ships as this repository's first formal-<area> skill under .claude/skills/.

The cost of carrying her scripts, raised on 2026-11-12:

  • Who reads them in full before they ship. chart runs about 3,900 lines of her TypeScript plus a large test folder. Before anything ships: a full read of her scripts by fresh-context agent reviewers, one per script group, and the plugin-acceptance security review recorded as a new section of ADR 0093.
  • How her updates are tracked. Copied scripts become ours to fix, including model-checker and Java version bumps. Once copied they diverge as ours; her changes are re-taken only through the pstack record's drift re-audit.
  • Copy or call. Copied, not called: her skill is disable-model-invocation: true.
  • Evidence. The chart's Mermaid text goes in the pull request's one evidence comment after the verifier passes.

Dogfood

This repository's work-item claim protocol (the work-item tracker contract's "Lease protocol"), charted by the new skill.

Owner step

The owner approves the agent-drafted invariants list before the claim-protocol model is checked. This is the one step in this slice that needs the owner.

Done when

  • /formal-verification:chart model-checks a TLA+ spec and writes a Mermaid chart of its states and a report giving the shortest path to each broken rule.
  • The /formal-verification:chart description and body declare "TLA+ with TLC".
  • The /formal-verification:chart body keeps her statechart text, including "Never weaken a rule to make it hold." and her "Write the spec" rules.
  • The /formal-verification:chart scripts take the model-checker jar, Java and bounds as arguments, and no script downloads or installs anything.
  • The formal-verification plugin declares the script runtime, Java, the model checker and the renderer as prerequisites, and /formal-verification:setup checks each one.
  • /formal-verification:setup installs the pinned, checksummed toolchain into the plugin data directory only after an interactive yes.
  • An unattended formal-verification run with the toolchain absent installs nothing and reports "blocked: toolchain missing".
  • The formal-verification plugin's jar, Java and script-runtime paths are userConfig options, and no script reads them from environment variables.
  • The formal-verification plugin reads version pins and default bounds from keys in the consumer's docs/conventions/formal-verification.yaml, with her values as the defaults.
  • A per-model bound for /formal-verification:chart is an argument in that model's run command.
  • The formal-verification image renderer is a setting defaulting to Pele 1.9.0, used only when an image is asked for.
  • Every number or pinned version the formal-verification skill text restates points at the script or carries an as-of date and a recheck trigger.
  • The formal-verification consumer contract requires frontmatter and the sections Property, Harm, Tool and why, Bounds and assumptions, Not covered, Run, Reading a result, Updating.
  • The formal-verification consumer contract requires one sync.yaml entry per check holding id, covered paths, sync commit, command and recorded seeded break.
  • The formal-verification consumer contract names the view file /formal-verification:chart reads.
  • The formal-verification consumer contract defaults model files to formal-<area>/model/ and allows a path outside it only when sync.yaml lists it.
  • The formal-verification consumer contract limits unattended writes to the formal-<area> folder and the paths its sync.yaml lists.
  • The claim-protocol model covers one item claimed at once by a default claim, a lease-only claim and a triage-phase claim on the GitHub adapter.
  • The claim-protocol model includes lease expiry, release and the completed-run rule, and leaves out renew, reclaim and other adapters.
  • Each invariant in the claim-protocol model quotes the tracker contract or code line it encodes.
  • The claim-protocol model ships as this repository's first formal-<area> skill under .claude/skills/ and meets the formal-verification consumer contract.
  • Every counterexample the claim-protocol model reports was replayed step by step against the GitHub adapter's claim script, and the replay result is in the pull request.
  • The tracker contract's three documented limits (clock skew, ttl-0, comment-id monotonicity) appear in the claim-protocol findings as known limits, not defects.
  • If a claim-protocol counterexample reproduces against the claim script, it is written up through /bugs:write.
  • No CI job runs the model checker.
  • Fresh-context agent reviewers, one per script group, each read every line of her copied scripts, and the pull request names each group and its reviewer's result.
  • ADR 0093 has a new section recording the plugin-acceptance security review of the formal-verification plugin.
  • The formal-verification plugin README carries the MIT notice for her scripts.
  • The cursor pstack upstream record's rows for her three formal-verification skills read adopted, with the pin and the note that copied scripts diverge and are re-taken only through the drift re-audit.
  • The parked TLA+ upstream record carries the revised decision and attribution rows at the new pin.
  • The pull request's one evidence comment holds the claim-protocol chart's Mermaid text, added after the verifier passes for the head.

Out of scope

  • A CI job that runs the model checker: revisit after the first model proves its value.
  • A tool interface for a second checker: not until a second tool exists.
  • Renew, reclaim and adapters other than GitHub in the first model.
  • specify (slices 2 and 3).

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 12, 2026
  2. added theissue type on Nov 12, 2026
  3. github-actions commented on Nov 12, 2026

    @github-actions
    Contributor

    Size: Large
    Complexity: High
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Understood: this adds the formal-verification plugin's chart and setup skills and the consumer contract. It is proven on the work-item claim protocol as this repository's first formal-<area> skill.

    The newest recorded decision (2026-11-16) is:

    Decision recorded.

    Owner, 2026-11-16, in session ("Go w/ recommendations"), posted by an agent session. Two done-when lines are accepted as #16458 delivers them:

    #16458 closes this issue on merge.

    #15648 (comment)

    That decision settles the open questions, so none are asked again. #16458 is named as the change that closes this issue; I did not find it in the recent merged pull requests, so it looks unmerged. The earlier patch-protected-path stop and the host-run label are unchanged by this triage.

    Automated triage. A maintainer reviews it.

  4. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 12, 2026
  5. melodic-automation-lanes commented on Nov 12, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  6. melodic-automation-lanes commented on Nov 12, 2026

    @melodic-automation-lanes
    Contributor

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: A bounded TLA+ model of the claim and lease protocol in plugins/work-items/tools/work-item-tracker/CONTRACT.md, with owner-written invariants; counterexamples are replayed against the real scripts, and a skill ships only if a real defect is confirmed.

    Related work

    Open questions

    1. Which protocol states and actions does the model cover? The lease protocol has two claim modes, claim steps 1 to 5, renew, reclaim, release, TTL expiry and run-completed supersession.
      RECOMMENDED: Cover the lease-only claim (issue lane) racing the default claim (local session): lease comment, assignee, earlier-live-lease arbitration, back-off, TTL expiry and run-completed supersession. Leave out renew, phase and the local-markdown adapter. Basis: plugins/work-items/tools/work-item-tracker/CONTRACT.md:637
    2. Who writes the invariants and where are they recorded?
      RECOMMENDED: The owner writes them before the model is built and the agent writes only the TLA+ mechanics. A first candidate is that at most one live, non-superseded lease holder owns an item at any time. Record them in the spec header. Basis: judgment
    3. What counts as a real defect that justifies shipping a skill?
      RECOMMENDED: A counterexample trace that the real scripts reproduce, such as two lanes both proceeding on the same item. A model-only violation that does not reproduce is a model bug and does not count. Basis: issue body, Decision section: replay every counterexample against the real scripts
    4. Where does the spec live and what is the done-when artifact, given no .tla file exists in the repository yet and no TLC run can be assumed in CI?
      RECOMMENDED: Put the spec and its TLC config under docs/ or a research directory, with a short findings note stating the outcome of the TLC run and of each replay. Do not add a CI check. Basis: judgment

    Sources read

    • plugins/work-items/tools/work-item-tracker/CONTRACT.md:637-710: claim modes, lease comment, claim steps 1 to 5, arbitration and run-completed supersession
    • docs/contracts/loop-lane/README.md:1493-1498: any assignee keeps an item off the frontier, a lease marker is never written by hand, and the tracker CONTRACT owns the lease protocol
    • docs/contracts/loop-lane/README.md:1000: session ids are distinct across concurrently running instances
    • repository glob **/*.tla: no TLA+ files exist yet

    To answer, post a comment that starts with this line:

    **Decision recorded.** <answer>
    

    Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.

  7. changed the title [-]Model the loop-lane claim protocol in TLA+ as one bounded proof of concept[/-] [+]Add the formal-verification plugin's chart skill, setup and consumer contract, proven on the loop-lane claim protocol[/+] on Nov 13, 2026
  8. added
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Nov 13, 2026
  9. kyle-sexton commented on Nov 14, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded. Interview complete, owner-confirmed 2026-11-14 ("Go with all recommendations.", then "Confirmed."). Decisions for this slice and the ones all three slices share:

    • chart is the plugin's only TLA+-specific skill; its description and body declare "TLA+ with TLC" and it keeps her "Write the spec" rules. specify names no tool. Her sentence "TLA+ is the default" is replaced, recorded as a change (ADR 0101).
    • Her scripts are copied, not called (her skill is disable-model-invocation: true). They take the jar, Java and bounds as arguments and never install. Once copied they diverge as ours; her changes are re-taken only through the pstack record's drift re-audit.
    • setup installs the pinned, checksummed toolchain into the plugin data directory only on an interactive yes; unattended runs never install and report "blocked: toolchain missing". No CI job runs the model checker in this slice.
    • Settings: machine paths (jar, Java, script runtime) are userConfig options; version pins and default bounds are keys in the consumer's docs/conventions/formal-verification.yaml, her values as defaults; a per-model bound is an argument in that model's run command. The image renderer is a setting (default: her Pele 1.9.0), used only when an image is asked for.
    • Contract (reference/formal-skill.md): frontmatter; sections Property, Harm, Tool and why, Bounds and assumptions, Not covered, Run, Reading a result, Updating; one sync.yaml entry per check (id, covered paths, sync commit, command, recorded seeded break). Model files default to formal-<area>/model/; paths outside it are allowed only when listed in sync.yaml, and unattended writes are limited to the folder plus those paths.
    • First model: one item claimed at once by a default claim, a lease-only claim and a triage-phase claim on the GitHub adapter, with lease expiry, release and the completed-run rule. Renew, reclaim and other adapters are left out. The race is not "two lanes": two issue-lane runs on one issue are already serialized.
    • Invariants: the agent drafts them, each quoting the contract or code line it encodes; the owner approves the list before the model is checked. This is the one step in this slice that needs the owner.
    • A real defect is a counterexample replayed step by step against claim.sh; the contract's three documented limits are reported as known limits; a defect is written up through /bugs:write.
    • The model ships as this repository's first formal-<area> skill under .claude/skills/.
    • Before anything ships: a full read of her scripts by fresh-context agent reviewers, one per script group, and the plugin-acceptance security review recorded as a new section of ADR 0093.
    • Evidence: the chart's Mermaid text goes in the pull request's one evidence comment after the verifier passes.

    Corrections to the body: the pstack record's three rows change from "Pending adoption" to adopted (they are not parked); the claim protocol is defined in plugins/work-items/tools/work-item-tracker/CONTRACT.md "Lease protocol", not in the loop-lane README.

  10. 9 remaining items

  11. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded.

    Owner, 2026-11-16, in session ("Go w/ recommendations"), posted by an agent session. Two done-when lines are accepted as #16458 delivers them:

    #16458 closes this issue on merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.

    Type

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions