From 20e3a76846f3fa2332b662ff0fe018808a5023ae Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 15 Nov 2026 19:50:20 -0500 Subject: [PATCH 01/13] fix(work-items): release by the claim arbitration rule and hand back the assignee release and renew-lease in the GitHub adapter accept the holder's live lease when it is the earliest live lease (claim step 4, the completed-run rule included), through one shared wit_lease_winner in lib/lease.sh, so a newer lease that has not backed off no longer blocks a finished session's hand-back. release now removes the holder's assignee unless another live lease of that login remains, and re-reads the leases after the removal to restore it; a winning claim restores its own assignee too. A same-login losing claim keeps @me only when the login was assigned before it started, and a default claim that fails after its post supersedes its lease. The attend-queue, triage and work skills rely on release for the assignee. Closes #16472 Co-Authored-By: Claude Opus 5.5 --- ...-work-items-lease-release-rule-30317b4d.md | 9 + .../aq-answer-write-back-and-flip/case.yaml | 2 +- .../aq-answer-write-back-and-flip/prompt.md | 4 +- .../aq-release-lease-after-flip/case.yaml | 2 +- .../aq-release-lease-after-flip/prompt.md | 4 +- .../work-items/skills/attend-queue/SKILL.md | 28 ++- plugins/work-items/skills/triage/SKILL.md | 7 +- plugins/work-items/skills/work/SKILL.md | 3 +- .../tools/work-item-tracker/CONTRACT.md | 32 ++- .../adapters/github/README.md | 4 +- .../adapters/github/claim.sh | 100 +++------ .../adapters/github/common.sh | 68 ++++++ .../github/lease-coordination.test.sh | 209 ++++++++++++++++-- .../adapters/github/release.sh | 57 +++-- .../adapters/github/renew-lease.sh | 26 +-- .../tools/work-item-tracker/lib/lease.sh | 28 +++ .../tools/work-item-tracker/lib/lease.test.sh | 19 ++ 17 files changed, 454 insertions(+), 148 deletions(-) create mode 100644 .changes/work-items/fix-work-items-lease-release-rule-30317b4d.md diff --git a/.changes/work-items/fix-work-items-lease-release-rule-30317b4d.md b/.changes/work-items/fix-work-items-lease-release-rule-30317b4d.md new file mode 100644 index 00000000000..e332727cd4f --- /dev/null +++ b/.changes/work-items/fix-work-items-lease-release-rule-30317b4d.md @@ -0,0 +1,9 @@ +--- +bump: patch +--- + +### Fixed + +- The GitHub adapter's `release` and `renew-lease` accept the holder's live lease while a newer lease that has not yet backed off is on the item, and refuse a lease an earlier live lease outranks. They use the rule claim arbitration uses (earliest live lease holds, a completed run's lease counts as released), now one shared `wit_lease_winner` function. Before, a holder's release under such a lease was refused and its lease blocked every later claim until its TTL. +- `release` now hands the assignee back itself: it removes the login unless another live lease held by that login remains, so one same-login session's hand-back no longer unassigns another session's live claim. `/work-items:attend-queue`, `/work-items:triage` and `/work-items:work` rely on it and no longer clear `@me` themselves; attend-queue releases right after the flip. +- A same-login `claim` that loses arbitration keeps `@me` only when the login was assigned before the claim, so it no longer leaves an assignee behind that no lease holds. A winning claim puts its assignee back if a same-login release removed it, and a default claim that fails after posting its lease supersedes that lease, as a lease-only claim already did. diff --git a/plugins/work-items/evals/aq-answer-write-back-and-flip/case.yaml b/plugins/work-items/evals/aq-answer-write-back-and-flip/case.yaml index 7ddc8e8814b..5c3a2ffabe6 100644 --- a/plugins/work-items/evals/aq-answer-write-back-and-flip/case.yaml +++ b/plugins/work-items/evals/aq-answer-write-back-and-flip/case.yaml @@ -8,4 +8,4 @@ graders: target: 'last_message' - name: 'meets-expectations' type: 'llm' - criteria: 'Read the reply as the account of what the /work-items:attend-queue session does with the stated state. PASS only if it satisfies every point: (1) Acquires the seam claim (assignee + lease, the same protocol as /work-items:work) on the [escalated] row #720 before any comment or label change on it, and leaves the [intake] row''s claim to /work-items:triage, which claims the item it writes to; (2) Uses /planning:interview behind an installed-ness gate, with inline focused questions, one at a time, as the fallback; (3) Writes the operator''s answer back as an issue comment on the escalated item rather than leaving the decision only in the session; (4) Flips the escalated item in ONE edit that applies the autonomous-eligible role and removes the human-gated role while still holding the claim, both resolved from config.role_labels, then clears @me via the adapter assignee edit; never clears the assignee before the flip; (5) Returns the flipped escalated item to the frontier without executing or dispatching it from this lane; (6) Runs intake triage through /work-items:triage''s interactive direction gate: recommend, wait for direction, then mutate; (7) Prefixes agent-authored comments with the AI disclaimer; (8) Never merges a PR and never executes a work item from this lane. A point counts only when the reply states it or plainly acts on it; it need not use the same words. FAIL if any point is missing or contradicted.' + criteria: 'Read the reply as the account of what the /work-items:attend-queue session does with the stated state. PASS only if it satisfies every point: (1) Acquires the seam claim (assignee + lease, the same protocol as /work-items:work) on the [escalated] row #720 before any comment or label change on it, and leaves the [intake] row''s claim to /work-items:triage, which claims the item it writes to; (2) Uses /planning:interview behind an installed-ness gate, with inline focused questions, one at a time, as the fallback; (3) Writes the operator''s answer back as an issue comment on the escalated item rather than leaving the decision only in the session; (4) Flips the escalated item in ONE edit that applies the autonomous-eligible role and removes the human-gated role while still holding the claim, both resolved from config.role_labels, then releases the claim, which clears @me; never releases before the flip; (5) Returns the flipped escalated item to the frontier without executing or dispatching it from this lane; (6) Runs intake triage through /work-items:triage''s interactive direction gate: recommend, wait for direction, then mutate; (7) Prefixes agent-authored comments with the AI disclaimer; (8) Never merges a PR and never executes a work item from this lane. A point counts only when the reply states it or plainly acts on it; it need not use the same words. FAIL if any point is missing or contradicted.' diff --git a/plugins/work-items/evals/aq-answer-write-back-and-flip/prompt.md b/plugins/work-items/evals/aq-answer-write-back-and-flip/prompt.md index 56179e0d9b4..394e9fa3824 100644 --- a/plugins/work-items/evals/aq-answer-write-back-and-flip/prompt.md +++ b/plugins/work-items/evals/aq-answer-write-back-and-flip/prompt.md @@ -1,10 +1,10 @@ --- -description: 'Legacy attend-queue 2. An [escalated] row and an [intake] row worked in one pass: the lane claims the escalated row, interviews, writes the answer back as a comment, flips it in one edit while claimed and clears @me; the intake row goes to /work-items:triage under its interactive gate; comments carry the AI disclaimer from the per-user layer. disclaimer-true reads the reply. Read-only tools, graded on the reply.' +description: 'Legacy attend-queue 2. An [escalated] row and an [intake] row worked in one pass: the lane claims the escalated row, interviews, writes the answer back as a comment, flips it in one edit while claimed and releases the claim, which clears @me; the intake row goes to /work-items:triage under its interactive gate; comments carry the AI disclaimer from the per-user layer. disclaimer-true reads the reply. Read-only tools, graded on the reply.' tags: [attend-queue, escalation, flip, disclaimer, hard] max_turns: 15 timeout_seconds: 420 allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: 'For #720 the session acquires the seam claim (assignee + lease) before any comment or label change. It drives the decision via /planning:interview when the planning plugin is installed (inline focused questions, one at a time, as the fallback) and writes the answer back as an issue comment on the item, then flips it in one edit while the claim is held: the autonomous-eligible role label applied and the human-gated role label removed, both resolved from config.role_labels, then @me cleared via the adapter assignee edit and the lease released. The item re-enters the worker loop''s frontier; this lane never dispatches or executes it. For #721 the lane takes no claim itself: /work-items:triage claims it and runs under its interactive direction gate, recommendation first, mutation only after the operator''s direction. ai_disclaimer resolves true from the per-user layer, so agent-authored comments carry the AI disclaimer. Nothing is merged.' +expected_outcome: 'For #720 the session acquires the seam claim (assignee + lease) before any comment or label change. It drives the decision via /planning:interview when the planning plugin is installed (inline focused questions, one at a time, as the fallback) and writes the answer back as an issue comment on the item, then flips it in one edit while the claim is held: the autonomous-eligible role label applied and the human-gated role label removed, both resolved from config.role_labels, then the lease released, which clears @me. The item re-enters the worker loop''s frontier; this lane never dispatches or executes it. For #721 the lane takes no claim itself: /work-items:triage claims it and runs under its interactive direction gate, recommendation first, mutation only after the operator''s direction. ai_disclaimer resolves true from the per-user layer, so agent-authored comments carry the AI disclaimer. Nothing is merged.' --- /work-items:attend-queue diff --git a/plugins/work-items/evals/aq-release-lease-after-flip/case.yaml b/plugins/work-items/evals/aq-release-lease-after-flip/case.yaml index 16e801f09f0..ee4ae4efb45 100644 --- a/plugins/work-items/evals/aq-release-lease-after-flip/case.yaml +++ b/plugins/work-items/evals/aq-release-lease-after-flip/case.yaml @@ -8,4 +8,4 @@ graders: target: 'last_message' - name: 'meets-expectations' type: 'llm' - criteria: 'Read the reply as the account of what the /work-items:attend-queue session does with the stated state. PASS only if it satisfies every point: (1) Performs the single-edit role flip while the claim is held, before clearing @me; (2) Clears @me through the adapter''s assignee edit after the flip; (3) Runs the seam release verb with the claim''s lease_comment_id after the assignee clear; (4) Never hand-writes or PATCHes lease-comment JSON directly. A point counts only when the reply states it or plainly acts on it; it need not use the same words. FAIL if any point is missing or contradicted.' + criteria: 'Read the reply as the account of what the /work-items:attend-queue session does with the stated state. PASS only if it satisfies every point: (1) Performs the single-edit role flip while the claim is held, before the release; (2) Runs the seam release verb with the claim''s lease_comment_id after the flip; (3) Relies on that release to clear @me rather than clearing the assignee itself before the release; (4) Never hand-writes or PATCHes lease-comment JSON directly. A point counts only when the reply states it or plainly acts on it; it need not use the same words. FAIL if any point is missing or contradicted.' diff --git a/plugins/work-items/evals/aq-release-lease-after-flip/prompt.md b/plugins/work-items/evals/aq-release-lease-after-flip/prompt.md index cf54df005d5..03a10fb5103 100644 --- a/plugins/work-items/evals/aq-release-lease-after-flip/prompt.md +++ b/plugins/work-items/evals/aq-release-lease-after-flip/prompt.md @@ -1,10 +1,10 @@ --- -description: 'Legacy attend-queue 7. After an answer removes the blocker on a claimed [escalated] row: flip in one edit while claimed, clear @me through the adapter, then end the lease with the seam release verb and the claim''s lease_comment_id rather than leaving it to its TTL or hand-writing lease JSON. release-with-lease-id reads the reply. Read-only tools, graded on the reply.' +description: 'Legacy attend-queue 7. After an answer removes the blocker on a claimed [escalated] row: flip in one edit while claimed, then end the lease with the seam release verb and the claim''s lease_comment_id, which also clears @me, rather than leaving it to its TTL or hand-writing lease JSON. release-with-lease-id reads the reply. Read-only tools, graded on the reply.' tags: [attend-queue, lease, release, routine] max_turns: 15 timeout_seconds: 420 allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: 'While still holding the claim, the session applies the autonomous-eligible role label and removes the human-gated one in a single edit, then clears @me through the adapter''s assignee edit, then runs the seam release verb, work-item-tracker.sh release "github:/#3711" --lease-comment-id 5858800001. It does not leave the lease to run out its TTL, because claim backs off from any earlier live lease, the same login''s included. It reads exit 0 with either released value as done, and never hand-writes lease-comment JSON.' +expected_outcome: 'While still holding the claim, the session applies the autonomous-eligible role label and removes the human-gated one in a single edit, then runs the seam release verb, work-item-tracker.sh release "github:/#3711" --lease-comment-id 5858800001, which also clears @me; it clears no assignee itself. It does not leave the lease to run out its TTL, because claim backs off from any earlier live lease, the same login''s included. It reads exit 0 with either released value as done, and never hand-writes lease-comment JSON.' --- /work-items:attend-queue diff --git a/plugins/work-items/skills/attend-queue/SKILL.md b/plugins/work-items/skills/attend-queue/SKILL.md index 9d150b50aa2..33fd90324aa 100644 --- a/plugins/work-items/skills/attend-queue/SKILL.md +++ b/plugins/work-items/skills/attend-queue/SKILL.md @@ -155,7 +155,7 @@ same idempotent stale-lease sweep `/work-items:work` Step 0 uses: enumerate assi each `number` to a fully-qualified id, `"$TRACKER" reclaim ""` on each. Exit `6` (capability-unsupported) skips the sweep for providers that declare `reclaim: false`. -**Clear assignee after disposition (flip while claimed), then release.** The claim is a +**Release after disposition (flip while claimed).** The claim is a coordination lock, not an execution assignment: hand it back as soon as the row is done, since `claim` backs off from any earlier live lease, the same login's included (the seam's `CONTRACT.md` "Lease protocol", claim step 4), and a lease left to run out its TTL keeps every @@ -163,25 +163,23 @@ worker lane off the flipped item. Once the row's answer is written (triage hands back the `[intake]` rows it claims itself): - **When the human blocker is removed:** perform the single-edit role-label flip **while this - session still holds the claim**, then clear `@me` from assignees via the bound adapter's assignee - edit (for GitHub, the adapter README "Edit labels / assignees"; the flag is read live at - [gh issue edit § Options](https://cli.github.com/manual/gh_issue_edit#options)), then end the lease with - `"$TRACKER" release "" --lease-comment-id `. Never clear the assignee before the flip: a - released row still reads as `[escalated]` in another attended session's view until the label - lands. A brief frontier delay after the flip is acceptable. + session still holds the claim**, then end the lease with + `"$TRACKER" release "" --lease-comment-id `, which also clears `@me` unless another live + lease of this login holds the item. Never release before the flip: a released row still reads as + `[escalated]` in another attended session's view until the label lands. A brief frontier delay + after the flip is acceptable. - **When disposition leaves the item human-gated** (decline, an answered escalation on a human-floor work class that was not reclassified; only `provenance: untrusted` and `provenance: signal` floor an item, never a structural class or a missing one, per [`${CLAUDE_PLUGIN_ROOT}/reference/work-class-labels.md`](${CLAUDE_PLUGIN_ROOT}/reference/work-class-labels.md) - "The human-floor class excludes the autonomous-eligible role label"): clear `@me` via the - same adapter assignee edit once disposition comments are written, still while holding the claim - through those writes, then `release` the same way. + "The human-floor class excludes the autonomous-eligible role label"): hold the claim through + the disposition comments, then `release` the same way. - **Release outcomes.** Exit `0` with either `released` value is done: `false` means the lease had already expired or been superseded. Exit `6` (the provider declares `release: false`) leaves the lease comment live until TTL expiry, and a later session-start `reclaim` clears it once expired. - Exit `7` means the handle is no longer this session's active lease; leave it. Never hand-roll - lease-comment JSON in place of the verb. -- A row skipped on exit `7` needs no assignee clear and no release. + Exit `7` means the handle is no longer this session's lease, or an earlier live lease outranks + it; leave it. Never hand-roll lease-comment JSON in place of the verb. +- A row skipped on exit `7` needs no release. **Long operator waits.** When an interview spans longer than the binding's lease TTL, renew the held lease via `"$TRACKER" renew-lease "" --lease-comment-id ` (the `claim` output carries @@ -362,7 +360,7 @@ Two further reader-contract rules apply alongside the floor (outside the byte-au parser; never string-interpolate them into a shell command, another interpreter, or a prompt. For this attended lane, when opted in, "stop claiming new work" means: finish the row in hand -(including the flip-while-claimed and assignee clear when disposition is complete), then stop +(including the flip-while-claimed and release when disposition is complete), then stop pulling further rows and report the pause to the operator, who may explicitly choose to continue (the "explicit user request" the hard-stop rule names). This lane keeps no durable state, so the latched account is held in the session only: apply the **Account switch** bullet on each Monitor @@ -406,7 +404,7 @@ placeholder before running a command or writing it into a brief. claim and the topology's single-authority rule. - **Claim before mutate, flip while claimed.** The seam `claim` arbitrates two attended sessions on one row (exit `7` → skip). The single-edit role-label flip must land while the claim is still - held; only then clear `@me` via the adapter assignee edit, and then `release` the lease. + held; only then `release` the lease, which clears `@me`. - **Do not re-triage routed items.** The `[intake]` source is `/work-items:triage`'s attention view by composition; items already carrying a routing outcome are out of scope by construction, and naming one explicitly gets triage's "already triaged" stop. diff --git a/plugins/work-items/skills/triage/SKILL.md b/plugins/work-items/skills/triage/SKILL.md index 48e4862af0f..30c1c4d041d 100644 --- a/plugins/work-items/skills/triage/SKILL.md +++ b/plugins/work-items/skills/triage/SKILL.md @@ -139,10 +139,9 @@ reporter-info listing, the sweep's list and count, and every read before the fir name the lost item in the report. 3. **Hand back** each claimed item after its last write, including on a run that stops partway, and each claimed item an aborted outcome never wrote to. First - `"$TRACKER" release "" --lease-comment-id `. Exit `0` with either `released` value: then - clear the session's own assignee through the adapter's assignee edit (GitHub: - `--remove-assignee "@me"`), because `claim` assigned it, `release` never touches assignees, and - the frontier lists only unassigned items. Exit `6` (the provider declares `release: false`, as + `"$TRACKER" release "" --lease-comment-id `. Exit `0` with either `released` value is + done: `release` also clears the assignee `claim` set, unless another live lease of this login + still holds the item. Exit `6` (the provider declares `release: false`, as local-markdown and linear do): leave both the lease and the assignee. Clearing the assignee under a live lease lists the item on the frontier while every `claim` on it fails with exit `7`; instead the lease runs out its TTL, after which local-markdown's `list-items` reports the item diff --git a/plugins/work-items/skills/work/SKILL.md b/plugins/work-items/skills/work/SKILL.md index e4c1b0633dc..670cd953f5c 100644 --- a/plugins/work-items/skills/work/SKILL.md +++ b/plugins/work-items/skills/work/SKILL.md @@ -340,5 +340,4 @@ no fix and no PR. When the diagnosis names no single cause, rank the candidates escalating. Then post the [report](#bug-report). A `not-reproduced` outcome then escalates, as step 2 says, citing the report. Then release the -claim (seam `release`), clear this session's own assignee through the adapter's assignee edit, -since release never touches assignees and the frontier excludes an assigned item, and end the run. +claim (seam `release`, which also clears this session's assignee) and end the run. diff --git a/plugins/work-items/tools/work-item-tracker/CONTRACT.md b/plugins/work-items/tools/work-item-tracker/CONTRACT.md index 681f9ada229..e976f3e850a 100644 --- a/plugins/work-items/tools/work-item-tracker/CONTRACT.md +++ b/plugins/work-items/tools/work-item-tracker/CONTRACT.md @@ -665,7 +665,8 @@ issue comment with a machine marker. The GitHub adapter's `claim` has two modes: (steps 1 to 5 below). Local sessions use this mode. - **Lease-only (`--lease-only`):** the claim posts the lease comment and assigns no one. It skips steps 1 and 2, and on a lost arbitration it supersedes its own comment and touches no assignee. - A failure after the comment is posted and before the emit also supersedes it. + In either mode, a failure after the comment is posted and before the emit also supersedes it + (and, in default mode, rolls back the assignment). `--holder ` names the holder instead of the authenticated login, for a caller whose token cannot read `GET /user` (a GitHub App installation token, which a CI job holds). The issue lane claims this way, holding `vars.AUTOMATION_LANES_APP_SENDER_ID`, before its model starts. @@ -697,6 +698,9 @@ backs off, and a lane run that finds a live local lease backs off without starti (non-superseded) lease but already **expired**, meaning `renewed_at + ttl_hours` elapsed with no back-off yet, is refused (exit `7`), never revived: a delayed holder must not undo a TTL-based handoff. Recovery from an expired lease is a fresh claim/reclaim, not a renew. + The GitHub adapter's `renew-lease` also refuses (exit `7`) a live lease that an earlier live + lease outranks under claim step 4; a newer live lease does not stop it, since that is a + claimant that has not backed off yet. - `session_id` is optional and collision-prone; a missing/duplicate `session_id` still counts as a competing lease. The one shape the GitHub adapter's claim step 4 reads is `run-`, a workflow run of the item's repository; any other value is diagnostic only. @@ -714,14 +718,16 @@ backs off, and a lane run that finds a live local lease backs off without starti Claim sequence (race-safe, same-identity aware): -1. Assign the authenticated user (`--add-assignee "@me"`, always the session identity, - never the bot). +1. Note whether the authenticated user is already assigned, then assign it + (`--add-assignee "@me"`, always the session identity, never the bot). 2. Re-read assignees. Any other login present → back off: unassign self, exit `7`. 3. Post the lease comment; capture its comment ID (comment identity, not `session_id`, discriminates same-login sessions). 4. Re-read all lease comments. If an earlier live lease exists that is not our own - comment, the foreign lease wins → supersede own comment, exit `7`. (Assignee is left - in place on a same-login race: it belongs to the winner.) + comment, the foreign lease wins → supersede own comment, exit `7`. On a same-login loss the + assignee stays only when step 1 found it already assigned, since it is then the winner's; + one this claim added is removed, so it cannot outlive a winner that has finished. On a win, + re-read assignees and assign self again if a same-login release removed it after step 2. The back-off message prints the winner's holder with control characters removed. GitHub adapter: an earlier live lease whose `session_id` is `run-` costs one `GET /repos/{owner}/{repo}/actions/runs/{id}` read (the token needs `actions: read`). A run @@ -736,15 +742,19 @@ elapses, so a session that has finished with an item hands it back at once inste every later claim, same-login ones included (claim step 4), until expiry. It acts only on a lease that is all of: on the addressed item, held by the authenticated login (or the `--holder` value, which, like `claim`'s, replaces the `gh api user` lookup and is asserted by the caller; the comment -edit still needs the token's write access), and the item's active -(newest non-superseded) lease. Anything else is refused with exit `7` and nothing is written: a -cross-item handle, another login's lease, or a stale handle under a newer claim. The login check +edit still needs the token's write access), and, for a live lease, not outranked by an earlier +live lease under claim step 4 (the completed-run rule included); a newer live lease is a claimant +that has not backed off yet and does not stop the release. Anything else is refused with exit `7` +and nothing is written: a cross-item handle, another login's lease, a live handle an earlier live +lease outranks, or an expired handle under a newer non-superseded lease. The login check exists because lease handles are public (a GitHub comment id), and within one login the handle tells sessions apart, as for `renew-lease`. An already-superseded or expired lease is an idempotent no-op (`released: false`, exit `0`), because it already blocks nobody. A live lease is -superseded in place (`released: true`). Release never touches assignees: the caller clears its -own assignee through the adapter's assignee edit, and `/work-items:attend-queue` flips the role -label, clears `@me`, then releases. The write carries the same TOCTOU caveat as reclaim's +superseded in place (`released: true`). On every exit `0` release then hands the assignee back: +it removes the login unless another live lease held by that login remains on the item. The GitHub +adapter re-reads the leases after the removal and assigns the login again if one has appeared, +the other half of claim step 4's win check. Callers rely on it and clear no assignee +themselves; `/work-items:attend-queue` flips the role label, then releases. The write carries the same TOCTOU caveat as reclaim's revalidation: GitHub's comment PATCH has no If-Match/CAS, so a concurrent writer can still interleave. The verb is capability-gated (`verbs.release`); an adapter that declares it `false`, or a manifest that predates it and has no key, exits `6`, and the caller falls back to waiting out diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/README.md b/plugins/work-items/tools/work-item-tracker/adapters/github/README.md index 3bfd7ae9ca4..2750e673e65 100644 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/README.md +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/README.md @@ -318,8 +318,8 @@ gh issue edit --repo / --remove-assignee "@me" # clear this s user (not a bot), so it runs on bare `gh`. Coordination claims go through the seam `claim` verb, which owns this. -**The lease protocol's own assignee ops are REST, not `gh issue`.** `claim` and `reclaim` do not -use `gh issue edit --add-assignee` / `gh issue view --json assignees`: those route through +**The lease protocol's own assignee ops are REST, not `gh issue`.** `claim`, `release` and +`reclaim` do not use `gh issue edit --add-assignee` / `gh issue view --json assignees`: those route through GitHub's GraphQL API, which made the lease protocol unrunnable in cloud sessions (see "View item" above). They use `gh api` against `…/issues//assignees` instead, through the `wit_read_assignees` / `wit_add_assignee` / `wit_remove_assignee` / `wit_try_remove_assignee` diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/claim.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/claim.sh index 873aa565c95..655502bcd2f 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/claim.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/claim.sh @@ -74,6 +74,12 @@ fi # A lease-only claim assigns no one, so steps 1-2 and their rollback do not apply. if [[ "$lease_only" == false ]]; then + # Whether the login was assigned before this claim decides, on a same-login loss in + # step 4, whether @me belongs to the winner or only to this claim. + wit_read_assignees "$owner" "$repo" "$number" + pre_assigned=false + jq -e --arg me "$login" 'any(.[]; . == $me)' <<<"$WIT_GH_OUT" >/dev/null && pre_assigned=true + # 1. Assign the session identity. `read` routes to bare gh, so the assignee is # the session user, not the bot (README "Edit labels / assignees" carve-out); # `@me` is resolved to $login explicitly because REST takes a literal login. @@ -125,88 +131,54 @@ wit_run_gh write api "repos/$owner/$repo/issues/$number/comments" \ -f body="${WIT_LEASE_MARKER}${lease} -->" --jq '.id' our_comment_id="$WIT_GH_OUT" -# supersede_lease_comment : best-effort edit adding -# superseded_at to that lease comment; never fails the caller. -supersede_lease_comment() { - local superseded - superseded="$(jq -c --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" '. + {superseded_at: $ts}' <<<"$2")" - gh_write api --method PATCH "repos/$owner/$repo/issues/comments/$1" \ - -f body="${WIT_LEASE_MARKER}${superseded} -->" >/dev/null 2>&1 || true -} - -# A lease-only claim that fails before its emit supersedes its own comment, so a -# lease whose handle never reached the caller does not block the item for its TTL. -[[ "$lease_only" == false ]] || trap 'supersede_lease_comment "$our_comment_id" "$lease"' EXIT - -# run_completed : 0 when that workflow run of the item's repository has -# completed (any conclusion); 1 for any other status or a failed read, which prints -# one line and keeps the lease winning. Bare gh, not wit_run_gh: a failed read must -# back off (exit 7), never fail the claim. -run_completed() { - local status - status="$(gh api "repos/$owner/$repo/actions/runs/$1" --jq '.status' 2>/dev/null)" || { - printf 'claim: could not read run %s status; lease kept\n' "$1" >&2 - return 1 - } - [[ "$(wit_strip_cr <<<"$status")" == completed ]] +# A claim that fails before its emit supersedes its own comment, so a lease whose +# handle never reached the caller does not block the item for its TTL; a default +# claim also rolls back its assignment. +_wit_claim_abort() { + wit_try_supersede_lease_comment "$owner" "$repo" "$our_comment_id" "$lease" + [[ "$lease_only" == true ]] || _wit_claim_rollback } +trap _wit_claim_abort EXIT # 4. Re-read all lease comments; the EARLIEST live lease wins, except a lease held -# by a completed workflow run: session_id run- costs one run_completed read, -# and a completed run's lease is superseded in place. The rollback -# guard STAYS armed through arbitration: a read failure here (exit "$?" below) -# would otherwise leave @me assigned with a live lease comment whose id was -# never returned to the caller — stranded exactly like the pre-lease window, -# just one step later. -now_epoch="$(date -u +%s)" +# by a completed workflow run (wit_lease_released_by_run). The abort guard STAYS +# armed through arbitration: a read failure here (exit "$?" below) would otherwise +# leave @me assigned with a live lease comment whose id was never returned to the +# caller. # $() swallows wit_run_gh's exit-on-error; propagate so a read failure here does # not silently arbitrate the claim against an empty lease set. leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || exit "$?" -winner_id="" +wit_lease_winner "$leases" "$(date -u +%s)" "$our_comment_id" wit_lease_released_by_run +winner_id="$WIT_WINNER_ID" winner_holder="" -while IFS= read -r row; do - [[ -n "$row" ]] || continue - lease_json="$(wit_lease_json "$(jq -r '.body' <<<"$row")")" - [[ -n "$lease_json" ]] || continue - if wit_lease_is_live "$lease_json" "$now_epoch"; then - row_id="$(jq -r '.id' <<<"$row")" - sid="$(jq -r '.session_id // empty' <<<"$lease_json")" - if [[ "$row_id" != "$our_comment_id" && "$sid" =~ ^run-([1-9][0-9]{0,19})$ ]]; then - run_id="${BASH_REMATCH[1]}" - if run_completed "$run_id"; then - supersede_lease_comment "$row_id" "$lease_json" - printf 'claim: lease comment %s was held by run %s, which has completed; treated as released\n' "$row_id" "$run_id" >&2 - continue - fi - fi - winner_id="$row_id" - winner_holder="$(jq -r '.holder // empty' <<<"$lease_json")" - break - fi -done < <(jq -c 'sort_by(.id) | .[]' <<<"$leases") +[[ -z "$WIT_WINNER_JSON" ]] || winner_holder="$(jq -r '.holder // empty' <<<"$WIT_WINNER_JSON")" # Earliest LIVE lease wins. No live lease at all (e.g. ttl 0) → our claim stands; # expiry is reclaim's concern. if [[ -n "$winner_id" && "$winner_id" != "$our_comment_id" ]]; then - supersede_lease_comment "$our_comment_id" "$lease" - # Keep the @me assignment only on a SAME-login race (the winner shares our - # login, so the assignee still matches the holder). If the winning lease - # belongs to a different holder, our @me would leave the item assigned to the - # loser while the holder no longer matches — drop it (the rollback trap - # would also catch this on exit; the explicit call keeps the disarm/removal - # decision made in this block, not left implicit to the trap). - if [[ "$lease_only" == true ]]; then - trap - EXIT # nothing assigned; our comment is already superseded - elif [[ "$winner_holder" != "$login" ]]; then + wit_try_supersede_lease_comment "$owner" "$repo" "$our_comment_id" "$lease" + trap - EXIT + # Keep @me only on a same-login loss where the login was assigned before this + # claim: then the assignee is the winner's. Assigned only by this claim, it would + # outlive a winner that has finished or is about to release. + if [[ "$lease_only" == false ]] && [[ "$winner_holder" != "$login" || "$pre_assigned" == false ]]; then wit_try_remove_assignee "$owner" "$repo" "$number" "$login" - else - trap - EXIT # same-login race — @me legitimately stays assigned fi printf 'claim: live lease (comment %s, holder %s) wins — backing off\n' \ "$winner_id" "$(tr -d '[:cntrl:]' <<<"${winner_holder:-unknown}")" >&2 exit "$EX_CONFLICT" fi +# A same-login release that ran between step 2 and our post can have removed @me +# (it saw no other live lease of ours yet); put it back. The release re-reads the +# leases after its removal, so between the two checks one sees the other's write. +if [[ "$lease_only" == false ]]; then + wit_read_assignees "$owner" "$repo" "$number" + if ! jq -e --arg me "$login" 'any(.[]; . == $me)' <<<"$WIT_GH_OUT" >/dev/null; then + wit_add_assignee read "$owner" "$repo" "$number" "$login" + fi +fi + trap - EXIT # arbitration confirms our lease is the winner — claim complete jq -cn --arg sv "$WIT_SCHEMA_VERSION" --arg id "$id" --arg holder "$login" \ --arg now "$now" --arg ttl "$ttl" --arg cid "$our_comment_id" --arg sid "$session_id" \ diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh index 4105fd802c6..741d29bf313 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh @@ -444,3 +444,71 @@ wit_list_lease_comments() { jq -cs 'add // [] | map(select(.author_association == "OWNER" or .author_association == "MEMBER" or .author_association == "COLLABORATOR" or .user.type == "Bot"))' <<<"$WIT_GH_OUT" } + +# wit_try_supersede_lease_comment : best-effort +# edit adding superseded_at to that lease comment; never fails the caller. +wit_try_supersede_lease_comment() { + local superseded + superseded="$(jq -c --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" '. + {superseded_at: $ts}' <<<"$4")" + gh_write api --method PATCH "repos/$1/$2/issues/comments/$3" \ + -f body="${WIT_LEASE_MARKER}${superseded} -->" >/dev/null 2>&1 || true +} + +# wit_lease_released_by_run : the every lease verb +# hands wit_lease_winner (claim step 4). 0 when the lease's session_id is run- and that +# workflow run of the parsed id's repository has completed (any conclusion); the lease is +# then superseded in place, best-effort. Any other session_id costs no read; another run +# status, or a failed read, returns 1 and keeps the lease winning. Bare gh, not wit_run_gh: +# a failed read must keep the lease, never fail the verb. +wit_lease_released_by_run() { + local sid run_id status verb + sid="$(jq -r '.session_id // empty' <<<"$2")" + [[ "$sid" =~ ^run-([1-9][0-9]{0,19})$ ]] || return 1 + run_id="${BASH_REMATCH[1]}" + verb="$(basename "$0" .sh)" + status="$(gh api "repos/$WIT_ID_OWNER/$WIT_ID_REPO/actions/runs/$run_id" --jq '.status' 2>/dev/null)" || { + printf '%s: could not read run %s status; lease kept\n' "$verb" "$run_id" >&2 + return 1 + } + [[ "$(wit_strip_cr <<<"$status")" == completed ]] || return 1 + wit_try_supersede_lease_comment "$WIT_ID_OWNER" "$WIT_ID_REPO" "$1" "$2" + printf '%s: lease comment %s was held by run %s, which has completed; treated as released\n' \ + "$verb" "$1" "$run_id" >&2 +} + +# wit_unassign_if_unheld : after a +# release, remove from the assignees unless another live lease held by remains +# (CONTRACT.md "Lease protocol", release). is a list read before the release's +# PATCH, so a lease posted after it is caught by the re-read after the removal, which puts the +# assignee back: a same-login claim posts its lease before it re-checks its assignee, and the +# release removes the assignee before it re-reads the leases, so one of the two sees the other. +wit_unassign_if_unheld() { + local owner="$1" repo="$2" number="$3" login="$4" released="$5" leases="$6" assignees + _wit_login_holds_other "$leases" "$login" "$released" && return 0 + assignees="$(gh api "repos/$owner/$repo/issues/$number" --jq '[.assignees[].login]' 2>/dev/null)" || { + printf '%s: could not read the assignees; %s left assigned\n' "$(basename "$0" .sh)" "$login" >&2 + return 0 + } + jq -e --arg l "$login" 'any(.[]; . == $l)' <<<"$assignees" >/dev/null || return 0 + wit_try_remove_assignee "$owner" "$repo" "$number" "$login" + leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || return 0 + if _wit_login_holds_other "$leases" "$login" "$released"; then + wit_write_pause + gh api --method POST "repos/$owner/$repo/issues/$number/assignees" \ + -f "assignees[]=$login" >/dev/null 2>&1 || true + fi +} + +# _wit_login_holds_other : 0 when a live lease other than +# is held by . +_wit_login_holds_other() { + local row cand now + now="$(date -u +%s)" + while IFS= read -r row; do + [[ -n "$row" && "$(jq -r '.id' <<<"$row")" != "$3" ]] || continue + cand="$(wit_lease_json "$(jq -r '.body' <<<"$row")")" + [[ -n "$cand" && "$(jq -r '.holder // empty' <<<"$cand")" == "$2" ]] || continue + wit_lease_is_live "$cand" "$now" && return 0 + done < <(jq -c '.[]' <<<"$1") + return 1 +} diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh index e4c1752f596..a0edbf4447a 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh @@ -12,14 +12,17 @@ # arbitration), which claim.test.sh does not reach; # (4) release ends the caller's own live lease, so a same-login claim that # backed off on it succeeds afterwards, and refuses any lease that is not -# the caller's active one; +# the caller's or that an earlier live lease outranks; # (5) a lane's lease-only claim and release under a --holder override: a local # claim backs off while the lane lease is live and holds once it is # released, and a lane claim backs off on a live local lease; # (6) claim --phase writes the lease's phase and renew-lease keeps it; # (7) a lease held by a completed workflow run (session_id run-) is # treated as released, while a live run, a non-run session or an -# unreadable run status keeps the lease winning. +# unreadable run status keeps the lease winning; +# (8) release and renew act on the holder's lease under a newer lease not yet +# arbitrated, release unassigns only when no other live lease of the login +# remains, and a same-login loser leaves no assignee that outlives the leases. # shellcheck disable=SC2154 # FAILED/CASE_NUM initialized by the sourced lib set -uo pipefail # In a CI job the adapter wraps its gh calls and records into the job's @@ -60,6 +63,15 @@ stub_login_arg() { case "$a" in "assignees[]="*) printf '%s' "${a#assignees[]=}" ;; *) : ;; esac done } +# stub_hook : run $d/hook- once, if present, so a scenario can land another +# verb inside a given call: the n-th lease-list read (list-, before it answers) or the +# k-th assignee read (assignees-, after it answers). +stub_hook() { + local h="${GH_STUB_DIR:?}/hook-$1" + [[ -f "$h" ]] || return 0 + mv "$h" "$h.ran" + bash "$h.ran" >/dev/null 2>&1 +} gh() { local d="${GH_STUB_DIR:?}" args="$*" a n cid body login case "$args" in @@ -98,7 +110,12 @@ gh() { fi cat "$d/assignees" 2>/dev/null || printf '[]\n' ;; - *".assignees[].login"*) cat "$d/assignees" ;; + *".assignees[].login"*) + n=$(($(cat "$d/assignees.count" 2>/dev/null || printf '0') + 1)) + printf '%s\n' "$n" >"$d/assignees.count" + cat "$d/assignees" + stub_hook "assignees-$n" + ;; # A workflow run's status read; no $d/run-status file is an unreadable status. *"/actions/runs/"*) printf 'RUN_READ %s\n' "${args##*/actions/runs/}" >>"$d/calls.log" @@ -115,6 +132,7 @@ gh() { n=$(($(cat "$d/lease-comments.count" 2>/dev/null || printf '0') + 1)) printf '%s\n' "$n" >"$d/lease-comments.count" [[ ! -f "$d/list-fails-$n" ]] || exit 1 + stub_hook "list-$n" if [[ -f "$d/lease-comments-$n" ]]; then cat "$d/lease-comments-$n"; else cat "$d/lease-comments"; fi ;; esac @@ -154,7 +172,25 @@ gh() { ;; esac } -export -f gh stub_body_arg stub_login_arg +export -f gh stub_body_arg stub_login_arg stub_hook + +# hook : arm stub_hook to run , writing its exit code to +# hook-.rc. next_read : the hook name for the read +# calls after the reads made so far. +hook() { + local name="$1" + shift + { + printf '%q ' "$@" + printf '\nprintf "%%s\\n" "$?" >%q\n' "$GH_STUB_DIR/hook-$name.rc" + } >"$GH_STUB_DIR/hook-$name" +} +hook_rc() { cat "$GH_STUB_DIR/hook-$1.rc" 2>/dev/null || printf 'hook did not run\n'; } +next_read() { + local file=lease-comments.count + [[ "$1" == list ]] || file=assignees.count + printf '%s-%s' "$1" "$(($(cat "$GH_STUB_DIR/$file" 2>/dev/null || printf '0') + $2))" +} # marker [session_id] — echo a lease marker comment body. marker() { @@ -367,19 +403,30 @@ assert_eq "release refuses (7) a lease another login holds" "7" "$rc" assert_not_contains "release does not supersede a foreign lease" "$(calls)" "PATCH" cleanup_scenario -# [not the active lease] an older non-superseded lease under a newer claim. +# [outranked] two live leases: the earlier one holds the item (claim step 4) and the +# newer one is a claimant that has not backed off. The newer handle is refused; the +# earlier one is released. +two_live_leases() { + jq -cn --arg o "$(marker alice "$FUTURE" 24)" --arg n "$(marker alice "$FUTURE" 24)" \ + '[{id:100, node_id:"MDEx", body:$o, created_at:"2020-01-01T00:00:00Z", author_association:"MEMBER"}, + {id:200, node_id:"MDEx", body:$n, created_at:"2020-01-01T00:00:00Z", author_association:"MEMBER"}]' >"$GH_STUB_DIR/lease-comments" + jq -cn --arg b "$(marker alice "$FUTURE" 24)" '{body:$b, issue:"1"}' >"$GH_STUB_DIR/comment" +} +new_scenario +printf 'alice\n' >"$GH_STUB_DIR/login" +two_live_leases +rc=0 +bash "$RELEASE" "$ID" --lease-comment-id 200 >/dev/null 2>&1 || rc=$? +assert_eq "release refuses (7) a handle outranked by an earlier live lease" "7" "$rc" +assert_not_contains "release writes nothing for an outranked handle" "$(calls)" "PATCH" +cleanup_scenario new_scenario printf 'alice\n' >"$GH_STUB_DIR/login" -OLD_MARKER="$(marker alice "$FUTURE" 24)" -NEW_MARKER="$(marker alice "$FUTURE" 24)" -jq -cn --arg o "$OLD_MARKER" --arg n "$NEW_MARKER" \ - '[{id:100, node_id:"MDEx", body:$o, created_at:"2020-01-01T00:00:00Z", author_association:"MEMBER"}, - {id:200, node_id:"MDEx", body:$n, created_at:"2020-01-01T00:00:00Z", author_association:"MEMBER"}]' >"$GH_STUB_DIR/lease-comments" -jq -cn --arg b "$OLD_MARKER" '{body:$b, issue:"1"}' >"$GH_STUB_DIR/comment" +two_live_leases rc=0 bash "$RELEASE" "$ID" --lease-comment-id 100 >/dev/null 2>&1 || rc=$? -assert_eq "release refuses (7) a handle that is not the active lease" "7" "$rc" -assert_not_contains "release writes nothing for a stale handle" "$(calls)" "PATCH" +assert_eq "release succeeds (0) for the holder's lease under a newer live lease" "0" "$rc" +assert_contains "release PATCHes the holder's lease under a newer live lease" "$(calls)" "PATCH 100" cleanup_scenario # [cross-issue handle] the comment belongs to a different issue. @@ -645,4 +692,140 @@ three_lease_case() { three_lease_case "a dead run's lease before a live lane lease: the lane lease wins" run-100 lane-3 7 1 three_lease_case "a live lane lease before a dead run's lease wins with no run read" lane-3 run-100 7 0 +# =========================================================================== +# Finding 8 — the holder's release and renew act under a newer, not yet arbitrated +# lease; a same-login hand-back unassigns only when no other live lease of that login +# remains; a same-login loser does not strand an assignee the winner no longer holds. +# Each scenario starts with alice's attended claim holding lease 1. +# =========================================================================== +attended_scenario() { + new_scenario + : >"$GH_STUB_DIR/stateful" + printf 'alice\n' >"$GH_STUB_DIR/login" + printf '[]\n' >"$GH_STUB_DIR/assignees" + printf '[]\n' >"$GH_STUB_DIR/lease-comments" + bash "$CLAIM" "$ID" --ttl-hours 24 >/dev/null 2>&1 + assert_eq "attended claim holds lease 1 assigned to alice" '["alice"]' "$(jq -c . "$GH_STUB_DIR/assignees")" +} +assignees_now() { jq -c . "$GH_STUB_DIR/assignees"; } +superseded() { stored_lease "$1" | jq -r 'has("superseded_at")'; } + +# [release under a lane lease] the lane has posted lease 2 and its re-read is next; the +# holder releases lease 1 in that window. The release ends lease 1 and unassigns alice, +# and the lane's arbitration then finds lease 2 earliest and holds. +attended_scenario +h="$(next_read list 1)" +hook "$h" bash "$RELEASE" "$ID" --lease-comment-id 1 +rc=0 +out="$(bash "$CLAIM" "$ID" --ttl-hours 2 --lease-only --holder "$LANE" 2>/dev/null)" || rc=$? +assert_eq "the holder's release under an unarbitrated lane lease succeeds (0)" "0" "$(hook_rc "$h")" +assert_contains "that release PATCHes the holder's own lease comment" "$(calls)" "PATCH 1" +assert_eq "the holder's lease is superseded" "true" "$(superseded 1)" +assert_eq "the lane's claim then wins (0)" "0" "$rc" +assert_eq "the lane holds lease 2" "2" "$(jq -r '.lease_comment_id' <<<"$out")" +assert_eq "the released holder is unassigned" "[]" "$(assignees_now)" +cleanup_scenario + +# [renew under a lane lease] the same window with a renew: it renews lease 1, which keeps +# winning, so the lane backs off. +attended_scenario +sleep 1 +h="$(next_read list 1)" +hook "$h" bash "$RENEW" "$ID" --lease-comment-id 1 +rc=0 +bash "$CLAIM" "$ID" --ttl-hours 2 --lease-only --holder "$LANE" >/dev/null 2>&1 || rc=$? +assert_eq "the holder's renew under an unarbitrated lane lease succeeds (0)" "0" "$(hook_rc "$h")" +assert_contains "that renew PATCHes the holder's own lease comment" "$(calls)" "PATCH 1" +assert_eq "the renewed lease moved renewed_at past acquired_at" "true" "$(stored_lease 1 | jq -r '.renewed_at > .acquired_at')" +assert_eq "the lane backs off (7) on the renewed lease" "7" "$rc" +assert_eq "the lane's lease is superseded" "true" "$(superseded 2)" +cleanup_scenario + +# [hand-back after the other claim posts] a same-login triage claim has posted lease 2 +# and its re-read is next when the attended session releases lease 1. The release sees +# lease 2 and leaves alice assigned, and triage holds lease 2 with alice assigned. +attended_scenario +h="$(next_read list 1)" +hook "$h" bash "$RELEASE" "$ID" --lease-comment-id 1 +rc=0 +out="$(bash "$CLAIM" "$ID" --ttl-hours 1 --phase triage 2>/dev/null)" || rc=$? +assert_eq "the attended release succeeds (0) while triage's lease is pending" "0" "$(hook_rc "$h")" +assert_eq "triage's claim holds (0)" "0" "$rc" +assert_eq "triage holds lease 2" "2" "$(jq -r '.lease_comment_id' <<<"$out")" +assert_eq "alice stays assigned for triage's live lease" '["alice"]' "$(assignees_now)" +assert_not_contains "the hand-back never removes alice" "$(calls)" "REMOVE_ASSIGNEE alice" +cleanup_scenario + +# [hand-back before the other claim posts] the release lands after triage's assignee +# check and before its post, so it sees no other lease and unassigns alice; triage then +# wins and puts its assignee back. +attended_scenario +h="$(next_read assignees 2)" +hook "$h" bash "$RELEASE" "$ID" --lease-comment-id 1 +rc=0 +bash "$CLAIM" "$ID" --ttl-hours 1 --phase triage >/dev/null 2>&1 || rc=$? +assert_eq "the attended release succeeds (0) before triage posts" "0" "$(hook_rc "$h")" +assert_eq "triage's claim holds (0)" "0" "$rc" +assert_eq "triage's claim restores alice after the release removed her" '["alice"]' "$(assignees_now)" +cleanup_scenario + +# [claim inside the release's unassign] triage posts and wins between the release's +# lease check and its unassign; the release's re-read finds triage's lease and puts +# alice back. +attended_scenario +h="$(next_read assignees 1)" +hook "$h" bash "$CLAIM" "$ID" --ttl-hours 1 --phase triage +out="$(bash "$RELEASE" "$ID" --lease-comment-id 1 2>/dev/null)" +rc=$? +assert_eq "the attended release succeeds (0)" "0" "$rc" +assert_eq "triage's claim inside the release holds (0)" "0" "$(hook_rc "$h")" +assert_eq "the release restores alice for triage's live lease" '["alice"]' "$(assignees_now)" +cleanup_scenario + +# [stranded, caller cleared first] the attended caller clears alice before releasing; a +# same-login triage claim made in between loses to lease 1. Triage assigned alice itself, +# so it removes her, and once lease 1 is released no assignee is left without a lease. +attended_scenario +gh api --method DELETE "repos/acme/widgets/issues/1/assignees" -f "assignees[]=alice" >/dev/null +rc=0 +bash "$CLAIM" "$ID" --ttl-hours 1 --phase triage >/dev/null 2>&1 || rc=$? +assert_eq "triage backs off (7) on the attended lease" "7" "$rc" +assert_eq "the losing triage claim removes the assignee it added" "[]" "$(assignees_now)" +bash "$RELEASE" "$ID" --lease-comment-id 1 >/dev/null 2>&1 +assert_eq "after the release no assignee is left" "[]" "$(assignees_now)" +out="$(bash "$RECLAIM" "$ID" 2>/dev/null)" +assert_eq "reclaim finds no active lease" "no active lease record" "$(jq -r '.reason' <<<"$out")" +assert_eq "and no assignee to strand" "[]" "$(assignees_now)" +cleanup_scenario + +# [stranded, release-only hand-back] triage loses to lease 1 while alice was already +# assigned, so it keeps her for the winner; the attended release then unassigns her, +# since no other live lease of alice's remains. +attended_scenario +rc=0 +bash "$CLAIM" "$ID" --ttl-hours 1 --phase triage >/dev/null 2>&1 || rc=$? +assert_eq "triage backs off (7) on the attended lease" "7" "$rc" +assert_eq "the losing triage claim keeps the winner's assignee" '["alice"]' "$(assignees_now)" +bash "$RELEASE" "$ID" --lease-comment-id 1 >/dev/null 2>&1 +assert_eq "the attended release unassigns alice" "[]" "$(assignees_now)" +out="$(bash "$RECLAIM" "$ID" 2>/dev/null)" +assert_eq "reclaim finds no active lease" "no active lease record" "$(jq -r '.reason' <<<"$out")" +assert_eq "and no assignee to strand" "[]" "$(assignees_now)" +cleanup_scenario + +# [default claim dies after its post] a failed re-list supersedes the comment just posted +# and rolls back the assignment, as a lease-only claim does. +new_scenario +: >"$GH_STUB_DIR/stateful" +printf 'alice\n' >"$GH_STUB_DIR/login" +printf '[]\n' >"$GH_STUB_DIR/assignees" +printf '[]\n' >"$GH_STUB_DIR/lease-comments" +: >"$GH_STUB_DIR/list-fails-1" +rc=0 +bash "$CLAIM" "$ID" --ttl-hours 24 >/dev/null 2>&1 || rc=$? +assert_eq "a default claim whose re-list fails exits non-zero" "true" "$([[ $rc -ne 0 ]] && echo true || echo false)" +assert_eq "the default claim supersedes its own comment on that failure" "true" "$(superseded 1)" +assert_eq "the default claim rolls back its assignment" "[]" "$(assignees_now)" +cleanup_scenario + [[ $FAILED -eq 0 ]] || exit 1 diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh index c358f39c512..6f99d511c9e 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh @@ -1,8 +1,8 @@ #!/usr/bin/env bash # release --lease-comment-id [--holder ] — CONTRACT.md "Lease protocol". Ends the -# caller's own live lease early by adding superseded_at in place. The assignee is -# left alone; the caller clears it through the adapter's assignee edit. The PATCH -# routes through the same writer that posted the lease comment (bot). +# caller's own live lease early by adding superseded_at in place, then unassigns the +# holder unless another live lease of that holder remains. The PATCH routes through the +# same writer that posted the lease comment (bot). set -uo pipefail # shellcheck source=common.sh source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/common.sh" @@ -57,10 +57,6 @@ if [[ -z "$lease_json" ]]; then printf 'release: comment %s is not a work-item lease\n' "$lease_comment_id" >&2 exit "$EX_CONFLICT" fi -if [[ -n "$(jq -r '.superseded_at // empty' <<<"$lease_json")" ]]; then - emit false "lease already superseded" - exit 0 -fi # Comment ids are public, so the handle alone would let any session end another # login's lease. Same-login sessions are told apart by the handle, as for renew-lease. @@ -72,6 +68,23 @@ else login="$WIT_GH_OUT" fi holder="$(jq -r '.holder // empty' <<<"$lease_json")" + +# Every exit-0 path hands the assignee back too, since the caller relies on release for it. +hand_back() { + [[ "$holder" == "$login" ]] && wit_unassign_if_unheld "$owner" "$repo" "$number" "$login" "$lease_comment_id" "$1" + emit "$2" "$3" + exit 0 +} + +if [[ -n "$(jq -r '.superseded_at // empty' <<<"$lease_json")" ]]; then + [[ "$holder" == "$login" ]] || { + emit false "lease already superseded" + exit 0 + } + leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || exit "$?" + hand_back "$leases" false "lease already superseded" +fi + if [[ "$holder" != "$login" ]]; then printf 'release: lease %s is held by %s, not %s\n' \ "$lease_comment_id" "${holder:-unknown}" "$login" >&2 @@ -79,21 +92,29 @@ if [[ "$holder" != "$login" ]]; then fi leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || exit "$?" -wit_select_active_lease "$leases" -if [[ "$WIT_ACTIVE_LEASE_ID" != "$lease_comment_id" ]]; then - printf 'release: comment %s is not the active lease (active: comment %s)\n' \ - "$lease_comment_id" "${WIT_ACTIVE_LEASE_ID:-none}" >&2 - exit "$EX_CONFLICT" -fi -# An expired lease already blocks nobody; superseding it is reclaim's business, -# together with the assignee it strands. +# An expired lease already blocks nobody; superseding it is reclaim's business. It is +# still refused under a newer non-superseded lease, as a stale handle. if ! wit_lease_is_live "$lease_json" "$(date -u +%s)"; then - emit false "lease already expired" - exit 0 + wit_select_active_lease "$leases" + if [[ "$WIT_ACTIVE_LEASE_ID" != "$lease_comment_id" ]]; then + printf 'release: comment %s is not the active lease (active: comment %s)\n' \ + "$lease_comment_id" "${WIT_ACTIVE_LEASE_ID:-none}" >&2 + exit "$EX_CONFLICT" + fi + hand_back "$leases" false "lease already expired" +fi + +# The holder is the earliest live lease (claim step 4); a newer live lease is a claimant +# that has not backed off yet and does not stop the holder's release. +wit_lease_winner "$leases" "$(date -u +%s)" "$lease_comment_id" wit_lease_released_by_run +if [[ "$WIT_WINNER_ID" != "$lease_comment_id" ]]; then + printf 'release: comment %s is outranked by the earlier live lease, comment %s\n' \ + "$lease_comment_id" "${WIT_WINNER_ID:-none}" >&2 + exit "$EX_CONFLICT" fi now="$(date -u +%Y-%m-%dT%H:%M:%SZ)" released="$(jq -c --arg ts "$now" '. + {superseded_at: $ts}' <<<"$lease_json")" wit_patch_lease_comment "$owner" "$repo" "$lease_comment_id" "$released" -emit true "live lease released" +hand_back "$leases" true "live lease released" diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh index 55222e55e74..6dbe4103a74 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh @@ -49,25 +49,25 @@ if [[ -n "$(jq -r '.superseded_at // empty' <<<"$lease_json")" ]]; then exit "$EX_CONFLICT" fi -# Non-superseded is not enough: an expired lease can coexist with a newer claim, and -# renewing it would leave two live leases. Require the newest non-superseded lease. -leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || exit "$?" -wit_select_active_lease "$leases" -if [[ "$WIT_ACTIVE_LEASE_ID" != "$lease_comment_id" ]]; then - printf 'renew-lease: comment %s is not the active lease (superseded by a newer claim, comment %s)\n' \ - "$lease_comment_id" "${WIT_ACTIVE_LEASE_ID:-none}" >&2 - exit "$EX_CONFLICT" -fi - -# The active lease can still be expired, e.g. a crashed or delayed holder that never -# backed off. Reviving it would undo a handoff another worker made on expiry; -# recovery is a fresh claim or reclaim. +# An expired lease is never revived, e.g. a crashed or delayed holder that never +# backed off: that would undo a handoff another worker made on expiry; recovery is a +# fresh claim or reclaim. if ! wit_lease_is_live "$lease_json" "$(date -u +%s)"; then printf 'renew-lease: lease %s is expired (renewed_at + ttl_hours elapsed); not renewing\n' \ "$lease_comment_id" >&2 exit "$EX_CONFLICT" fi +# Live is not enough: the holder is the earliest live lease (claim step 4). A newer +# live lease is a claimant that has not backed off yet and does not stop the renew. +leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || exit "$?" +wit_lease_winner "$leases" "$(date -u +%s)" "$lease_comment_id" wit_lease_released_by_run +if [[ "$WIT_WINNER_ID" != "$lease_comment_id" ]]; then + printf 'renew-lease: comment %s is outranked by the earlier live lease, comment %s\n' \ + "$lease_comment_id" "${WIT_WINNER_ID:-none}" >&2 + exit "$EX_CONFLICT" +fi + now="$(date -u +%Y-%m-%dT%H:%M:%SZ)" renewed="$(jq -c --arg ts "$now" '. + {renewed_at: $ts}' <<<"$lease_json")" wit_patch_lease_comment "$owner" "$repo" "$lease_comment_id" "$renewed" diff --git a/plugins/work-items/tools/work-item-tracker/lib/lease.sh b/plugins/work-items/tools/work-item-tracker/lib/lease.sh index a4384a949df..1a2b99d8c15 100755 --- a/plugins/work-items/tools/work-item-tracker/lib/lease.sh +++ b/plugins/work-items/tools/work-item-tracker/lib/lease.sh @@ -55,6 +55,34 @@ wit_select_active_lease() { export WIT_ACTIVE_LEASE_ID WIT_ACTIVE_LEASE_JSON } +# wit_lease_winner []: set +# WIT_WINNER_ID / WIT_WINNER_JSON to the EARLIEST live lease, the one claim arbitration +# (CONTRACT.md "Lease protocol", claim step 4) says holds the item; both empty when no lease +# is live. A newer live lease is a claimant that has not yet backed off and never holds. +# , when given, runs as ` ` for each live lease other +# than ; its exit 0 counts that lease as released and arbitration moves on. +WIT_WINNER_ID="" +WIT_WINNER_JSON="" +wit_lease_winner() { + local leases="$1" now_epoch="$2" self="$3" released_fn="${4:-}" row cand row_id + WIT_WINNER_ID="" + WIT_WINNER_JSON="" + while IFS= read -r row; do + [[ -n "$row" ]] || continue + cand="$(wit_lease_json "$(jq -r '.body' <<<"$row")")" + [[ -n "$cand" ]] || continue + wit_lease_is_live "$cand" "$now_epoch" || continue + row_id="$(jq -r '.id' <<<"$row")" + if [[ -n "$released_fn" && "$row_id" != "$self" ]] && "$released_fn" "$row_id" "$cand"; then + continue + fi + WIT_WINNER_ID="$row_id" + WIT_WINNER_JSON="$cand" + break + done < <(jq -c 'sort_by(.id) | .[]' <<<"$leases") + export WIT_WINNER_ID WIT_WINNER_JSON +} + # wit_lease_ttl_seconds — echo total TTL in seconds from ttl_hours and # optional ttl_minutes. Returns 1 when ttl_hours is missing or non-numeric. wit_lease_ttl_seconds() { diff --git a/plugins/work-items/tools/work-item-tracker/lib/lease.test.sh b/plugins/work-items/tools/work-item-tracker/lib/lease.test.sh index b56dd905368..7826b8787da 100755 --- a/plugins/work-items/tools/work-item-tracker/lib/lease.test.sh +++ b/plugins/work-items/tools/work-item-tracker/lib/lease.test.sh @@ -99,4 +99,23 @@ wit_select_active_lease "$LEASES_NONE" assert_eq "select: no active lease → empty id" "" "$WIT_ACTIVE_LEASE_ID" assert_eq "select: no active lease → empty json" "" "$WIT_ACTIVE_LEASE_JSON" +# Winner: the EARLIEST live lease holds; superseded and expired leases are skipped, and a +# lease the released-fn reports released moves arbitration on, though never the self id. +LIVE_A="$(mk "$(jq -cn --arg t "$NOW_ISO" '{holder:"a", renewed_at:$t, ttl_hours:24}')")" +LIVE_B="$(mk "$(jq -cn --arg t "$NOW_ISO" '{holder:"b", renewed_at:$t, ttl_hours:24}')")" +DEAD_C="$(mk "$(jq -cn '{holder:"c", renewed_at:"2000-01-01T00:00:00Z", ttl_hours:24}')")" +SUP_D="$(mk "$(jq -cn --arg t "$NOW_ISO" '{holder:"d", renewed_at:$t, ttl_hours:24, superseded_at:$t}')")" +LEASES_MIXED="$(jq -cn --arg c "$DEAD_C" --arg d "$SUP_D" --arg a "$LIVE_A" --arg b "$LIVE_B" \ + '[{id:40, body:$b}, {id:10, body:$c}, {id:20, body:$d}, {id:30, body:$a}]')" +wit_lease_winner "$LEASES_MIXED" "$NOW" 40 +assert_eq "winner: earliest live lease, past an expired and a superseded one" "30" "$WIT_WINNER_ID" +assert_eq "winner: its lease json" "a" "$(jq -r '.holder' <<<"$WIT_WINNER_JSON")" +released_a() { [[ "$(jq -r '.holder' <<<"$2")" == a ]]; } +wit_lease_winner "$LEASES_MIXED" "$NOW" 40 released_a +assert_eq "winner: a lease the released-fn releases is passed over" "40" "$WIT_WINNER_ID" +wit_lease_winner "$LEASES_MIXED" "$NOW" 30 released_a +assert_eq "winner: the self id is never passed to the released-fn" "30" "$WIT_WINNER_ID" +wit_lease_winner "$LEASES_NONE" "$NOW" 100 +assert_eq "winner: no live lease → empty id" "" "$WIT_WINNER_ID" + [[ $FAILED -eq 0 ]] || exit 1 From 30c9fb1fb03b30565909c5d6d24197a531ec4754 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:57:31 -0500 Subject: [PATCH 02/13] test(work-items): answer release's lease list and issue read in the broker stub The already-released-lease case now reaches the lease-comment list and the issue read that release's assignee hand-back makes; without stub cases for them the case exited 1 on an unhandled gh call. Refs: #16472 Co-Authored-By: Claude Opus 5.5 --- scripts/read-lane-broker-check.test.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/read-lane-broker-check.test.sh b/scripts/read-lane-broker-check.test.sh index 8b1df13efad..c72e898ddf1 100755 --- a/scripts/read-lane-broker-check.test.sh +++ b/scripts/read-lane-broker-check.test.sh @@ -295,6 +295,8 @@ cat >"$STUB_BIN/gh" <<'STUB' case "$*" in *"--method PATCH"*) echo PATCH >>"$STUB_LOG"; echo 41 ;; *"issues/comments/41"*) printf '%s\n' "$COMMENT" ;; +*"issues/7/comments?per_page=100"*) printf '[]\n' ;; +*".assignees[].login"*) printf '[]\n' ;; *) echo "gh-stub: unhandled: $*" >&2; exit 90 ;; esac STUB From 925fd82c43028fec9ee829afece8a486ee5737bc Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:57:50 -0500 Subject: [PATCH 03/13] fix(work-items): refuse a renew near expiry while a newer lease is pending Within WIT_LEASE_SKEW_SECONDS (300 s, now one named constant that wit_lease_is_live also uses) of a lease's expiry, a newer claimant may already have found it expired and started. renew-lease there also requires the newest non-superseded lease, so it never revives a lease that would then outrank a claimant already working. Refs: #16472 Co-Authored-By: Claude Opus 5.5 --- .../adapters/github/renew-lease.sh | 16 +++++++++++++++- .../tools/work-item-tracker/lib/lease.sh | 7 +++++-- 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh index 6dbe4103a74..774a5b80b25 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh @@ -61,13 +61,27 @@ fi # Live is not enough: the holder is the earliest live lease (claim step 4). A newer # live lease is a claimant that has not backed off yet and does not stop the renew. leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || exit "$?" -wit_lease_winner "$leases" "$(date -u +%s)" "$lease_comment_id" wit_lease_released_by_run +now_epoch="$(date -u +%s)" +wit_lease_winner "$leases" "$now_epoch" "$lease_comment_id" wit_lease_released_by_run if [[ "$WIT_WINNER_ID" != "$lease_comment_id" ]]; then printf 'renew-lease: comment %s is outranked by the earlier live lease, comment %s\n' \ "$lease_comment_id" "${WIT_WINNER_ID:-none}" >&2 exit "$EX_CONFLICT" fi +# Within the skew margin of expiry, a newer claimant may already have found this lease +# expired and started; renewing would then revive the lease that outranks it. There the +# lease must also be the newest non-superseded one. +expires_epoch=$(($(wit_iso_to_epoch "$(jq -r '.renewed_at' <<<"$lease_json")") + $(wit_lease_ttl_seconds "$lease_json"))) +if ((expires_epoch - now_epoch <= WIT_LEASE_SKEW_SECONDS)); then + wit_select_active_lease "$leases" + if [[ "$WIT_ACTIVE_LEASE_ID" != "$lease_comment_id" ]]; then + printf 'renew-lease: lease %s expires within %s s and a newer lease (comment %s) is pending; not renewing\n' \ + "$lease_comment_id" "$WIT_LEASE_SKEW_SECONDS" "$WIT_ACTIVE_LEASE_ID" >&2 + exit "$EX_CONFLICT" + fi +fi + now="$(date -u +%Y-%m-%dT%H:%M:%SZ)" renewed="$(jq -c --arg ts "$now" '. + {renewed_at: $ts}' <<<"$lease_json")" wit_patch_lease_comment "$owner" "$repo" "$lease_comment_id" "$renewed" diff --git a/plugins/work-items/tools/work-item-tracker/lib/lease.sh b/plugins/work-items/tools/work-item-tracker/lib/lease.sh index 1a2b99d8c15..e497cdcce5e 100755 --- a/plugins/work-items/tools/work-item-tracker/lib/lease.sh +++ b/plugins/work-items/tools/work-item-tracker/lib/lease.sh @@ -8,6 +8,8 @@ readonly _WIT_LEASE_LOADED=1 readonly WIT_LEASE_MARKER='' "$(jq -cn --arg h "$1" --arg r "$FUTURE" '{schema_version:"1.0", holder:$h, acquired_at:$r, renewed_at:$r, ttl_hours:1, superseded_at:$r}')"; } +# short_marker : a lease renewed now with a one-minute TTL, inside the skew margin. +short_marker() { printf '' "$(jq -cn --arg h "$1" --arg r "$(date -u +%Y-%m-%dT%H:%M:%SZ)" '{schema_version:"1.0", holder:$h, acquired_at:$r, renewed_at:$r, ttl_hours:0, ttl_minutes:1}')"; } + +# [renew near expiry, newer lease pending] lease 1 expires within the skew margin and a +# lane has posted lease 2. The lane may already have found lease 1 expired and started, +# so a renew that revived lease 1 would put two claimants on the item: refused. +new_scenario +SHORT_MARKER="$(short_marker alice)" +jq -cn --arg a "$SHORT_MARKER" --arg b "$(marker "$LANE" "$FUTURE" 2)" \ + '[{id:1, node_id:"MDEx", body:$a, created_at:"2020-01-01T00:00:00Z", author_association:"MEMBER"}, + {id:2, node_id:"MDEx", body:$b, created_at:"2020-01-01T00:00:00Z", author_association:"MEMBER"}]' >"$GH_STUB_DIR/lease-comments" +jq -cn --arg b "$SHORT_MARKER" '{body:$b, issue:"1"}' >"$GH_STUB_DIR/comment" +rc=0 +err="$(bash "$RENEW" "$ID" --lease-comment-id 1 2>&1 >/dev/null)" || rc=$? +assert_eq "renew refuses (7) a lease within the skew margin under a newer lease" "7" "$rc" +assert_contains "the refusal names the skew margin, not expiry" "$err" "expires within 300 s" +assert_not_contains "that renew writes nothing" "$(calls)" "PATCH" +cleanup_scenario +# [renew near expiry, alone] the same lease with no newer one still renews. +new_scenario +SHORT_MARKER="$(short_marker alice)" +lease_array 1 "$SHORT_MARKER" >"$GH_STUB_DIR/lease-comments" +jq -cn --arg b "$SHORT_MARKER" '{body:$b, issue:"1"}' >"$GH_STUB_DIR/comment" +rc=0 +bash "$RENEW" "$ID" --lease-comment-id 1 >/dev/null 2>&1 || rc=$? +assert_eq "renew within the skew margin succeeds (0) with no newer lease" "0" "$rc" +assert_contains "that renew PATCHes lease 1" "$(calls)" "PATCH 1" +cleanup_scenario + +# [renew outranked] renewing the newer of two live leases is refused. +new_scenario +two_live_leases +rc=0 +bash "$RENEW" "$ID" --lease-comment-id 200 >/dev/null 2>&1 || rc=$? +assert_eq "renew refuses (7) a handle outranked by an earlier live lease" "7" "$rc" +assert_not_contains "renew writes nothing for an outranked handle" "$(calls)" "PATCH" +cleanup_scenario + +# [stale snapshot in the release] a same-login claim posts lease 2 before the release's +# first lease read and supersedes it, losing, right after that read. The release must not +# keep alice for lease 2 on the strength of that read. +attended_scenario +h="$(next_read list 1)" +hook "$h" put_lease 2 "$(marker alice "$FUTURE" 1)" +hook "$h-after" put_lease 2 "$(superseded_marker alice)" +bash "$RELEASE" "$ID" --lease-comment-id 1 >/dev/null 2>&1 +assert_eq "the release succeeds with a lease 2 that was live in its first read" "true" "$(superseded 1)" +assert_eq "lease 2 is superseded by its losing claim" "true" "$(superseded 2)" +assert_eq "no assignee is left once every lease is superseded" "[]" "$(assignees_now)" +out="$(bash "$RECLAIM" "$ID" 2>/dev/null)" +assert_eq "reclaim finds no active lease" "no active lease record" "$(jq -r '.reason' <<<"$out")" +cleanup_scenario + +# [stale snapshot in the loser] triage's arbitration read shows lease 1 live, but the +# attended release has superseded it by the time triage backs off. The release saw +# triage's lease 2 live and kept alice; triage's own read after its supersede finds no +# other live lease of alice's and removes her. +attended_scenario +h="$(next_read list 1)" +jq -c --arg b "$(marker alice "$FUTURE" 1)" \ + '. + [{id:2, node_id:"MDEx", body:$b, created_at:"2020-01-01T00:00:00Z", author_association:"MEMBER"}]' \ + "$GH_STUB_DIR/lease-comments" >"$GH_STUB_DIR/lease-comments-${h#list-}" +hook "$h" bash "$RELEASE" "$ID" --lease-comment-id 1 +rc=0 +bash "$CLAIM" "$ID" --ttl-hours 1 --phase triage >/dev/null 2>&1 || rc=$? +assert_eq "the attended release inside triage's arbitration succeeds (0)" "0" "$(hook_rc "$h")" +assert_eq "triage backs off (7) on the lease 1 its read showed" "7" "$rc" +assert_eq "every lease is superseded" "true true" "$(superseded 1) $(superseded 2)" +assert_eq "no assignee is left once every lease is superseded" "[]" "$(assignees_now)" +cleanup_scenario + +# [two same-login claims, both unassigned] the winner's claim runs entirely inside the +# loser's, after the loser's assign; the loser must leave alice for the winner's lease, +# and the winner's release then leaves no assignee. +new_scenario +: >"$GH_STUB_DIR/stateful" +printf 'alice\n' >"$GH_STUB_DIR/login" +printf '[]\n' >"$GH_STUB_DIR/assignees" +printf '[]\n' >"$GH_STUB_DIR/lease-comments" +h="$(next_read assignees 1)" +hook "$h" bash "$CLAIM" "$ID" --ttl-hours 24 +rc=0 +bash "$CLAIM" "$ID" --ttl-hours 1 --phase triage >/dev/null 2>&1 || rc=$? +assert_eq "the inner claim wins (0)" "0" "$(hook_rc "$h")" +assert_eq "the outer claim backs off (7)" "7" "$rc" +assert_eq "the winner holds lease 1" "false" "$(superseded 1)" +assert_eq "alice stays assigned while the winner holds" '["alice"]' "$(assignees_now)" +bash "$RELEASE" "$ID" --lease-comment-id 1 >/dev/null 2>&1 +assert_eq "after the winner's release no assignee is left" "[]" "$(assignees_now)" +cleanup_scenario + [[ $FAILED -eq 0 ]] || exit 1 diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh index 6f99d511c9e..e20a0d95831 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh @@ -117,4 +117,7 @@ fi now="$(date -u +%Y-%m-%dT%H:%M:%SZ)" released="$(jq -c --arg ts "$now" '. + {superseded_at: $ts}' <<<"$lease_json")" wit_patch_lease_comment "$owner" "$repo" "$lease_comment_id" "$released" +# The hand-back decides on a list read after the PATCH: a same-login loser that superseded +# its lease since the read above must not keep the assignee alive. +leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || exit "$?" hand_back "$leases" true "live lease released" From 59d1569077033ce3614fd994cac1f382ce7154eb Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 15 Nov 2026 22:19:48 -0500 Subject: [PATCH 05/13] test(work-items): pin renew near expiry, same-login rollback and refused-release hand-back A renew that lands near expiry under a newer lease, a claim failure path that must keep a same-login holder's assignee, an assign that lands but reports failure, a refused expired release, release's single clock read, and the linear renew of an outranked lease. Co-Authored-By: Claude Opus 5.5 --- .../github/lease-coordination.test.sh | 147 +++++++++++++++++- .../tools/work-item-tracker/lib/lease.test.sh | 6 + .../seeds/linear/renew-lease.test.sh | 16 ++ 3 files changed, 166 insertions(+), 3 deletions(-) diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh index 49ad93263d7..091526f4144 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh @@ -108,6 +108,8 @@ gh() { jq -c --arg l "$login" 'if index($l) then . else . + [$l] end' "$d/assignees" >"$d/assignees.tmp" && mv "$d/assignees.tmp" "$d/assignees" fi + # $d/assign-fails: the assignment lands but the call reports failure, as on a timeout. + [[ ! -f "$d/assign-fails" ]] || return 1 cat "$d/assignees" 2>/dev/null || printf '[]\n' ;; *".assignees[].login"*) @@ -321,15 +323,15 @@ assert_contains "claim assigns via REST" "$(calls)" "ADD_ASSIGNEE alice" cleanup_scenario # [foreign assignee] our assignment lands, but bob already holds the item → -# conflict (7), and our own @me assignment is rolled back. +# conflict (7), and our own @me assignment is rolled back once, alice holding no lease. new_scenario printf 'alice\n' >"$GH_STUB_DIR/login" jq -cn '["alice","bob"]' >"$GH_STUB_DIR/assignees" -lease_array 1 "$(marker alice "$FUTURE" 24)" >"$GH_STUB_DIR/lease-comments" +printf '[]\n' >"$GH_STUB_DIR/lease-comments" rc=0 bash "$CLAIM" "$ID" --ttl-hours 24 >/dev/null 2>&1 || rc=$? assert_eq "claim returns conflict (7) when another login is assigned" "7" "$rc" -assert_contains "claim rolls its own assignment back on conflict" "$(calls)" "REMOVE_ASSIGNEE alice" +assert_eq "claim rolls its own assignment back once on conflict" "1" "$(grep -c '^REMOVE_ASSIGNEE alice' "$GH_STUB_DIR/calls.log")" cleanup_scenario # [silently dropped assignment] REST POST /assignees returns 201 but drops a login @@ -932,4 +934,143 @@ bash "$RELEASE" "$ID" --lease-comment-id 1 >/dev/null 2>&1 assert_eq "after the winner's release no assignee is left" "[]" "$(assignees_now)" cleanup_scenario +# =========================================================================== +# Finding 10 — a renew that lands near expiry yields to a newer claimant, a claim's +# failure path keeps a same-login holder's assignee, and a refused release hands back. +# =========================================================================== +# short_scenario: alice's lease 1 expires within the skew margin, alice assigned. +short_scenario() { + new_scenario + : >"$GH_STUB_DIR/stateful" + printf 'alice\n' >"$GH_STUB_DIR/login" + printf '["alice"]\n' >"$GH_STUB_DIR/assignees" + lease_array 1 "$(short_marker alice)" >"$GH_STUB_DIR/lease-comments" +} + +# [renew lands near expiry] the lane posts lease 2 after the renew's lease read, so every +# check before the PATCH passes; the lane may have found lease 1 expired, so the renew +# must not leave lease 1 live over it. +short_scenario +hook "$(next_read list 1)-after" put_lease 2 "$(marker "$LANE" "$FUTURE" 2)" +rc=0 +err="$(bash "$RENEW" "$ID" --lease-comment-id 1 2>&1 >/dev/null)" || rc=$? +assert_eq "a renew landing near expiry under a newer lease exits conflict (7)" "7" "$rc" +assert_contains "the refusal says the renew landed too close to expiry" "$err" "too close to expiry" +assert_eq "the renewed lease 1 is superseded" "true" "$(superseded 1)" +assert_eq "the lane's lease 2 stays live" "false" "$(superseded 2)" +cleanup_scenario + +# [renew near expiry, re-list fails twice] a failed read keeps the lease: the renew +# retries the list once, then keeps lease 1 and says so. +short_scenario +: >"$GH_STUB_DIR/list-fails-2" +: >"$GH_STUB_DIR/list-fails-3" +rc=0 +err="$(bash "$RENEW" "$ID" --lease-comment-id 1 2>&1 >/dev/null)" || rc=$? +assert_eq "a renew near expiry whose re-lists fail succeeds (0)" "0" "$rc" +assert_contains "that renew says the lease is kept" "$err" "renew-lease: could not re-read the leases after the renew; lease kept" +assert_eq "that renew retries the list once" "3" "$(cat "$GH_STUB_DIR/lease-comments.count")" +assert_eq "that renew writes only its renew PATCH" "1" "$(grep -c '^PATCH 1' "$GH_STUB_DIR/calls.log")" +cleanup_scenario +# [renew near expiry, re-list fails once] the retry sees the newer lease and yields. +short_scenario +: >"$GH_STUB_DIR/list-fails-2" +hook "list-3" put_lease 2 "$(marker "$LANE" "$FUTURE" 2)" +rc=0 +bash "$RENEW" "$ID" --lease-comment-id 1 >/dev/null 2>&1 || rc=$? +assert_eq "a renew whose retried re-list shows a newer lease exits conflict (7)" "7" "$rc" +assert_eq "that renew supersedes lease 1" "true" "$(superseded 1)" +cleanup_scenario + +# [renew far from expiry] a normal renew reads the lease list once. +attended_scenario +sleep 1 +printf '0\n' >"$GH_STUB_DIR/lease-comments.count" +bash "$RENEW" "$ID" --lease-comment-id 1 >/dev/null 2>&1 +assert_eq "a renew far from expiry succeeds (0)" "0" "$?" +assert_eq "that renew reads the lease list once" "1" "$(cat "$GH_STUB_DIR/lease-comments.count")" +cleanup_scenario + +# [same-login claim dies after its post] alice's attended lease 1 holds; a second alice +# session posts lease 2 and fails on its arbitration read. Its rollback must leave alice +# assigned for lease 1. +attended_scenario +: >"$GH_STUB_DIR/list-fails-$(($(cat "$GH_STUB_DIR/lease-comments.count") + 1))" +rc=0 +bash "$CLAIM" "$ID" --ttl-hours 1 --phase triage >/dev/null 2>&1 || rc=$? +assert_eq "the second session's claim exits non-zero" "true" "$([[ $rc -ne 0 ]] && echo true || echo false)" +assert_eq "its lease 2 is superseded" "true" "$(superseded 2)" +assert_eq "alice stays assigned for live lease 1" '["alice"]' "$(assignees_now)" +cleanup_scenario + +# [same-login step-2 conflict] bob was added by hand next to the attended alice; a second +# alice session backs off at step 2 and must leave alice assigned for lease 1. +attended_scenario +printf '["alice","bob"]\n' >"$GH_STUB_DIR/assignees" +rc=0 +bash "$CLAIM" "$ID" --ttl-hours 1 --phase triage >/dev/null 2>&1 || rc=$? +assert_eq "the second session backs off (7) on bob" "7" "$rc" +assert_eq "alice stays assigned for live lease 1" '["alice","bob"]' "$(assignees_now)" +cleanup_scenario + +# [assign lands but reports failure] the claim exits on the failed assign; the rollback +# removes the assignment that landed, since alice holds no lease. +new_scenario +: >"$GH_STUB_DIR/stateful" +printf 'alice\n' >"$GH_STUB_DIR/login" +printf '[]\n' >"$GH_STUB_DIR/assignees" +printf '[]\n' >"$GH_STUB_DIR/lease-comments" +: >"$GH_STUB_DIR/assign-fails" +rc=0 +bash "$CLAIM" "$ID" --ttl-hours 24 >/dev/null 2>&1 || rc=$? +assert_eq "a claim whose assign reports failure exits non-zero" "true" "$([[ $rc -ne 0 ]] && echo true || echo false)" +assert_eq "the assignment that landed is rolled back" "[]" "$(assignees_now)" +cleanup_scenario + +# [refused expired release] alice's lease 1 expired and a newer lease 2 is pending: the +# release is refused (7) and hands back alice unless lease 2 is alice's own. +refused_release_case() { + new_scenario + : >"$GH_STUB_DIR/stateful" + printf 'alice\n' >"$GH_STUB_DIR/login" + printf '["alice"]\n' >"$GH_STUB_DIR/assignees" + jq -cn --arg a "$(marker alice "$PAST" 24)" --arg b "$(marker "$2" "$FUTURE" 2)" \ + '[{id:1, node_id:"MDEx", body:$a, created_at:"2020-01-01T00:00:00Z", author_association:"MEMBER"}, + {id:2, node_id:"MDEx", body:$b, created_at:"2020-01-01T00:00:00Z", author_association:"MEMBER"}]' >"$GH_STUB_DIR/lease-comments" + rc=0 + bash "$RELEASE" "$ID" --lease-comment-id 1 >/dev/null 2>&1 || rc=$? + assert_eq "$1: exit" "7" "$rc" + assert_eq "$1: assignees" "$3" "$(assignees_now)" + cleanup_scenario +} +refused_release_case "a refused release under a lane lease unassigns alice" "$LANE" '[]' +refused_release_case "a refused release under alice's own newer lease keeps her" alice '["alice"]' + +# [release clock] the lease expires between two clock reads; release decides on one read, +# so it releases the lease that was live at that read. +new_scenario +: >"$GH_STUB_DIR/stateful" +printf 'alice\n' >"$GH_STUB_DIR/login" +printf '["alice"]\n' >"$GH_STUB_DIR/assignees" +T0=1783828800 +lease_array 1 "$(marker alice "2026-07-12T04:00:00Z" 1)" >"$GH_STUB_DIR/lease-comments" +printf '%s\n' "$((T0 + 3599))" "$((T0 + 3600))" >"$GH_STUB_DIR/clock" +# date: `date -u +%s` answers the next line of $GH_STUB_DIR/clock, repeating the last. +date() { + if [[ "$*" == "-u +%s" && -f "${GH_STUB_DIR:?}/clock" ]]; then + head -n 1 "$GH_STUB_DIR/clock" + [[ "$(wc -l <"$GH_STUB_DIR/clock")" -le 1 ]] || + { tail -n +2 "$GH_STUB_DIR/clock" >"$GH_STUB_DIR/clock.tmp" && mv "$GH_STUB_DIR/clock.tmp" "$GH_STUB_DIR/clock"; } + else + command date "$@" + fi +} +export -f date +rc=0 +bash "$RELEASE" "$ID" --lease-comment-id 1 >/dev/null 2>&1 || rc=$? +unset -f date +assert_eq "release of a lease live at its clock read succeeds (0)" "0" "$rc" +assert_eq "that release supersedes lease 1" "true" "$(superseded 1)" +cleanup_scenario + [[ $FAILED -eq 0 ]] || exit 1 diff --git a/plugins/work-items/tools/work-item-tracker/lib/lease.test.sh b/plugins/work-items/tools/work-item-tracker/lib/lease.test.sh index 7826b8787da..a8945c38e53 100755 --- a/plugins/work-items/tools/work-item-tracker/lib/lease.test.sh +++ b/plugins/work-items/tools/work-item-tracker/lib/lease.test.sh @@ -118,4 +118,10 @@ assert_eq "winner: the self id is never passed to the released-fn" "30" "$WIT_WI wit_lease_winner "$LEASES_NONE" "$NOW" 100 assert_eq "winner: no live lease → empty id" "" "$WIT_WINNER_ID" +# Newer live lease: only a live lease with a higher comment id than self counts. +assert_succeeds "newer: a live lease after self is found" "found" "not found" wit_newer_live_lease "$LEASES_MIXED" "$NOW" 30 +assert_fails "newer: no lease after the newest" "not found" "found" wit_newer_live_lease "$LEASES_MIXED" "$NOW" 40 +LEASES_SUP_AFTER="$(jq -cn --arg a "$LIVE_A" --arg d "$SUP_D" '[{id:10, body:$a}, {id:20, body:$d}]')" +assert_fails "newer: a superseded lease after self does not count" "not found" "found" wit_newer_live_lease "$LEASES_SUP_AFTER" "$NOW" 10 + [[ $FAILED -eq 0 ]] || exit 1 diff --git a/plugins/work-items/tools/work-item-tracker/seeds/linear/renew-lease.test.sh b/plugins/work-items/tools/work-item-tracker/seeds/linear/renew-lease.test.sh index 8d02a8cef71..3fa2c8949ad 100755 --- a/plugins/work-items/tools/work-item-tracker/seeds/linear/renew-lease.test.sh +++ b/plugins/work-items/tools/work-item-tracker/seeds/linear/renew-lease.test.sh @@ -72,6 +72,22 @@ rc="$(lin_run "$S" "linear:acme/ENG#12" --lease-comment-id "$HANDLE")" assert_eq "superseded lease → exit 7" "7" "$rc" assert_contains "and says it was superseded" "$(lin_err)" "superseded" +# --- a live lease an EARLIER live lease outranks is refused (claim step 4) --- +EARLIER="$(jq -cn --arg b "$(lin_lease_body $((HANDLE - 1000)) kyle "$NOW" 24)" \ + '[{id: "uuid-comment-earlier", body: $b, createdAt: "2026-08-20T12:00:00.500Z"}]')" +seed_lease "$(jq -cn --argjson a "$EARLIER" --argjson b "$(lin_lease_node "$HANDLE" kyle "$NOW" 24)" '$a + $b')" +rc="$(lin_run "$S" "linear:acme/ENG#12" --lease-comment-id "$HANDLE")" +assert_eq "an outranked live lease → exit 7" "7" "$rc" +assert_contains "and names the earlier lease" "$(lin_err)" "outranked" +if [[ "$(lin_requests)" == *"commentUpdate"* ]]; then + fail "an outranked lease is not written to" "no commentUpdate" "commentUpdate issued" +else + pass "an outranked lease is not written to" +fi +# The earlier lease itself still renews under the newer one. +rc="$(lin_run "$S" "linear:acme/ENG#12" --lease-comment-id "$((HANDLE - 1000))")" +assert_eq "the earlier lease renews (0) under a newer live lease" "0" "$rc" + # --- a ttl-0 lease is born expired --- seed_lease "$(lin_lease_node "$HANDLE" kyle "$NOW" 0)" rc="$(lin_run "$S" "linear:acme/ENG#12" --lease-comment-id "$HANDLE")" From 5843e49572d6933c8f78b827168b2331447e1325 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 15 Nov 2026 22:26:09 -0500 Subject: [PATCH 06/13] fix(work-items): yield a near-expiry renew to a newer lease and keep a same-login holder's assignee renew-lease re-reads the leases after its PATCH when the clock is within the skew margin of the old expiry; a newer live lease there makes it supersede its own lease and exit 7, and a list it cannot read twice keeps the lease. Every claim back-off and failure path removes the login once, only when no other live lease of that login remains, with the rollback armed before the assign. A release refused under a newer lease hands the assignee back, and release reads the clock once. The Linear renew-lease refuses an outranked lease. The work, triage and attend-queue skills say what a session does on a renew exit 7. Co-Authored-By: Claude Opus 5.5 --- ...-work-items-lease-release-rule-30317b4d.md | 4 ++ .../work-items/skills/attend-queue/SKILL.md | 3 +- plugins/work-items/skills/triage/SKILL.md | 4 +- plugins/work-items/skills/work/SKILL.md | 2 +- .../tools/work-item-tracker/CONTRACT.md | 33 ++++++++++---- .../adapters/github/claim.sh | 43 ++++++++----------- .../adapters/github/common.sh | 10 +++++ .../github/lease-coordination.test.sh | 1 + .../adapters/github/release.sh | 9 ++-- .../adapters/github/renew-lease.sh | 19 ++++++++ .../tools/work-item-tracker/lib/lease.sh | 11 +++++ .../seeds/linear/renew-lease.sh | 9 ++++ 12 files changed, 107 insertions(+), 41 deletions(-) diff --git a/.changes/work-items/fix-work-items-lease-release-rule-30317b4d.md b/.changes/work-items/fix-work-items-lease-release-rule-30317b4d.md index 2802feea323..65e6d45beca 100644 --- a/.changes/work-items/fix-work-items-lease-release-rule-30317b4d.md +++ b/.changes/work-items/fix-work-items-lease-release-rule-30317b4d.md @@ -7,3 +7,7 @@ bump: patch - The GitHub adapter's `release` and `renew-lease` accept the holder's live lease while a newer lease that has not yet backed off is on the item, and refuse a lease an earlier live lease outranks. They use the rule claim arbitration uses (earliest live lease holds, a completed run's lease counts as released), now one shared `wit_lease_winner` function. Before, a holder's release under such a lease was refused and its lease blocked every later claim until its TTL. Within 300 s of expiry `renew-lease` still requires the newest non-superseded lease, so it never revives a lease a newer claimant may already have taken over. - `release` now hands the assignee back itself: it removes the login unless another live lease held by that login remains, so one same-login session's hand-back no longer unassigns another session's live claim. `/work-items:attend-queue`, `/work-items:triage` and `/work-items:work` rely on it and no longer clear `@me` themselves; attend-queue releases right after the flip. - A same-login `claim` that loses arbitration re-reads the leases after superseding its own and keeps `@me` only while another live lease of the login remains; `release` decides on a list read after its own supersede. Between the two, no assignee is left behind that no lease holds. A winning claim puts its assignee back if a same-login release removed it, a claim whose own assignment is missing assigns once more before failing, and a default claim that fails after posting its lease supersedes that lease, as a lease-only claim already did. `release` also removes a same-login assignment made by hand. +- A `renew-lease` whose write lands within 300 s of the old expiry re-reads the leases, and when a newer live lease is there supersedes its own lease and exits `7`, so a renew can no longer revive a lease a newer claimant already took over; a list it cannot read keeps the lease. `/work-items:work`, `/work-items:triage` and `/work-items:attend-queue` say what a session does on that exit: stop writing to the item and claim again to continue. +- Every `claim` back-off and failure path (a lost arbitration, another login at step 2, any failure from the assign on) removes the login once and only when no other live lease of that login remains, so a failing same-login session no longer unassigns the session that holds the item. The rollback is armed before the assign, so an assign that lands but reports failure is undone. +- A `release` refused because its expired lease sits under a newer one now hands the assignee back, and `release` reads the clock once. +- The Linear adapter's `renew-lease` refuses a lease an earlier live lease outranks. diff --git a/plugins/work-items/skills/attend-queue/SKILL.md b/plugins/work-items/skills/attend-queue/SKILL.md index 33fd90324aa..b8e051236a5 100644 --- a/plugins/work-items/skills/attend-queue/SKILL.md +++ b/plugins/work-items/skills/attend-queue/SKILL.md @@ -183,7 +183,8 @@ worker lane off the flipped item. Once the row's answer is written **Long operator waits.** When an interview spans longer than the binding's lease TTL, renew the held lease via `"$TRACKER" renew-lease "" --lease-comment-id ` (the `claim` output carries -`lease_comment_id`). +`lease_comment_id`). On exit `7` the session no longer holds the item: stop writing to it, and +claim again to continue (that claim backs off if another session holds it). ## Working the queue diff --git a/plugins/work-items/skills/triage/SKILL.md b/plugins/work-items/skills/triage/SKILL.md index 30c1c4d041d..42bb914885d 100644 --- a/plugins/work-items/skills/triage/SKILL.md +++ b/plugins/work-items/skills/triage/SKILL.md @@ -130,7 +130,9 @@ reporter-info listing, the sweep's list and count, and every read before the fir declares `claim: false`, as gitea and jira do): write unclaimed and report that no claim was possible. Any other exit leaves the item unwritten and is reported. The one-hour TTL frees the item soon on a provider with no `release`; when verification may run longer, renew with - `"$TRACKER" renew-lease "" --lease-comment-id ` before it lapses. + `"$TRACKER" renew-lease "" --lease-comment-id ` before it lapses. On exit `7` the + session no longer holds the item: stop writing to it, and claim again to continue (that claim + backs off if another session holds it). 2. **Other items** an outcome writes to, such as a cluster member's edge and comment or an umbrella's membership comment, are claimed the same way, all of them before the outcome's first write, since an umbrella fold is one indivisible sequence diff --git a/plugins/work-items/skills/work/SKILL.md b/plugins/work-items/skills/work/SKILL.md index 670cd953f5c..c1917524baf 100644 --- a/plugins/work-items/skills/work/SKILL.md +++ b/plugins/work-items/skills/work/SKILL.md @@ -199,7 +199,7 @@ On user confirmation ("yes"): "$TRACKER" claim "" ``` - `` must be fully-qualified (`claim` rejects a bare number): frontier candidates (tiers 2/3) already carry it from `list-frontier`; a recurring candidate matched to an open item by `number` (Step 2) is first qualified via adapter "Resolve item ID". Exit `0` → claim held. Exit `7` → another session won: advance to the next candidate (do not retry the same item). Claim identity is the authenticated session user, never the bot. Record `lease_comment_id` from the claim object and put it in the dispatch brief's enumerated payload (next to the PR contract list): a worker that only pushes commits looks idle to reclaim (activity is comments + open PR cross-references, not branch pushes). The **worker** is the durable renew actor while implement-dispatch is blocked on it: at each phase boundary, and before returning if the run may outlast TTL, run `"$TRACKER" renew-lease "" --lease-comment-id ` on the live lease. If a phase can outlast the remaining TTL, renew **before** the deadline with a safety margin; a late renew is exit 7 and another lane may already hold the item. The orchestrator renews only after the worker returns, before a long monitor/review wait. `renew-lease` on an already-expired lease is exit `7`; recover with a fresh claim, not a renew. + `` must be fully-qualified (`claim` rejects a bare number): frontier candidates (tiers 2/3) already carry it from `list-frontier`; a recurring candidate matched to an open item by `number` (Step 2) is first qualified via adapter "Resolve item ID". Exit `0` → claim held. Exit `7` → another session won: advance to the next candidate (do not retry the same item). Claim identity is the authenticated session user, never the bot. Record `lease_comment_id` from the claim object and put it in the dispatch brief's enumerated payload (next to the PR contract list): a worker that only pushes commits looks idle to reclaim (activity is comments + open PR cross-references, not branch pushes). The **worker** is the durable renew actor while implement-dispatch is blocked on it: at each phase boundary, and before returning if the run may outlast TTL, run `"$TRACKER" renew-lease "" --lease-comment-id ` on the live lease. If a phase can outlast the remaining TTL, renew **before** the deadline with a safety margin; a late renew is exit 7 and another lane may already hold the item. The orchestrator renews only after the worker returns, before a long monitor/review wait. `renew-lease` on an already-expired lease is exit `7`; recover with a fresh claim, not a renew. On any `renew-lease` exit `7` the session no longer holds the item: stop writing to it, and claim again to continue (that claim backs off if another session holds it). 1. **Suggest branch name.** A bug item in [investigate mode](#investigate-mode) skips this sub-step. Propose `/-`, `` being the bound adapter's `branch_ref` for the item, so `/source-control:pull-request create`, when it is among the available skills, can resolve the item from the branch and write the adapter's link line; otherwise the orchestrator writes the adapter's link line into the PR body itself. Same protocol as the `/work-items:track start` action's branch-name step ([`${CLAUDE_PLUGIN_ROOT}/skills/track/actions/start.md`](${CLAUDE_PLUGIN_ROOT}/skills/track/actions/start.md) "Suggest branch name"). Branch `` vocabulary derived from the item's issue type (native Issue Type preferred, `type:*` label fallback), slug from title (kebab-case, 40-char cap), existing-branch detection, multi-claim 3-option (switch / stay+cover-both / skip). Agent emits `git checkout -b ...` for the user; never executes itself. Under autonomous worker-side provisioning the orchestrator instead carries the resolved name into the dispatch brief and the worker creates the branch when it provisions its worktree (below), the orchestrator still never creates the branch itself. An owning item's docs, exploration notes and draft plan go on its dossier branch, `-`, by default `dossier/-`, `` being the same `branch_ref` and the slug the same as above. Read `` from `dossier_branch_prefix` in the repository's `docs/conventions/source-control.yaml`, else use `dossier/` (the source-control setting, [config-resolution.md](https://raw.githubusercontent.com/melodic-software/claude-code-plugins/main/plugins/source-control/reference/config-resolution.md) "YAML keys"). One dossier per owning item: a parent or spec container gets its own, and a child item adds its own only when it needs one. diff --git a/plugins/work-items/tools/work-item-tracker/CONTRACT.md b/plugins/work-items/tools/work-item-tracker/CONTRACT.md index c44e7b08924..8d67c09156b 100644 --- a/plugins/work-items/tools/work-item-tracker/CONTRACT.md +++ b/plugins/work-items/tools/work-item-tracker/CONTRACT.md @@ -666,7 +666,7 @@ issue comment with a machine marker. The GitHub adapter's `claim` has two modes: - **Lease-only (`--lease-only`):** the claim posts the lease comment and assigns no one. It skips steps 1 and 2, and on a lost arbitration it supersedes its own comment and touches no assignee. In either mode, a failure after the comment is posted and before the emit also supersedes it - (and, in default mode, rolls back the assignment). + (and, in default mode, rolls back the assignment by the step 4 rule). `--holder ` names the holder instead of the authenticated login, for a caller whose token cannot read `GET /user` (a GitHub App installation token, which a CI job holds). The issue lane claims this way, holding `vars.AUTOMATION_LANES_APP_SENDER_ID`, before its model starts. @@ -702,7 +702,16 @@ backs off, and a lane run that finds a live local lease backs off without starti lease outranks under claim step 4; a newer live lease does not stop it, since that is a claimant that has not backed off yet, except within 300 s (the skew a lease reader tolerates) of the lease's expiry. There the newer claimant may already have found the lease expired and - started, so the lease must also be the newest non-superseded one, else exit `7`. + started, so the lease must also be the newest non-superseded one, else exit `7`. The Linear + adapter's `renew-lease` applies the outranked refusal only. When the + clock read after the PATCH is within that margin of the old expiry, the GitHub adapter re-reads + the leases (one retry on a failed read); a newer live lease there makes it supersede its own + lease and exit `7`, and a list it still cannot read keeps the renewed lease (exit `0`, with a + warning). On exit `7` the session no longer holds the item, so it stops writing to it, and to + continue it claims again, which backs off if another session holds it. That check has no + compare-and-swap either: it covers a claimant that posted before the re-list, while a renew + stalled between the re-list and its return stays open, and GitHub's read-after-write + consistency for comments is assumed. - `session_id` is optional and collision-prone; a missing/duplicate `session_id` still counts as a competing lease. The one shape the GitHub adapter's claim step 4 reads is `run-`, a workflow run of the item's repository; any other value is diagnostic only. @@ -723,15 +732,16 @@ Claim sequence (race-safe, same-identity aware): 1. Assign the authenticated user (`--add-assignee "@me"`, always the session identity, never the bot). 2. Re-read assignees. Self missing → assign once more and re-read (a same-login release may - have removed it), still missing → exit `4`. Any other login present → back off: unassign - self, exit `7`. + have removed it), still missing → exit `4`. Any other login present → back off: exit `7`, + removing self by the step 4 rule. 3. Post the lease comment; capture its comment ID (comment identity, not `session_id`, discriminates same-login sessions). 4. Re-read all lease comments. If an earlier live lease exists that is not our own - comment, the foreign lease wins → supersede own comment, exit `7`. On a same-login loss, - re-read the leases after that supersede and keep the assignee only while another live lease - of this login remains; a release re-reads them after its own supersede, so between the two - one removes an assignee no lease holds. On a win, + comment, the foreign lease wins → supersede own comment, exit `7`. On that loss, on the + step 2 back-off, and on any failure from step 1 until the emit, the GitHub adapter removes its + login once, unless a lease list read at that moment (after the supersede) shows another live + lease of this login; a failed read removes. A release re-reads the leases after its own + supersede, so between the two one removes an assignee no lease holds. On a win, re-read assignees and assign self again if a same-login release removed it after step 2. The back-off message prints the winner's holder with control characters removed. GitHub adapter: an earlier live lease whose `session_id` is `run-` costs one @@ -751,7 +761,9 @@ edit still needs the token's write access), and, for a live lease, not outranked live lease under claim step 4 (the completed-run rule included); a newer live lease is a claimant that has not backed off yet and does not stop the release. Anything else is refused with exit `7` and nothing is written: a cross-item handle, another login's lease, a live handle an earlier live -lease outranks, or an expired handle under a newer non-superseded lease. The login check +lease outranks, or an expired handle under a newer non-superseded lease. That last refusal still +hands the assignee back as below, judged on the release's one lease read, since its lease no +longer holds the item. The login check exists because lease handles are public (a GitHub comment id), and within one login the handle tells sessions apart, as for `renew-lease`. An already-superseded or expired lease is an idempotent no-op (`released: false`, exit `0`), because it already blocks nobody. A live lease is @@ -792,6 +804,9 @@ claim's `lease_comment_id`) rather than relying on push activity (`/work-items:w - **Comment-id monotonicity.** Same-login race arbitration treats an earlier numeric lease handle as the winner. GitHub lists issue comments by ascending ID by default; adapters MUST emit ordered unique numeric `lease_comment_id` handles. +- **Ambiguous lease post.** A lease comment POST that GitHub applies but reports as failed (a + timeout) returns no comment id, so the claim cannot supersede it: that lease blocks the item + for its TTL, and the claim's rollback keeps the assignee for it. ## Containers and state diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/claim.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/claim.sh index f2059aa34b1..ce7bb2515d9 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/claim.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/claim.sh @@ -74,23 +74,24 @@ fi # A lease-only claim assigns no one, so steps 1-2 and their rollback do not apply. if [[ "$lease_only" == false ]]; then + # Guard the partial-claim window, armed before the assign so an assign that lands + # but reports failure is covered: any failure before a successful lease-comment + # write (step 3) would otherwise strand the issue assigned with no lease record + # (list-frontier excludes assigned items; reclaim ignores items with no lease). + # wit_run_gh's failure path calls `exit` directly rather than returning, and an + # explicit `exit` inside a function does not run the caller's ERR trap (only its + # EXIT trap), so this must be an EXIT trap, not ERR. A same-login session may hold + # the item, so the rollback keeps @me while another live lease of this login does. + _wit_claim_rollback() { + wit_unassign_unless_held "$owner" "$repo" "$number" "$login" "${our_comment_id:-}" + } + trap _wit_claim_rollback EXIT + # 1. Assign the session identity. `read` routes to bare gh, so the assignee is # the session user, not the bot (README "Edit labels / assignees" carve-out); # `@me` is resolved to $login explicitly because REST takes a literal login. wit_add_assignee read "$owner" "$repo" "$number" "$login" - # Guard the partial-claim window: any failure between this successful - # assignment and a successful lease-comment write (step 3) would otherwise - # strand the issue assigned with no lease record (list-frontier excludes - # assigned items; reclaim ignores items with no lease). wit_run_gh's failure - # path calls `exit` directly rather than returning, and an explicit `exit` - # inside a function does not run the caller's ERR trap (only its EXIT trap) — - # verified empirically — so this must be an EXIT trap, not ERR. - _wit_claim_rollback() { - wit_try_remove_assignee "$owner" "$repo" "$number" "$login" - } - trap _wit_claim_rollback EXIT - # 2. Sole-assignee check — a different login present means an established claim. wit_read_assignees "$owner" "$repo" "$number" assignees="$WIT_GH_OUT" @@ -112,7 +113,6 @@ if [[ "$lease_only" == false ]]; then fi other="$(jq -r --arg me "$login" '[.[] | select(. != $me)] | first // empty' <<<"$assignees")" if [[ -n "$other" ]]; then - wit_try_remove_assignee "$owner" "$repo" "$number" "$login" printf 'claim: item already claimed by %s\n' "$other" >&2 exit "$EX_CONFLICT" fi @@ -155,20 +155,11 @@ winner_holder="" # Earliest LIVE lease wins. No live lease at all (e.g. ttl 0) → our claim stands; # expiry is reclaim's concern. +# The abort trap backs off: it supersedes our lease, then the rollback keeps @me only +# while another live lease of this login holds the item, read after that supersede: a +# winner that has released by then is seen as gone, and one releasing later re-reads +# the leases after its own PATCH and sees ours superseded, so one of the two removes @me. if [[ -n "$winner_id" && "$winner_id" != "$our_comment_id" ]]; then - wit_try_supersede_lease_comment "$owner" "$repo" "$our_comment_id" "$lease" - trap - EXIT - # On a same-login loss, @me stays only while another live lease of this login holds - # the item, read after our supersede: a winner that has released by then is seen as - # gone, and one releasing later re-reads the leases after its own PATCH and sees ours - # superseded, so one of the two removes @me. - if [[ "$lease_only" == false ]]; then - if [[ "$winner_holder" != "$login" ]] || - ! leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || - ! _wit_login_holds_other "$leases" "$login" "$our_comment_id"; then - wit_try_remove_assignee "$owner" "$repo" "$number" "$login" - fi - fi printf 'claim: live lease (comment %s, holder %s) wins — backing off\n' \ "$winner_id" "$(tr -d '[:cntrl:]' <<<"${winner_holder:-unknown}")" >&2 exit "$EX_CONFLICT" diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh index ddd296e081a..3dbd458b445 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh @@ -513,6 +513,16 @@ wit_unassign_if_unheld() { fi } +# wit_unassign_unless_held []: a claim's +# back-off and failure paths remove unless a lease list read now shows a live lease +# of other than . A failed read removes, since an assignee that no +# lease holds keeps the item off the frontier for good. +wit_unassign_unless_held() { + local leases + leases="$(wit_list_lease_comments "$1" "$2" "$3")" && _wit_login_holds_other "$leases" "$4" "${5:-}" && return 0 + wit_try_remove_assignee "$1" "$2" "$3" "$4" +} + # _wit_login_holds_other : 0 when a live lease other than # is held by . _wit_login_holds_other() { diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh index 091526f4144..1283d24a0a0 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh @@ -1056,6 +1056,7 @@ T0=1783828800 lease_array 1 "$(marker alice "2026-07-12T04:00:00Z" 1)" >"$GH_STUB_DIR/lease-comments" printf '%s\n' "$((T0 + 3599))" "$((T0 + 3600))" >"$GH_STUB_DIR/clock" # date: `date -u +%s` answers the next line of $GH_STUB_DIR/clock, repeating the last. +# shellcheck disable=SC2329 # exported to the release.sh child date() { if [[ "$*" == "-u +%s" && -f "${GH_STUB_DIR:?}/clock" ]]; then head -n 1 "$GH_STUB_DIR/clock" diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh index e20a0d95831..ea9fccde85b 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh @@ -92,12 +92,15 @@ if [[ "$holder" != "$login" ]]; then fi leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || exit "$?" +now_epoch="$(date -u +%s)" # An expired lease already blocks nobody; superseding it is reclaim's business. It is -# still refused under a newer non-superseded lease, as a stale handle. -if ! wit_lease_is_live "$lease_json" "$(date -u +%s)"; then +# still refused under a newer non-superseded lease, as a stale handle, and the assignee +# goes back unless that lease or another live one is this login's. +if ! wit_lease_is_live "$lease_json" "$now_epoch"; then wit_select_active_lease "$leases" if [[ "$WIT_ACTIVE_LEASE_ID" != "$lease_comment_id" ]]; then + wit_unassign_if_unheld "$owner" "$repo" "$number" "$login" "$lease_comment_id" "$leases" printf 'release: comment %s is not the active lease (active: comment %s)\n' \ "$lease_comment_id" "${WIT_ACTIVE_LEASE_ID:-none}" >&2 exit "$EX_CONFLICT" @@ -107,7 +110,7 @@ fi # The holder is the earliest live lease (claim step 4); a newer live lease is a claimant # that has not backed off yet and does not stop the holder's release. -wit_lease_winner "$leases" "$(date -u +%s)" "$lease_comment_id" wit_lease_released_by_run +wit_lease_winner "$leases" "$now_epoch" "$lease_comment_id" wit_lease_released_by_run if [[ "$WIT_WINNER_ID" != "$lease_comment_id" ]]; then printf 'release: comment %s is outranked by the earlier live lease, comment %s\n' \ "$lease_comment_id" "${WIT_WINNER_ID:-none}" >&2 diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh index 774a5b80b25..fbb0d8b7b9b 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh @@ -86,6 +86,25 @@ now="$(date -u +%Y-%m-%dT%H:%M:%SZ)" renewed="$(jq -c --arg ts "$now" '. + {renewed_at: $ts}' <<<"$lease_json")" wit_patch_lease_comment "$owner" "$repo" "$lease_comment_id" "$renewed" +# The checks above see only leases posted before their read. A claimant that found this +# lease expired before the PATCH landed posted before it, so a list read now shows it; +# one posting later sees the renewed lease and backs off. Outside the margin nobody can +# have found the lease expired. A failed read keeps the lease, as wit_lease_released_by_run does. +after_epoch="$(date -u +%s)" +if ((after_epoch >= expires_epoch - WIT_LEASE_SKEW_SECONDS)); then + if leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || + leases="$(wit_list_lease_comments "$owner" "$repo" "$number")"; then + if wit_newer_live_lease "$leases" "$after_epoch" "$lease_comment_id"; then + wit_try_supersede_lease_comment "$owner" "$repo" "$lease_comment_id" "$renewed" + printf 'renew-lease: the renew of lease %s landed too close to expiry and a newer claimant holds; lease superseded\n' \ + "$lease_comment_id" >&2 + exit "$EX_CONFLICT" + fi + else + printf 'renew-lease: could not re-read the leases after the renew; lease kept\n' >&2 + fi +fi + jq -c --arg sv "$WIT_SCHEMA_VERSION" --arg id "$id" --arg cid "$lease_comment_id" \ '{schema_version: $sv, id: $id, holder: .holder, acquired_at: .acquired_at, renewed_at: .renewed_at, ttl_hours: .ttl_hours, diff --git a/plugins/work-items/tools/work-item-tracker/lib/lease.sh b/plugins/work-items/tools/work-item-tracker/lib/lease.sh index e497cdcce5e..fbe1d429c6c 100755 --- a/plugins/work-items/tools/work-item-tracker/lib/lease.sh +++ b/plugins/work-items/tools/work-item-tracker/lib/lease.sh @@ -85,6 +85,17 @@ wit_lease_winner() { export WIT_WINNER_ID WIT_WINNER_JSON } +# wit_newer_live_lease : 0 when a live lease +# has a higher comment id than . +wit_newer_live_lease() { + local row cand + while IFS= read -r row; do + cand="$(wit_lease_json "$(jq -r '.body' <<<"$row")")" + [[ -n "$cand" ]] && wit_lease_is_live "$cand" "$2" && return 0 + done < <(jq -c --argjson self "$3" '.[] | select(.id > $self)' <<<"$1") + return 1 +} + # wit_lease_ttl_seconds — echo total TTL in seconds from ttl_hours and # optional ttl_minutes. Returns 1 when ttl_hours is missing or non-numeric. wit_lease_ttl_seconds() { diff --git a/plugins/work-items/tools/work-item-tracker/seeds/linear/renew-lease.sh b/plugins/work-items/tools/work-item-tracker/seeds/linear/renew-lease.sh index fcc20e2fba8..2e9c0460a36 100755 --- a/plugins/work-items/tools/work-item-tracker/seeds/linear/renew-lease.sh +++ b/plugins/work-items/tools/work-item-tracker/seeds/linear/renew-lease.sh @@ -65,6 +65,15 @@ if ! wit_linear_lease_live "$LEASE"; then exit "$EX_CONFLICT" fi +# Live is not enough: an earlier live lease holds the item, in claim's step-4 order. +while IFS=$'\t' read -r ehandle elease; do + if [[ -z "$ehandle" ]] || ! wit_linear_lease_live "$elease"; then continue; fi + printf 'linear: lease %s on %s is outranked by the earlier live lease %s; not renewing\n' \ + "$LEASE_COMMENT_ID" "$ID" "$ehandle" >&2 + exit "$EX_CONFLICT" +done < <(jq -r --argjson h "$LEASE_COMMENT_ID" --arg c "$COMMENT_UUID" \ + '.[] | select(.handle < $h or (.handle == $h and .comment_id < $c)) | "\(.handle)\t\(.lease | tojson)"' <<<"$ENTRIES") + RENEWED="$(jq -c --arg t "$(wit_linear_now_iso)" '. + {renewed_at: $t}' <<<"$LEASE")" wit_linear_update_comment "$COMMENT_UUID" "$(wit_linear_lease_marker "$RENEWED")" From 7f524f700a2a618c6b7d45e9ecb90de6b5dcf985 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 15 Nov 2026 22:27:57 -0500 Subject: [PATCH 07/13] fix(work-items): name the Linear renew's earlier lease variable plainly Co-Authored-By: Claude Opus 5.5 --- .../tools/work-item-tracker/seeds/linear/renew-lease.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plugins/work-items/tools/work-item-tracker/seeds/linear/renew-lease.sh b/plugins/work-items/tools/work-item-tracker/seeds/linear/renew-lease.sh index 2e9c0460a36..2dda175bde5 100755 --- a/plugins/work-items/tools/work-item-tracker/seeds/linear/renew-lease.sh +++ b/plugins/work-items/tools/work-item-tracker/seeds/linear/renew-lease.sh @@ -66,8 +66,8 @@ if ! wit_linear_lease_live "$LEASE"; then fi # Live is not enough: an earlier live lease holds the item, in claim's step-4 order. -while IFS=$'\t' read -r ehandle elease; do - if [[ -z "$ehandle" ]] || ! wit_linear_lease_live "$elease"; then continue; fi +while IFS=$'\t' read -r ehandle earlier_lease; do + if [[ -z "$ehandle" ]] || ! wit_linear_lease_live "$earlier_lease"; then continue; fi printf 'linear: lease %s on %s is outranked by the earlier live lease %s; not renewing\n' \ "$LEASE_COMMENT_ID" "$ID" "$ehandle" >&2 exit "$EX_CONFLICT" From 61c52d1f90bf0f3ca158b12048a5995d8b531b21 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 15 Nov 2026 22:33:45 -0500 Subject: [PATCH 08/13] test(work-items): pin the assignee hand-back after a removal and a renew back-off A claim rollback that removes the login re-adds it when a same-login lease has appeared since, a renew that backs off near expiry hands back the holder's assignee unless another live lease of the login remains, and a failed assignee read still removes the login. Co-Authored-By: Claude Opus 5.5 --- .../github/lease-coordination.test.sh | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh index 1283d24a0a0..d69ceee2ecf 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh @@ -115,6 +115,7 @@ gh() { *".assignees[].login"*) n=$(($(cat "$d/assignees.count" 2>/dev/null || printf '0') + 1)) printf '%s\n' "$n" >"$d/assignees.count" + [[ ! -f "$d/assignees-fails-$n" ]] || return 1 cat "$d/assignees" stub_hook "assignees-$n" ;; @@ -958,6 +959,42 @@ assert_eq "a renew landing near expiry under a newer lease exits conflict (7)" " assert_contains "the refusal says the renew landed too close to expiry" "$err" "too close to expiry" assert_eq "the renewed lease 1 is superseded" "true" "$(superseded 1)" assert_eq "the lane's lease 2 stays live" "false" "$(superseded 2)" +assert_eq "the backed-off renew hands back alice" "[]" "$(assignees_now)" +cleanup_scenario +# [renew lands near expiry, own newer lease] the newer lease is alice's, so the back-off +# keeps her assigned for it. +short_scenario +hook "$(next_read list 1)-after" put_lease 2 "$(marker alice "$FUTURE" 1)" +rc=0 +bash "$RENEW" "$ID" --lease-comment-id 1 >/dev/null 2>&1 || rc=$? +assert_eq "a renew yielding to alice's own newer lease exits conflict (7)" "7" "$rc" +assert_eq "that back-off keeps alice for lease 2" '["alice"]' "$(assignees_now)" +cleanup_scenario + +# [claim rollback re-adds] alice loses to a lane lease; between her rollback's lease read +# and its removal a same-login claim posts lease 3 and wins. The rollback's re-list after +# the removal sees lease 3 and puts alice back. +new_scenario +: >"$GH_STUB_DIR/stateful" +printf 'alice\n' >"$GH_STUB_DIR/login" +printf '[]\n' >"$GH_STUB_DIR/assignees" +lease_array 1 "$(marker "$LANE" "$FUTURE" 2)" >"$GH_STUB_DIR/lease-comments" +hook list-3 put_lease 3 "$(marker alice "$FUTURE" 1)" +rc=0 +bash "$CLAIM" "$ID" --ttl-hours 24 >/dev/null 2>&1 || rc=$? +assert_eq "the claim losing to the lane backs off (7)" "7" "$rc" +assert_eq "its rollback removes alice, then adds her back" "REMOVE_ASSIGNEE alice ADD_ASSIGNEE alice" \ + "$(grep -E '^(REMOVE|ADD)_ASSIGNEE' "$GH_STUB_DIR/calls.log" | tail -n 2 | paste -sd ' ' -)" +assert_eq "alice stays assigned for the same-login winner's lease 3" '["alice"]' "$(assignees_now)" +cleanup_scenario + +# [hand-back, assignee read fails] a release whose assignee read fails still removes +# alice, since an assignee no lease holds would block the item for good. +attended_scenario +: >"$GH_STUB_DIR/assignees-fails-$(($(cat "$GH_STUB_DIR/assignees.count") + 1))" +bash "$RELEASE" "$ID" --lease-comment-id 1 >/dev/null 2>&1 +assert_contains "the release still removes alice" "$(calls)" "REMOVE_ASSIGNEE alice" +assert_eq "no assignee is left" "[]" "$(assignees_now)" cleanup_scenario # [renew near expiry, re-list fails twice] a failed read keeps the lease: the renew From bda5b4da56528dd1eee2a386a30a91da29666e85 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 15 Nov 2026 22:36:28 -0500 Subject: [PATCH 09/13] test(work-items): pin a renew landing after expiry and a refused release on a stale list Co-Authored-By: Claude Opus 5.5 --- .../github/lease-coordination.test.sh | 35 ++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh index d69ceee2ecf..58df4b9bc18 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh @@ -1106,9 +1106,42 @@ date() { export -f date rc=0 bash "$RELEASE" "$ID" --lease-comment-id 1 >/dev/null 2>&1 || rc=$? -unset -f date assert_eq "release of a lease live at its clock read succeeds (0)" "0" "$rc" assert_eq "that release supersedes lease 1" "true" "$(superseded 1)" cleanup_scenario +# [renew lands after expiry] the PATCH returns after lease 1's old expiry: a same-login +# claimant may have removed the shared assignee meanwhile, so the renew never keeps it. +new_scenario +: >"$GH_STUB_DIR/stateful" +printf 'alice\n' >"$GH_STUB_DIR/login" +printf '["alice"]\n' >"$GH_STUB_DIR/assignees" +lease_array 1 "$(marker alice "2026-07-12T04:00:00Z" 1)" >"$GH_STUB_DIR/lease-comments" +printf '%s\n' "$((T0 + 3590))" "$((T0 + 3590))" "$((T0 + 3600))" >"$GH_STUB_DIR/clock" +rc=0 +err="$(bash "$RENEW" "$ID" --lease-comment-id 1 2>&1 >/dev/null)" || rc=$? +assert_eq "a renew whose PATCH lands after the old expiry exits conflict (7)" "7" "$rc" +assert_contains "the refusal says the lease expired before the renew landed" "$err" "expired before the renew landed" +assert_eq "that renew PATCHes lease 1 twice, the second a supersede" "2" "$(grep -c '^PATCH 1' "$GH_STUB_DIR/calls.log")" +assert_eq "lease 1 ends superseded" "true" "$(superseded 1)" +assert_eq "alice is handed back" "[]" "$(assignees_now)" +cleanup_scenario +unset -f date + +# [expired release, stale list] a same-login loser supersedes lease 2 right after the +# release's lease read; the refused release must not keep alice for lease 2. +new_scenario +: >"$GH_STUB_DIR/stateful" +printf 'alice\n' >"$GH_STUB_DIR/login" +printf '["alice"]\n' >"$GH_STUB_DIR/assignees" +jq -cn --arg a "$(marker alice "$PAST" 24)" --arg b "$(marker alice "$FUTURE" 1)" \ + '[{id:1, node_id:"MDEx", body:$a, created_at:"2020-01-01T00:00:00Z", author_association:"MEMBER"}, + {id:2, node_id:"MDEx", body:$b, created_at:"2020-01-01T00:00:00Z", author_association:"MEMBER"}]' >"$GH_STUB_DIR/lease-comments" +hook list-1-after put_lease 2 "$(superseded_marker alice)" +rc=0 +bash "$RELEASE" "$ID" --lease-comment-id 1 >/dev/null 2>&1 || rc=$? +assert_eq "the release of an expired lease under lease 2 is refused (7)" "7" "$rc" +assert_eq "it hands back alice once lease 2 is superseded" "[]" "$(assignees_now)" +cleanup_scenario + [[ $FAILED -eq 0 ]] || exit 1 From ac1ebc06aa7d6651c5074f86bf5a6dd54b2307fd Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 15 Nov 2026 22:39:19 -0500 Subject: [PATCH 10/13] fix(work-items): hand back the assignee one way on every lease path wit_hand_back_assignee replaces both earlier helpers. It lists the leases itself after the caller's own lease is dead, keeps the login while another live lease of it holds, removes it otherwise (a failed assignee read removes too), and re-adds it when a same-login lease appears after the removal. The claim rollback, every release path and the renew back-off call it. A renew whose PATCH returns after the old expiry always supersedes its lease and exits 7. Co-Authored-By: Claude Opus 5.5 --- ...-work-items-lease-release-rule-30317b4d.md | 4 +- .../tools/work-item-tracker/CONTRACT.md | 34 ++++++++------ .../adapters/github/claim.sh | 12 +++-- .../adapters/github/common.sh | 45 ++++++++----------- .../adapters/github/release.sh | 16 +++---- .../adapters/github/renew-lease.sh | 31 ++++++++----- 6 files changed, 71 insertions(+), 71 deletions(-) diff --git a/.changes/work-items/fix-work-items-lease-release-rule-30317b4d.md b/.changes/work-items/fix-work-items-lease-release-rule-30317b4d.md index 65e6d45beca..52ad7d281da 100644 --- a/.changes/work-items/fix-work-items-lease-release-rule-30317b4d.md +++ b/.changes/work-items/fix-work-items-lease-release-rule-30317b4d.md @@ -7,7 +7,7 @@ bump: patch - The GitHub adapter's `release` and `renew-lease` accept the holder's live lease while a newer lease that has not yet backed off is on the item, and refuse a lease an earlier live lease outranks. They use the rule claim arbitration uses (earliest live lease holds, a completed run's lease counts as released), now one shared `wit_lease_winner` function. Before, a holder's release under such a lease was refused and its lease blocked every later claim until its TTL. Within 300 s of expiry `renew-lease` still requires the newest non-superseded lease, so it never revives a lease a newer claimant may already have taken over. - `release` now hands the assignee back itself: it removes the login unless another live lease held by that login remains, so one same-login session's hand-back no longer unassigns another session's live claim. `/work-items:attend-queue`, `/work-items:triage` and `/work-items:work` rely on it and no longer clear `@me` themselves; attend-queue releases right after the flip. - A same-login `claim` that loses arbitration re-reads the leases after superseding its own and keeps `@me` only while another live lease of the login remains; `release` decides on a list read after its own supersede. Between the two, no assignee is left behind that no lease holds. A winning claim puts its assignee back if a same-login release removed it, a claim whose own assignment is missing assigns once more before failing, and a default claim that fails after posting its lease supersedes that lease, as a lease-only claim already did. `release` also removes a same-login assignment made by hand. -- A `renew-lease` whose write lands within 300 s of the old expiry re-reads the leases, and when a newer live lease is there supersedes its own lease and exits `7`, so a renew can no longer revive a lease a newer claimant already took over; a list it cannot read keeps the lease. `/work-items:work`, `/work-items:triage` and `/work-items:attend-queue` say what a session does on that exit: stop writing to the item and claim again to continue. -- Every `claim` back-off and failure path (a lost arbitration, another login at step 2, any failure from the assign on) removes the login once and only when no other live lease of that login remains, so a failing same-login session no longer unassigns the session that holds the item. The rollback is armed before the assign, so an assign that lands but reports failure is undone. +- A `renew-lease` whose write lands within 300 s of the old expiry re-reads the leases, and when a newer live lease is there supersedes its own lease, hands back the holder's assignee and exits `7`, so a renew can no longer revive a lease a newer claimant already took over; a list it cannot read keeps the lease. A renew whose write returns after the old expiry always supersedes its lease, hands back the assignee and exits `7`, since an expired lease is never revived. A refused or no-op `release` decides its hand-back on a lease list read after its own lease is dead. `/work-items:work`, `/work-items:triage` and `/work-items:attend-queue` say what a session does on that exit: stop writing to the item and claim again to continue. +- Every `claim` back-off and failure path (a lost arbitration, another login at step 2, any failure from the assign on) hands the assignee back the way `release` does: it removes the login only when no other live lease of that login remains, and after the removal re-reads the leases and assigns the login again if a same-login lease has appeared, so a failing same-login session no longer unassigns the session that holds the item. A failed assignee read in that hand-back now still removes the login instead of leaving it assigned with no lease. The rollback is armed before the assign, so an assign that lands but reports failure is undone. - A `release` refused because its expired lease sits under a newer one now hands the assignee back, and `release` reads the clock once. - The Linear adapter's `renew-lease` refuses a lease an earlier live lease outranks. diff --git a/plugins/work-items/tools/work-item-tracker/CONTRACT.md b/plugins/work-items/tools/work-item-tracker/CONTRACT.md index 8d67c09156b..70e6f436cf2 100644 --- a/plugins/work-items/tools/work-item-tracker/CONTRACT.md +++ b/plugins/work-items/tools/work-item-tracker/CONTRACT.md @@ -703,10 +703,14 @@ backs off, and a lane run that finds a live local lease backs off without starti claimant that has not backed off yet, except within 300 s (the skew a lease reader tolerates) of the lease's expiry. There the newer claimant may already have found the lease expired and started, so the lease must also be the newest non-superseded one, else exit `7`. The Linear - adapter's `renew-lease` applies the outranked refusal only. When the - clock read after the PATCH is within that margin of the old expiry, the GitHub adapter re-reads + adapter's `renew-lease` applies the outranked refusal only. When the GitHub adapter's clock + read after the PATCH is at or past the old expiry, the PATCH may have revived an expired lease, + so it supersedes its own lease, runs claim step 4's hand-back for the lease's holder, and exits + `7`, whatever the leases show. When that clock read is within the margin before the old + expiry, it re-reads the leases (one retry on a failed read); a newer live lease there makes it supersede its own - lease and exit `7`, and a list it still cannot read keeps the renewed lease (exit `0`, with a + lease, run claim step 4's hand-back for the lease's holder on a fresh list (a superseded lease + is beyond reclaim), and exit `7`, and a list it still cannot read keeps the renewed lease (exit `0`, with a warning). On exit `7` the session no longer holds the item, so it stops writing to it, and to continue it claims again, which backs off if another session holds it. That check has no compare-and-swap either: it covers a claimant that posted before the re-list, while a renew @@ -738,10 +742,14 @@ Claim sequence (race-safe, same-identity aware): discriminates same-login sessions). 4. Re-read all lease comments. If an earlier live lease exists that is not our own comment, the foreign lease wins → supersede own comment, exit `7`. On that loss, on the - step 2 back-off, and on any failure from step 1 until the emit, the GitHub adapter removes its - login once, unless a lease list read at that moment (after the supersede) shows another live - lease of this login; a failed read removes. A release re-reads the leases after its own - supersede, so between the two one removes an assignee no lease holds. On a win, + step 2 back-off, and on any failure from step 1 until the emit, the GitHub adapter runs the + **hand-back** after superseding its own lease: it lists the leases (a failed list counts as + empty) and stops if one shows a live lease of the login other than its own; else it reads the + assignees and stops only if that read succeeds without the login; else it removes the login, + re-lists the leases (stopping on a failed list), and assigns the login again if a live lease + of it other than its own has appeared. Release and renew hand back the same way after their + own supersede; the hand-back always lists the leases itself, after the caller's own lease is + dead, so between any two sessions of one login one sees the other's write. On a win, re-read assignees and assign self again if a same-login release removed it after step 2. The back-off message prints the winner's holder with control characters removed. GitHub adapter: an earlier live lease whose `session_id` is `run-` costs one @@ -762,16 +770,14 @@ live lease under claim step 4 (the completed-run rule included); a newer live le that has not backed off yet and does not stop the release. Anything else is refused with exit `7` and nothing is written: a cross-item handle, another login's lease, a live handle an earlier live lease outranks, or an expired handle under a newer non-superseded lease. That last refusal still -hands the assignee back as below, judged on the release's one lease read, since its lease no -longer holds the item. The login check +runs the hand-back below, since its lease no longer holds the item. The login check exists because lease handles are public (a GitHub comment id), and within one login the handle tells sessions apart, as for `renew-lease`. An already-superseded or expired lease is an idempotent no-op (`released: false`, exit `0`), because it already blocks nobody. A live lease is -superseded in place (`released: true`). On every exit `0` release then hands the assignee back: -it removes the login unless another live lease held by that login remains on the item, judged on -a lease list read after the release's own supersede. That removal includes an assignment of the -same login a person made by hand. The GitHub adapter re-reads the leases after the removal and -assigns the login again if one has appeared, the other half of claim step 4's win check. Callers rely on it and clear no assignee +superseded in place (`released: true`). On every exit `0` release then runs claim step 4's +hand-back for the holder, on a lease list it reads after its own lease is dead (superseded by +its PATCH, already superseded, or expired). That removal includes an assignment of the same login a +person made by hand. Callers rely on it and clear no assignee themselves; `/work-items:attend-queue` flips the role label, then releases. The write carries the same TOCTOU caveat as reclaim's revalidation: GitHub's comment PATCH has no If-Match/CAS, so a concurrent writer can still interleave. The verb is capability-gated (`verbs.release`); an adapter that declares it `false`, diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/claim.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/claim.sh index ce7bb2515d9..17b65850284 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/claim.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/claim.sh @@ -83,7 +83,7 @@ if [[ "$lease_only" == false ]]; then # EXIT trap), so this must be an EXIT trap, not ERR. A same-login session may hold # the item, so the rollback keeps @me while another live lease of this login does. _wit_claim_rollback() { - wit_unassign_unless_held "$owner" "$repo" "$number" "$login" "${our_comment_id:-}" + wit_hand_back_assignee "$owner" "$repo" "$number" "$login" "${our_comment_id:-}" } trap _wit_claim_rollback EXIT @@ -155,10 +155,8 @@ winner_holder="" # Earliest LIVE lease wins. No live lease at all (e.g. ttl 0) → our claim stands; # expiry is reclaim's concern. -# The abort trap backs off: it supersedes our lease, then the rollback keeps @me only -# while another live lease of this login holds the item, read after that supersede: a -# winner that has released by then is seen as gone, and one releasing later re-reads -# the leases after its own PATCH and sees ours superseded, so one of the two removes @me. +# The abort trap backs off: it supersedes our lease, then hands back @me +# (wit_hand_back_assignee). if [[ -n "$winner_id" && "$winner_id" != "$our_comment_id" ]]; then printf 'claim: live lease (comment %s, holder %s) wins — backing off\n' \ "$winner_id" "$(tr -d '[:cntrl:]' <<<"${winner_holder:-unknown}")" >&2 @@ -166,8 +164,8 @@ if [[ -n "$winner_id" && "$winner_id" != "$our_comment_id" ]]; then fi # A same-login release that ran between step 2 and our post can have removed @me -# (it saw no other live lease of ours yet); put it back. The release re-reads the -# leases after its removal, so between the two checks one sees the other's write. +# or a same-login back-off (each saw no other live lease of ours yet); put it back. +# Each re-lists the leases after its removal, so one of the two sees the other's write. if [[ "$lease_only" == false ]]; then wit_read_assignees "$owner" "$repo" "$number" if ! jq -e --arg me "$login" 'any(.[]; . == $me)' <<<"$WIT_GH_OUT" >/dev/null; then diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh index 3dbd458b445..876384a394e 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh @@ -490,39 +490,30 @@ wit_lease_released_by_run() { "$verb" "$1" "$run_id" >&2 } -# wit_unassign_if_unheld : after a -# release, remove from the assignees unless another live lease held by remains -# (CONTRACT.md "Lease protocol", release). is read after the release's PATCH, so -# a same-login loser that superseded its own lease before that read is seen as gone, and its own -# re-read after superseding sees this release. A lease posted after the read is caught by the -# re-read after the removal, which puts the assignee back: a same-login claim posts its lease -# before it re-checks its assignee, and the release removes the assignee before it re-reads the -# leases, so one of the two sees the other. -wit_unassign_if_unheld() { - local owner="$1" repo="$2" number="$3" login="$4" released="$5" leases="$6" - _wit_login_holds_other "$leases" "$login" "$released" && return 0 - wit_try_read_assignees "$owner" "$repo" "$number" || { - printf '%s: could not read the assignees; %s left assigned\n' "$(basename "$0" .sh)" "$login" >&2 - return 0 - } - jq -e --arg l "$login" 'any(.[]; . == $l)' <<<"$WIT_GH_OUT" >/dev/null || return 0 +# wit_hand_back_assignee : after a release, a claim's +# back-off or failure, or a renew's back-off, remove from the assignees unless a live +# lease of other than holds the item (CONTRACT.md "Lease protocol", claim step +# 4). It lists the leases itself; a failed list counts as empty. Every caller's own lease is +# dead (superseded or expired) before that list, so a same-login session that +# superseded its lease earlier is seen as gone, and its own list after superseding sees this +# one's. A lease posted after the list is caught by the re-list after the removal, which puts +# the assignee back: a claim posts its lease before it re-checks its assignee, and this removes +# the assignee before it re-lists, so one of the two sees the other. A failed assignee read +# still removes, since an assignee that no lease holds keeps the item off the frontier for good. +wit_hand_back_assignee() { + local owner="$1" repo="$2" number="$3" login="$4" own="$5" leases + leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || leases='[]' + _wit_login_holds_other "$leases" "$login" "$own" && return 0 + if wit_try_read_assignees "$owner" "$repo" "$number"; then + jq -e --arg l "$login" 'any(.[]; . == $l)' <<<"$WIT_GH_OUT" >/dev/null || return 0 + fi wit_try_remove_assignee "$owner" "$repo" "$number" "$login" leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || return 0 - if _wit_login_holds_other "$leases" "$login" "$released"; then + if _wit_login_holds_other "$leases" "$login" "$own"; then wit_try_add_assignee "$owner" "$repo" "$number" "$login" fi } -# wit_unassign_unless_held []: a claim's -# back-off and failure paths remove unless a lease list read now shows a live lease -# of other than . A failed read removes, since an assignee that no -# lease holds keeps the item off the frontier for good. -wit_unassign_unless_held() { - local leases - leases="$(wit_list_lease_comments "$1" "$2" "$3")" && _wit_login_holds_other "$leases" "$4" "${5:-}" && return 0 - wit_try_remove_assignee "$1" "$2" "$3" "$4" -} - # _wit_login_holds_other : 0 when a live lease other than # is held by . _wit_login_holds_other() { diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh index ea9fccde85b..9d2a3a18560 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/release.sh @@ -71,8 +71,8 @@ holder="$(jq -r '.holder // empty' <<<"$lease_json")" # Every exit-0 path hands the assignee back too, since the caller relies on release for it. hand_back() { - [[ "$holder" == "$login" ]] && wit_unassign_if_unheld "$owner" "$repo" "$number" "$login" "$lease_comment_id" "$1" - emit "$2" "$3" + [[ "$holder" == "$login" ]] && wit_hand_back_assignee "$owner" "$repo" "$number" "$login" "$lease_comment_id" + emit "$1" "$2" exit 0 } @@ -81,8 +81,7 @@ if [[ -n "$(jq -r '.superseded_at // empty' <<<"$lease_json")" ]]; then emit false "lease already superseded" exit 0 } - leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || exit "$?" - hand_back "$leases" false "lease already superseded" + hand_back false "lease already superseded" fi if [[ "$holder" != "$login" ]]; then @@ -100,12 +99,12 @@ now_epoch="$(date -u +%s)" if ! wit_lease_is_live "$lease_json" "$now_epoch"; then wit_select_active_lease "$leases" if [[ "$WIT_ACTIVE_LEASE_ID" != "$lease_comment_id" ]]; then - wit_unassign_if_unheld "$owner" "$repo" "$number" "$login" "$lease_comment_id" "$leases" + wit_hand_back_assignee "$owner" "$repo" "$number" "$login" "$lease_comment_id" printf 'release: comment %s is not the active lease (active: comment %s)\n' \ "$lease_comment_id" "${WIT_ACTIVE_LEASE_ID:-none}" >&2 exit "$EX_CONFLICT" fi - hand_back "$leases" false "lease already expired" + hand_back false "lease already expired" fi # The holder is the earliest live lease (claim step 4); a newer live lease is a claimant @@ -120,7 +119,4 @@ fi now="$(date -u +%Y-%m-%dT%H:%M:%SZ)" released="$(jq -c --arg ts "$now" '. + {superseded_at: $ts}' <<<"$lease_json")" wit_patch_lease_comment "$owner" "$repo" "$lease_comment_id" "$released" -# The hand-back decides on a list read after the PATCH: a same-login loser that superseded -# its lease since the read above must not keep the assignee alive. -leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || exit "$?" -hand_back "$leases" true "live lease released" +hand_back true "live lease released" diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh index fbb0d8b7b9b..aabecdc83a5 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/renew-lease.sh @@ -86,20 +86,29 @@ now="$(date -u +%Y-%m-%dT%H:%M:%SZ)" renewed="$(jq -c --arg ts "$now" '. + {renewed_at: $ts}' <<<"$lease_json")" wit_patch_lease_comment "$owner" "$repo" "$lease_comment_id" "$renewed" -# The checks above see only leases posted before their read. A claimant that found this -# lease expired before the PATCH landed posted before it, so a list read now shows it; -# one posting later sees the renewed lease and backs off. Outside the margin nobody can -# have found the lease expired. A failed read keeps the lease, as wit_lease_released_by_run does. +# A superseded lease is out of reclaim's reach, so a back-off hands its holder's assignee back. +back_off() { + wit_try_supersede_lease_comment "$owner" "$repo" "$lease_comment_id" "$renewed" + wit_hand_back_assignee "$owner" "$repo" "$number" "$(jq -r '.holder // empty' <<<"$renewed")" "$lease_comment_id" + printf 'renew-lease: lease %s: %s; lease superseded\n' "$lease_comment_id" "$1" >&2 + exit "$EX_CONFLICT" +} + +# A PATCH that returns after the old expiry may have revived an expired lease, and a +# same-login claimant that lost to it meanwhile removes the shared assignee: never kept. +# Within the margin the checks above saw only leases posted before their read; a claimant +# that found this lease expired before the PATCH landed posted before it, so a list read +# now shows it, and one posting later sees the renewed lease and backs off. Earlier than +# that nobody can have found the lease expired. A failed read keeps the lease, as +# wit_lease_released_by_run does. after_epoch="$(date -u +%s)" -if ((after_epoch >= expires_epoch - WIT_LEASE_SKEW_SECONDS)); then +if ((after_epoch >= expires_epoch)); then + back_off "it expired before the renew landed" +elif ((after_epoch >= expires_epoch - WIT_LEASE_SKEW_SECONDS)); then if leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || leases="$(wit_list_lease_comments "$owner" "$repo" "$number")"; then - if wit_newer_live_lease "$leases" "$after_epoch" "$lease_comment_id"; then - wit_try_supersede_lease_comment "$owner" "$repo" "$lease_comment_id" "$renewed" - printf 'renew-lease: the renew of lease %s landed too close to expiry and a newer claimant holds; lease superseded\n' \ - "$lease_comment_id" >&2 - exit "$EX_CONFLICT" - fi + ! wit_newer_live_lease "$leases" "$after_epoch" "$lease_comment_id" || + back_off "the renew landed too close to expiry and a newer claimant holds" else printf 'renew-lease: could not re-read the leases after the renew; lease kept\n' >&2 fi From 2b76e8eeb3553e1fbe017b2b7ba4398af561ccb2 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 15 Nov 2026 22:48:18 -0500 Subject: [PATCH 11/13] test(work-items): pin a hand-back whose first lease list fails once Co-Authored-By: Claude Opus 5.5 --- .../adapters/github/lease-coordination.test.sh | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh index 58df4b9bc18..f53b7140c62 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh @@ -988,6 +988,18 @@ assert_eq "its rollback removes alice, then adds her back" "REMOVE_ASSIGNEE alic assert_eq "alice stays assigned for the same-login winner's lease 3" '["alice"]' "$(assignees_now)" cleanup_scenario +# [hand-back, lease list fails once] a second alice session fails on its arbitration read +# and its rollback's first list fails too; the retry shows lease 1, so alice is never removed. +attended_scenario +: >"$GH_STUB_DIR/calls.log" +n="$(cat "$GH_STUB_DIR/lease-comments.count")" +: >"$GH_STUB_DIR/list-fails-$((n + 1))" +: >"$GH_STUB_DIR/list-fails-$((n + 2))" +bash "$CLAIM" "$ID" --ttl-hours 1 --phase triage >/dev/null 2>&1 +assert_not_contains "the rollback never removes alice for live lease 1" "$(calls)" "REMOVE_ASSIGNEE alice" +assert_eq "alice stays assigned" '["alice"]' "$(assignees_now)" +cleanup_scenario + # [hand-back, assignee read fails] a release whose assignee read fails still removes # alice, since an assignee no lease holds would block the item for good. attended_scenario From 007db628355e1bd49506daf64beb65099f863564 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 15 Nov 2026 22:49:24 -0500 Subject: [PATCH 12/13] test(work-items): pin a hand-back re-add that outlives every lease Co-Authored-By: Claude Opus 5.5 --- .../adapters/github/lease-coordination.test.sh | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh index f53b7140c62..324fc1cb748 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/lease-coordination.test.sh @@ -988,6 +988,22 @@ assert_eq "its rollback removes alice, then adds her back" "REMOVE_ASSIGNEE alic assert_eq "alice stays assigned for the same-login winner's lease 3" '["alice"]' "$(assignees_now)" cleanup_scenario +# [claim rollback re-add, then the lease ends] as above, but lease 3 is superseded before the +# rollback's check after its re-add; the re-add must not outlive every lease. +new_scenario +: >"$GH_STUB_DIR/stateful" +printf 'alice\n' >"$GH_STUB_DIR/login" +printf '[]\n' >"$GH_STUB_DIR/assignees" +lease_array 1 "$(marker "$LANE" "$FUTURE" 2)" >"$GH_STUB_DIR/lease-comments" +hook list-3 put_lease 3 "$(marker alice "$FUTURE" 1)" +hook list-4 put_lease 3 "$(superseded_marker alice)" +rc=0 +bash "$CLAIM" "$ID" --ttl-hours 24 >/dev/null 2>&1 || rc=$? +assert_eq "the claim losing to the lane backs off (7)" "7" "$rc" +assert_eq "its rollback removes alice again after the re-add" "2" "$(grep -c '^REMOVE_ASSIGNEE alice' "$GH_STUB_DIR/calls.log")" +assert_eq "no assignee outlives every lease" "[]" "$(assignees_now)" +cleanup_scenario + # [hand-back, lease list fails once] a second alice session fails on its arbitration read # and its rollback's first list fails too; the retry shows lease 1, so alice is never removed. attended_scenario From 191d631a3e9faf311d820ccf8ea794e79f9165eb Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 15 Nov 2026 22:51:44 -0500 Subject: [PATCH 13/13] fix(work-items): retry the hand-back's lease lists and undo a stale re-add Each lease list in wit_hand_back_assignee retries once, so a throttled list no longer removes a same-login holder's assignee. After a re-add the hand-back lists once more and removes the login again when the lease that earned the re-add has ended. CONTRACT.md states the three post-PATCH renew cases, which refusal edits the assignee, and when the rollback keeps an assignee for an ambiguous lease post. The triage skill's renew sentence is shortened to stay under the 500-line cap. Co-Authored-By: Claude Opus 5.5 --- ...-work-items-lease-release-rule-30317b4d.md | 2 +- plugins/work-items/skills/triage/SKILL.md | 3 +- .../tools/work-item-tracker/CONTRACT.md | 29 ++++++++++--------- .../adapters/github/common.sh | 19 ++++++++---- 4 files changed, 31 insertions(+), 22 deletions(-) diff --git a/.changes/work-items/fix-work-items-lease-release-rule-30317b4d.md b/.changes/work-items/fix-work-items-lease-release-rule-30317b4d.md index 52ad7d281da..5d07cb4d576 100644 --- a/.changes/work-items/fix-work-items-lease-release-rule-30317b4d.md +++ b/.changes/work-items/fix-work-items-lease-release-rule-30317b4d.md @@ -8,6 +8,6 @@ bump: patch - `release` now hands the assignee back itself: it removes the login unless another live lease held by that login remains, so one same-login session's hand-back no longer unassigns another session's live claim. `/work-items:attend-queue`, `/work-items:triage` and `/work-items:work` rely on it and no longer clear `@me` themselves; attend-queue releases right after the flip. - A same-login `claim` that loses arbitration re-reads the leases after superseding its own and keeps `@me` only while another live lease of the login remains; `release` decides on a list read after its own supersede. Between the two, no assignee is left behind that no lease holds. A winning claim puts its assignee back if a same-login release removed it, a claim whose own assignment is missing assigns once more before failing, and a default claim that fails after posting its lease supersedes that lease, as a lease-only claim already did. `release` also removes a same-login assignment made by hand. - A `renew-lease` whose write lands within 300 s of the old expiry re-reads the leases, and when a newer live lease is there supersedes its own lease, hands back the holder's assignee and exits `7`, so a renew can no longer revive a lease a newer claimant already took over; a list it cannot read keeps the lease. A renew whose write returns after the old expiry always supersedes its lease, hands back the assignee and exits `7`, since an expired lease is never revived. A refused or no-op `release` decides its hand-back on a lease list read after its own lease is dead. `/work-items:work`, `/work-items:triage` and `/work-items:attend-queue` say what a session does on that exit: stop writing to the item and claim again to continue. -- Every `claim` back-off and failure path (a lost arbitration, another login at step 2, any failure from the assign on) hands the assignee back the way `release` does: it removes the login only when no other live lease of that login remains, and after the removal re-reads the leases and assigns the login again if a same-login lease has appeared, so a failing same-login session no longer unassigns the session that holds the item. A failed assignee read in that hand-back now still removes the login instead of leaving it assigned with no lease. The rollback is armed before the assign, so an assign that lands but reports failure is undone. +- Every `claim` back-off and failure path (a lost arbitration, another login at step 2, any failure from the assign on) hands the assignee back the way `release` does: it removes the login only when no other live lease of that login remains, and after the removal re-reads the leases and assigns the login again if a same-login lease has appeared, so a failing same-login session no longer unassigns the session that holds the item. A failed assignee read in that hand-back now still removes the login instead of leaving it assigned with no lease, each lease list in it retries once, and a re-add is undone when a further list shows the lease that earned it has ended. The rollback is armed before the assign, so an assign that lands but reports failure is undone. - A `release` refused because its expired lease sits under a newer one now hands the assignee back, and `release` reads the clock once. - The Linear adapter's `renew-lease` refuses a lease an earlier live lease outranks. diff --git a/plugins/work-items/skills/triage/SKILL.md b/plugins/work-items/skills/triage/SKILL.md index 42bb914885d..f6cff64f54f 100644 --- a/plugins/work-items/skills/triage/SKILL.md +++ b/plugins/work-items/skills/triage/SKILL.md @@ -131,8 +131,7 @@ reporter-info listing, the sweep's list and count, and every read before the fir possible. Any other exit leaves the item unwritten and is reported. The one-hour TTL frees the item soon on a provider with no `release`; when verification may run longer, renew with `"$TRACKER" renew-lease "" --lease-comment-id ` before it lapses. On exit `7` the - session no longer holds the item: stop writing to it, and claim again to continue (that claim - backs off if another session holds it). + session no longer holds the item: stop writing to it, and claim again to continue. 2. **Other items** an outcome writes to, such as a cluster member's edge and comment or an umbrella's membership comment, are claimed the same way, all of them before the outcome's first write, since an umbrella fold is one indivisible sequence diff --git a/plugins/work-items/tools/work-item-tracker/CONTRACT.md b/plugins/work-items/tools/work-item-tracker/CONTRACT.md index 70e6f436cf2..02922c952ad 100644 --- a/plugins/work-items/tools/work-item-tracker/CONTRACT.md +++ b/plugins/work-items/tools/work-item-tracker/CONTRACT.md @@ -707,11 +707,10 @@ backs off, and a lane run that finds a live local lease backs off without starti read after the PATCH is at or past the old expiry, the PATCH may have revived an expired lease, so it supersedes its own lease, runs claim step 4's hand-back for the lease's holder, and exits `7`, whatever the leases show. When that clock read is within the margin before the old - expiry, it re-reads - the leases (one retry on a failed read); a newer live lease there makes it supersede its own + expiry, it re-reads the leases (one retry on a failed read); a newer live lease there makes it supersede its own lease, run claim step 4's hand-back for the lease's holder on a fresh list (a superseded lease - is beyond reclaim), and exit `7`, and a list it still cannot read keeps the renewed lease (exit `0`, with a - warning). On exit `7` the session no longer holds the item, so it stops writing to it, and to + is beyond reclaim), and exit `7`, and a list it still cannot read keeps the renewed lease (exit + `0`, with a warning). A clock read earlier than the margin adds no read. On exit `7` the session no longer holds the item, so it stops writing to it, and to continue it claims again, which backs off if another session holds it. That check has no compare-and-swap either: it covers a claimant that posted before the re-list, while a renew stalled between the re-list and its return stays open, and GitHub's read-after-write @@ -743,11 +742,14 @@ Claim sequence (race-safe, same-identity aware): 4. Re-read all lease comments. If an earlier live lease exists that is not our own comment, the foreign lease wins → supersede own comment, exit `7`. On that loss, on the step 2 back-off, and on any failure from step 1 until the emit, the GitHub adapter runs the - **hand-back** after superseding its own lease: it lists the leases (a failed list counts as - empty) and stops if one shows a live lease of the login other than its own; else it reads the - assignees and stops only if that read succeeds without the login; else it removes the login, - re-lists the leases (stopping on a failed list), and assigns the login again if a live lease - of it other than its own has appeared. Release and renew hand back the same way after their + **hand-back** after superseding its own lease. Each of its lease lists retries once before it + counts as failed. It lists the leases (a failed list counts as empty) and stops if one shows a + live lease of the login other than its own; else it reads the assignees and stops only if that + read succeeds without the login; else it removes the login, re-lists the leases (stopping on a + failed list), and assigns the login again if a live lease of it other than its own has + appeared. After such a re-add it lists the leases once more and removes the login again if + that list shows no such lease, so a re-add never outlives the lease that earned it (a failed + list keeps the re-add). Release and renew hand back the same way after their own supersede; the hand-back always lists the leases itself, after the caller's own lease is dead, so between any two sessions of one login one sees the other's write. On a win, re-read assignees and assign self again if a same-login release removed it after step 2. @@ -768,9 +770,9 @@ which, like `claim`'s, replaces the `gh api user` lookup and is asserted by the edit still needs the token's write access), and, for a live lease, not outranked by an earlier live lease under claim step 4 (the completed-run rule included); a newer live lease is a claimant that has not backed off yet and does not stop the release. Anything else is refused with exit `7` -and nothing is written: a cross-item handle, another login's lease, a live handle an earlier live -lease outranks, or an expired handle under a newer non-superseded lease. That last refusal still -runs the hand-back below, since its lease no longer holds the item. The login check +and no lease is written: a cross-item handle, another login's lease, a live handle an earlier +live lease outranks, or an expired handle under a newer non-superseded lease. Only that last +refusal edits anything: it runs the hand-back below, since its lease no longer holds the item. The login check exists because lease handles are public (a GitHub comment id), and within one login the handle tells sessions apart, as for `renew-lease`. An already-superseded or expired lease is an idempotent no-op (`released: false`, exit `0`), because it already blocks nobody. A live lease is @@ -812,7 +814,8 @@ claim's `lease_comment_id`) rather than relying on push activity (`/work-items:w MUST emit ordered unique numeric `lease_comment_id` handles. - **Ambiguous lease post.** A lease comment POST that GitHub applies but reports as failed (a timeout) returns no comment id, so the claim cannot supersede it: that lease blocks the item - for its TTL, and the claim's rollback keeps the assignee for it. + for its TTL. The claim's rollback keeps the assignee for it when its lease list shows that + lease; when the list fails twice the rollback removes the assignee. ## Containers and state diff --git a/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh b/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh index 876384a394e..d4cb10e46c8 100755 --- a/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh +++ b/plugins/work-items/tools/work-item-tracker/adapters/github/common.sh @@ -493,7 +493,8 @@ wit_lease_released_by_run() { # wit_hand_back_assignee : after a release, a claim's # back-off or failure, or a renew's back-off, remove from the assignees unless a live # lease of other than holds the item (CONTRACT.md "Lease protocol", claim step -# 4). It lists the leases itself; a failed list counts as empty. Every caller's own lease is +# 4). It lists the leases itself, each list retried once; a list failed twice counts as empty +# at the first read and ends the hand-back at the re-list. Every caller's own lease is # dead (superseded or expired) before that list, so a same-login session that # superseded its lease earlier is seen as gone, and its own list after superseding sees this # one's. A lease posted after the list is caught by the re-list after the removal, which puts @@ -502,16 +503,22 @@ wit_lease_released_by_run() { # still removes, since an assignee that no lease holds keeps the item off the frontier for good. wit_hand_back_assignee() { local owner="$1" repo="$2" number="$3" login="$4" own="$5" leases - leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || leases='[]' + leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || + leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || leases='[]' _wit_login_holds_other "$leases" "$login" "$own" && return 0 if wit_try_read_assignees "$owner" "$repo" "$number"; then jq -e --arg l "$login" 'any(.[]; . == $l)' <<<"$WIT_GH_OUT" >/dev/null || return 0 fi wit_try_remove_assignee "$owner" "$repo" "$number" "$login" - leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || return 0 - if _wit_login_holds_other "$leases" "$login" "$own"; then - wit_try_add_assignee "$owner" "$repo" "$number" "$login" - fi + leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || + leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || return 0 + _wit_login_holds_other "$leases" "$login" "$own" || return 0 + wit_try_add_assignee "$owner" "$repo" "$number" "$login" + # The lease that earned the re-add may have ended meanwhile (a renew that revived it for a + # moment backs off and finds the login already gone), so the re-add must not outlive it. + leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || + leases="$(wit_list_lease_comments "$owner" "$repo" "$number")" || return 0 + _wit_login_holds_other "$leases" "$login" "$own" || wit_try_remove_assignee "$owner" "$repo" "$number" "$login" } # _wit_login_holds_other : 0 when a live lease other than