Skip to content

Record a decided tab retirement even when the write outlasts its budget - #20

Merged
EquityBot merged 4 commits into
mainfrom
ryan/retirement-deadline-flake
Oct 30, 2026
Merged

EquityBot merged 4 commits into
mainfrom
ryan/retirement-deadline-flake

Conversation

@ryanduguid

Copy link
Copy Markdown
Collaborator

Summary

A slow disk write at the end of the retirement budget made a finished retirement report failure. This was a production defect, not a tight test: the 100 ms reserved for the write is fixed, so a longer budget doesn't help. A decided outcome is now always recorded, and the record's lock gets its own deadline to release.

 retire_record_until(creator, deadline)
   lock the record               # bounded by deadline
   run the operation             # bounded by deadline - 100 ms
-  save_until(record, deadline)  # slow write: Err, record left undecided
-  lock.release()                # inherits the spent deadline
+  save(record)                  # slow write: still recorded
+  lock.release_until(now + 5 s)

What went wrong before:

  • A write that missed the 100 ms returned 'Atomic publication deadline reached' and left the record at its earlier state, with the tab possibly already closed.
  • A write that landed late returned 'Target disposition deadline reached' with the outcome already on disk.
  • Either way the caller saw a false failure, and the lease and session clean-up waited for a retry or a later reconcile. No wrong decision was recorded: this failed closed.
  • If the recovery mutex was contended during a late release, the release failed and was not retried, stranding the lock.

The overrun past the deadline is bounded by the existing rename retries (about 1.6 s) plus the I/O itself. A budget that has already expired before the operation still publishes nothing, and expired_retirement_budget_cannot_publish_a_disposition still passes.

Evidence

Stall injected temporarily into the durable write, running a_stalled_retirement_preserves_its_commit_bound_identity_for_retry:

Stall Before After
none, or 50 ms pass pass
150 ms or 1000 ms before rename Atomic publication deadline reached pass
150 ms or 1000 ms after rename Target disposition deadline reached pass
150 ms after rename, test budget 30 s Target disposition deadline reached pass
  • Before: a_decision_reached_as_the_budget_ends_is_published_despite_the_stall fails with Atomic publication deadline reached; a_late_publication_releases_its_lock_despite_contention fails with Unable to acquire directory recovery mutex.
    After: both pass.
  • Windows, on the final head: cargo fmt --check and cargo clippy --all-targets --locked -- -D warnings pass; browser_disposition, storage, browser_retirement and the close retry test give 37 passed, 0 failed.
  • Windows, one commit earlier: library suite 667 passed, 0 failed, 14 ignored.

Merge danger

Door: two-way. Blast radius: tab retirement

retire_record_until can now return after its deadline when the disk is slow. Callers' own later steps keep their own deadlines; a settlement that misses one is retried by the next reconcile.

Unverified

  • The ubuntu-latest failure at a5ff8be being the after-rename path is inferred: the log gives only the error text, which matches, and the other sources of that text need about 2.9 s of stall before the operation starts.
  • Not run locally on Linux, including the contention test; this PR's CI covers it.
  • The full library suite was not rerun after the final commit, which adds the lock release fix and its test.
  • bridge_doctor_accepts_and_ignores_verbose_like_commander and local_browser_cli_and_mcp_admit_foreground_and_detached_rust_workers fail locally, as they do on main. No browser is installed here, which is the likely cause (not confirmed).

Retirement ran its operation to 100 ms before the deadline, then published
the outcome under the same deadline. A durable write that missed the tail
returned an error, leaving the record at its earlier state; one that landed
but finished late returned an error with the outcome already on disk. Neither
depends on the size of the budget, because the tail is fixed.

Publish the decided outcome without a deadline, under the record's lock. The
deadline still bounds lock acquisition and the operation, and a budget that
has already expired still publishes nothing.
The lock kept the deadline it was acquired under, so after a late publication
its release had no time left. If another process held the recovery mutex at
that moment, the release failed and was not retried, stranding the lock and
returning an error for a disposition already on disk.
The blocking sleep sat inside async code, which SonarCloud reports as a
reliability bug. It now runs in the closure, so the operation still
completes on its first poll.
SonarCloud also reads a closure inside an async test as async, so the
blocking stall moves to a plain helper.
@sonarqubecloud

sonarqubecloud Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

@EquityBot
EquityBot merged commit 81692a5 into main Oct 30, 2026
11 checks passed
@EquityBot
EquityBot deleted the ryan/retirement-deadline-flake branch October 30, 2026 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants