Repository navigation
Record a decided tab retirement even when the write outlasts its budget - #20
Merged
Merged
Conversation
Retirement ran its operation to 100 ms before the deadline, then published the outcome under the same deadline. A durable write that missed the tail returned an error, leaving the record at its earlier state; one that landed but finished late returned an error with the outcome already on disk. Neither depends on the size of the budget, because the tail is fixed. Publish the decided outcome without a deadline, under the record's lock. The deadline still bounds lock acquisition and the operation, and a budget that has already expired still publishes nothing.
The lock kept the deadline it was acquired under, so after a late publication its release had no time left. If another process held the recovery mutex at that moment, the release failed and was not retried, stranding the lock and returning an error for a disposition already on disk.
The blocking sleep sat inside async code, which SonarCloud reports as a reliability bug. It now runs in the closure, so the operation still completes on its first poll.
SonarCloud also reads a closure inside an async test as async, so the blocking stall moves to a plain helper.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
A slow disk write at the end of the retirement budget made a finished retirement report failure. This was a production defect, not a tight test: the 100 ms reserved for the write is fixed, so a longer budget doesn't help. A decided outcome is now always recorded, and the record's lock gets its own deadline to release.
What went wrong before:
The overrun past the deadline is bounded by the existing rename retries (about 1.6 s) plus the I/O itself. A budget that has already expired before the operation still publishes nothing, and
expired_retirement_budget_cannot_publish_a_dispositionstill passes.Evidence
Stall injected temporarily into the durable write, running
a_stalled_retirement_preserves_its_commit_bound_identity_for_retry:Atomic publication deadline reachedTarget disposition deadline reachedTarget disposition deadline reacheda_decision_reached_as_the_budget_ends_is_published_despite_the_stallfails withAtomic publication deadline reached;a_late_publication_releases_its_lock_despite_contentionfails withUnable to acquire directory recovery mutex.After: both pass.
cargo fmt --checkandcargo clippy --all-targets --locked -- -D warningspass;browser_disposition,storage,browser_retirementand the close retry test give 37 passed, 0 failed.Merge danger
Door: two-way. Blast radius: tab retirement
retire_record_untilcan now return after its deadline when the disk is slow. Callers' own later steps keep their own deadlines; a settlement that misses one is retried by the next reconcile.Unverified
bridge_doctor_accepts_and_ignores_verbose_like_commanderandlocal_browser_cli_and_mcp_admit_foreground_and_detached_rust_workersfail locally, as they do onmain. No browser is installed here, which is the likely cause (not confirmed).