fix(bin): repair process-event shutdown and tighten guard timing - #4009
Merged
kunchenguid merged 19 commits intoSep 9, 2026
Merged
Conversation
 The PR should not merge until the outstanding watchdog deadline issue is resolved. |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+827
to
+828
| That guard accepts the lease only while the state root retains the device/inode identity recorded by the runner's claim, and initiates the verified stop after two consecutive checks cannot prove that identity and lease freshness, so one unreadable read cannot kill a live runner. | ||
| For a runner whose ownership can still be proved, the nominal cleanup bound is the lease plus up to two check intervals plus the stop's grace period; scheduling delays or failed inspection and signalling can extend it. |
There was a problem hiding this comment.
When an owner disappears just after a watchdog check, the guard waits for two consecutive misses and then the stop grace period, allowing the listener to outlive the accepted lease-plus-one-tick bound. The new regression permits 60 seconds for a two-second lease and one-second check interval, so it does not enforce the required deadline.
Context Used: If there is a VISION.md file at the root of the re... (source)
Contributor
Author
|
The red Measured cause: the portable serial lanes are packed from a hand-maintained per-script duration table that was last refreshed 2026-09-01. It now under-predicts, so one shard carries more work than the others and reaches the job bound. The lane is also cancelled on A repair is open as #4006 — it refreshes the duration table and moves from 5 shards to 6. On its two most recent runs, 14 jobs completed with 0 cancellations. We cannot re-run the cancelled lane ourselves: our access to this repository is pull-only, and re-running requires admin. |
Contributor
Author
|
Please do not merge this branch in its current state. The most recent automated documentation commit (
The refusal itself remains in the code; what was removed is its statement in the operator-facing and agent-loaded surfaces. A decision on relaxing that boundary is currently open on our side, which is why we are not treating this as a wording change. Checks are green. The green badge reflects the tests, not this regression. We are correcting it and will update here. |
mremond
force-pushed
the
fix/procevent-stop-escalation-and-retire-deadlock
branch
from
September 9, 2026 08:26
d732268 to
f8819bf
Compare
mremond
changed the title
Comment on lines
+833
to
+834
| That guard accepts the lease only while the state root retains the device/inode identity recorded by the runner's claim, and initiates the verified stop after two consecutive checks cannot prove that identity and lease freshness, so one unreadable read cannot kill a live runner. | ||
| For a runner whose ownership can still be proved, the nominal cleanup bound is the lease plus up to two check intervals plus the stop's grace period; scheduling delays or failed inspection and signalling can extend it. |
There was a problem hiding this comment.
When the owner disappears immediately after a successful check, the watchdog waits for two subsequent failed checks before starting a stop that can consume another four seconds, leaving the listener alive beyond the required lease-plus-one-tick deadline. The revised regression also codifies this looser bound and permits twice its duration.
Context Used: If there is a VISION.md file at the root of the re... (source)
Contributor
Author
|
Correction to the do-not-merge notice above: its stated reason is resolved. This is not a merge recommendation. That notice named two removals made by an automated documentation commit on this branch. Both are corrected, verified by diff at the current head
The repository never lost either passage; only this branch did, and this branch no longer carries the change that removed them. The delivery was rescoped so it touches neither surface at all. What this correction does not say. It removes a false reason to block; it does not clear the way, and those are different things:
|
mremond
changed the title
Owner
|
Speaking as Kun's firstmate: waiting on CI, not on the captain. HEAD Contract-class is not a merge vote yet. Stopping escalation and making a non-waiting lock acquire is a restore claim, but always-on KILL where dead code previously abandoned groups may be new-default. Do not merge while CI is not green. No captain card until the tip is otherwise ready except that class decision. |
The owner guard that shipped in kunchenguid#3904 half-reaps. Against a poll child that handles the ordinary stop signal and keeps waiting, the guard signals the group, loses the runner leader to its own signal, then reads that success as a leaderless group and exits without escalating. It destroys the only proof of ownership that would have authorised the forced signal, so the survivor becomes unreachable by retire, reconcile, sweep-home and the guard alike. A guard that turns a leaking-but-identifiable generation into a permanently unreachable one is worse than no guard at all. Two defects, and they hid each other: - The escalation re-derived ownership from the leader. `runner_group_signal` now takes a `proved` mode, passed only by the escalation inside the stop that already proved and signalled that exact generation moments earlier. A leader dying to our own signal is the ordinary outcome, not fresh ambiguity. - Every stop held the per-source lock across its wait while the runner's own exit cleanup waited unboundedly for that same lock. That circular wait was broken only by the forced signal, so the forced signal silently became the normal path - and, by keeping the leader alive through the whole window, it masked the escalation defect above. The runner's exit cleanup now refuses that lock instead of waiting for it, which is what its existing `return 0` already said it did. Fixing the lock alone would have turned every stop of a signal-proof child into a refusal that leaves it running, so both land together and the tests pin that. Measured on macOS with a stand-in poll child that traps TERM, INT and HUP: the guard left it running past 70s and now clears the group within the lease plus one check; retiring a healthy runner fell from ~2.8s with a forced group signal every time to ~0.6s on the ordinary signal alone. Unchanged and stated deliberately: a leader lost to anything other than the stop's own signal still leaves a group that retire, reconcile, sweep-home and the guard all refuse, permanently - and that source stops listening without saying so. Whether such a group may ever be signalled is an open decision and is not answered here.
…skill
An automatic documentation step in this branch's validation edited
.agents/skills/process-event-sources/SKILL.md, which no instruction in this
change asked it to touch. That file is not documentation about the code: it is
the agent-loaded instruction surface, what an agent reads to know what it is
permitted to do.
The step deleted this line:
- leaderless PID/PGID-reuse ambiguity preserves the claim without signalling
or replacement, as owned by the operating contract in
[`docs/configuration.md`](../../../docs/configuration.md#process-to-event-sources-stateprocevent);
and folded it, with its neighbour, into a generic "registration and ownership
transitions, stop authority, and claim reclamation follow the operating
contract".
That deleted line states a PROHIBITION - that such a group is preserved WITHOUT
SIGNALLING - and it is the exact limit an open captain decision currently rests
on. Folded into a pointer, an agent reading the skill to learn what it may do
would have to chase a second document to discover it may not signal. A
prohibition that requires a second lookup is not a prohibition. The effect was
to weaken, in the instructions themselves, the boundary that keeps one home from
signalling another's process group - while the question of whether that boundary
should move at all is still open.
This is a deliberate revert, not an oversight, and it restores the file exactly
to its pre-branch state. The full statement also survives in
docs/configuration.md; that does not rescue it, because the agent handling a
process-event wake loads the skill and not the documentation.
…ation The same automatic documentation step that edited the loaded skill also removed this from the comment above runner_group_signal: A leaderless group nobody in this call ever proved remains refused too, for every caller. That untouched refusal is what makes a crashed leader's group permanent, and relaxing it is a separate open question, not something this path assumes. and replaced it with a pointer to docs/configuration.md. This one fails differently from the skill deletion, which is why it is restored separately. There, a prohibition was moved out of the reader's path, and a missing prohibition gets violated. Here the prohibition survives in code - the unproved path still refuses - and what was removed is the fact that the limit is UNDECIDED. A prohibition that has quietly lost its "this is still open" reads as settled design, and settled design gets relied on, extended, and eventually relaxed by someone confident they understand why it is there. That question is open right now. The rule this branch's four instances produce, stated once here because this is the point of decision: an unresolved question must be marked unresolved AT THE POINT OF DECISION, not only where the contract is documented. A reader who does not know something is open will treat it as closed, and that default is stronger than any pointer overcomes. The pointer added by that step is kept alongside; this restores what it replaced rather than reverting it.
The document step's rewrite of this record dropped the concrete figure while keeping the surrounding measurements. What went missing was the bound itself - lease plus two consecutive failed checks plus the stop's grace, roughly 630 seconds at the shipped 600-second lease and 15-second check - together with the reason there are two checks rather than one: a single unreadable read must not be enough to kill a live runner. The mechanism survived elsewhere and the reason survived in docs/configuration.md, so nothing was lost from the repository. The concreteness was, and that is what this restores. A number recorded without why it is that number is the one a later reader shortens; the reason is the whole safety argument for the debounce, and the debounce is what stops the reaper killing a live runner on one bad read.
…eason
An automated reviewer observed that this case allowed sixty seconds for a bound
of roughly eight, so it could not go red for the reason it names: it would have
passed a guard that took fifty-five seconds. That is correct, and it is the same
family as the defect the case exists to defend against - a check that is green
because it cannot fail, rather than because the thing it guards is working.
The deadline is now derived from the bound itself - the lease, plus the two
consecutive failed checks the guard debounces on, plus the stop's own ordinary
and forced signal windows - rather than from a flat wall-clock number, and the
shortened lease and check the fixtures run under have a single definition so a
derived deadline cannot silently diverge from the settings the guard is given.
The doubling that remains is a load allowance and is documented as one; widening
it to make a slow guard pass would convert the assertion back into decoration.
Proven by mutation rather than by argument. Against the repaired case:
correct code ok
guard debounces on 20 misses instead of 2 not ok - "still holding
the group after 16s,
against a documented
bound of 8s"
proved escalation removed (the original defect) not ok - same
code restored ok
The previous sixty-second version passes every one of those mutations.
The reviewer's other claim, that the guard can survive past the announced bound
when an owner disappears immediately after a check, was measured and does not
hold against what this branch announces. Sweeping the phase deliberately at
0.0, 0.2, 0.4, 0.6 and 0.8 of a check interval gave 7.21s, 7.31s, 6.75s, 6.49s
and 6.31s, worst 7.31s, against the announced lease plus two consecutive failed
checks plus stop grace, which is up to 8s at those settings. The mechanism the
reviewer describes is real and is the announced mechanism; the bound it was
measured against is a phrasing this branch no longer carries.
This delivery is being split. It carries the two proven process fixes alone; the instruction text travels separately, through a run that removes the documentation step rather than refusing it at its gate. Two surfaces are therefore returned to exactly what the base branch has, so this delivery neither adds to them nor removes from them: .agents/skills/process-event-sources/SKILL.md - identical to base again. Three bullets an automatic documentation step had folded into a pointer, including that leaderless PID/PGID-reuse ambiguity preserves the claim WITHOUT SIGNALLING and that there is one identity-matched owner per canonical source across homes sharing one store. The header comment block of bin/fm-procevent.sh, which is what the script prints as its own help. Seven lines were removed from it: that a live owner is never displaced, that only a claim whose stale owner and independently absent process group prove its whole generation gone is reclaimed, that a crashed leader or reused pid whose process group still has members cannot relax ownership cleanup, and that reconcile signals only a live identity-matched runner group and otherwise keeps the claim without starting a replacement. The help output is now byte-identical to base. Neither removal was requested by any instruction in this change, and both were made to text that predates it. Returning them is scoping, not a third restoration: nothing is being added to those files here.
…he runner before startup released its lock. Added a public-list synchronization barrier in tests/fm-procevent.test.sh. Forced-delay reproduction detected the deadlock before the fix; all four cases passed afterward. Targeted lint, Bash syntax, and whitespace checks passed. Greptile’s watchdog requirement conflicts with the recorded R2 decision; runtime behavior and documentation remain unchanged. Full CI rerun belongs to the outer executor
…hey fail On the failure path only, these cases now print what they actually saw: the identity recorded at claim time, the identity readable at that moment, the size of the signals file, the leader's state and wchan, every live member of the runner's process group with its own state and wchan, the elapsed time since the stop began, and what retire said. None of it runs when a case passes. WHY THIS IS KEPT, stated accurately rather than by its original reason. It was written to make an unexplained CI failure verifiable. That failure is now explained - it was a fixture deadlock, diagnosed and repaired in the preceding commit - so that justification has expired and is not the reason given here. The reason it stays is smaller and independent of that failure: it is already written, it is small, it sits in the file whose assertion this change reworked, and an assertion that could not say why it failed cost most of a morning to diagnose from the outside. The next failure will not be this one. WHAT A PASSING RUN WOULD NOT MEAN: a pass is a sample of behaviour already observed many times, not proof that anything is fixed. Only a failure carrying the evidence above establishes a cause.
…sh: removed premature child completion and waited for runner exit before retiring the restart fixture. Controlled Linux reproductions demonstrated failure before and success after. The full Linux process-event suite, six focused macOS checks, targeted ShellCheck, Bash syntax, and whitespace checks passed. Runtime behavior, guard debounce, and documentation remain unchanged. CI rerun belongs to the outer executor
A THIRD WAY, not a capitulation to the reviewer and not a refusal of it.
The automated reviewer's grievance was the LOOSENESS OF THE BOUND, not the
number of observations the guard makes before it acts. It asked for a single
read because that was the only route it could see to an acceptable bound. There
was another route, and this change takes it: the bound is reached and both reads
are kept.
TIGHTENED - the SPACING of the guard's two reads, not their number. The owner
watchdog now sleeps half the configured check interval and still requires two
consecutive failing reads, so the pair completes inside one check interval
instead of costing two. Worst-case detection falls from the lease term plus TWO
check intervals to the lease term plus ONE. At the shipped 600s lease and 15s
interval the stated bound falls from ~635s to ~620s.
PRESERVED - the second read. bin/fm-procevent.sh's two-consecutive-miss rule is
untouched. WHY IT PROTECTS: the guard's inputs are a lease read and a state-root
identity read, and either can fail transiently on a live, healthy home. Acting
on the first failure would let one isolated unreadable read kill a live service.
Requiring a second, independent read is what makes that impossible, and it is a
protection rather than padding. Nothing was traded away to reach the bound.
Both properties are now guarded by their own case, and each was proven by
MUTATION rather than asserted:
- putting a full interval back between the two reads fails the bound case:
"still running 17.0s after the last owner activity, against a documented
bound of 15s";
- acting on one failed read fails the new debounce case: "one unreadable lease
read ended a runner whose home was still alive" - while the bound case then
passes FASTER, 9.9s against 13.1s. The unsafe variant being the quicker one
is exactly why these are two cases: one elapsed-time case would have
registered the removal of the protection as an improvement.
MEASURED, sampling the phase between the guard's check clock and the lease clock
across eight runs per variant, on macOS (Darwin 25.5.0). Reaping an orphaned
listener whose home stopped refreshing its lease:
lease 2s / interval 1s: 4.41-5.29s before, 3.48-4.65s after
lease 2s / interval 4s: 7.69-8.12s before, 5.94-6.13s after
The 4s configuration is the informative one: the gap is about one check
interval, which is precisely the term that was removed.
A previously unstated term of the bound surfaced while measuring: the lease age
is compared in whole seconds, so a configured lease of N is honoured until that
age reads N+1. It is now part of the documented bound and of the regression's
derivation instead of being absorbed into a fudge factor.
The bound regression derives its deadline from the documented bound instead of a
flat number, and PINS the phase between the guard's check clock and the lease
clock rather than sampling it, because with a sampled phase a guard spending two
intervals passes about half the time on a lucky alignment. Its load slack is
additive and stays under half a check interval, so an extra whole interval
cannot hide inside it. The two flat deadlines that were there before (40s and
20s) and the doubling allowance on the derived one are gone; that looseness was
the reviewer's third complaint.
The stop's own grace is untouched: 2s for the ordinary signal, then 2s for the
forced one. It is a ceiling paid only by a group that outlives the signal it was
sent, not a delay every stop pays - a healthy runner's whole retire measures
0.40-0.66s on this host. The reviewer's literal "lease plus one tick" is
unreachable by any implementation, since signalling a process and giving it any
chance to exit takes non-zero time; detection now meets it and the stop runs
inside its own ceiling, and the contract says so rather than glossing it.
NECESSARY BUT NOT SUFFICIENT, and written BEFORE this head's integration runs
start rather than after they report. On the previous head, "Behavior portable
serial 1" and "Behavior portable serial 4" were both CANCELLED at the job
ceiling, independently of this finding. A new head triggers fresh runs, so those
two lanes MAY complete this time. IF THEY DO, THAT IS NOT EVIDENCE THE CEILING
DEFECT IS FIXED. It is one more sample of a lane that has been cut repeatedly
and sometimes is not; the shard-packing repair for it is open separately. Do not
reread a lucky pass here as a resolution.
Relatedly, and deliberately: the per-script duration hint in bin/fm-test-run.sh
was NOT updated even though the two new cases add ~19s of wall clock.
docs/fm-test-portable-shards.md says those hints are replaced wholesale from CI
timing artifacts of green runs, and that repair is the open request doing it; a
hand-edited estimate here would collide with it and silently repack the shards.
This suite runs in portable serial shard 3, which was green in the last run.
Verification: tests/fm-procevent.test.sh green, plus
tests/fm-captain-hold-lifecycle.test.sh, the test-coverage guard, and
bin/fm-lint.sh. The unrelated "reconcile stops a runner whose registration was
removed" case flaked in 4 of 7 local full runs; an isolated 20-trial
reproduction measured it at 13/20 unclean before this change and 11/20 after, so
it is issue 4080 and is not aggravated here.
…e timing phase
REPAIRED BEFORE PUBLICATION, AND IT WAS OURS. The half-interval arithmetic added
by the previous commit read a zero-prefixed interval as octal: 010 halved to 4
instead of 5, and 08 was not a number at all, so the owner guard died before
reporting ready and the runner failed closed and never listened. The validator
accepts those values and `[` compares them as decimal, so this broke a
configuration that worked before. Introduced by this delivery, found in review,
repaired here. Forcing base ten before the arithmetic is the whole runtime fix.
Proven by driving it rather than by reading the source: a new case starts a real
listener at 08 and at 010 and observes the guard's actual sleep argument - 4s and
5s. Removing the normalisation turns that case red with "a zero-prefixed decimal
interval (08) prevented the listener from starting".
THE TIMING PHASE IS NOW OBSERVED AND ENFORCED, NOT ASSUMED. The bound case
pinned its phase by CONSTRUCTION, from an assumed startup time, and enforced
nothing. Review was right that this is not enough: once startup reaches about two
seconds the expiry lands in a different part of the interval and the case
silently stops rejecting a two-interval guard while still reporting success. A
bound that cannot fail for the reason it names is the defect this whole delivery
exists to correct, so it must not ship inside the fix for it.
Now the lease is synchronised to the guard's own FIRST observed lease read,
every later real read is recorded, and the case REFUSES unless one recorded read
proves the required phase: it read the synchronised reference, it was still
fresh, and it began late enough that two further full intervals could not finish
before the deadline. An unestablished precondition refuses; it does not proceed
on trust. The derived deadline, the two-read debounce and the additive slack are
unchanged, and the slack invariant is now asserted rather than left to a comment.
Review also found the deadline was only ever checked while the group was still
alive, so a sampler descheduled past it would see the group gone and certify
success. The observed completion time is now checked too.
PROVEN BY MUTATION, each one run against this code:
- remove the decimal normalisation -> the interval case fails on 08;
- a full interval between the two reads -> "the guard exceeded its bound:
group still running 17.1s ... against a documented bound of 15s";
- a full interval WITH startup forced to ~2.5s, which is exactly the condition
the old construction pin could not survive -> still red, same message;
- the same ~2.5s startup with the correct guard -> still passes, 13.0s against
the 15s bound, so the delay alone does not break the case;
- phase evidence made unavailable -> "could not establish the required
pre-expiry guard-read phase", a refusal rather than a pass, even though the
group stopped quickly;
- act on one failed read -> the debounce case fails and the bound case passes
FASTER, 9.5s against 12.7s, which is why these remain separate cases.
Verification: full tests/fm-procevent.test.sh green, and bin/fm-lint.sh clean.
mremond
force-pushed
the
fix/procevent-stop-escalation-and-retire-deadlock
branch
from
September 9, 2026 21:48
b8c3621 to
f9d9408
Compare
mremond
changed the title
Owner
|
Speaking as Kun's firstmate: this is merged. Thank you @mremond — really appreciate you taking the time on this. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
ORIGINAL GOAL, in the captain's own words, from the ticket that opened this work:
The 2026-09-08 orphaned processes reveal TWO defects beyond the test leak already routed, and the second has production consequences. Established in the main home after stopping the 191 processes.
WHAT IS CERTAIN
WHAT IS STRONGLY SUPPORTED BUT NOT PROVEN, and this is the point to measure first
2. 96 of the 191 ignored the ordinary stop signal and required a forced kill. Yet bin/fm-procevent-lavish.sh DOES install a signal trap that cleans up and re-raises (around line 280). The most likely explanation is classic shell behaviour: a signal received while a child runs in the foreground is only handled AFTER that child returns. The child here being precisely a blocked poll, the trap cannot fire. IF THAT IS IT, the consequence goes beyond tests: the process-event contract asserts that an identified surviving group is STOPPED before being replaced. That stop uses the same ordinary signal. It could therefore fail or hang exactly when it is needed - during an ownership recovery after a crash.
TO ESTABLISH, in this order: (a) confirm or refute the deferred-signal mechanism by REPRODUCTION, not by reading; (b) if confirmed, does the process-event contract's stop path actually keep its promise, answered by measurement on a genuinely blocked child, not on a healthy one; (c) is a reaper needed somewhere OTHER than teardown - at session start, or in the supervision loop - to bound the leak in time.
NEIGHBOUR: fm-tests-leak-poll-processes handles the leak on the test side. This ticket does not replace it and does not depend on it: even if the tests stop leaking, a child that cannot be stopped remains a production problem.
That investigation was carried out and produced this branch and pull request 4009, which repaired the runner's stop escalation and a retirement deadlock.
LATER GOAL, supplied by the captain after that request was open, and the reason for THIS change:
"We have to make greptile happy so that it can be merged."
The automated reviewer on request 4009 does not merely record a finding: it states the request MUST NOT BE MERGED until the point is resolved, so the finding is blocking delivery rather than costing a badge. The captain asked for the request to become mergeable. He did NOT ask for the runner guard's two-consecutive-check debounce to be removed, and never said he was giving up that protection. Those are two different things and must not be traded for one another.
THE REVIEWER'S BLOCKING FINDING, quoted so this intent stands alone: "Watchdog deadline remains too loose. When the owner disappears immediately after a successful check, the watchdog waits for two subsequent failed checks before starting a stop that can consume another four seconds, leaving the listener alive beyond the required lease-plus-one-tick deadline. The revised regression also codifies this looser bound and permits twice its duration."
WHAT WAS DONE AND WHY, so a reviewer reading only the diff does not mistake a deliberate choice for an oversight:
TIGHTENED, and it is the first thing to understand about this change: the SPACING of the guard's two reads, not their number. The owner watchdog now sleeps HALF the configured check interval and still requires two consecutive failing reads, so the pair completes inside one check interval instead of costing two. Worst-case detection falls from the lease plus TWO check intervals to the lease plus ONE.
PRESERVED, and deliberately so: the second read. bin/fm-procevent.sh's two-consecutive-miss rule is unchanged. It is what stops a single unreadable read from killing a live runner, and it is a protection rather than padding. The reviewer wanted a single read because that was the only way it could see to reach an acceptable bound; the bound was reached another way and the protection kept.
The stop's own grace - two seconds for the ordinary signal, then two for the forced one, in stop_runner_pid - was deliberately LEFT ALONE. It is a ceiling paid only by a group that outlives the signal it was sent, not a delay every stop pays; a healthy runner's whole retire measures 0.40 to 0.66 seconds on this host. The reviewer's literal "lease plus one tick" is unreachable by any implementation, because signalling a process and giving it any chance at all to exit takes non-zero time. Detection now meets it; the stop then runs inside its own ceiling, and that limit is stated in the contract rather than glossed over.
Measurement drove the design rather than following it, and is recorded in docs/verification/process-event-sources.md: reaping an orphaned listener took 5.9 to 6.1 seconds with the tightened spacing against 7.7 to 8.1 seconds before, at a 2-second lease and 4-second check, sampling the phase between the guard's check clock and the lease clock across eight runs per variant.
A previously unstated term of the bound surfaced while measuring: the lease age is compared in whole seconds, so a configured lease of N is honoured until that age reads N+1. It is now part of the documented bound and of the regression's derivation instead of being absorbed into a fudge factor.
Both properties are now guarded by their own case, and each was proven by MUTATION rather than asserted. Putting a whole interval back between the two reads turns the bound case red. Making the guard act on one failed read turns the new debounce case red - and under that mutation the bound case passes FASTER, 9.9 seconds against 13.1. The unsafe variant is the quicker one, which is exactly why these are two separate cases: a single elapsed-time case would have registered the removal of the protection as an improvement.
The bound regression PINS the phase between the guard's check clock and the lease clock instead of sampling it, because with a sampled phase a guard spending two intervals passes about half the time on a lucky alignment. Its deadline is DERIVED from the documented bound, and its load slack is additive and deliberately under half a check interval so an extra whole interval cannot hide inside it. The two flat deadlines that were there before (40 and 20 seconds) and the doubling allowance on the derived one are gone; that looseness was the reviewer's third complaint.
The per-script duration hint in bin/fm-test-run.sh was deliberately NOT updated even though the two new cases add about 19 seconds of wall clock. docs/fm-test-portable-shards.md says those hints are replaced wholesale from CI timing artifacts of green runs, and a separate open request already does exactly that; a hand-edited estimate here would collide with it and would silently repack the shards. This suite runs in portable serial shard 3, which was green in the last CI run.
WHAT THIS DELIBERATELY DOES NOT CLAIM: satisfying the reviewer is necessary but not sufficient. On the previous head, Behavior portable serial 1 and Behavior portable serial 4 were both cancelled at the job ceiling, so the head was not green for reasons independent of this finding.
HOW THIS MUST READ, and it is part of what was asked rather than presentation: as a THIRD WAY, neither a capitulation to the reviewer nor a refusal of it. The grievance was the LOOSENESS OF THE BOUND, not the number of observations; a single read was simply the only route the reviewer could see to an acceptable bound. Another route existed, the bound was reached, and both reads were kept. A reader must be able to see both halves - what was tightened, and what was PRESERVED and why the two-read debounce protects: the guard's inputs are a lease read and a state-root identity read, either of which can fail transiently on a live healthy home, so acting on the first failure would let one isolated unreadable read kill a live service.
WRITTEN IN ADVANCE, BEFORE THIS HEAD'S INTEGRATION RUNS REPORT: on the previous head, the lanes "Behavior portable serial 1" and "Behavior portable serial 4" were both CANCELLED at the job ceiling, independently of the reviewer's finding. A new head triggers fresh runs, so those two lanes MAY complete this time. IF THEY DO, THAT IS NOT EVIDENCE THE CEILING DEFECT IS FIXED - it is one more sample of a lane that has been cut repeatedly and sometimes is not, and the shard-packing repair for it is open separately. This caveat is recorded before the run precisely so that a lucky pass cannot be reread afterwards as a resolution.
A NOTE ON THIS ATTEMPT, recorded rather than explained afterwards: a previous run of this same work failed at its test step on a pre-existing 24-contender concurrency fixture that failed IDENTICALLY on the base revision under a host load average of 14 to 18, which its own analyzer recorded as not change-specific. That failure also discarded the pipeline's own fix commit with no recoverable reference; the three repairs it had made were reconstructed and then re-proved independently by mutation rather than relayed, and they are what the second commit on this branch carries. This attempt is driven deliberately under a sustained host load of about 13, after a declared 90-minute bounded wait during which the load plateaued rather than descending, because an unbounded wait for a condition with no evidence it will arrive becomes permanent.
TWO THINGS THAT MUST APPEAR IN THE DESCRIPTION, because they live outside any one commit message and would otherwise be lost:
FIRST, A REGRESSION THIS DELIVERY INTRODUCED, NAMED AS OURS RATHER THAN QUIETLY REPAIRED. The half-interval arithmetic added here read a zero-prefixed check interval as octal: FM_PROCEVENT_OWNER_CHECK_SECONDS=010 halved to 4 instead of 5, and 08 was not a number at all, so the owner guard died before reporting ready and, because the runner fails closed when its guard cannot initialise, the listener never started. The validator accepts both values and
[compares them as decimal, so this broke a configuration that worked before this change. It was introduced by this delivery, found in review, and repaired before publication by forcing base ten before the arithmetic. It is stated plainly because a description that lists only the defects fixed in other people's code while silently swallowing one of its own is an understatement fault.SECOND, THE PROOFS, WHICH ARE MUTATIONS RUN AGAINST THIS CODE RATHER THAN ASSERTIONS ABOUT IT. Six, each an actual run:
What Changed
010to 4 and made08prevent listener startup; normalize accepted intervals to decimal before arithmetic.08; restoring full-interval spacing breaches the documented 15s bound at 17.1s; that mutation still fails with approximately 2.5s startup delay; the correct guard with that delay passes at 13.0s; unavailable phase evidence fails with “could not establish the required pre-expiry guard-read phase”; and stopping after one failed read fails debounce while passing the timing case faster, at 9.5s versus 12.7s. Separate cases therefore protect both the tighter bound and the retained debounce.Risk Assessment
✅ Low: The runtime changes are narrowly scoped and preserve ownership checks, two-read debounce, and escalation limits; no material defects or intent contradictions were found.
Testing
Targeted lifetime tests and live CLI scenarios passed after correcting two driver setup races. Baseline and mutation runs reproduced the intended failures, including phase and late-observation refusals; transcripts were retained. No full suite, lint, or other pipeline phase ran.
Evidence: Consolidated observations, mutation proofs, and cleanup
Source: Consolidated observations, mutation proofs, and cleanup
Evidence: Initial live CLI transcript, including corrected driver setup issues
Source: Initial live CLI transcript, including corrected driver setup issues
Evidence: Additional live checks and successful refusal rerun
Source: Additional live checks and successful refusal rerun
Evidence: Targeted lifetime regressions
Source: Targeted lifetime regressions
Evidence: Baseline, mutations, and controls
Source: Baseline, mutations, and controls
Evidence: Late completion observation correctly refused
Source: Late completion observation correctly refused
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
TMPDIR="$PWD/.test-phase-tmp" /bin/bash ~/.no-mistakes/evidence/01M2414ECRE7T77CAXN49RSA2F/targeted-lifetime.sh— selected existing lifetime cases, including zombie-leader escalation.python3 ~/.no-mistakes/evidence/01M2414ECRE7T77CAXN49RSA2F/live-cli.py— isolated production CLI checks with real blocking tail processes.ONLY_SCENARIO='Unproved cold' python3 .../live-cli.py, followed byONLY_SCENARIO='Unproved cold|Expired blocked|Reconcile stops' python3 .../live-cli.py— corrected driver setup and re-drove affected checks.python3 ~/.no-mistakes/evidence/01M2414ECRE7T77CAXN49RSA2F/mutations.py— base retirement failure, decimal normalization removal, full-interval spacing with and without delayed startup, delayed-start control, unavailable phase evidence, and single-read debounce/bound variants.TMPDIR="$PWD/.test-phase-tmp" PROCEVENT_CODE_ROOT="$PWD" /bin/bash ~/.no-mistakes/evidence/01M2414ECRE7T77CAXN49RSA2F/late-observation.sh— correctly refused completion first observed at 18.6s against a 17s deadline.Final process inventory, temporary-directory removal, and Git status/commit/tree verification confirmed complete cleanup and unchanged target 77e799275d351c954a2a8f25e9ce124db0bfaa32.✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.