Skip to content

fix(workflow-host): a lone recall function is never shown waiting behind others - #137

Merged
rami-hatoum merged 5 commits into
mainfrom
fix/deterministic-trigger-and-recall-tests
Oct 9, 2026
Merged

rami-hatoum merged 5 commits into
mainfrom
fix/deterministic-trigger-and-recall-tests

Conversation

@rami-hatoum

@rami-hatoum rami-hatoum commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Two tests passed alone and failed when every package's tests ran at once. Each now waits for, or sets up, the state it asserts, so how busy the machine is no longer changes the result.

The start rate of a workflow with several triggers

packages/workflow-host/src/triggers/several-triggers.test.ts proved that a workflow's event trigger starts it 60 times a minute and that its schedule's start is not counted. To get there it published 61 events one after another into the host's SQLite file, while the host's follower started a run for each over a second connection to the same file. So the test's length grew with the limit and with the load on the machine: 94 ms alone, 507 to 643 ms beside 64 busy threads, and 5,049 ms against the 5,000 ms limit in the full gate. It never waited on the clock; the work was the 61 deliveries.

The test now records that the workflow was already started 59 times in the current minute, the way start-rates.test.ts records starts already waiting in its backlog test. It then saves the workflow and publishes two events. The every schedule starts its run, the first event makes the 60th start and the second waits for a later minute. If a schedule start were counted, both events would wait. If event starts were not counted, both would start. Each of those faults, put into reactions/reacting.ts or reactions/start-rates.ts, fails the assertion. start-rates.test.ts still proves that 60 starts are admitted one by one. The test takes 37 ms alone and 166 to 338 ms beside 64 busy threads.

A recall function whose view is being built

packages/server/src/recall/recall-versions.test.ts waited for the recall function's standing to read rebuilding. The standing also reads rebuilding before the projector has added the view's row (primitives/recollection/src/primitive/recall-standing.ts). The projector added the row as waiting and moved it to rebuilding with a second write (packages/workflow-host/src/projector/view-reconciling.ts). So the recall the test made next could answer any of three details, depending on how far the projector had got: not begun, waits behind other recall functions, or is building. Alone, one run in eight returned from the wait before the row existed. Beside 64 busy threads, one run in twelve failed on the waiting detail.

The test now waits until the test's fold gate holds the rebuild's first fold. The projector asks for that fold only after it has set the row to rebuilding, and while the fold is held the view stays at 0 folded events, so the "is building" detail is the only one possible. The waiting detail is a real state too: it is what a function shows when another function in its brain holds the only rebuild slot. The test for that state was already deterministic in its order. It now waits for the held fold through the same helper, foldsHeld, bounded like the standing waits instead of by vi.waitFor's default of one second.

The same test also checked that the answer came within 2,000 ms. The held fold already proves what that bound stood for: the rebuild cannot finish while its first fold is held, so a recall that waited for its view would never answer, and the answer says the view has folded 0 events. The wall-clock bound is removed.

A lone recall function is never shown waiting

Between the projector's two writes, a brain's only recall function read as waiting, and a run of it answered that it "waits to build its view of version 1, behind other recall functions of this brain being built", when there were none. The projector now writes a new or renewed view in the phase its place gives it. It counts the slots over the views still building and the new ones, in the order saved, before it writes anything. It removes a dropped view first, so a new view never takes a slot that a dropped one still holds. The rows suite, on SQLite and PostgreSQL, reads the row after every write a reconcile makes. It sees a view built alone as rebuilding from its first write, both new and renewed; on the earlier code it sees waiting after each. The README's account of the projector says so.

The same holds when the view holding a slot is retired or renewed while another waits. The waiting view is moved to rebuilding before the holder's row is deleted or reset, and before any new row is written. Otherwise it would read waiting beside a free slot until a later write. The rows suite covers both cases: the holder retired while another view is saved, and the holder renewed while another waits. After every write it finds no row waiting while fewer views rebuild than there are slots. Deleting the holder first would fail the retirement case, because the waiting view reads waiting with nothing rebuilding between the delete and its own move. Moving it first means the retired holder and the promoted view both read rebuilding until the delete. A retired function shows no standing, so no one sees that moment. The writes after the rows are read again now only move rows past the slots back to waiting, as when the brain may build fewer views at once than before. The rows suite covers that case as well.

Checks

  • Each changed test, 20 runs alone: all passed.
  • pnpm check without PostgreSQL: passed, 55 of 55 tasks, with 5,188 tests passing and the 194 PostgreSQL tests skipped.

rami-hatoum and others added 4 commits October 9, 2026 09:56
…o event starts, not 61

The test published 61 events one after another into the host's SQLite
file while the host's follower started a run for each on a second
connection to the same file, so its length grew with the start rate and
with the machine's load: 94 ms alone, 507 to 643 ms beside 64 busy
threads, and 5,049 ms against the 5,000 ms limit in the full gate.

The test now records that the workflow was started 59 times in the
current minute, as the backlog test of start-rates.test.ts records the
starts already waiting, and then publishes two events: the every
schedule still starts its run, the first event makes the 60th start and
the second waits for a later minute. A schedule start counted in the
rate leaves both events waiting, and an event start left out of it
starts both, and each fails the test. That 60 starts are admitted one by
one stays proven by start-rates.test.ts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…its first fold

The test waited for the recall function's standing to read rebuilding,
which the standing also reads before the projector has added the view's
row (recall-standing.ts answers rebuilding when no view is kept). The
projector adds the row as waiting and moves it to rebuilding in a second
write (view-reconciling.ts), so the recall the test made next answered
whichever of three details held at that moment: not begun, waits behind
other recall functions, or is building. Alone, one run in eight saw the
standing before the row existed, and beside 64 busy threads one run in
twelve failed on the waiting detail.

The test now waits until the gate holds the rebuild's first fold, which
the projector asks for only after it has set the row to rebuilding and
which holds the view at 0 folded events, so the building detail is the
one the situation gives. The test of a function that waits behind
another, which shows the waiting detail, waits the same way, with the
bound the standing waits use rather than vi.waitFor's default second.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ind others

The projector wrote a new or renewed view as waiting and then, in a
second write, moved the views that took a slot to rebuilding. Between
the two writes a brain's only recall function read as waiting, and a
run of it answered that it waits behind other recall functions of the
brain being built, when there were none.

A new or renewed view is now written in the phase its place gives it:
the slots are counted over the views still building and the fresh ones,
in the order saved, before anything is written, and a dropped view is
removed first, so a fresh view never takes a slot a dropped one still
holds. The rows suite watches the row after every write a reconcile
makes and sees a view built alone as rebuilding from its first write,
new and renewed; on the earlier code it sees waiting after each.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d fold, not a clock

The test measured that the recall answered within 2,000 ms. The gate
already holds the rebuild's first fold until after the answer, so the
rebuild cannot finish before it: a recall that waited for its view
would never answer, and the answer's detail says the view has folded 0
events. The wall-clock bound added only a dependence on the machine's
load, and is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rami-hatoum rami-hatoum changed the title fix(workflow-host): a recall function built alone is never shown waiting, and two tests no longer depend on timing fix(workflow-host): a lone recall function is never shown waiting behind others Oct 9, 2026
…er's row changes

When the view holding a slot was retired or renewed in the reconcile
that freed it, the view waiting behind it kept reading waiting beside
the free slot until the phase writes after the next read of the rows: a
retired holder's row was deleted first, and a renewed holder left both
rows waiting for a moment.

A still-building view that takes a slot is now moved to rebuilding
first, before the drops and before the new and renewed rows. With the
drops first, the rows suite sees the waiting view alone and waiting
between the delete and its move; moving it first leaves the retired
holder and it both rebuilding until the delete, which no standing
shows, since a retired function has none. The suite records the rows
after every write of a reconcile, with the holder retired while another
view is saved and renewed while another waits, and no row reads waiting
while fewer rows rebuild than there are slots.

Since a view that takes a slot now moves with the other changes, the
writes after the rows are read again only move back to waiting the rows
past the slots, as when the brain may build fewer views at once than it
did; the suite now covers that case too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rami-hatoum
rami-hatoum merged commit 3df2373 into main Oct 9, 2026
18 checks passed
@rami-hatoum
rami-hatoum deleted the fix/deterministic-trigger-and-recall-tests branch October 9, 2026 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant