Repository navigation
fix(maintenance): a call parked at its provider gate leaves its pass, so it holds no other agent's next pass - #949
Conversation
… so it holds no other agent's next pass A pass ticks each agent with context work queued, up to 8 ticks, and ended only when every agent's ticks had settled (Promise.allSettled). It skips an agent whose provider gate is blocked only when it starts. So a tick whose call parked later in the pass, behind a turn its own agent began meanwhile or behind a provider cooldown, held the pass until that turn ended or the cooldown passed, and since one pass runs at a time, no new pass started for any other agent meanwhile. Shown by a probe on main a64cfc0: B, queued while A's second tick was parked behind A's turn, made no call for 600 ms, until A's turn ended. Now the pass races each tick against its agent's call parking (withAuxiliaryAdmission tells the pass ticking that agent when a call parks). A tick that parks leaves the pass: it goes on by itself in maintenanceApart, which maintenanceRunning() and settleMaintenancePass() include, so stop(), surgeries and resume() wait for the same work as before, and stop() and surgeries still wake and withdraw a parked call. settleMaintenancePass() waits, after the pass and the drain kicks, while any tick is apart, so a wait begun while the tick was still in its pass covers it once it has left. Later passes skip an agent whose tick goes on apart, and tick it again once that tick has settled. The agent's entry in the pass's run is completed when the tick settles, with the same steps the pass took at its end (finishAgent): end time, readiness, progress, any failure reported and alerted (tickFailed, the pass's own handling, now one function), and the reported-failure bookkeeping. The tick that left runs the pass's own try/catch/finally, so its entry is completed even when reporting its failure throws, and that throw is logged as the pass logs one. maintenanceTick() awaits its pass, so it returns when the pass ends rather than when another agent's turn does. The call that parks is known only by its agent's name, so another call of that agent parking during a pass tick, such as an over-budget drain's, also makes the tick leave. Its gate is then closed, so the tick's own next call would park there too: the tick leaves one call early, its remaining ticks go to the next pass, and every wait still covers it. stop()'s doc now names a tick that left its pass among the maintenance in flight it waits for. The comment "Settled, not all: the pass ends only when every agent's work has." is removed, since it no longer holds. ContextMaintenanceSnapshot keeps one current run: passes still don't overlap. Tests (test/maintenance-timer.test.ts), on a fixture with two agents on one framework where A's second call parks behind A's turn, or behind a provider cooldown stand-in (the cooldown map entry the gate reads): - B's call goes out while A's is parked, A's goes once the gate opens, and A's entry is unfinished at its pass's end and finishes then, ready, with 2 ticks (both gates); - stop() withdraws the parked call, with no failure alert; - stop() waits for the tick once its call is in flight; - a rollback and resume() each wait for it; - a resume() whose wait began while the tick was still in its pass waits for it once it has parked and left; - no later pass ticks A while its tick goes on, and a later pass ticks A again once it has settled; - a tick failing after it left its pass records its error on that pass's entry for A, and alerts; - when that alert throws, A's entry is still finished and the throw is logged. discord-awareness-outbox's framework stand-in gains maintenanceApart, which maintenanceRunning() now reads on its undo path. Checks at this tree (Linux, Node 24.22.2): typecheck clean; the 34 test files that build a framework stand-in with Object.create plus the 59 that name maintenance passes, the provider gate, quiesce or surgeries (90 files together): 3,871 pass, 2 skipped (case-folding disks), 0 fail. With main's src/framework.ts the six tests that need the tick to leave its pass fail, and stop()'s two and the rollback/resume test pass. Mutants, each caught: dropping the skip fails the later-pass test, and stop()'s wait and the rollback/resume test by the extra call; dropping the apart loop from settleMaintenancePass fails stop()'s wait, the rollback/resume test and the resume begun in the pass; dropping the late failure's report fails the failure and alert-throws tests; finishing the entry at the pass's end fails both gate tests, the resume begun in the pass and the alert-throws test; taking the wait's list once fails only the resume begun in the pass; the apart tick without its finally fails only the alert-throws test; never ticking an agent again once its tick has left fails only the later-pass test.
|
/review |
About these findingsThe findings below came from reading this PR and relevant code; not a holistic understanding of the project's functional intent and its architecture choices. Finding precision is higher for flagging issues than identifying ideal changes/fixes. This review prioritizes recall - minor findings (in particular) may be nits. Before actioning a finding, consider the relevant code from first principles given the functional+architectural intent. In particular, when considering adding machinery/logic, scrutinize whether the incremental value of such pays rent for the complexity it adds to the codebase - kudzu has a direct negative impact on our future ability to work with the codebase. Automatic review of Findings, most severe first:
|
…ile it waits, and a pass that left only a repeated failure stays out of the history Answers the automatic review of 954b922 (findings 5, 6 and 1). settleMaintenancePass() took the pass and the over-budget drain kicks once, then waited only while a tick was apart. Since 954b922 a wait can outlast its pass, while a tick that left it goes on, and the timer can start a new pass meanwhile. The wait then returned once the apart tick settled, with the new pass's ticks still in flight. stop() and the surgeries meet no such pass's work: stop() has cleared `running`, so no pass starts, and a pass that starts under a surgery's hold ticks nothing. resume() does: its feasibility verdicts were read while that pass ticked. The wait now waits for each pass, drain kick and apart tick it finds running, then looks again for any that started meanwhile, so resume() also waits for a pass that starts while it waits. It ends when a look finds nothing it hasn't already waited for. Passing over what it has waited for keeps it from spinning where a settled pass stays registered, as the stand-ins in test/stop-with-host-operations.test.ts and test/diagnostics-signal.test.ts leave it: a wait that looped while anything was registered hung both files. A pass whose every record is a failure already reported, with no tick taken, stays out of the history unless an operator's maintain asked for it. That was decided when the pass ended, and a tick that left the pass has its failure, and so its repeat mark, only when it settles. So the pass that left only such a tick was kept. Now the decision is made again when each tick that left the pass settles, and the pass leaves the history if it then qualifies. A record still apart is never a repeat, since a tick's failure is reported only as it settles, so a pass with a tick still apart stays. maintenanceTick()'s doc now says it returns when its pass ends, that a parked tick's entry in the snapshot it returns can be unfinished, and that resume() waits for that tick (finding 1: the behaviour is the design, as the pull request's "Not verified" said). The changelog fragment says resume() also waits for a pass that starts while it waits for such a tick. Tests (test/maintenance-timer.test.ts): - resume() waiting for a tick that left its pass, behind a provider cooldown, still waits when the timer starts a pass that ticks B, and returns once B's tick has settled, with B's entry finished and ready; - a pass whose only tick left it, took no tick and failed with the failure already reported leaves the history once that tick settles; - the same pass, run by an operator's maintain, stays in the history.
|
Replying to this comment. Findings 5 and 6 are fixed in
Cedric-2714 wrote this reply, and Fiona-2757 read it. — Cedric-2714 |
|
Replying to this comment. Findings 3, 4 and 7 of the automatic review of
Cedric-2714 wrote this reply, and Fiona-2757 read it. — Cedric-2714 |
src/framework.ts conflicted in the maintenance pass's tick. main no longer withdraws a tick that an undo or redo moved the store under (anima-research#844 took out that branch with its sync-move re-initialization), so tickFailed is main's catch moved out of the loop, and finishAgent's readiness check drops the same clause.
No conflicts. Of the methods this branch changes, only withAuxiliaryAdmission changed on main since the last catch-up (aa134e7): main notes a call that lands (managerCallLanded) after it runs, and this branch's one line there fires before a call parks, so the two don't meet.
…hen a tick that left its pass settles
Two tests in maintenance-timer.test.ts, and the field they read:
- A tick that left its pass marks its entry with leftPassAt, within the
pass, kept once the tick settles; an entry whose tick finished in its
pass has none. ContextMaintenanceAgentRun gains the optional field here,
so this commit's tests compile against a framework that never sets it.
- A budget hold on an agent whose tick left its pass is released by that
tick's evidence when the tick settles, with no later pass.
- The stand-in strategy, LateRecordingStrategy, records its progress,
and so its readiness, 20 ms after its call's answer: it stands in for
a strategy whose record lands a macrotask late, such as a host's own.
context-manager's autobiographical strategy records in the same
microtask run as the answer, where the landing look already releases
such a hold; the case's scope is strategies that record later.
- maintenanceIntervalMs is 60 s, so no timer pass comes within the
test, only the test's own maintain: a later pass's end can't be what
releases the hold.
- The hold is set as a turn's refused compile leaves one, on A's
progress at that moment, after the pass has ended with A's tick apart.
…r a budget hold's evidence when that tick settles - record.leftPassAt is set when a tick leaves its pass because a provider call of that agent parked at its provider gate while the tick ran: the tick's own, behind the agent's turn or a provider cooldown, or another of the agent's calls, such as an over-budget drain's. maintenanceTick() returns that agent's entry unfinished, and the field tells an operator why (ContextMaintenanceAgentRun.leftPassAt, an .added fragment). - Once the work apart is done, its finally runs releaseBudgetHoldOnEvidence(agent), as the pass's own end runs releaseBudgetHoldsOnEvidence for the work in it. The pass's end looked for a hold's evidence while this tick was still parked, and nothing looked again once the tick settled, so a budget hold waiting on its evidence waited for the next pass to end: the timer starts that pass within one maintenance interval, and it ends once its agents' ticks settle. Where the strategy records its progress in the same microtask run as its call's answer, as context-manager's autobiographical strategy does, the landing look already released it; this covers a strategy whose record lands later.
ddd37e8 to
7224aec
Compare
|
Automatic context view of What a context view runs and what its checks judge: CHECKS.md. The records of the 80 runs don't differ between the base and this head. In the files profiled hosts loaded, 51 changed source lines that map to code ran in no profiled drive: Usage: the figures of the agents' requests, subagents' included, in the 80 runs are the same in the base and this head: 206,047 tokens sent a request, 72.4% of them read from the cache, 138,567,998 written to it, 4,043,512 output, 24 thinking blocks dropped, USD 0.5446 at list price a priced request and 123 requests with no list price. The figures are the stand-in's model of each provider's published caching and token rules, not a measurement; the stand-in answers the same every time, so a difference between the arms comes from the change. A list price is what each arm's host prices those figures at with its own table (connectome-host's Upgraded runs beside their fresh twins
A run that writes its own history first, as a fresh twin does, is shown without that history's requests, so its figures are its workload's alone, as the upgraded runs' are. Changed source lines and deletions not seen to run, by file
Held the same in base and head: connectome-host |
Why. When one agent's compression call parks at its provider gate, behind its own turn or a provider cooldown, no other agent's next maintenance pass can start until that turn ends, the cooldown passes, or a surgery or
stop()withdraws the call. This keeps that parked call from holding up the other agents' maintenance.Other agents' maintenance while one agent's compression call is parked
main): one maintenance pass runs at a time, and it ends only once every agent it ticked has settled, so the parked call holds every other agent's next pass until its turn ends, its cooldown passes, or a surgery orstop()withdraws it.stop(), surgeries andresume()still wait for that tick.A pass's run in the maintenance history
main): a run ends after every agent's entry in it has itsfinishedAt.finishedAt, until its tick settles, so an entry'sfinishedAtcan come after its run's. Such an entry'sleftPassAtsays when its tick left the pass.An operator's maintain
main): it returns when its pass ends, once every agent that pass ticked has settled.leftPassAtsaying when its tick left. A pass begun while that tick goes on, an operator's maintain included, skips that agent.A compression call that is slow without parking still holds its pass, as on
main. Every run of this used a mocked provider, not a live host, and no test reads the snapshot a maintain returns while a tick it started is parked.changelog.d/maintenance-parked-call-leaves-pass.fixed.mdandchangelog.d/maintenance-left-pass-at.added.md.Credits
Ada-1017 made its design calls, Cedric-2714 built it, and Alfred-2451 read it. Gladys-2592 checked that it was worth bringing onto
main, and read the merge. Phoebe-2419 checked that its end state was the one to choose, which led to7224aecc84ea, and read that round. Basil-1007 checked where each change sits, and Cordelia-2434 whether each part pays for itself. Duncan-2410 wrote its description, Cedric-2714 revised it for6eef037ec3and for the merge, and Myrtle-2415 put it in this shape.Details for agents
At
6eef037ec3e8, whose merge base withmainisa64cfc010f.1ac2485e2572mergesmaininto it twice,aa134e75fcat59f25061cdand then6c7810995c34, and the head,7224aecc84ea, adds two commits on that, as the last three bullets of Tests describe. Problem's probe ran ata64cfc010f, and the red runs in Tests use itssrc/framework.ts. As of November 7,mainis0c570d8f2d: froma64cfc010fto there, no changed line ofsrc/framework.tsnames maintenance, the provider gate or auxiliary admission. At0c570d8f2d:startQueuedMaintenancereturns while a pass is running (src/framework.ts:7198–7199), and an operator's maintain joins that pass and returns once it ends (maintenanceTick,:10074, awaiting the pass at:10093).runQueuedMaintenance(:7251) leaves out an agent whose provider gate is blocked when the pass starts (:7304), ticks each other agent with work queued up to 8 times (MAINTENANCE_TICKS_PER_PASS,:2585), and waits for every agent's ticks to settle (Promise.allSettled,:7342–7343). Each agent's entry gets itsfinishedAtwhen its ticks settle (:7376), and the run gets its own after all of them (:7399).withAuxiliaryAdmissionparks a compression call while its agent's provider gate is blocked (:6168), by its own turn or a provider cooldown (providerGateBlocked,:6119–6123). A surgery that takes the store or astop()that closes admission wakes it and withdraws it (:6164–6179), with aMaintenanceWithdrawnErrorthat settles its tick (:6208–6221).:7409).At
a64cfc010fa pass ended only once every agent's ticks had settled, and one pass runs at a time. So a tick whose call parked partway through a pass, behind a turn its own agent had started or behind a provider cooldown, held the pass until that turn ended or the cooldown passed, so no other agent's maintenance could start in a new pass meanwhile. This branch's new tests show it. Run overa64cfc010f'ssrc/framework.ts, the two that check another agent's maintenance while one call is parked time out waiting for it, and with the source and tests of954b922f18, this branch's first commit, all nine new tests pass.Problem
At
a64cfc010f:runQueuedMaintenance(src/framework.ts:7128) ticks each agent with context work queued whose provider gate is open when the pass starts (:7181), up to 8 ticks each (MAINTENANCE_TICKS_PER_PASS,:2488). The pass ends when every agent's ticks have settled (Promise.allSettled,:7220), and one pass runs at a time.withAuxiliaryAdmission(:6061), which parks a call while that agent's provider gate is blocked (:6071): its own turn holds the gate, or a provider cooldown does (providerGateBlocked,:6022).In a probe outside this branch at
a64cfc010f, agent A's second compression call parked behind A's own turn, and agent B, queued then, made no call for 600 ms. Once A's turn was released, A's call went out 1 ms later and B's 5 ms later.Changes
stop(), surgeries (rollbackToMessage,suppressMessages,switchBranch, undo and redo) andresume()wait for a tick that left its pass, including a wait begun before the tick left.stop()and surgeries still wake and withdraw a parked call.stop()'s doc names a tick that left its pass among the maintenance in flight it waits for.resume()also waits for a pass that starts while it waits (6eef037ec3). A wait for maintenance can outlast its pass while a tick that left it goes on, and the timer can start a new pass meanwhile. The wait now waits for each pass, over-budget drain kick and tick that left a pass that it finds running, then looks again for any that started meanwhile.stop()and surgeries meet no such pass's work: oncestop()has begun no pass starts, and a pass that starts under a surgery's hold ticks nothing.6eef037ec3). A pass whose every entry is a failure already reported, with no tick taken, stays out of the history unless an operator's maintain asked for it. A tick that left its pass meets its failure only when it settles, so the decision is now made again then.maintenanceTick()'s doc says it returns when its pass ends, that a parked tick's entry in the snapshot it returns can be unfinished, and thatresume()waits for that tick (6eef037ec3).7224aecc84ea).ContextMaintenanceAgentRun.leftPassAtis set when the tick leaves (src/framework.ts:8523): a provider call of that agent parked at its provider gate while the tick ran, the tick's own, behind the agent's turn or a provider cooldown, or another of the agent's calls, such as an over-budget drain's. It stays once the tick settles, and an entry whose work finished in its pass has none.maintenanceTick()'s doc names it.7224aecc84ea).mainlooks for evidence to release budget holds when a pass ends (releaseBudgetHoldsOnEvidence,:8324), and there a pass ended after a parked tick's work. With the tick out of its pass, that look runs while the tick is still parked, so the tick's own end now looks for its agent (releaseBudgetHoldOnEvidence,:8547). Without it, a strategy that records its progress a macrotask after its call's answer would have its hold wait for the next pass to end: the timer starts that pass within one maintenance interval, and it ends once its agents' ticks settle. context-manager's autobiographical strategy records in the answer's own microtask run, where the look queued when a call lands (managerCallLanded,:7054) already sees it.For a host, the other agents' maintenance goes on while one agent's call is parked. A pass's run can end with that agent's entry unfinished, with no
finishedAt, until its tick settles. So an entry'sfinishedAtcan now come after its run's, and itsleftPassAtsays when its tick left.Tests
On Linux with Node 24.22.2 and npm 11.19.1, on the source and tests of
954b922f18, afternpm installinstalled@animalabs/membrane0.8.0,@animalabs/chronicle0.5.1 and@animalabs/context-manager0.24.2:test/maintenance-timer.test.ts: nine, on one fixture of two agents on one framework, where A's second call parks behind A's own turn or behind a provider cooldown. For each gate, B's call goes out while A's is parked, A's goes once the gate opens, and A's entry in the pass's run is unfinished when the pass ends and finished after, ready, with 2 ticks. The others coverstop()withdrawing the parked call with no failure alert,stop()waiting for the tick once its call is in flight, a rollback andresume()each waiting for it, aresume()begun before the tick left its pass, the skip and the later tick, a failure after the tick left, and an alert that throws.a64cfc010f'ssrc/framework.tsand this branch's tests, 6 of the 9 new tests fail. Four time out waiting for maintenance: both gate tests, theresume()begun before the tick left, and the skip and the later tick. The two that check a failure after the tick left find the pass that ticked A not ended, with A's call still parked. The twostop()tests and the rollback andresume()test pass there as well.src/framework.ts, each caught by the tests predicted. They drop the skip, drop the waits' loop over ticks that left, drop a late failure's report, finish the entry at the pass's end, take the waits' list once, run the late tick without itsfinally, and never tick an agent again once its tick has left.6eef037ec3, with the same versions: three more tests intest/maintenance-timer.test.ts.resume(), waiting for a tick that left its pass behind a provider cooldown, still waits when the timer starts a pass that ticks B, and returns once B's tick has settled. A pass whose only tick left it, took no tick and failed with the failure already reported leaves the history once that tick settles. The same pass, run by an operator's maintain, stays. With954b922f18'ssrc/framework.ts, the first two fail and the file's other 64 tests pass. With6eef037ec3the typecheck is clean, and the 98 test files that build a framework stand-in or name maintenance passes, the provider gate, quiesce,resume()or a surgery, each run alone, give 4,163 passing, 2 skipped and 0 failing. The two skipped need a disk that folds case.6eef037ec3: six, each changing one piece of it insrc/framework.ts, each caught. Putting back the wait's old form fails theresume()test. Removing the second decision fails the history test, and dropping its exemption for an operator's maintain fails the maintain test. Leaving drain kicks out of what the wait looks for fails threestop()tests with a drain kick. Removing the second decision's check of the run's entries fails nine tests in which a tick leaves its pass. A wait that loops while anything is registered doesn't end within 120 seconds intest/stop-with-host-operations.test.tsortest/diagnostics-signal.test.ts, whose stand-in passes stay registered once settled.59f25061cd, the merge ofmainaa134e75fcinto6eef037ec3: one conflict, in the maintenance pass's tick.mainno longer withdraws a tick that an undo or redo moved the store under, a branch feat(presentation): one record, a presentation index per agent — the deferral queue replaced (#291 step 3) #844 took out, so the merge takes that branch out oftickFailedand the same clause out of the readiness check.tickFailedis nowmain'scatch, and the readiness check ismain's. Otherwise this branch's diff ontomainis its diff ontoa64cfc010f, with its tests and changelog fragment unchanged. Withmain'ssrc/framework.tsand the merge's tests, on Linux with Node 24.22.2 and context-manager 0.25.0, 9 of the 66 tests intest/maintenance-timer.test.tsfail: both gate tests, and the seven tests of a tick that left its pass. At59f25061cd, CI's whole suite on ubuntu-latest with Node 22.12.0 gives 10,300 passing, 63 skipped and 0 failing, and the pull request's other 11 checks pass.1ac2485e2572, the merge ofmain6c7810995c34into59f25061cd: no conflicts, and this branch's diff ontomainhas the same four files and the same lines as at59f25061cd. Withmain'ssrc/framework.tsand the merge's tests, on Linux with Node 24.22.2, the same 9 of the 66 tests intest/maintenance-timer.test.tsfail. CI ran on GitHub's test merge of1ac2485e2572intomain, whose tree is1ac2485e2572's. On ubuntu-latest with Node 22.12.0, its whole suite gives 10,718 passing, 63 skipped, 1 todo and 0 failing, its unforced run gives 131 passing, and the pull request's other 15 checks pass.7224aecc84ea, two commits on1ac2485e2572: two more tests intest/maintenance-timer.test.ts. A tick that left its pass has its entry marked withleftPassAtwithin its pass, kept once the tick settles, and an entry finished in its pass has none. A budget hold on an agent whose tick left its pass is released by that tick's evidence when the tick settles, with no later pass, under a stand-in strategy that records its progress 20 ms after its call's answer. On Linux with Node 24.22.2, with1ac2485e2572'ssrc/framework.tsand these tests (f8fb89298660), the typecheck is clean and exactly those two of the file's 68 tests fail. On the head's tree, run before its commit message alone was corrected, all 68 pass, and three mutants there are each caught by its one test: without the mark, without the hold's look, and with the mark set when the work apart finishes instead of when it leaves. CI ran on GitHub's test merge of7224aecc84eaintomain6c7810995c34, whose tree is7224aecc84ea's. On ubuntu-latest with Node 22.12.0, its whole suite gives 10,720 passing, 63 skipped, 1 todo and 0 failing, its unforced run gives 131 passing, and the pull request's other 15 checks pass.Not verified
maintenanceTick()returns with a parked tick. It returns when its pass ends, as its doc now says (src/framework.ts:10003). One test has a maintain whose tick parks, the maintain test above, and no test reads the snapshot that maintain returns.maintenanceParkedis keyed by name (src/framework.ts:3714), and any callwithAuxiliaryAdmissionparks for that agent calls it (:6076). By those lines, another of that agent's calls parking during a pass tick also makes the tick leave. No test covers that.:7345), and a snapshot gives that history (:10054). By those lines, a run pushed out before its parked tick settles has its entry completed in a run the history no longer holds. No test covers that.callProviderreturns the membrane's answer (autobiographical.ts:3306),ladderStepawaits only that call (:12279), and on the success path nothing awaits between the step's answer and the chunk's mark (:13347). So the look queued when a call lands sees that progress. The hold test uses a stand-in strategy, and no test runs this one with a hold.src/framework.ts:7290), and the pass still waits for every agent's ticks (:7235). By those lines, a call that is slow without parking still holds its pass, as ata64cfc010f, so until that pass ends, no new pass starts for an agent queued since, or for one still not ready after its ticks. This branch doesn't change that case, and no test covers it.