Repository navigation
fix(control-plane,mocactl): count completed turns, so sessions stop reporting 0 (#420) - #672
Conversation
…eporting 0 `turns` was on the runtime hash's allow-list (ownership.ts) and read by both the session summary (handlers.ts) and the resources view (resources.ts), but nothing ever wrote it: `runtimeFieldsForTurn` stamps turnStartedAt / turnEndedAt / lastTurnAt and no other putRuntime caller exists. So every real deployment reported 0 turns, visible in the mocactl TUI's Sessions overlay -- and rossoctl#416 left the field out of `mocactl sessions` because of it. Counted with HINCRBY, not a read-modify-write. The server does not serialize concurrent turns of one session (ADR-0036), so reading the old value would have both turns read N and both write N+1, losing a count. This adds hIncrBy to CpRedisLike and a bumpTurns() that goes through the same guard as every other index write, so a transport failure is still redis_unavailable. `turns` is now readable but NOT settable through putRuntime (RUNTIME_COUNTERS). It is the only runtime field whose new value depends on its old one, and the tier that self-reports this hash is untrusted (spec §7.4) -- a `turns` passed to putRuntime would let a compromised harness claim any count. It is dropped on write, as a field off the allow-list is, rather than stored and then ignored. The reporter bumps when the fields carry turnEndedAt, rather than at each of the six call sites: a completed turn is exactly the report that stamps that timestamp, so no call site can report an end and forget to count it, and the counter cannot drift from the timestamp beside it. With a writer in place, `mocactl sessions` brings the field back: a TURNS column in the table and `turns` in `--json`, both straight from the server's summary. The local transcript store is deliberately not the source -- a headless `mocactl run` writes no transcript and a session resumed on another machine has none, yet both have run turns (rossoctl#417's other half). The TUI already renders `s.turns` (Sessions.tsx), so it shows real counts with no change. Eight mutants, all caught: a read-modify-write instead of HINCRBY, a dropped RUNTIME_COUNTERS guard (turns forgeable through putRuntime), a reporter that never bumps, one that bumps unconditionally (counting started turns), a summary hardcoded to 0, a fake-redis op log that cannot tell HINCRBY from hSet, a TURNS column fed from the local transcript, and a header with no cell. Closes rossoctl#420. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: cwiklik <cwiklikj@gmail.com>
ingpaolodettori-dev
left a comment
There was a problem hiding this comment.
Summary
This PR gives the runtime hash's turns field its first writer. OwnershipIndex.bumpTurns() is a server-side HINCRBY +1 through the same guard. putRuntime now drops turns, and the knative-server runtime reporter bumps the count on every report that carries turnEndedAt. mocactl sessions gets a TURNS column and turns in --json, read from the server summary. The TUI Sessions overlay already renders s.turns, so it picks up real counts with no change. I checked the three end-report sites in server.ts (JSON :395, SSE :653, detachable :789). Each one reports end once per turn, in a finally, and turns refused before the start report (429 quota, 409) are never counted. So the "rides turnEndedAt" design holds, and no call site double-counts. The failure modes are fine for display-only data. If putRuntime throws, the bump is skipped. If the bump throws, the reporter's existing catch swallows it and rebuilds the client. node-redis 6.3 hIncrBy(key, field, n) matches the new CpRedisLike signature. Tests are assertive, and the op-log check on the fake proves the HINCRBY path is used.
Author: cwiklik (MEMBER, maintainer)
Areas reviewed: TypeScript (control-plane, knative-server, mocactl), tests, docs/API contract
Agent/IDE config (.claude/.vscode): none
Commits: 1 commit, all signed-off: yes
CI status: passing (all 17 checks green, including k8s-kind-e2e, check, lint, codeql, trivy)
| // rather than a flag of its own: no call site can report an end and forget to count it, and | ||
| // the counter cannot drift from the timestamp beside it (#420). Separate from the putRuntime | ||
| // above because it is an HINCRBY -- `turns` is never settable through the merging hSet. | ||
| if (fields.turnEndedAt !== undefined) await index?.bumpTurns(sessionId); |
There was a problem hiding this comment.
suggestion: The PR calls this a count of "completed" turns. In practice, every end-report site in server.ts (:395, :653, :789) runs in a finally, so a turn whose executeTurn threw, an aborted turn, and a client-gone detachable turn all stamp turnEndedAt and bump the counter. Only turns refused before the start report (429/409) are skipped. That matches the TUI's local count (transcripts.ts:281 counts done and error terminals), so I think it's the right behaviour. But the contract doesn't say so: turns: { type: integer } has no description at docs/api/openapi.yaml:146 or :887. Consider adding one, e.g. "turns that ran to an end, including errored or aborted ones; counted by the harness at turn end; sessions from before this change start at 0". Then API clients and the new mocactl sessions column have a documented meaning.
| * (spec §7.4), which a `fields.turns` passed to `putRuntime` would be. `bumpTurns` -- a server-side | ||
| * HINCRBY of exactly +1 -- is the only writer; everything else is dropped on write. | ||
| */ | ||
| const RUNTIME_COUNTERS: readonly string[] = ['turns']; |
There was a problem hiding this comment.
nit: This comment and the test at ownership.test.ts:310 say this makes turns unforgeable by "a compromised harness". But the harness is the tier that calls bumpTurns, over a Redis connection that can HSET/HINCRBY the runtime key directly. So the allow-list stops well-behaved reporter code from setting an arbitrary value. It doesn't stop a compromised harness. That's the same limit as the existing owner allow-list, and display-only data makes it acceptable. Consider wording it as "the reporter cannot set it", so later readers don't treat turns as trusted.
Closes #420
The bug
turnswas on the runtime hash's allow-list (ownership.ts:103) and read in three places — the session summary (handlers.ts:375), the resources view (resources.ts:80) and the TUI's Sessions overlay (Sessions.tsx:84) — but nothing ever wrote it.runtimeFieldsForTurnstampsturnStartedAt/turnEndedAt/lastTurnAt, and no otherputRuntimecaller exists. So every real deployment reported0 turns, and #416 left the field out ofmocactl sessionsbecause of it.The fix
hIncrByadded toCpRedisLikecontrol-plane/src/ownership.tsbumpTurns()— one server-sideHINCRBY, through the sameguardas every other index writeRUNTIME_COUNTERS—turnsstays readable but is no longer settable viaputRuntimeturnEndedAtknative-server/src/turn-auth.tsTURNScolumn andturnsin--jsonformocactl sessionsmocactl/src/headless.tsHINCRBY, not a read-modify-write. The server does not serialize concurrent turns of one session (ADR-0036, Negative/accepted-cost bullet), so reading the old value would have both turns read N and both write N+1, losing a count.turnsis not settable throughputRuntime. It is the only runtime field whose new value depends on its old one, and the tier that self-reports this hash is untrusted (spec §7.4) — afields.turnswould let a compromised harness claim any count. It is dropped on write, as a field off the allow-list is, rather than stored and then ignored.The bump rides
turnEndedAtrather than sitting at each of the six reporter call sites. A completed turn is exactly the report that stamps that timestamp, so no call site can report an end and forget to count it, and the counter cannot drift from the timestamp beside it.mocactl sessionsbrings the field backThe issue notes that once there is a writer, the field can return. There is now, so it does: a
TURNScolumn in the table andturnsin--json.Both read the server's summary. The local transcript store is deliberately not the source — a headless
mocactl runwrites no transcript, and a session resumed on another machine has none at all, yet both have run turns. That is the other half of what #417 reports (0 turns · local historyafter a headless run). The TUI already renderss.turns, so it shows real counts with no client change.Tests
12 new assertions across 5 files:
bumpTurnscounts, increments server-side (asserting on the command, since a fake that merely ends at the right number would also pass for a racy implementation run sequentially), is unforgeable throughputRuntime, and maps a transport failure toredis_unavailablemocactl sessionsreports the server's count, not the local transcript's; a session with no turns reads0, not a blankThe fake Redis's op log records the field name, because
turnsmust move byHINCRBYand never by the merginghSet— a log saying onlyhIncrBy <key>could not tell those apart.Mutation testing
Eight mutants, all caught:
HINCRBYredis_unavailableRUNTIME_COUNTERSguard dropped (turnsforgeable)0TURNScolumn fed from the local transcriptVerification
tsc --noEmit× control-plane, knative-server, mocactlownership+handlers-sessionsturn-authheadless-manage+cliprettier --checkon the edited source filesTwo things I could not verify locally, both pre-existing:
ECONNREFUSED 127.0.0.1:6379) —refresh-redis,sandbox-redis,detachable-turn-route,workload-runs,workloads. CI suppliesredis:7as a service, so they run there. The 1500 runnable tests in those packages pass.mainfails 5 of the same files, the failing set changes between identical runs, and all pass in isolation both with and without this change. Same class as vmpool tests: waitFor has a hard 2s budget and an undiagnosable failure message, so load flakes read as regressions #297, but in TS.make typecheckandmake lintboth invokepnpm install, which fails here onERR_PNPM_IGNORED_BUILDS; I rantscandprettierdirectly against the same configs instead.Assisted-By: Claude