From 9f3279ab25eb89ac4c2e8d5bf9f245976e34b7c7 Mon Sep 17 00:00:00 2001 From: Alex J Lennon Date: Sun, 20 Sep 2026 12:47:57 +0100 Subject: [PATCH 1/6] Add measured baseline to event-driven draft Assisted-by: Codex --- content/review-drafts/stop-watching-the-build.json | 2 +- docs/review/stop-watching-the-build.html | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/content/review-drafts/stop-watching-the-build.json b/content/review-drafts/stop-watching-the-build.json index 78b1572..3b392b0 100644 --- a/content/review-drafts/stop-watching-the-build.json +++ b/content/review-drafts/stop-watching-the-build.json @@ -21,5 +21,5 @@ "reviewImage": "../assets/review/stop-watching-the-build.png", "imageAlt": "An empty railway junction at dawn with a green signal releasing the next movement while the infrastructure waits quietly.", "imageDisclosure": "Image generated with OpenAI from an editorial brief; CWCW typography and layout were applied deterministically.", - "bodyMarkdown": "On 16th August, one of our CI and bench chats generated 92 Task workers. When we\nadded the parent conversation and those workers together, the cost-equivalent\nusage came to about $183.\n\nCall it roughly $200 for one chat.\n\nThat needs a qualification. It was a combined usage estimate, not necessarily\na $183 charge landing on a card. Some of the work sat within bundled product\nusage. Nor was every dollar caused by CI polling alone. The thread mixed build\nmonitoring, bench work, Task fan-out and large tool results.\n\nBut the shape of the waste was clear. We had made a reasoning system behave\nlike a sensor.\n\nThe agent would ask GitHub whether a workflow had finished. It had not. A later\nturn would ask again. The same thing happened with embedded Linux builds that\ncould run for hours. Each check looked small in isolation, but the chat kept\ngrowing. Tool results were added to context and could be carried into later\nturns. Worker conversations accumulated outside the parent total. We were\npaying an increasingly capable model to discover that nothing had happened.\n\nThe build was doing useful work. The agent was mostly watching it do that work.\n\n### First response: poll less often\n\nOur first improvement was straightforward. We stopped continuous checking and\nmoved long waits to scheduled heartbeats.\n\nInstead of keeping an agent running, a heartbeat woke at a slower interval,\nmade one bounded observation and went quiet again when the state had not\nchanged. That was materially better. It removed frantic status loops and made\nthe cost visible as a deliberate schedule rather than an accidental habit.\n\nIt also helped us establish some important disciplines:\n\n- one bounded status read per scheduled run;\n- no full CI logs unless a failure required them;\n- no `gh run watch` inside an agent chat;\n- no repeated progress narration when the external state was unchanged.\n\nThis was a useful intermediate design. It was not the final one.\n\nA fifteen-minute heartbeat still wakes up every fifteen minutes. Most of those\nwake-ups may say nothing more interesting than \"still running\". Slower polling\nreduces waste, but it does not remove the underlying mistake. Time is still\ndriving the reasoning system when an event should be driving it.\n\n### The change in question\n\nThe better question was not \"how often should the agent check?\"\n\nIt was \"who already knows that the state changed?\"\n\nGitHub knows when a workflow completes. Foundries knows when an embedded Linux\nbuild succeeds or fails. Those systems should emit an event. The harness should\nretain it durably, correlate it with the exact waiting task and wake that task\nonce.\n\nThat led us towards a genuinely event-driven continuation path:\n\n1. The task launching a build registers the immutable provider and build ID,\n its own task ID and an expiry.\n2. The task stops. There is no model waiting in the background.\n3. CI sends a signed success or failure webhook.\n4. A small gateway stores the event before trying to deliver it.\n5. A private tunnel carries it back to the workstation when available.\n6. A local dispatcher matches the exact correlation and resumes only the task\n that launched that build.\n7. The task receives bounded evidence and continues with the next useful test\n or a targeted diagnosis.\n\nEither side can arrive first. A very short build may complete before its wait\nregistration reaches the dispatcher. A laptop may be asleep when the webhook\narrives. A task may still be active when its event is delivered. Those are\ntransport and concurrency problems, not reasons to make an agent poll. The\nevent is retained and reconciled when the other side becomes available.\n\nSuccess events matter as much as failures. A successful image build often\nunblocks the next physical-board or runtime test. If success does not wake the\ntask, someone still has to watch the build.\n\n### Do not wake the model with a whole log\n\nEvent-driven delivery solved when to wake the agent. It did not by itself solve\nwhat to put into context.\n\nA raw BitBake, Soong or Ninja log can contain megabytes of routine progress.\nPulling all of it into a chat recreates much of the cost in a different form.\nIt can also bury the useful failure line.\n\nThe harness now reduces logs programmatically before model reasoning begins.\nFor BitBake, the first `ERROR:` record is significant even when unrelated\ntasks continue afterwards. For Android, we look for bounded, actionable Ninja,\nSoong, compiler, `lpmake` or `avbtool` failures. The extractor sanitises the\nfirst useful signal and leaves the ordinary log outside model context.\n\nWhere we control the build process, the same rules can fail fast. There is\nlittle value in allowing hours of dependent work to continue after a decisive\nfailure if the harness can stop safely, wake the task and begin a repair.\n\n### Where Jev and Preloop fit\n\nWe are also testing Jev through a Preloop adapter as an observe-only semantic\nsensor. It can help classify a failure, judge whether the first error appears\nactionable and suggest the cheapest next proof.\n\nIt is deliberately not in the critical wake path.\n\nThe deterministic event and bounded failure envelope are stored and routed\nfirst. Jev is supplemental. It cannot mark CI green, authorise a change, retry\na build or weaken compiler, test, hash or human approval gates. If Jev or\nPreloop is unavailable, the task still wakes. A later explicit event may make\none deferred advisory attempt; there is no watcher checking when the watcher is\navailable.\n\nThis distinction matters. Event-driven should not mean handing control to a\nprobabilistic component. It means using deterministic events to decide when\nreasoning is worth paying for.\n\n### What this should change about cost\n\nThe expected saving is not mysterious:\n\n- unchanged external state should cost zero model turns;\n- success should carry a small typed event, not a log or an LLM summary;\n- failure should carry the earliest bounded evidence needed for diagnosis;\n- disconnected infrastructure should queue events rather than provoke retries\n from an agent;\n- duplicate delivery should be absorbed by idempotent transport;\n- one task should wake once for the build it actually owns.\n\nThis does not make CI free. Builds still consume runner time, storage and\nnetwork traffic. Webhooks, durable storage and private delivery have an\nengineering cost. A difficult failure may still justify substantial model\nwork.\n\nThe aim is narrower and more defensible: do not spend reasoning tokens on\nwaiting.\n\n### When we can call it truly event-driven\n\nAt the time of this draft, the Foundries kiosk and Android FRDM lanes provide\nthe reference implementation. A real Foundries failure has already traversed\nthe webhook, durable outbox, private tunnel and exact-task continuation path.\nThat is useful evidence, but it is not yet a claim that every external wait in\nthe harness is event-driven.\n\nBefore publishing this as a completed journey, I want evidence that:\n\n- every material CI and long-running build lane uses a provider callback or an\n equivalent completion event;\n- superseded scheduled status heartbeats remain disabled;\n- success and failure both resume the correct task;\n- offline delivery and restart recovery have been exercised in practice;\n- duplicate, late and event-first delivery do not create a second turn;\n- before-and-after usage data shows fewer model turns and lower charged or\n cost-equivalent usage for comparable waits.\n\nThe most important design principle is already clear, though.\n\nThe agent should reason when there is something to reason about. The transport\nshould do the waiting.\n\nChop wood. Carry water.\n" + "bodyMarkdown": "On 16th August, one of our CI and bench chats generated 92 Task workers. When we\nadded the parent conversation and those workers together, the cost-equivalent\nusage came to about $183.\n\nCall it roughly $200 for one chat.\n\nThat needs a qualification. It was a combined usage estimate, not necessarily\na $183 charge landing on a card. Some of the work sat within bundled product\nusage. Nor was every dollar caused by CI polling alone. The thread mixed build\nmonitoring, bench work, Task fan-out and large tool results.\n\nBut the shape of the waste was clear. We had made a reasoning system behave\nlike a sensor.\n\nThe agent would ask GitHub whether a workflow had finished. It had not. A later\nturn would ask again. The same thing happened with embedded Linux builds that\ncould run for hours. Each check looked small in isolation, but the chat kept\ngrowing. Tool results were added to context and could be carried into later\nturns. Worker conversations accumulated outside the parent total. We were\npaying an increasingly capable model to discover that nothing had happened.\n\nThe build was doing useful work. The agent was mostly watching it do that work.\n\n### First response: poll less often\n\nOur first improvement was straightforward. We stopped continuous checking and\nmoved long waits to scheduled heartbeats.\n\nInstead of keeping an agent running, a heartbeat woke at a slower interval,\nmade one bounded observation and went quiet again when the state had not\nchanged. That was materially better. It removed frantic status loops and made\nthe cost visible as a deliberate schedule rather than an accidental habit.\n\nIt also helped us establish some important disciplines:\n\n- one bounded status read per scheduled run;\n- no full CI logs unless a failure required them;\n- no `gh run watch` inside an agent chat;\n- no repeated progress narration when the external state was unchanged.\n\nThis was a useful intermediate design. It was not the final one.\n\nA fifteen-minute heartbeat still wakes up every fifteen minutes. Most of those\nwake-ups may say nothing more interesting than \"still running\". Slower polling\nreduces waste, but it does not remove the underlying mistake. Time is still\ndriving the reasoning system when an event should be driving it.\n\n### The change in question\n\nThe better question was not \"how often should the agent check?\"\n\nIt was \"who already knows that the state changed?\"\n\nGitHub knows when a workflow completes. Foundries knows when an embedded Linux\nbuild succeeds or fails. Those systems should emit an event. The harness should\nretain it durably, correlate it with the exact waiting task and wake that task\nonce.\n\nThat led us towards a genuinely event-driven continuation path:\n\n1. The task launching a build registers the immutable provider and build ID,\n its own task ID and an expiry.\n2. The task stops. There is no model waiting in the background.\n3. CI sends a signed success or failure webhook.\n4. A small gateway stores the event before trying to deliver it.\n5. A private tunnel carries it back to the workstation when available.\n6. A local dispatcher matches the exact correlation and resumes only the task\n that launched that build.\n7. The task receives bounded evidence and continues with the next useful test\n or a targeted diagnosis.\n\nEither side can arrive first. A very short build may complete before its wait\nregistration reaches the dispatcher. A laptop may be asleep when the webhook\narrives. A task may still be active when its event is delivered. Those are\ntransport and concurrency problems, not reasons to make an agent poll. The\nevent is retained and reconciled when the other side becomes available.\n\nSuccess events matter as much as failures. A successful image build often\nunblocks the next physical-board or runtime test. If success does not wake the\ntask, someone still has to watch the build.\n\n### Do not wake the model with a whole log\n\nEvent-driven delivery solved when to wake the agent. It did not by itself solve\nwhat to put into context.\n\nA raw BitBake, Soong or Ninja log can contain megabytes of routine progress.\nPulling all of it into a chat recreates much of the cost in a different form.\nIt can also bury the useful failure line.\n\nThe harness now reduces logs programmatically before model reasoning begins.\nFor BitBake, the first `ERROR:` record is significant even when unrelated\ntasks continue afterwards. For Android, we look for bounded, actionable Ninja,\nSoong, compiler, `lpmake` or `avbtool` failures. The extractor sanitises the\nfirst useful signal and leaves the ordinary log outside model context.\n\nWhere we control the build process, the same rules can fail fast. There is\nlittle value in allowing hours of dependent work to continue after a decisive\nfailure if the harness can stop safely, wake the task and begin a repair.\n\n### Where Jev and Preloop fit\n\nWe are also testing Jev through a Preloop adapter as an observe-only semantic\nsensor. It can help classify a failure, judge whether the first error appears\nactionable and suggest the cheapest next proof.\n\nIt is deliberately not in the critical wake path.\n\nThe deterministic event and bounded failure envelope are stored and routed\nfirst. Jev is supplemental. It cannot mark CI green, authorise a change, retry\na build or weaken compiler, test, hash or human approval gates. If Jev or\nPreloop is unavailable, the task still wakes. A later explicit event may make\none deferred advisory attempt; there is no watcher checking when the watcher is\navailable.\n\nThis distinction matters. Event-driven should not mean handing control to a\nprobabilistic component. It means using deterministic events to decide when\nreasoning is worth paying for.\n\n### What this should change about cost\n\nThe expected saving is not mysterious:\n\n- unchanged external state should cost zero model turns;\n- success should carry a small typed event, not a log or an LLM summary;\n- failure should carry the earliest bounded evidence needed for diagnosis;\n- disconnected infrastructure should queue events rather than provoke retries\n from an agent;\n- duplicate delivery should be absorbed by idempotent transport;\n- one task should wake once for the build it actually owns.\n\nThis does not make CI free. Builds still consume runner time, storage and\nnetwork traffic. Webhooks, durable storage and private delivery have an\nengineering cost. A difficult failure may still justify substantial model\nwork.\n\nThe aim is narrower and more defensible: do not spend reasoning tokens on\nwaiting.\n\n### When we can call it truly event-driven\n\nAt the time of this draft, the Foundries kiosk and Android FRDM lanes provide\nthe reference implementation. A real Foundries failure has already traversed\nthe webhook, durable outbox, private tunnel and exact-task continuation path.\nThat is useful evidence, but it is not yet a claim that every external wait in\nthe harness is event-driven.\n\nThe scale of the old pattern is now easier to see. Before we paused the kiosk\nheartbeat, its task had recorded 811 completed turns and 668,928,917 locally\ncounted tokens. The Android task had recorded 797 completed turns and\n831,302,773 tokens. In both cases 98.9% of input was cached. Those are whole-task\ncounters, not a measure of polling alone, and they are not an invoice. They are\na baseline for the before-and-after comparison, not a cost-saving claim.\n\nBefore publishing this as a completed journey, I want evidence that:\n\n- every material CI and long-running build lane uses a provider callback or an\n equivalent completion event;\n- superseded scheduled status heartbeats remain disabled;\n- success and failure both resume the correct task;\n- offline delivery and restart recovery have been exercised in practice;\n- duplicate, late and event-first delivery do not create a second turn;\n- before-and-after usage data shows fewer model turns and lower charged or\n cost-equivalent usage for comparable waits.\n\nThe most important design principle is already clear, though.\n\nThe agent should reason when there is something to reason about. The transport\nshould do the waiting.\n\nChop wood. Carry water.\n" } diff --git a/docs/review/stop-watching-the-build.html b/docs/review/stop-watching-the-build.html index 77529c1..b1fe6a5 100644 --- a/docs/review/stop-watching-the-build.html +++ b/docs/review/stop-watching-the-build.html @@ -96,6 +96,7 @@

What this should change about cost

The aim is narrower and more defensible: do not spend reasoning tokens on waiting.

When we can call it truly event-driven

At the time of this draft, the Foundries kiosk and Android FRDM lanes provide the reference implementation. A real Foundries failure has already traversed the webhook, durable outbox, private tunnel and exact-task continuation path. That is useful evidence, but it is not yet a claim that every external wait in the harness is event-driven.

+

The scale of the old pattern is now easier to see. Before we paused the kiosk heartbeat, its task had recorded 811 completed turns and 668,928,917 locally counted tokens. The Android task had recorded 797 completed turns and 831,302,773 tokens. In both cases 98.9% of input was cached. Those are whole-task counters, not a measure of polling alone, and they are not an invoice. They are a baseline for the before-and-after comparison, not a cost-saving claim.

Before publishing this as a completed journey, I want evidence that: