+ Working practice · draft 2026-09-20
+Stop Watching the Build
+ +How costly CI polling led us from constant checking, through slower scheduled heartbeats, to signed webhooks and exact agent continuation.
+On 16th August, one of our CI and bench chats generated 92 Task workers. When we added the parent conversation and those workers together, the cost-equivalent usage came to about $183.
+Call it roughly $200 for one chat.
+That needs a qualification. It was a combined usage estimate, not necessarily a $183 charge landing on a card. Some of the work sat within bundled product usage. Nor was every dollar caused by CI polling alone. The thread mixed build monitoring, bench work, Task fan-out and large tool results.
+But the shape of the waste was clear. We had made a reasoning system behave like a sensor.
+The agent would ask GitHub whether a workflow had finished. It had not. A later turn would ask again. The same thing happened with embedded Linux builds that could run for hours. Each check looked small in isolation, but the chat kept growing. Tool results were added to context and could be carried into later turns. Worker conversations accumulated outside the parent total. We were paying an increasingly capable model to discover that nothing had happened.
+The build was doing useful work. The agent was mostly watching it do that work.
+First response: poll less often
+Our first improvement was straightforward. We stopped continuous checking and moved long waits to scheduled heartbeats.
+Instead of keeping an agent running, a heartbeat woke at a slower interval, made one bounded observation and went quiet again when the state had not changed. That was materially better. It removed frantic status loops and made the cost visible as a deliberate schedule rather than an accidental habit.
+It also helped us establish some important disciplines:
+-
+
- one bounded status read per scheduled run; +
- no full CI logs unless a failure required them; +
- no
gh run watchinside an agent chat;
+ - no repeated progress narration when the external state was unchanged. +
This was a useful intermediate design. It was not the final one.
+A fifteen-minute heartbeat still wakes up every fifteen minutes. Most of those wake-ups may say nothing more interesting than "still running". Slower polling reduces waste, but it does not remove the underlying mistake. Time is still driving the reasoning system when an event should be driving it.
+The change in question
+The better question was not "how often should the agent check?"
+It was "who already knows that the state changed?"
+GitHub knows when a workflow completes. Foundries knows when an embedded Linux build succeeds or fails. Those systems should emit an event. The harness should retain it durably, correlate it with the exact waiting task and wake that task once.
+That led us towards a genuinely event-driven continuation path:
+-
+
- The task launching a build registers the immutable provider and build ID, its own task ID and an expiry. +
- The task stops. There is no model waiting in the background. +
- CI sends a signed success or failure webhook. +
- A small gateway stores the event before trying to deliver it. +
- A private tunnel carries it back to the workstation when available. +
- A local dispatcher matches the exact correlation and resumes only the task that launched that build. +
- The task receives bounded evidence and continues with the next useful test or a targeted diagnosis. +
Either side can arrive first. A very short build may complete before its wait registration reaches the dispatcher. A laptop may be asleep when the webhook arrives. A task may still be active when its event is delivered. Those are transport and concurrency problems, not reasons to make an agent poll. The event is retained and reconciled when the other side becomes available.
+Success events matter as much as failures. A successful image build often unblocks the next physical-board or runtime test. If success does not wake the task, someone still has to watch the build.
+Do not wake the model with a whole log
+Event-driven delivery solved when to wake the agent. It did not by itself solve what to put into context.
+A raw BitBake, Soong or Ninja log can contain megabytes of routine progress. Pulling all of it into a chat recreates much of the cost in a different form. It can also bury the useful failure line.
+The harness now reduces logs programmatically before model reasoning begins. For BitBake, the first ERROR: record is significant even when unrelated tasks continue afterwards. For Android, we look for bounded, actionable Ninja, Soong, compiler, lpmake or avbtool failures. The extractor sanitises the first useful signal and leaves the ordinary log outside model context.
Where we control the build process, the same rules can fail fast. There is little value in allowing hours of dependent work to continue after a decisive failure if the harness can stop safely, wake the task and begin a repair.
+Where Jev and Preloop fit
+We are also testing Jev through a Preloop adapter as an observe-only semantic sensor. It can help classify a failure, judge whether the first error appears actionable and suggest the cheapest next proof.
+It is deliberately not in the critical wake path.
+The deterministic event and bounded failure envelope are stored and routed first. Jev is supplemental. It cannot mark CI green, authorise a change, retry a build or weaken compiler, test, hash or human approval gates. If Jev or Preloop is unavailable, the task still wakes. A later explicit event may make one deferred advisory attempt; there is no watcher checking when the watcher is available.
+This distinction matters. Event-driven should not mean handing control to a probabilistic component. It means using deterministic events to decide when reasoning is worth paying for.
+What this should change about cost
+The expected saving is not mysterious:
+-
+
- unchanged external state should cost zero model turns; +
- success should carry a small typed event, not a log or an LLM summary; +
- failure should carry the earliest bounded evidence needed for diagnosis; +
- disconnected infrastructure should queue events rather than provoke retries from an agent; +
- duplicate delivery should be absorbed by idempotent transport; +
- one task should wake once for the build it actually owns. +
This does not make CI free. Builds still consume runner time, storage and network traffic. Webhooks, durable storage and private delivery have an engineering cost. A difficult failure may still justify substantial model work.
+The aim is narrower and more defensible: do not spend reasoning tokens on waiting.
+When we can call it truly event-driven
+At the time of this draft, the Foundries kiosk and Android FRDM lanes provide the reference implementation. A real Foundries failure has already traversed the webhook, durable outbox, private tunnel and exact-task continuation path. That is useful evidence, but it is not yet a claim that every external wait in the harness is event-driven.
+Before publishing this as a completed journey, I want evidence that:
+-
+
- every material CI and long-running build lane uses a provider callback or an equivalent completion event; +
- superseded scheduled status heartbeats remain disabled; +
- success and failure both resume the correct task; +
- offline delivery and restart recovery have been exercised in practice; +
- duplicate, late and event-first delivery do not create a second turn; +
- before-and-after usage data shows fewer model turns and lower charged or cost-equivalent usage for comparable waits. +
The most important design principle is already clear, though.
+The agent should reason when there is something to reason about. The transport should do the waiting.
+Chop wood. Carry water.
+