Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,12 @@ research notes, raw material and social copy stay in the private package. Review
pages are visibly marked as drafts, carry `noindex,nofollow`, and are excluded
from Insights, RSS, the sitemap, `llms.txt` and social-card generation. They are
unlisted, not confidential: anyone with the URL or repository access can read
them. Publishing remains a separate, deliberate edit to `content/blog.json`.
them. Every review requires a newly generated, post-specific 1200×627 editorial
image derived from that draft's image brief; reusable generic cards are not an
acceptable substitute. The generated-image disclosure, meaningful alt text and
the exact `../assets/review/<slug>.png` path must be present in the manifest
before `insight:review` will promote it. Publishing remains a separate,
deliberate edit to `content/blog.json`.

`content/blog.json` is the insight source of truth. `npm run sync` renders static
`docs/notes/*.html`, `feed.xml`, `sitemap.xml` and `llms.txt`; GitHub Pages
Expand Down
4 changes: 4 additions & 0 deletions content/review-drafts/stop-watching-the-build.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,9 @@
"Jev",
"Preloop"
],
"imageBrief": "Documentary photograph of an engineer who has stepped away from a workstation\nwhile a real embedded build runs. The screen should not contain legible fake\ncode. Show a quiet bench with a development board, ordinary tools and a small\nphysical status light or notification device. Leave deliberate dark negative\nspace on the left for the CWCW headline. Avoid robots, holograms, neon and\nstaged stock-photo expressions. The visual idea is not \"AI watching a screen\";\nit is trusted infrastructure waiting quietly until human or agent attention is\nuseful.\n",
"reviewImage": "../assets/review/stop-watching-the-build.png",
"imageAlt": "An engineer away from a quiet embedded systems bench while build infrastructure waits for a meaningful event.",
"imageDisclosure": "Image generated with OpenAI from an editorial brief; CWCW typography and layout were applied deterministically.",
"bodyMarkdown": "On 16th August, one of our CI and bench chats generated 92 Task workers. When we\nadded the parent conversation and those workers together, the cost-equivalent\nusage came to about $183.\n\nCall it roughly $200 for one chat.\n\nThat needs a qualification. It was a combined usage estimate, not necessarily\na $183 charge landing on a card. Some of the work sat within bundled product\nusage. Nor was every dollar caused by CI polling alone. The thread mixed build\nmonitoring, bench work, Task fan-out and large tool results.\n\nBut the shape of the waste was clear. We had made a reasoning system behave\nlike a sensor.\n\nThe agent would ask GitHub whether a workflow had finished. It had not. A later\nturn would ask again. The same thing happened with embedded Linux builds that\ncould run for hours. Each check looked small in isolation, but the chat kept\ngrowing. Tool results were added to context and could be carried into later\nturns. Worker conversations accumulated outside the parent total. We were\npaying an increasingly capable model to discover that nothing had happened.\n\nThe build was doing useful work. The agent was mostly watching it do that work.\n\n### First response: poll less often\n\nOur first improvement was straightforward. We stopped continuous checking and\nmoved long waits to scheduled heartbeats.\n\nInstead of keeping an agent running, a heartbeat woke at a slower interval,\nmade one bounded observation and went quiet again when the state had not\nchanged. That was materially better. It removed frantic status loops and made\nthe cost visible as a deliberate schedule rather than an accidental habit.\n\nIt also helped us establish some important disciplines:\n\n- one bounded status read per scheduled run;\n- no full CI logs unless a failure required them;\n- no `gh run watch` inside an agent chat;\n- no repeated progress narration when the external state was unchanged.\n\nThis was a useful intermediate design. It was not the final one.\n\nA fifteen-minute heartbeat still wakes up every fifteen minutes. Most of those\nwake-ups may say nothing more interesting than \"still running\". Slower polling\nreduces waste, but it does not remove the underlying mistake. Time is still\ndriving the reasoning system when an event should be driving it.\n\n### The change in question\n\nThe better question was not \"how often should the agent check?\"\n\nIt was \"who already knows that the state changed?\"\n\nGitHub knows when a workflow completes. Foundries knows when an embedded Linux\nbuild succeeds or fails. Those systems should emit an event. The harness should\nretain it durably, correlate it with the exact waiting task and wake that task\nonce.\n\nThat led us towards a genuinely event-driven continuation path:\n\n1. The task launching a build registers the immutable provider and build ID,\n its own task ID and an expiry.\n2. The task stops. There is no model waiting in the background.\n3. CI sends a signed success or failure webhook.\n4. A small gateway stores the event before trying to deliver it.\n5. A private tunnel carries it back to the workstation when available.\n6. A local dispatcher matches the exact correlation and resumes only the task\n that launched that build.\n7. The task receives bounded evidence and continues with the next useful test\n or a targeted diagnosis.\n\nEither side can arrive first. A very short build may complete before its wait\nregistration reaches the dispatcher. A laptop may be asleep when the webhook\narrives. A task may still be active when its event is delivered. Those are\ntransport and concurrency problems, not reasons to make an agent poll. The\nevent is retained and reconciled when the other side becomes available.\n\nSuccess events matter as much as failures. A successful image build often\nunblocks the next physical-board or runtime test. If success does not wake the\ntask, someone still has to watch the build.\n\n### Do not wake the model with a whole log\n\nEvent-driven delivery solved when to wake the agent. It did not by itself solve\nwhat to put into context.\n\nA raw BitBake, Soong or Ninja log can contain megabytes of routine progress.\nPulling all of it into a chat recreates much of the cost in a different form.\nIt can also bury the useful failure line.\n\nThe harness now reduces logs programmatically before model reasoning begins.\nFor BitBake, the first `ERROR:` record is significant even when unrelated\ntasks continue afterwards. For Android, we look for bounded, actionable Ninja,\nSoong, compiler, `lpmake` or `avbtool` failures. The extractor sanitises the\nfirst useful signal and leaves the ordinary log outside model context.\n\nWhere we control the build process, the same rules can fail fast. There is\nlittle value in allowing hours of dependent work to continue after a decisive\nfailure if the harness can stop safely, wake the task and begin a repair.\n\n### Where Jev and Preloop fit\n\nWe are also testing Jev through a Preloop adapter as an observe-only semantic\nsensor. It can help classify a failure, judge whether the first error appears\nactionable and suggest the cheapest next proof.\n\nIt is deliberately not in the critical wake path.\n\nThe deterministic event and bounded failure envelope are stored and routed\nfirst. Jev is supplemental. It cannot mark CI green, authorise a change, retry\na build or weaken compiler, test, hash or human approval gates. If Jev or\nPreloop is unavailable, the task still wakes. A later explicit event may make\none deferred advisory attempt; there is no watcher checking when the watcher is\navailable.\n\nThis distinction matters. Event-driven should not mean handing control to a\nprobabilistic component. It means using deterministic events to decide when\nreasoning is worth paying for.\n\n### What this should change about cost\n\nThe expected saving is not mysterious:\n\n- unchanged external state should cost zero model turns;\n- success should carry a small typed event, not a log or an LLM summary;\n- failure should carry the earliest bounded evidence needed for diagnosis;\n- disconnected infrastructure should queue events rather than provoke retries\n from an agent;\n- duplicate delivery should be absorbed by idempotent transport;\n- one task should wake once for the build it actually owns.\n\nThis does not make CI free. Builds still consume runner time, storage and\nnetwork traffic. Webhooks, durable storage and private delivery have an\nengineering cost. A difficult failure may still justify substantial model\nwork.\n\nThe aim is narrower and more defensible: do not spend reasoning tokens on\nwaiting.\n\n### When we can call it truly event-driven\n\nAt the time of this draft, the Foundries kiosk and Android FRDM lanes provide\nthe reference implementation. A real Foundries failure has already traversed\nthe webhook, durable outbox, private tunnel and exact-task continuation path.\nThat is useful evidence, but it is not yet a claim that every external wait in\nthe harness is event-driven.\n\nBefore publishing this as a completed journey, I want evidence that:\n\n- every material CI and long-running build lane uses a provider callback or an\n equivalent completion event;\n- superseded scheduled status heartbeats remain disabled;\n- success and failure both resume the correct task;\n- offline delivery and restart recovery have been exercised in practice;\n- duplicate, late and event-first delivery do not create a second turn;\n- before-and-after usage data shows fewer model turns and lower charged or\n cost-equivalent usage for comparable waits.\n\nThe most important design principle is already clear, though.\n\nThe agent should reason when there is something to reason about. The transport\nshould do the waiting.\n\nChop wood. Carry water.\n"
}
Binary file added docs/assets/review/stop-watching-the-build.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 6 additions & 0 deletions docs/review/stop-watching-the-build.html
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,12 @@ <h1>Stop Watching the Build</h1>
<p class="note-byline">Alex Lennon</p>
<p class="note-lede">How costly CI polling led us from constant checking, through slower scheduled heartbeats, to signed webhooks and exact agent continuation.</p>
</header>

<figure class="note-hero review-hero">
<img src="../assets/review/stop-watching-the-build.png" width="1200" height="627"
alt="An engineer away from a quiet embedded systems bench while build infrastructure waits for a meaningful event." fetchpriority="high">
<figcaption>Image generated with OpenAI from an editorial brief; CWCW typography and layout were applied deterministically.</figcaption>
</figure>
<div class="note-body review-body">
<p>On 16th August, one of our CI and bench chats generated 92 Task workers. When we added the parent conversation and those workers together, the cost-equivalent usage came to about $183.</p>
<p>Call it roughly $200 for one chat.</p>
Expand Down
1 change: 1 addition & 0 deletions scripts/new_insight.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ def create_package(
"customCard": False,
"imageAlt": "",
"imageDisclosure": "",
"reviewImage": "",
"body": [],
"sources": [],
"discussionUrl": "",
Expand Down
39 changes: 30 additions & 9 deletions scripts/publish_insight_review.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
ROOT = Path(__file__).resolve().parents[1]
REVIEW_SOURCE = ROOT / "content" / "review-drafts"
ARTICLE_HEADING = "## CWCW insight draft"
IMAGE_BRIEF_HEADING = "## Image brief"


def validate_slug(value: str) -> str:
Expand All @@ -20,31 +21,47 @@ def validate_slug(value: str) -> str:
return value


def extract_article(markdown: str) -> str:
def extract_section(markdown: str, heading: str) -> str:
lines = markdown.splitlines()
try:
start = lines.index(ARTICLE_HEADING) + 1
start = lines.index(heading) + 1
except ValueError as error:
raise ValueError(f"Draft is missing {ARTICLE_HEADING!r}") from error
raise ValueError(f"Draft is missing {heading!r}") from error

end = len(lines)
for index in range(start, len(lines)):
if lines[index].startswith("## "):
end = index
break
article = "\n".join(lines[start:end]).strip()
if not article:
raise ValueError("CWCW insight draft section is empty")
return article + "\n"
value = "\n".join(lines[start:end]).strip()
if not value:
raise ValueError(f"Draft section is empty: {heading}")
return value + "\n"


def extract_article(markdown: str) -> str:
return extract_section(markdown, ARTICLE_HEADING)


def prepare_review(package: Path, output_root: Path, replace: bool = False) -> Path:
package = package.expanduser().resolve()
manifest = json.loads((package / "manifest.json").read_text())
article = extract_article((package / "draft-package.md").read_text())
package_markdown = (package / "draft-package.md").read_text()
article = extract_article(package_markdown)
image_brief = extract_section(package_markdown, IMAGE_BRIEF_HEADING)
if image_brief.lstrip().startswith("["):
raise ValueError("Replace the image brief placeholder before creating a review")
slug = validate_slug(manifest["id"])

required = ("title", "summary", "section", "created")
required = (
"title",
"summary",
"section",
"created",
"reviewImage",
"imageAlt",
"imageDisclosure",
)
missing = [key for key in required if not str(manifest.get(key, "")).strip()]
if missing:
raise ValueError(f"Draft manifest is missing: {', '.join(missing)}")
Expand All @@ -59,6 +76,10 @@ def prepare_review(package: Path, output_root: Path, replace: bool = False) -> P
"summary": manifest["summary"],
"section": manifest["section"],
"tags": manifest.get("tags", []),
"imageBrief": image_brief,
"reviewImage": manifest["reviewImage"],
"imageAlt": manifest["imageAlt"],
"imageDisclosure": manifest["imageDisclosure"],
"bodyMarkdown": article,
}
output_root.mkdir(parents=True, exist_ok=True)
Expand Down
19 changes: 19 additions & 0 deletions scripts/render_review_drafts.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,24 @@ def render_review(draft: dict) -> str:
section = html.escape(draft["section"])
created = html.escape(draft["created"])
body = render_markdown(draft["bodyMarkdown"])
hero = ""
review_image = draft.get("reviewImage", "").strip()
if review_image:
expected_image = f"../assets/review/{draft['id']}.png"
if review_image != expected_image:
raise ValueError(
f"Review image must use the draft asset path {expected_image}: {review_image}"
)
if not draft.get("imageAlt", "").strip():
raise ValueError("Review image requires imageAlt")
if not draft.get("imageDisclosure", "").strip():
raise ValueError("Review image requires imageDisclosure")
hero = f"""
<figure class="note-hero review-hero">
<img src="{html.escape(review_image)}" width="1200" height="627"
alt="{html.escape(draft['imageAlt'])}" fetchpriority="high">
<figcaption>{html.escape(draft['imageDisclosure'])}</figcaption>
</figure>"""
return f"""<!DOCTYPE html>
<html lang="en-GB">
<head>
Expand Down Expand Up @@ -116,6 +134,7 @@ def render_review(draft: dict) -> str:
<p class="note-byline">Alex Lennon</p>
<p class="note-lede">{summary}</p>
</header>
{hero}
<div class="note-body review-body">
{body}
</div>
Expand Down
10 changes: 10 additions & 0 deletions tests/review.spec.js
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,16 @@ test.describe("unlisted insight reviews", () => {
await expect(page.locator('meta[property^="og:"]')).toHaveCount(0);
await expect(page.locator('script[type="application/ld+json"]')).toHaveCount(0);
await expect(page.getByText("Share this insight", { exact: true })).toHaveCount(0);
if (review.reviewImage) {
const image = page.locator(".review-hero img");
await expect(image).toHaveAttribute("src", review.reviewImage);
await expect(image).toHaveAttribute("alt", review.imageAlt);
await expect(image).toHaveJSProperty("naturalWidth", 1200);
await expect(image).toHaveJSProperty("naturalHeight", 627);
await expect(page.locator(".review-hero figcaption")).toHaveText(
review.imageDisclosure,
);
}

const overflow = await page.evaluate(
() => document.documentElement.scrollWidth > document.documentElement.clientWidth + 1,
Expand Down
43 changes: 43 additions & 0 deletions tests/test_insight_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,9 @@ def test_review_import_copies_only_article(self):
"title": "Safe review",
"summary": "A review summary.",
"section": "Working practice",
"reviewImage": "../assets/review/safe-review.png",
"imageAlt": "A safe review image.",
"imageDisclosure": "Generated image disclosure.",
}
)
)
Expand All @@ -94,6 +97,10 @@ def test_review_import_copies_only_article(self):

The public article with `code`.

## Image brief

A newly generated scene specific to this article.

## Research and caveats

PRIVATE CORRESPONDENCE
Expand All @@ -107,9 +114,40 @@ def test_review_import_copies_only_article(self):
review = json.loads(target.read_text())
self.assertEqual(review["status"], "review")
self.assertIn("The public article", review["bodyMarkdown"])
self.assertEqual(review["reviewImage"], "../assets/review/safe-review.png")
self.assertIn("specific to this article", review["imageBrief"])
self.assertNotIn("PRIVATE CORRESPONDENCE", target.read_text())
self.assertNotIn("PRIVATE RAW MATERIAL", target.read_text())

def test_review_import_requires_post_specific_image_metadata(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
package = root / "incomplete-package"
package.mkdir()
(package / "manifest.json").write_text(
json.dumps(
{
"id": "incomplete-review",
"created": "2026-09-20",
"title": "Incomplete review",
"summary": "A review without its own image.",
"section": "Working practice",
}
)
)
(package / "draft-package.md").write_text(
"""## CWCW insight draft

Article text.

## Image brief

A scene made specifically for this post.
"""
)
with self.assertRaisesRegex(ValueError, "reviewImage"):
publish_review.prepare_review(package, root / "reviews")

def test_review_page_is_unlisted_and_escapes_html(self):
rendered = render_reviews.render_review(
{
Expand All @@ -119,6 +157,9 @@ def test_review_page_is_unlisted_and_escapes_html(self):
"title": "Safe review",
"summary": "A review summary.",
"section": "Working practice",
"reviewImage": "../assets/review/safe-review.png",
"imageAlt": "A safe review image.",
"imageDisclosure": "Generated image disclosure.",
"bodyMarkdown": "### Heading\n\n<script>alert(1)</script>",
}
)
Expand All @@ -129,6 +170,8 @@ def test_review_page_is_unlisted_and_escapes_html(self):
self.assertNotIn('property="og:', rendered)
self.assertNotIn("application/ld+json", rendered)
self.assertNotIn("Share this insight", rendered)
self.assertIn('../assets/review/safe-review.png', rendered)
self.assertIn("Generated image disclosure.", rendered)

def test_review_markdown_keeps_wrapped_list_items_together(self):
rendered = render_reviews.render_markdown(
Expand Down
Loading