Repository navigation
Beta framing, today's workflow, a call trace, and three corrections against source - #8
Merged
Merged
Conversation
Four changes, in the order they matter. The launch is a beta and the article never said so. Expectations are set in three places a reader actually reaches — a marker above the title, a paragraph in the introduction, and the opening of Limitations — and the call to action now asks for the failures rather than the successes, because gathering feedback (starting with our own policy team) is part of why we are publishing at this stage. The subtitle is left alone: its wording is the author's and has been reverted once already, so the beta status is carried around it rather than through it. The narrative was missing its first leg. The piece opened on people prompting a language model and getting wrong numbers, which argues against a generic chatbot rather than for a better research workflow. "How policy analysis is done today" describes what the work actually looks like now — Python or the API against the PolicyEngine UK model, a parameter path, a hand-built reform object, a notebook loop — so that the rest of the article reads as a response to a real cost. It is explicit that UK Chat does not replace that workflow; it removes the setup for the questions the typed tools already cover. Pavel asked for more descriptive methodology around tool calls. Each of the 21 tools had a one-line summary and there was no example of a call or a return, so "the figures come from tools" was an assertion. The worked Child Benefit example now carries the five calls it makes, tied to that same reform rather than a second invented one, and mined from the earlier draft of this article. The two claims that need showing rather than telling are visible in the trace: run_society_simulation returns a handle plus dataset and year, and does not return household rows, survey weights, or any serialisable simulation object; and every argument came from a preceding call or from the user. Built on CardTabs and the legacy .example-* vocabulary so it stays inside the existing design language. David's four fixes: the flow segments said "AI model" while the body says "language model" throughout; "stack these questions together" was redundant; the gateway sentence and the Households caption were split; and "because models can read" and the boundary diagram's doubled "deterministic PolicyEngine engine" are tightened. content/current-article.md is regenerated to match the page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Checked the article against PolicyEngine/policyengine-uk-chat. Three claims did not hold: - Limitations said a mixed request proceeds with what it can. The gateway does the opposite: the partial directive is "do not run anything yet", the route is set to lightweight, and the orchestrator passes an empty tool array, so it confirms first and runs nothing that turn. The Gate section already had this right; only Limitations was wrong. - The boundary section said the model receives a machine-readable description of what the deployed engine can compute. It does not. The system prompt is hand-written and so are the tool schemas; engine-derived truth reaches the model only when it calls discovery tools. Rewritten to what actually holds — the model has to ask — with the drift-proof claim moved to the gateway, where the tool vocabulary really is derived from the schemas. - "Charts are constructed from stored results" holds for the seven policy presets, which require a matching result id, but not for the generic kinds, which accept model-supplied data. Scoped both statements to policy presets. Added the two things the check turned up that the article should have said: - The approved-reform binding, which is the hardest guarantee in the system and went unmentioned: the simulation tool refuses any reform that does not match the gateway-approved construction, so a different one cannot be substituted after the check passes. - The gate fails open by design — a classifier error or an unreadable plan is treated as ready rather than refused. A piece arguing for a strict boundary should say where the boundary is not the gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…oduction The button is now a shared component so both placements stay identical, and it carries the beta marker rather than relying on the surrounding prose. The hero marker moves inline to the head of the subtitle, where it reads as part of the standfirst instead of a separate line above the title. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The section said a researcher writes Python or calls the API, which skips the route most people actually take: entering the reform by hand in the web app. Now names both, and the closing caveat says UK Chat replaces neither. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Same points, roughly half the words: two routes, the shared barrier being setup rather than arithmetic, the pivot to prompting, and the caveat that UK Chat replaces neither route. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tool explorer named 21 tools and described each in a sentence, which told a reader what a tool was for but not what it is: an argument list with fixed names and a return with fixed fields. The six stages under "Under the hood" had the same gap at a larger grain. Both now carry a short example, derived from the service source rather than written to sound plausible — argument names come from the input schemas in backend/tools/definitions.py and backend/gateway, return fields from what the handlers actually build. Values that depend on the deployed policyengine-uk package are elided descriptively, as the call trace already does, so nothing here goes stale when a rate changes. The three example surfaces are divided so they do not restate each other. The call trace keeps the Child Benefit reform as the single end-to-end tool sequence. The tool cards each illustrate one tool in isolation and deliberately vary the policy — personal allowance, Universal Credit, deciles, poverty — so clicking through shows range, and the five tools that also appear in the trace use different examples there. The stage examples show what the gateway produces rather than which tool ran: the emitted plan, the catalogue match and its match_type, the verdict, the approved reform construction, the loop's tool-result envelope, and the server-sent frames. None of those appear anywhere else. Examples are stored as literal source strings and highlighted by a shared tokeniser rather than hand-tagged as JSX, which is what keeps 27 of them short enough to compare. The tool panel lays call beside return so it stays shorter than the family cards above it; the narrative column is too narrow to split, so its examples stack. Removals, because adding these made three things redundant: - The worked example's six-phase timeline repeated the six stages it followed, and after this change repeated their examples too. Cut, along with its "User request" block, which quoted the request the paragraph above already quotes. The uk-chat.css rules written for it go with it. - The beta pill on both call-to-action buttons was the only beta marker that explained nothing; the hero tag, the introduction, Limitations, and "Try it yourself" all still say it, in words. Three corrections against the source while in here. run_society_simulation returns result_id, not simulation_id (simulation_id is what the derivatives take), and its dataset is the resolved DatasetSpec rather than a bare name; the trace and the paragraph below it said otherwise. And gov.hmrc.child_benefit.amount.eldest has no entry in COMMON_REFORM_TARGETS, so "eldest-child rate" resolves by catalogue label, not by an alias. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removed in the redundancy pass as the one beta signal that explained nothing. Restoring it: it was asked for explicitly, and a marker on the button is what a reader skimming to the button actually sees. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"One reform, end to end" implied broader coverage than the section above it, when the difference is depth: Under the hood is what the gateway decides, this is what the tools then do with the same request. The new title names that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Acts on the Max/Vahid 1:1 decisions, Pavel's feedback, and David's optional nits — then corrects three claims that turned out not to match the service source.
From the 1:1
Beta framing, which the article had nowhere. A
Betamarker above the title, a paragraph in the introduction, a new opening paragraph in Limitations separating the permanent boundary from the current state of a beta, and a paragraph before the call to action saying which failures are most useful to report. The subtitle is untouched.The missing leg of the narrative. The agreed arc is how policy analysis is done today → why a new tool is needed → what UK Chat adds, and the article skipped the first: it opened on "people ask language models and get wrong numbers", so it argued against a generic chatbot rather than for a better research workflow. A new section, "How policy analysis is done today", describes the existing route — find the parameter path, build a reform object, run baseline and reform, read off the measure — and states plainly that UK Chat does not replace it; custom variables, bespoke datasets, and reproducible scripts still belong in code.
From Pavel
Concrete tool calls. Each tool had a one-line summary and there was no example of a call or a return. The worked example now carries a five-step trace —
list_reform_targets→get_parameter→validate_reform→run_society_simulation→compute_budgetary_impact— tied to the article's own Child Benefit reform, showing the literal call, the literal return including the real reform JSON, and what is not returned: household rows, survey weights, any serialisable simulation object.From David
Segment tags unified on "language model"; "stack these questions together" trimmed; the gateway sentence and the Households caption split; "because these models can read"; and the doubled "deterministic PolicyEngine engine" in the diagram's alt-text.
Corrections against the service source
Verified the article against
PolicyEngine/policyengine-uk-chat. Three claims did not hold:lightweight, and the orchestrator passes an empty tool array. It confirms first and runs nothing that turn. The Gate section already had this right.Two things the check said the article should have claimed and didn't:
The same check confirmed the rest: 21 tools with every name correct and the 9/2/2/7/1 split exact, the five gate outcomes, all four loop bounds, FastAPI SSE calling the Anthropic SDK with no agent framework and no
run_pythonfallback, the five streamed event types, and every handle and privacy claim.Two figures remain unverified and need a human: the worked example's £0.9bn requires the private dataset to reproduce, and the parameter path is not in the curated alias list. The arithmetic and sign convention do match the engine.
content/current-article.mdis in step.bun run checkpasses.