diff --git a/content/current-article.md b/content/current-article.md index 9756037..257bee5 100644 --- a/content/current-article.md +++ b/content/current-article.md @@ -1,11 +1,23 @@ +**Beta** + # PolicyEngine UK Chat: an AI interface for tax and benefits _Introducing PolicyEngine's newest AI-powered tool to help users understand UK tax and benefit policy_ -People are increasingly turning to AI language models to answer complex questions about tax and benefit policy and reform because models can read loosely worded questions and answer them in plain language. This presents a problem: for many users, tax and benefit questions directly affect their lives and livelihoods, so they require correct, verifiable answers. Language models generate responses from learned statistical patterns, and an answer may be partly or wholly wrong. +People are increasingly turning to AI language models to answer complex questions about tax and benefit policy and reform because these models can read loosely worded questions and answer them in plain language. This presents a problem: for many users, tax and benefit questions directly affect their lives and livelihoods, so they require correct, verifiable answers. Language models generate responses from learned statistical patterns, and an answer may be partly or wholly wrong. PolicyEngine therefore built UK Chat, an AI interface to PolicyEngine's deterministic rules and simulation engine. A language model interprets an open-ended request and proposes a structured plan. The gateway checks the proposed inputs and output against the current PolicyEngine catalogue; only then do deterministic tools compute quantitative policy results from actual tax and benefit rules. +UK Chat is released as a **beta**. The pipeline described here is in place and running, but the range of reforms it covers, the wording of its answers, and its judgement about what it cannot compute are all still being tested. We are publishing it at this stage to gather feedback on where it is useful and where it falls short, beginning with PolicyEngine's own policy team. + +[Try PolicyEngine UK Chat (beta)](https://policyengine.org/uk/chat) + +## How policy analysis is done today + +PolicyEngine already answers questions of this kind, but not in conversation. Someone costing a Child Benefit change either builds the reform by hand in the web app, or writes code against the model: find the parameter path, construct a reform, run baseline and reform over the same microdata, read off the measure. Both routes are precise and fully inspectable, and both call the engine UK Chat calls underneath. Both also ask the same thing of the person using them: knowing which parameter carries the policy, how to express the change, and which output answers the question. + +For a policy analyst, a journalist, or a household working out what a reform would mean for them, the barrier is the setup rather than the arithmetic — and prompting a general-purpose language model looks like the way around it. The rest of this article is about why that shortcut does not produce a figure anyone can rely on. UK Chat replaces neither route, but for the questions its tools already cover it removes the setup and leaves the calculation where it was. + ## Separating calculation A language model generates text by predicting continuations, not by applying tax and benefit rules. Ask one what Universal Credit (UC) a lone parent receives and it may return a number without factoring in UC's income-based taper rate, work allowance, or the rates that apply to the household's circumstances. @@ -24,13 +36,13 @@ The predictive parts are interpreting wording, deciding which kind of analysis i The deterministic parts are the ones users need to be able to check: validating the request, looking up parameters, constructing household inputs, running simulations, computing weighted outputs, and producing chart data. None of these depends on the language model's memory. The runtime enforces the boundary rather than asking the language model to respect it in a prompt. -When a question is ready for computation, the language model receives the calculation tools plus a machine-readable description of what the deployed engine can compute: its capabilities and parameter schema. What the language model can be asked to compute is therefore tied to the deployed version of PolicyEngine, not to a hand-written prompt that drifts out of date. +When a question is ready for computation, the language model receives the calculation tools, but not a list of what exists in the model. It has to ask: names are confirmed by calling discovery tools against the deployed package, so a policy the model half-remembers either resolves to a real parameter path or fails to resolve at all. The gateway works the same way from the other side, describing the tools from their own schemas so its vocabulary cannot drift from what the tools actually accept. -The gateway sits on that line. It takes the plan the language model proposes and decides whether it may cross into the deterministic side, which is why the system has three parts rather than two: the language model, the gateway, and the tools a plan reaches once the gateway admits it. +The gateway sits on that line. It takes the plan the language model proposes and decides whether it may cross into the deterministic side. That is why the system has three parts rather than two: the language model, the gateway, and the tools a plan reaches once the gateway admits it. ## The language model proposes a plan -UK Chat breaks the user pathway into three segments: the AI model, the gateway, and supporting tools. +UK Chat breaks the user pathway into three segments: the language model, the gateway, and supporting tools. First, the language model does the predictive work. It takes the user's open-ended prompt and develops a simulation or analysis plan. The plan has a required format and must use one or more of the deterministic tools connected to the PolicyEngine UK tax and benefit simulation engine. @@ -46,31 +58,318 @@ The tools UK Chat exposes to the language model are limited in scope. They fall Find exact variables, parameters, entities, reform targets, and supported outputs. Discovery keeps the language model from guessing the names of things in PolicyEngine. -`list_entities`, `search_variables`, `get_variable`, `search_parameters`, `get_parameter`, `list_reform_targets`, `list_household_input_variables`, `list_society_output_variables`, `list_supported_outputs` +**`list_entities`** — Lists the PolicyEngine UK model entities and the number of variables defined for each one. + +``` +list_entities() +# returns +{ + "status": "success", + "entities": [{ "name": "person", "variable_count": … }, …] +} +``` + +**`search_variables`** — Searches the UK variable registry and identifies each match’s entity and default-output status. + +``` +search_variables( + query="employment income", + entity="person" +) +# returns +{ + "variables": [ + { "name": "employment_income", "entity": "person", + "value_type": "float", "definition_period": "year" } + ] +} +``` + +**`get_variable`** — Verifies one exact UK variable and returns its metadata, entity, and default-output status. + +``` +get_variable(name="household_net_income") +# returns +{ + "variable": { + "name": "household_net_income", "entity": "household", + "value_type": "float", "is_default_society_output": true + } +} +``` + +**`search_parameters`** — Searches policy parameters by path, label, description, or known alias. + +``` +search_parameters(query="personal allowance") +# returns +{ + "parameters": [ + { "path": "gov.hmrc.income_tax.allowances.personal_allowance.amount", + "unit": "currency-GBP", "aliases": ["personal allowance", …] } + ] +} +``` + +**`get_parameter`** — Looks up one exact policy parameter and returns its metadata and value for a selected year. + +``` +get_parameter( + path="gov.hmrc.income_tax.allowances.personal_allowance.amount", + year=2026 +) +# returns +{ + "parameter": { + "unit": "currency-GBP", "year": 2026, + "value": …the 2026 allowance in the deployed package… + } +} +``` + +**`list_reform_targets`** — Searches the current catalogue for parameter paths that can be used in a reform. + +``` +list_reform_targets(query="universal credit standard allowance") +# returns +{ + "targets": [ + { "path": "gov.dwp.universal_credit.standard_allowance.amount.SINGLE_OLD", + "aliases": ["universal credit standard allowance", …] } + ] +} +``` + +**`list_household_input_variables`** — Lists known variables that can be supplied as overrides for an illustrative household. + +``` +list_household_input_variables(entity="person") +# returns +{ + "entity": "person", + "variables": [{ "name": "employment_income", … }, …], + "input_contract": "policyengine.py accepts any known variable on its declared entity…" +} +``` + +**`list_society_output_variables`** — Lists the variables a society simulation materialises by default, grouped by entity. + +``` +list_society_output_variables(entity="household") +# returns +{ + "default_variables_by_entity": { + "household": ["household_net_income", "household_tax", "household_benefits", …] + }, + "default_variable_count": … +} +``` + +**`list_supported_outputs`** — Lists the household, society, derivative, and chart outputs supported by the chat runtime. + +``` +list_supported_outputs(scope="derivative") +# returns +{ + "scope": "derivative", + "outputs": [ + { "scope": "derivative", "name": "budgetary_impact", "description": … }, … + ] +} +``` ### Validation (2 tools) Check reform JSON and synthetic household inputs before calculation. Validation turns malformed inputs into explicit errors rather than plausible-looking results. -`validate_reform`, `validate_household` +**`validate_reform`** — Validates flat PolicyEngine reform JSON for a selected year without running a simulation. + +``` +validate_reform( + reform={ "gov.hmrc.income_tax.allowances.personal_allowance.amount": 15000 }, + year=2026 +) +# returns +{ + "valid": true, + "parameter_paths": ["gov.hmrc.income_tax.allowances.personal_allowance.amount"], + "warnings": [] +} +``` + +**`validate_household`** — Validates an illustrative synthetic UK household against PolicyEngine variable metadata. + +``` +validate_household( + people=[{ "age": 35, "employment_income": 28000 }, { "age": 6 }, { "age": 3 }], + household={ "country": "ENGLAND" } +) +# returns +{ + "valid": true, + "year": 2026, + "people_count": 3, "warnings": [] +} +``` ### Simulation (2 tools) Run one illustrative household or a reform across the modelled UK population. These tools apply the tax and benefit rules and return turn-local result handles. -`run_household_simulation`, `run_society_simulation` +**`run_household_simulation`** — Runs an illustrative synthetic household through the PolicyEngine UK tax-benefit model. + +``` +run_household_simulation( + people=[{ "age": 35, "employment_income": 28000 }, { "age": 6 }, { "age": 3 }], + household={ "country": "ENGLAND" } +) +# returns +{ + "reform_applied": false, + "household": { "household_net_income": …, "household_tax": … }, + "result_id": "household_simulation_…" +} +``` + +**`run_society_simulation`** — Runs baseline and reform UK simulations and returns metadata plus a turn-local result handle. + +``` +run_society_simulation( + reform={ "gov.hmrc.income_tax.allowances.personal_allowance.amount": 15000 }, + year=2026 +) +# returns +{ + "year": 2026, "reform_applied": true, + "dataset": { "label": "Enhanced FRS 2024-25", "row_level_access": false }, + "result_id": "society_simulation_…" +} +``` ### Analysis (7 tools) Turn a society simulation into specific, weighted policy results. Dedicated derivatives calculate budget, programme, decile, poverty, inequality, and gain-or-loss results. -`compute_budgetary_impact`, `compute_program_breakdown`, `compute_decile_impacts`, `compute_winners_losers`, `compute_poverty_metrics`, `compute_inequality_metrics`, `aggregate_result` +**`compute_budgetary_impact`** — Calculates changes in tax revenue, benefit spending, and the net budgetary impact of a reform. + +``` +compute_budgetary_impact(simulation_id="society_simulation_…") +# returns +{ + "tax_revenue": { "baseline": …, "reform": …, "change": … }, + "benefit_spending": { "baseline": …, "reform": …, "change": … }, + "net_budgetary_impact": …, "result_id": "budgetary_impact_…" +} +``` + +**`compute_program_breakdown`** — Calculates programme-level totals, caseloads, winners, and losers from a society simulation. + +``` +compute_program_breakdown( + simulation_id="society_simulation_…", + programs=["child_benefit", "universal_credit"] +) +# returns +{ + "programs": [ + { "program": "universal_credit", "is_tax": false, + "change": …, "baseline_count": …, "winners": …, "losers": … }, … + ] +} +``` + +**`compute_decile_impacts`** — Calculates changes in mean household income across a selected income- or wealth-decile concept. + +``` +compute_decile_impacts( + simulation_id="society_simulation_…", + decile_concept="household_net_income" +) +# returns +{ + "measure_label": "household net income", + "grouping_label": "Household net income decile", + "deciles": [{ "decile": 1, "absolute_change": …, "relative_change": … }, …] +} +``` + +**`compute_winners_losers`** — Calculates people-weighted gain, loss, and no-change shares within income or wealth deciles. + +``` +compute_winners_losers( + simulation_id="society_simulation_…", + basis="income" +) +# returns +{ + "grouping_label": "Income decile", + "deciles": [ + { "decile": 1, "gain_more_than_5pct": …, "no_change": …, … }, … + ] +} +``` + +**`compute_poverty_metrics`** — Calculates UK poverty rates and headcounts overall and by age under baseline and reform. + +``` +compute_poverty_metrics(simulation_id="society_simulation_…") +# returns +{ + "rates": [ + { "poverty_type": "relative_bhc", "group": "all", "baseline_rate": …, + "reform_rate": …, "rate_change": …, "baseline_headcount": … }, … + ] +} +``` + +**`compute_inequality_metrics`** — Calculates the Gini coefficient and changes in the top 10%, top 1%, and bottom 50% income shares. + +``` +compute_inequality_metrics(simulation_id="society_simulation_…") +# returns +{ + "metrics": { + "gini": { "baseline": …, "reform": …, "change": …, "relative_change": … }, + "top_10_share": { … }, "top_1_share": { … }, "bottom_50_share": { … } + } +} +``` + +**`aggregate_result`** — Calculates a weighted sum, mean, or count for a verified variable without returning survey rows. + +``` +aggregate_result( + simulation_id="society_simulation_…", + entity="household", variable="household_id", operation="count", + filter_variable="benunit_count_children", filter_variable_geq=3 +) +# returns +{ + "result": { "operation": "count", "target": "reform", + "filter_variable_geq": 3, "value": … }, + "privacy": "Aggregate only; no row-level records returned." +} +``` ### Presentation (1 tool) -Build a chart from a calculation result using a constrained chart schema. Charts are constructed from stored results, so the displayed artefact stays tied to the calculation. +Build a chart from a calculation result using a constrained chart schema. Policy charts are built from a stored result, so the displayed artefact stays tied to the calculation it came from. -`generate_chart` +**`generate_chart`** — Generates frontend-renderable chart markdown from a stored result or constrained chart data. + +``` +generate_chart( + chart_kind="decile_relative_bar", + result_id="decile_impacts_…", + title="Change in household net income by decile" +) +# returns +{ + "chart_markdown": …a fenced chart block, included verbatim in the answer…, + "spec": { "type": "preset", "preset": "decile_relative_bar", + "groupLabel": "Household net income decile", "data": [ … ] } +} +``` ## Under the hood @@ -78,10 +377,25 @@ Here's what happens inside a single request. UK Chat's core runtime is a FastAPI ### 1. Ground — A structured opening plan -_AI model_ +_Language model_ A fast language model reads the user's words and proposes a structured description of the request. It separates inputs the user supplied from the output they asked for and suggests which tools may be needed. This is an interpretation, not yet permission to calculate. +_The user's message_ + +``` +last_user_message="For 2026, set the Child Benefit eldest-child rate + to £30 a week and show the annual budgetary impact" +``` + +_Proposed plan_ + +``` +{ "domain_status": "uk_or_unspecified", "capability_status": "supported", + "tool": "compute_budgetary_impact", + "catalogue_queries": [{ "kind": "reform_target", "query": "eldest-child rate" }] } +``` + ### 2. Resolve — Current catalogue evidence _Gateway_ @@ -90,24 +404,81 @@ The gateway checks named policies, variables, and requested outputs against Poli It then refines the plan with the relevant tool inputs and declared defaults — for example, using the current year when the user does not name one. Any choice the system cannot resolve safely stays explicit instead of being silently invented. +_Catalogue query_ + +``` +resolve_catalogue_queries([ + { "kind": "reform_target", "query": "eldest-child rate" } +]) +``` + +_Match_ + +``` +{ + "identifier": "gov.hmrc.child_benefit.amount.eldest", + "match_type": "strong_phrase", "score": 0.9, "authoritative": true +} +``` + ### 3. Gate — Five deterministic outcomes _Gateway_ The gateway applies fixed policy to the grounded, catalogue-backed plan and assigns one of five outcomes. Only a ready request may enter the calculation loop. +The gate is deliberately biased towards computing. If the classifier errors or returns something the server cannot read, the request is treated as ready rather than refused, on the view that a wrong refusal costs more than a wrong attempt. The guarantees that follow come from the tools, not from the gate. + - Ready: the request is complete and supported, so calculation can begin. - Needs plan: a required choice is missing, so the chat asks a clarifying question. - Partial: the request mixes supported and unsupported outputs, so the chat offers the supported part. - Out of scope: the requested policy output cannot be modelled, so calculation tools stay closed. - Irrelevant: the request is unrelated to UK tax and benefit policy and takes a lightweight response path. +_Gated plan_ + +``` +gate( + in_domain=true, tool="compute_budgetary_impact", + unmodellable_outputs=[] +) +``` + +_Verdict_ + +``` +{ + "outcome": "ready", + "gating_reasons": [] +} +``` + ### 4. Verify — Exact reform construction _Gateway_ A society-wide reform receives a second bounded check. The resolver searches the rules engine's current reform targets, binds the user's wording to an exact parameter path and date, and constructs reform JSON. The validator must accept that construction before a simulation can run. +What the gateway approves is then the only reform that can run. The simulation tool compares the reform it is handed against the approved construction and refuses anything that does not match, so a different reform cannot be substituted after the check has passed. + +_Resolver construction_ + +``` +emit_reform_assessment( + reform={ "gov.hmrc.child_benefit.amount.eldest": 30.0 }, + confidence=95 +) +``` + +_Approved for this turn_ + +``` +{ + "approved_reform": { "gov.hmrc.child_benefit.amount.eldest": 30.0 }, + "require_approved_reform": true +} +``` + ### 5. Calculate — A bounded 21-tool runtime _Supporting tools_ @@ -116,6 +487,23 @@ A language model selected for the request's type and size executes the verified The language model chooses and sequences tools, but it does not supply the figures. Each number comes back as tool data produced by the PolicyEngine UK model. +_Plan handed to the tool loop_ + +``` +Required tool order: run_society_simulation -> compute_budgetary_impact. +Runtime handoff: pass the result_id returned by run_society_simulation + as simulation_id to the target derivative. +``` + +_Each tool result_ + +``` +{ + "type": "tool_result", "tool_use_id": …, + "content": …the tool’s JSON, capped at 15,000 characters… +} +``` + ### 6. Stream — An inspectable response _Gateway_ @@ -124,20 +512,123 @@ The gateway streams text chunks, tool starts, tool inputs, completion summaries, The tool loop remains bounded by a 30-round cap, repeated-call detection, result-size limits, and explicit terminal events. Tool errors return as data so the language model can recover without inventing a result. -## One reform, end to end +_Turn events_ + +``` +ToolStarted → ToolUsed → ToolCompleted → TextChunk → TurnCompleted +``` + +_Server-sent frames_ + +``` +data: { "type": "tool_use", "tool_name": "compute_budgetary_impact", + "status": "pending" } +data: { "type": "done", "content": …, "stop_reason": …, "usage": { … } } +``` + +## The same reform, call by call Consider: “For 2026, set the Child Benefit eldest-child rate to £30 a week and show the annual budgetary impact.” The request contains a policy, a final value, a year, and an output. It does not contain a PolicyEngine parameter path, a reform object, or the sequence of tools needed to answer it. -### User request +The six stages above decide what the system does; underneath, the turn is a short sequence of typed calls, each one taking an argument that an earlier call produced. This is that sequence for the request. + +### Tool-call trace — 5 calls, one turn + +**1. Find the target** — _Discovery_ + +``` +list_reform_targets( + query="child benefit eldest" +) +``` + +- ✓ gov.hmrc.child_benefit.amount.eldest +- · returned from the installed UK package, not from memory +- · “eldest-child rate” matches the parameter’s catalogue label + +**2. Read current law** — _Discovery_ + +``` +get_parameter( + path="gov.hmrc.child_benefit.amount.eldest", + year=2026 +) +``` + +Returned: + +``` +{ + "unit": "currency-GBP", + "period": "week", + "value": …the 2026 rate in the deployed package… +} +``` + +- · the comparator is whatever the package holds, not a remembered rate +- · the unit and period fix what “£30 a week” has to be written as + +**3. Validate** — _Validation_ -> For 2026, set the Child Benefit eldest-child rate to £30 a week and show the annual budgetary impact +``` +validate_reform( + reform={ + "gov.hmrc.child_benefit.amount.eldest": { + "2026-01-01.2026-12-31": 30.0 + } + }, + year=2026 +) +``` -1. **Ground** — _AI model_ — £30 a week, 2026, budgetary impact. Extract the requested value, policy year, and output from the user’s exact words, while keeping the proposed plan separate from verified facts. -2. **Resolve** — _Gateway_ — catalogue search, exact parameter. Bind “eldest-child rate” to gov.hmrc.child_benefit.amount.eldest in the current catalogue and confirm that budgetary impact is supported. -3. **Gate** — _Gateway_ — ready, all inputs explicit. Classify the plan as ready because its required policy, value, year, and output are present and supported. Only this outcome opens the calculation loop. -4. **Verify** — _Gateway_ — effective 1 January 2026, validate_reform. Construct an exact dated reform that sets the parameter to 30.0 pounds per week, then validate it against the PolicyEngine UK rules engine before simulation. -5. **Calculate** — _Supporting tools_ — run_society_simulation, compute_budgetary_impact. Run current law and the approved reform on the same population, store the result, and pass its handle to the budgetary-impact derivative. -6. **Stream** — _Gateway_ — tool events, method and result. Stream the calculation trace and final answer with the comparator, dataset, method, and sign convention attached so the result can be inspected. +- ✓ the path exists in the parameter schema +- ✓ 30.0 is the right type for a currency-GBP parameter +- ✓ the date range falls inside the package's coverage +- Note: an unrecognised path is rejected here. + +**4. Simulate** — _Simulation_ + +``` +run_society_simulation( + reform={ …validated above… }, + year=2026 +) +``` + +Returned to the language model: + +``` +{ + "result_id": "society_simulation_…", + "dataset": { "label": "Enhanced FRS 2024-25", … }, + "year": 2026 +} +``` + +Not returned: + +- ✗ household rows +- ✗ survey weights +- ✗ any serialisable simulation object + +**5. Compute** — _Analysis_ + +``` +compute_budgetary_impact( + simulation_id="society_simulation_…" +) +``` + +Computed inside the engine: + +- ✓ baseline and reform totals for tax revenue and benefit spending +- ✓ the net budgetary impact reported below +- ✓ survey weights applied inside the PolicyEngine UK model +- Stored: `result_id` → typed handle, turn-local. + +Two features of the sequence carry most of the design. The first is that `run_society_simulation` hands back a handle rather than data: the language model receives a `result_id`, a dataset reference, and a year, and never receives household rows, survey weights, or any serialisable simulation object. Weighting happens inside the PolicyEngine UK model, and the seven analysis tools are the only route from a stored simulation to a number. + +The second is provenance. Every parameter path, value, and handle in the sequence was either supplied by the user or produced by a preceding call, so each argument can be traced to its source. Nothing in it was recalled. ### Computed result: About £0.9 billion a year in additional government cost @@ -155,13 +646,13 @@ If the catalogue returned more than one materially plausible interpretation, or ## What you can ask -The Child Benefit reform above is one of four kinds of questions supported today, and users can stack these questions together within one conversation without starting a new chat. +The Child Benefit reform above is one of four kinds of questions supported today, and users can stack these questions within one conversation without starting a new chat. ### Households — Illustrative calculations > A single parent, two children, earning £28,000 — what do they take home? -The household is constructed as a synthetic input, validated, and run through the PolicyEngine UK model to produce taxes, benefits, and net income under stated assumptions. One household containing one benefit unit per call; more complex arrangements need separate calls. +The household is constructed as a synthetic input, validated, and run through the PolicyEngine UK model to produce taxes, benefits, and net income under stated assumptions. Each call covers one household containing one benefit unit. More complex arrangements need separate calls. `list_household_input_variables`, `validate_household`, `run_household_simulation` @@ -177,7 +668,7 @@ Parametric reforms run across the modelled UK population: budgetary impact, prog > Show me that by decile. -When a comparison reads better as a picture than as prose, the computed output is mapped onto a fixed chart preset. The chart is drawn from the same typed result as the text, so the two cannot disagree. +When a comparison reads better as a picture than as prose, the computed output is mapped onto a fixed chart preset. A policy preset has to name the stored result it draws, so that chart and the text around it cannot disagree. `generate_chart` @@ -189,9 +680,11 @@ Follow-ups stay in the same thread. The conversation carries the context; the si ## Limitations +Some of what follows is a deliberate boundary that will not change. The rest is the current state of a beta: reform coverage is incomplete, answers vary in how fully they state their assumptions, and gateway behaviour will move as we act on what users report. + The chat is a modelling tool, not advice. It reports what the engine calculates under stated assumptions, and it is not a substitute for professional guidance on an individual's circumstances. -Society results are direct static microsimulation estimates. They do not estimate behavioural responses, employment effects, inflation, GDP, market reactions, or general-equilibrium effects. When a request mixes a supported policy result with one of those effects, the gateway notifies the user that it cannot calculate these effects, then proceeds with what it can. +Society results are direct static microsimulation estimates. They do not estimate behavioural responses, employment effects, inflation, GDP, market reactions, or general-equilibrium effects. When a request mixes a supported policy result with one of those effects, the gateway says which part it cannot calculate and asks whether to run the part it can, rather than running anything that turn. Results depend on the dataset, year, and modelling assumptions, and the chat states these dependencies rather than presenting figures as universal. They should still be checked against independent estimates from organisations such as the Institute for Fiscal Studies, Resolution Foundation, or Office for Budget Responsibility. @@ -203,4 +696,6 @@ We want to widen the range of reforms covered by typed tools, improve the speed UK Chat's answers can be cited and reproduced because the figures come from the same open engine that powers the rest of PolicyEngine. Try it with a reform, then inspect the stated year, dataset, comparator, and method alongside the answer. -[Try PolicyEngine UK Chat](https://policyengine.org/uk/chat) +Because this is a beta, the cases where it fails are more useful to us than the cases where it works: a request refused that the tools should support, an answer that omits an assumption it should state, or a figure that does not match what the engine returns for the same reform. PolicyEngine's policy team is working through the same exercise, and what they and other early users report will decide what we change first. + +[Try PolicyEngine UK Chat (beta)](https://policyengine.org/uk/chat) diff --git a/src/App.jsx b/src/App.jsx index 105bedc..e285bad 100644 --- a/src/App.jsx +++ b/src/App.jsx @@ -13,41 +13,107 @@ const toolFamilies = [ { name: 'list_entities', summary: 'Lists the PolicyEngine UK model entities and the number of variables defined for each one.', + call: 'list_entities()', + returns: `{ + "status": "success", + "entities": [{ "name": "person", "variable_count": … }, …] +}`, }, { name: 'search_variables', summary: 'Searches the UK variable registry and identifies each match’s entity and default-output status.', + call: `search_variables( + query="employment income", + entity="person" +)`, + returns: `{ + "variables": [ + { "name": "employment_income", "entity": "person", + "value_type": "float", "definition_period": "year" } + ] +}`, }, { name: 'get_variable', summary: 'Verifies one exact UK variable and returns its metadata, entity, and default-output status.', + call: 'get_variable(name="household_net_income")', + returns: `{ + "variable": { + "name": "household_net_income", "entity": "household", + "value_type": "float", "is_default_society_output": true + } +}`, }, { name: 'search_parameters', summary: 'Searches policy parameters by path, label, description, or known alias.', + call: 'search_parameters(query="personal allowance")', + returns: `{ + "parameters": [ + { "path": "gov.hmrc.income_tax.allowances.personal_allowance.amount", + "unit": "currency-GBP", "aliases": ["personal allowance", …] } + ] +}`, }, { name: 'get_parameter', summary: 'Looks up one exact policy parameter and returns its metadata and value for a selected year.', + call: `get_parameter( + path="gov.hmrc.income_tax.allowances.personal_allowance.amount", + year=2026 +)`, + returns: `{ + "parameter": { + "unit": "currency-GBP", "year": 2026, + "value": …the 2026 allowance in the deployed package… + } +}`, }, { name: 'list_reform_targets', summary: 'Searches the current catalogue for parameter paths that can be used in a reform.', + call: 'list_reform_targets(query="universal credit standard allowance")', + returns: `{ + "targets": [ + { "path": "gov.dwp.universal_credit.standard_allowance.amount.SINGLE_OLD", + "aliases": ["universal credit standard allowance", …] } + ] +}`, }, { name: 'list_household_input_variables', summary: 'Lists known variables that can be supplied as overrides for an illustrative household.', + call: 'list_household_input_variables(entity="person")', + returns: `{ + "entity": "person", + "variables": [{ "name": "employment_income", … }, …], + "input_contract": "policyengine.py accepts any known variable on its declared entity…" +}`, }, { name: 'list_society_output_variables', summary: 'Lists the variables a society simulation materialises by default, grouped by entity.', + call: 'list_society_output_variables(entity="household")', + returns: `{ + "default_variables_by_entity": { + "household": ["household_net_income", "household_tax", "household_benefits", …] + }, + "default_variable_count": … +}`, }, { name: 'list_supported_outputs', summary: 'Lists the household, society, derivative, and chart outputs supported by the chat runtime.', + call: 'list_supported_outputs(scope="derivative")', + returns: `{ + "scope": "derivative", + "outputs": [ + { "scope": "derivative", "name": "budgetary_impact", "description": … }, … + ] +}`, }, ], }, @@ -59,10 +125,28 @@ const toolFamilies = [ { name: 'validate_reform', summary: 'Validates flat PolicyEngine reform JSON for a selected year without running a simulation.', + call: `validate_reform( + reform={ "gov.hmrc.income_tax.allowances.personal_allowance.amount": 15000 }, + year=2026 +)`, + returns: `{ + "valid": true, + "parameter_paths": ["gov.hmrc.income_tax.allowances.personal_allowance.amount"], + "warnings": [] +}`, }, { name: 'validate_household', summary: 'Validates an illustrative synthetic UK household against PolicyEngine variable metadata.', + call: `validate_household( + people=[{ "age": 35, "employment_income": 28000 }, { "age": 6 }, { "age": 3 }], + household={ "country": "ENGLAND" } +)`, + returns: `{ + "valid": true, + "year": 2026, + "people_count": 3, "warnings": [] +}`, }, ], }, @@ -74,11 +158,29 @@ const toolFamilies = [ { name: 'run_household_simulation', summary: 'Runs an illustrative synthetic household through the PolicyEngine UK tax-benefit model.', + call: `run_household_simulation( + people=[{ "age": 35, "employment_income": 28000 }, { "age": 6 }, { "age": 3 }], + household={ "country": "ENGLAND" } +)`, + returns: `{ + "reform_applied": false, + "household": { "household_net_income": …, "household_tax": … }, + "result_id": "household_simulation_…" +}`, }, { name: 'run_society_simulation', summary: 'Runs baseline and reform UK simulations and returns metadata plus a turn-local result handle.', + call: `run_society_simulation( + reform={ "gov.hmrc.income_tax.allowances.personal_allowance.amount": 15000 }, + year=2026 +)`, + returns: `{ + "year": 2026, "reform_applied": true, + "dataset": { "label": "Enhanced FRS 2024-25", "row_level_access": false }, + "result_id": "society_simulation_…" +}`, }, ], }, @@ -92,35 +194,94 @@ const toolFamilies = [ name: 'compute_budgetary_impact', summary: 'Calculates changes in tax revenue, benefit spending, and the net budgetary impact of a reform.', + call: 'compute_budgetary_impact(simulation_id="society_simulation_…")', + returns: `{ + "tax_revenue": { "baseline": …, "reform": …, "change": … }, + "benefit_spending": { "baseline": …, "reform": …, "change": … }, + "net_budgetary_impact": …, "result_id": "budgetary_impact_…" +}`, }, { name: 'compute_program_breakdown', summary: 'Calculates programme-level totals, caseloads, winners, and losers from a society simulation.', + call: `compute_program_breakdown( + simulation_id="society_simulation_…", + programs=["child_benefit", "universal_credit"] +)`, + returns: `{ + "programs": [ + { "program": "universal_credit", "is_tax": false, + "change": …, "baseline_count": …, "winners": …, "losers": … }, … + ] +}`, }, { name: 'compute_decile_impacts', summary: 'Calculates changes in mean household income across a selected income- or wealth-decile concept.', + call: `compute_decile_impacts( + simulation_id="society_simulation_…", + decile_concept="household_net_income" +)`, + returns: `{ + "measure_label": "household net income", + "grouping_label": "Household net income decile", + "deciles": [{ "decile": 1, "absolute_change": …, "relative_change": … }, …] +}`, }, { name: 'compute_winners_losers', summary: 'Calculates people-weighted gain, loss, and no-change shares within income or wealth deciles.', + call: `compute_winners_losers( + simulation_id="society_simulation_…", + basis="income" +)`, + returns: `{ + "grouping_label": "Income decile", + "deciles": [ + { "decile": 1, "gain_more_than_5pct": …, "no_change": …, … }, … + ] +}`, }, { name: 'compute_poverty_metrics', summary: 'Calculates UK poverty rates and headcounts overall and by age under baseline and reform.', + call: 'compute_poverty_metrics(simulation_id="society_simulation_…")', + returns: `{ + "rates": [ + { "poverty_type": "relative_bhc", "group": "all", "baseline_rate": …, + "reform_rate": …, "rate_change": …, "baseline_headcount": … }, … + ] +}`, }, { name: 'compute_inequality_metrics', summary: 'Calculates the Gini coefficient and changes in the top 10%, top 1%, and bottom 50% income shares.', + call: 'compute_inequality_metrics(simulation_id="society_simulation_…")', + returns: `{ + "metrics": { + "gini": { "baseline": …, "reform": …, "change": …, "relative_change": … }, + "top_10_share": { … }, "top_1_share": { … }, "bottom_50_share": { … } + } +}`, }, { name: 'aggregate_result', summary: 'Calculates a weighted sum, mean, or count for a verified variable without returning survey rows.', + call: `aggregate_result( + simulation_id="society_simulation_…", + entity="household", variable="household_id", operation="count", + filter_variable="benunit_count_children", filter_variable_geq=3 +)`, + returns: `{ + "result": { "operation": "count", "target": "reform", + "filter_variable_geq": 3, "value": … }, + "privacy": "Aggregate only; no row-level records returned." +}`, }, ], }, @@ -128,12 +289,22 @@ const toolFamilies = [ name: 'Presentation', summary: 'Build a chart from a calculation result using a constrained chart schema.', purpose: - 'Charts are constructed from stored results, so the displayed artefact stays tied to the calculation.', + 'Policy charts are built from a stored result, so the displayed artefact stays tied to the calculation it came from.', tools: [ { name: 'generate_chart', summary: 'Generates frontend-renderable chart markdown from a stored result or constrained chart data.', + call: `generate_chart( + chart_kind="decile_relative_bar", + result_id="decile_impacts_…", + title="Change in household net income by decile" +)`, + returns: `{ + "chart_markdown": …a fenced chart block, included verbatim in the answer…, + "spec": { "type": "preset", "preset": "decile_relative_bar", + "groupLabel": "Household net income decile", "data": [ … ] } +}`, }, ], }, @@ -146,7 +317,7 @@ const questionKinds = [ summary: 'Illustrative calculations', question: 'A single parent, two children, earning £28,000 — what do they take home?', description: - 'The household is constructed as a synthetic input, validated, and run through the PolicyEngine UK model to produce taxes, benefits, and net income under stated assumptions. One household containing one benefit unit per call; more complex arrangements need separate calls.', + 'The household is constructed as a synthetic input, validated, and run through the PolicyEngine UK model to produce taxes, benefits, and net income under stated assumptions. Each call covers one household containing one benefit unit. More complex arrangements need separate calls.', tools: ['list_household_input_variables', 'validate_household', 'run_household_simulation'], }, { @@ -164,7 +335,7 @@ const questionKinds = [ summary: 'Computed outputs, drawn', question: 'Show me that by decile.', description: - 'When a comparison reads better as a picture than as prose, the computed output is mapped onto a fixed chart preset. The chart is drawn from the same typed result as the text, so the two cannot disagree.', + 'When a comparison reads better as a picture than as prose, the computed output is mapped onto a fixed chart preset. A policy preset has to name the stored result it draws, so that chart and the text around it cannot disagree.', tools: ['generate_chart'], }, { @@ -178,6 +349,248 @@ const questionKinds = [ }, ]; +/* The five calls the Child Benefit request in "The same reform, call by call" actually + makes, one per tool family bar Presentation. Identifiers are shortened and the + current-law rate is elided, because both come from whichever PolicyEngine UK + package is deployed rather than from anything fixed at publication. */ +const traceSteps = [ + { + id: 'find-target', + stage: 'Find the target', + family: 'Discovery', + body: ( + <> +
+
tool call
+ 
+            list_reform_targets
+            {'(\n  query='}
+            "child benefit eldest"
+            {'\n)'}
+
+
+
+
result
+
+
+ ✓ + gov.hmrc.child_benefit.amount.eldest +
+
+ · + returned from the installed UK package, not from memory +
+
+ · + “eldest-child rate” matches the parameter’s catalogue label +
+
+
+ + ), + }, + { + id: 'read-current-law', + stage: 'Read current law', + family: 'Discovery', + body: ( + <> +
+
tool call
+ 
+            get_parameter
+            {'(\n  path='}
+            "gov.hmrc.child_benefit.amount.eldest"
+            {',\n  year='}
+            2026
+            {'\n)'}
+
+
+
+
returned
+ 
+            {'{\n  '}
+            "unit"
+            {': '}
+            "currency-GBP"
+            {',\n  '}
+            "period"
+            {': '}
+            "week"
+            {',\n  '}
+            "value"
+            {': '}
+            …the 2026 rate in the deployed package…
+            {'\n}'}
+
+
+
+
why it matters
+
+
+ · + the comparator is whatever the package holds, not a remembered rate +
+
+ · + the unit and period fix what “£30 a week” has to be written as +
+
+
+ + ), + }, + { + id: 'validate', + stage: 'Validate', + family: 'Validation', + body: ( + <> +
+
tool call
+ 
+            validate_reform
+            {'(\n  reform={\n    '}
+            "gov.hmrc.child_benefit.amount.eldest"
+            {': {\n      '}
+            "2026-01-01.2026-12-31"
+            {': '}
+            30.0
+            {'\n    }\n  },\n  year='}
+            2026
+            {'\n)'}
+
+
+
+
result
+
+
+ ✓ + the path exists in the parameter schema +
+
+ ✓ + 30.0 is the right type for a currency-GBP parameter +
+
+ ✓ + the date range falls inside the package's coverage +
+
+
+
+
note
+
+ an unrecognised path + rejected here +
+
+ + ), + }, + { + id: 'simulate', + stage: 'Simulate', + family: 'Simulation', + body: ( + <> +
+
tool call
+ 
+            run_society_simulation
+            {'(\n  reform={ '}
+            …validated above…
+            {' },\n  year='}
+            2026
+            {'\n)'}
+
+
+
+
returned to the language model
+ 
+            {'{\n  '}
+            "result_id"
+            {': '}
+            "society_simulation_…"
+            {',\n  '}
+            "dataset"
+            {': { '}
+            "label"
+            {': '}
+            "Enhanced FRS 2024-25"
+            {', '}
+            …
+            {' },\n  '}
+            "year"
+            {': '}
+            2026
+            {'\n}'}
+
+
+
+
not returned
+
+
+ ✗ + household rows +
+
+ ✗ + survey weights +
+
+ ✗ + any serialisable simulation object +
+
+
+ + ), + }, + { + id: 'compute', + stage: 'Compute', + family: 'Analysis', + body: ( + <> +
+
tool call
+ 
+            compute_budgetary_impact
+            {'(\n  simulation_id='}
+            "society_simulation_…"
+            {'\n)'}
+
+
+
+
computed inside the engine
+
+
+ ✓ + baseline and reform totals for tax revenue and benefit spending +
+
+ ✓ + the net budgetary impact reported below +
+
+ ✓ + survey weights applied inside the PolicyEngine UK model +
+
+
+
+
stored
+
+ result_id → typed handle + turn-local +
+
+ + ), + }, +]; + const authors = [ { id: 'vahid-ahmadi', @@ -198,7 +611,7 @@ const requestSteps = [ number: 1, title: 'Ground', subtitle: 'A structured opening plan', - segment: 'AI model', + segment: 'Language model', body: ( <>

@@ -208,6 +621,15 @@ const requestSteps = [

), + example: { + callTitle: 'the user’s message', + call: `last_user_message="For 2026, set the Child Benefit eldest-child rate + to £30 a week and show the annual budgetary impact"`, + returnsTitle: 'proposed plan', + returns: `{ "domain_status": "uk_or_unspecified", "capability_status": "supported", + "tool": "compute_budgetary_impact", + "catalogue_queries": [{ "kind": "reform_target", "query": "eldest-child rate" }] }`, + }, }, { number: 2, @@ -228,6 +650,17 @@ const requestSteps = [

), + example: { + callTitle: 'catalogue query', + call: `resolve_catalogue_queries([ + { "kind": "reform_target", "query": "eldest-child rate" } +])`, + returnsTitle: 'match', + returns: `{ + "identifier": "gov.hmrc.child_benefit.amount.eldest", + "match_type": "strong_phrase", "score": 0.9, "authoritative": true +}`, + }, }, { number: 3, @@ -240,6 +673,12 @@ const requestSteps = [ The gateway applies fixed policy to the grounded, catalogue-backed plan and assigns one of five outcomes. Only a ready request may enter the calculation loop.

+

+ The gate is deliberately biased towards computing. If the classifier errors or returns something the + server cannot read, the request is treated as ready rather than refused, on the view that a wrong + refusal costs more than a wrong attempt. The guarantees that follow come from the tools, not from + the gate. +

), + example: { + callTitle: 'gated plan', + call: `gate( + in_domain=true, tool="compute_budgetary_impact", + unmodellable_outputs=[] +)`, + returnsTitle: 'verdict', + returns: `{ + "outcome": "ready", + "gating_reasons": [] +}`, + }, }, { number: 4, @@ -275,8 +726,25 @@ const requestSteps = [ current reform targets, binds the user's wording to an exact parameter path and date, and constructs reform JSON. The validator must accept that construction before a simulation can run.

+

+ What the gateway approves is then the only reform that can run. The simulation tool compares the + reform it is handed against the approved construction and refuses anything that does not match, so a + different reform cannot be substituted after the check has passed. +

), + example: { + callTitle: 'resolver construction', + call: `emit_reform_assessment( + reform={ "gov.hmrc.child_benefit.amount.eldest": 30.0 }, + confidence=95 +)`, + returnsTitle: 'approved for this turn', + returns: `{ + "approved_reform": { "gov.hmrc.child_benefit.amount.eldest": 30.0 }, + "require_approved_reform": true +}`, + }, }, { number: 5, @@ -297,6 +765,17 @@ const requestSteps = [

), + example: { + callTitle: 'plan handed to the tool loop', + call: `Required tool order: run_society_simulation -> compute_budgetary_impact. +Runtime handoff: pass the result_id returned by run_society_simulation + as simulation_id to the target derivative.`, + returnsTitle: 'each tool result', + returns: `{ + "type": "tool_result", "tool_use_id": …, + "content": …the tool’s JSON, capped at 15,000 characters… +}`, + }, }, { number: 6, @@ -317,9 +796,66 @@ const requestSteps = [

), + example: { + callTitle: 'turn events', + call: 'ToolStarted → ToolUsed → ToolCompleted → TextChunk → TurnCompleted', + returnsTitle: 'server-sent frames', + returns: `data: { "type": "tool_use", "tool_name": "compute_budgetary_impact", + "status": "pending" } +data: { "type": "done", "content": …, "stop_reason": …, "usage": { … } }`, + }, }, ]; +/* The tool and stage examples are stored as literal source strings rather than + as hand-tagged JSX, so that all 27 of them stay short and comparable. This + applies the same token classes the call trace above marks up by hand: an + elision written between ellipses reads as a comment, and everything else is a + string, a literal, or the name of the thing being called. A phrase elision + may not span JSON punctuation, or two separate elisions on one line would be + read as a single comment running through the fields between them. */ +const EXAMPLE_TOKENS = + /(…[^…\n{}":]*…|…)|("(?:[^"\\]|\\.)*")|\b(true|false|null|\d+(?:\.\d+)?)\b|\b([a-z_][a-z0-9_]*)(?=\()/g; + +function ExampleCode({ children }) { + const nodes = []; + let cursor = 0; + + for (const match of children.matchAll(EXAMPLE_TOKENS)) { + if (match.index > cursor) { + nodes.push(children.slice(cursor, match.index)); + } + const [token, comment, string, literal] = match; + nodes.push( + + {token} + , + ); + cursor = match.index + token.length; + } + nodes.push(children.slice(cursor)); + + return 
{nodes}
; +} + +function ExampleExchange({ className, callTitle, call, returnsTitle, returns }) { + return ( +
+
+
{callTitle}
+ {call} +
+
+
{returnsTitle}
+ {returns} +
+
+ ); +} + function FadeIn({ children, delay = 0, className }) { const prefersReducedMotion = useReducedMotion(); @@ -341,6 +877,7 @@ function Hero() {

PolicyEngine UK Chat: an AI interface for tax and benefits

+ Beta Introducing PolicyEngine's newest AI-powered tool to help users understand UK tax and benefit policy

@@ -353,10 +890,10 @@ function Introduction() {

People are increasingly turning to AI language models to answer complex questions about tax and - benefit policy and reform because models can read loosely worded questions and answer them in plain - language. This presents a problem: for many users, tax and benefit questions directly affect their - lives and livelihoods, so they require correct, verifiable answers. Language models generate responses - from learned statistical patterns, and an answer may be partly or wholly wrong. + benefit policy and reform because these models can read loosely worded questions and answer them in + plain language. This presents a problem: for many users, tax and benefit questions directly affect + their lives and livelihoods, so they require correct, verifiable answers. Language models generate + responses from learned statistical patterns, and an answer may be partly or wholly wrong.

PolicyEngine therefore built UK Chat, an AI interface to PolicyEngine's deterministic rules and @@ -364,6 +901,36 @@ function Introduction() { The gateway checks the proposed inputs and output against the current PolicyEngine catalogue; only then do deterministic tools compute quantitative policy results from actual tax and benefit rules.

+

+ UK Chat is released as a beta. The pipeline described here is in place and running, + but the range of reforms it covers, the wording of its answers, and its judgement about what it cannot + compute are all still being tested. We are publishing it at this stage to gather feedback on where it + is useful and where it falls short, beginning with PolicyEngine's own policy team. +

+ + + ); +} + +function TodayWorkflow() { + return ( + +

How policy analysis is done today

+

+ PolicyEngine already answers questions of this kind, but not in conversation. Someone costing a Child + Benefit change either builds the reform by hand in the web app, or writes code against the model: find + the parameter path, construct a reform, run baseline and reform over the same microdata, read off the + measure. Both routes are precise and fully inspectable, and both call the engine UK Chat calls + underneath. Both also ask the same thing of the person using them: knowing which parameter carries the + policy, how to express the change, and which output answers the question. +

+

+ For a policy analyst, a journalist, or a household working out what a reform would mean for them, the + barrier is the setup rather than the arithmetic — and prompting a general-purpose language model looks + like the way around it. The rest of this article is about why that shortcut does not produce a figure + anyone can rely on. UK Chat replaces neither route, but for the questions its tools already cover it + removes the setup and leaves the calculation where it was. +

); } @@ -418,7 +985,7 @@ function BoundaryDiagram() { className="agent-flow-svg boundary-flow-svg" viewBox="0 0 900 300" role="img" - aria-label="The boundary between the predictive AI layer and the deterministic PolicyEngine engine: the language model interprets the question and calls typed tools; the engine validates, simulates, and computes; the language model then explains the returned numbers." + aria-label="The boundary between the predictive AI layer and the deterministic engine: the language model interprets the question and calls typed tools; the engine validates, simulates, and computes; the language model then explains the returned numbers." > {/* A slightly concave, offset head reads as an arrow rather than a @@ -622,14 +1189,15 @@ function Boundary() {

- When a question is ready for computation, the language model receives the calculation tools plus a - machine-readable description of what the deployed engine can compute: its capabilities and parameter - schema. What the language model can be asked to compute is therefore tied to the deployed version of - PolicyEngine, not to a hand-written prompt that drifts out of date. + When a question is ready for computation, the language model receives the calculation tools, but not a + list of what exists in the model. It has to ask: names are confirmed by calling discovery tools + against the deployed package, so a policy the model half-remembers either resolves to a real parameter + path or fails to resolve at all. The gateway works the same way from the other side, describing the + tools from their own schemas so its vocabulary cannot drift from what the tools actually accept.

The gateway sits on that line. It takes the plan the language model proposes and decides whether it - may cross into the deterministic side, which is why the system has three parts rather than two: the + may cross into the deterministic side. That is why the system has three parts rather than two: the language model, the gateway, and the tools a plan reaches once the gateway admits it.

@@ -733,7 +1301,8 @@ function ToolExplorer() {

The language model proposes a plan

- UK Chat breaks the user pathway into three segments: the AI model, the gateway, and supporting tools. + UK Chat breaks the user pathway into three segments: the language model, the gateway, and supporting + tools.

First, the language model does the predictive work. It takes the user's open-ended prompt and @@ -796,6 +1365,13 @@ function ToolExplorer() {

{activeTool.name}

{activeTool.summary}

+
@@ -991,7 +1567,10 @@ function ArchitectureScrolly() { {step.segment} -
{step.body}
+
+ {step.body} + +
))} @@ -1015,96 +1594,60 @@ function ArchitectureScrolly() { ); } -function WorkedExample() { - const phases = [ - { - number: 1, - title: 'Ground', - segment: 'AI model', - agents: ['£30 a week', '2026', 'budgetary impact'], - description: - 'Extract the requested value, policy year, and output from the user’s exact words, while keeping the proposed plan separate from verified facts.', - }, - { - number: 2, - title: 'Resolve', - segment: 'Gateway', - agents: ['catalogue search', 'exact parameter'], - description: - 'Bind “eldest-child rate” to gov.hmrc.child_benefit.amount.eldest in the current catalogue and confirm that budgetary impact is supported.', - }, - { - number: 3, - title: 'Gate', - segment: 'Gateway', - agents: ['ready', 'all inputs explicit'], - description: - 'Classify the plan as ready because its required policy, value, year, and output are present and supported. Only this outcome opens the calculation loop.', - }, - { - number: 4, - title: 'Verify', - segment: 'Gateway', - agents: ['effective 1 January 2026', 'validate_reform'], - description: - 'Construct an exact dated reform that sets the parameter to 30.0 pounds per week, then validate it against the PolicyEngine UK rules engine before simulation.', - }, - { - number: 5, - title: 'Calculate', - segment: 'Supporting tools', - agents: ['run_society_simulation', 'compute_budgetary_impact'], - description: - 'Run current law and the approved reform on the same population, store the result, and pass its handle to the budgetary-impact derivative.', - }, - { - number: 6, - title: 'Stream', - segment: 'Gateway', - agents: ['tool events', 'method and result'], - description: - 'Stream the calculation trace and final answer with the comparator, dataset, method, and sign convention attached so the result can be inspected.', - }, - ]; +const traceItems = traceSteps.map((step) => ({ key: step.id, step })); + +function CallTrace() { + const [activeIndex, setActiveIndex] = useState(0); + + return ( + setActiveIndex(index)} + cardsClassName="tool-cards trace-cards" + renderCard={(item, index) => ( + <> + {index + 1} + {item.step.stage} + {item.step.family} + + )} + > +
{traceSteps[activeIndex].body}
+ + ); +} +function WorkedExample() { return ( -

One reform, end to end

+

The same reform, call by call

Consider: “For 2026, set the Child Benefit eldest-child rate to £30 a week and show the annual budgetary impact.” The request contains a policy, a final value, a year, and an output. It does not contain a PolicyEngine parameter path, a reform object, or the sequence of tools needed to answer it.

-
-
-
User request
-
- For 2026, set the Child Benefit eldest-child rate to £30 a week and show the annual budgetary - impact -
-
-
- {phases.map((phase) => ( -
-
-
{phase.number}
-
{phase.title}
-
{phase.segment}
-
-
-
- {phase.agents.map((agent) => ( - - {agent} - - ))} -
-

{phase.description}

-
-
- ))} -
-
+

+ The six stages above decide what the system does; underneath, the turn is a short sequence of typed + calls, each one taking an argument that an earlier call produced. This is that sequence for the + request. +

+ +

+ Two features of the sequence carry most of the design. The first is that{' '} + run_society_simulation hands back a handle rather than data: the language model receives + a result_id, a dataset reference, and a year, and never receives household rows, survey + weights, or any serialisable simulation object. Weighting happens inside the PolicyEngine UK model, + and the seven analysis tools are the only route from a stored simulation to a number. +

+

+ The second is provenance. Every parameter path, value, and handle in the sequence was either supplied + by the user or produced by a preceding call, so each argument can be traced to its source. Nothing in + it was recalled. +

@@ -1168,7 +1711,7 @@ function WhatYouCanAsk() {

What you can ask

The Child Benefit reform above is one of four kinds of questions supported today, and users can stack - these questions together within one conversation without starting a new chat. + these questions within one conversation without starting a new chat.

Limitations

+

+ Some of what follows is a deliberate boundary that will not change. The rest is the current state of a + beta: reform coverage is incomplete, answers vary in how fully they state their assumptions, and + gateway behaviour will move as we act on what users report. +

The chat is a modelling tool, not advice. It reports what the engine calculates under stated assumptions, and it is not a substitute for professional guidance on an individual's @@ -1219,8 +1767,8 @@ function Limitations() {

Society results are direct static microsimulation estimates. They do not estimate behavioural responses, employment effects, inflation, GDP, market reactions, or general-equilibrium effects. When - a request mixes a supported policy result with one of those effects, the gateway notifies the user - that it cannot calculate these effects, then proceeds with what it can. + a request mixes a supported policy result with one of those effects, the gateway says which part it + cannot calculate and asks whether to run the part it can, rather than running anything that turn.

Results depend on the dataset, year, and modelling assumptions, and the chat states these dependencies @@ -1248,10 +1796,26 @@ function NextStepsAndTryIt() { that powers the rest of PolicyEngine. Try it with a reform, then inspect the stated year, dataset, comparator, and method alongside the answer.

+

+ Because this is a beta, the cases where it fails are more useful to us than the cases where it works: + a request refused that the tools should support, an answer that omits an assumption it should state, + or a figure that does not match what the engine returns for the same reform. PolicyEngine's + policy team is working through the same exercise, and what they and other early users report will + decide what we change first. +

+ + + ); +} + +function ChatCta() { + return ( + ); } @@ -1281,6 +1845,7 @@ export default function App() {
+ diff --git a/src/uk-chat.css b/src/uk-chat.css index 0c1d47b..9b17585 100644 --- a/src/uk-chat.css +++ b/src/uk-chat.css @@ -1,11 +1,31 @@ .article-wrapper p, .article-wrapper li, .article-wrapper .subtitle, -.article-wrapper .step-title, -.article-wrapper .workflow-command { +.article-wrapper .step-title { font-size: 18px; } +/* A release marker above the title. It sits outside .subtitle deliberately: the + subtitle wording is fixed, so the beta status is flagged around it. */ +/* Sits inline at the head of the subtitle, so the status reads as part of the + standfirst rather than as a separate line above the title. */ +.hero-beta { + background: var(--accent-light); + border: 1px solid var(--accent-mid); + border-radius: 999px; + color: var(--accent); + display: inline-block; + font-family: 'JetBrains Mono', monospace; + font-size: 0.62em; + font-style: normal; + font-weight: 700; + letter-spacing: 0.14em; + margin-right: 12px; + padding: 4px 12px; + text-transform: uppercase; + vertical-align: 0.14em; +} + .site-header, .site-header * { -webkit-font-smoothing: antialiased; @@ -119,10 +139,6 @@ margin: 0; } -.timeline-agent { - overflow-wrap: anywhere; -} - .gateway-outcomes { margin-top: 20px; padding-left: 26px; @@ -272,69 +288,27 @@ padding: 7px 9px; } -.timeline-phases { - grid-template-columns: repeat(3, minmax(0, 1fr)); -} - -/* The legacy connector is a single absolutely-positioned line across the whole - grid, which only ever covers the first row. With six stages on three columns - that leaves rows 2 (stages 4-6) unconnected, so draw the connector per stage - instead: each one reaches from its own number to the next column's. */ -.timeline-phases::before { - display: none; -} - -.timeline-phase::before { - background: var(--accent); - content: ''; - height: 3px; - left: 50%; - position: absolute; - top: 25px; - width: calc(100% + 20px); - z-index: 0; +/* Five calls, one row, following .question-cards. Like that rule this has to + stay above the breakpoints below, or the 2- and 1-column fallbacks lose. */ +.trace-cards { + grid-template-columns: repeat(5, minmax(0, 1fr)); } -/* The last stage in a row runs on to the grid edge and the first stage of the - next row picks the line up from the left edge, so the connector reads as - wrapping onto the following row rather than stopping at stage 3. */ -.timeline-phase:nth-child(3n)::before { - width: 50%; +.trace-cards .iteration-card { + min-height: 152px; } -.timeline-phase:nth-child(3n + 1):not(:first-child)::after { - background: var(--accent); - content: ''; - height: 3px; - left: 0; - position: absolute; - top: 25px; - width: 50%; - z-index: 0; -} - -/* Source order matters: this has to beat the :nth-child(3n) width above so the - final stage terminates instead of trailing off the edge. */ -.timeline-phase:last-child::before { - display: none; -} - -.timeline-phase-num { - position: relative; - z-index: 1; +.trace-cards .iteration-num { + margin: 0 auto 12px; } -.timeline-phase-card { - min-height: 300px; +.trace-cards .iteration-title, +.trace-cards .iteration-subtitle { + display: block; } -.timeline-phase-segment { - color: var(--text-light); - font-family: 'JetBrains Mono', monospace; - font-size: 0.64rem; - letter-spacing: 0.05em; - margin-top: 2px; - text-transform: uppercase; +.trace-cards .iteration-title { + margin-bottom: 6px; } .worked-result { @@ -442,6 +416,31 @@ margin: 0; } +/* Both placements centre the button and carry the beta marker, so the offer + reads the same in the introduction and at the end. */ +.article-cta-wrap { + display: flex; + justify-content: center; + margin: 26px 0 8px; +} + +.article-cta-beta { + background: rgba(255, 255, 255, 0.22); + border-radius: 999px; + font-family: 'JetBrains Mono', monospace; + font-size: 0.68rem; + font-weight: 700; + letter-spacing: 0.1em; + margin-left: 10px; + padding: 3px 8px; + text-transform: uppercase; +} + +.article-cta:hover .article-cta-beta, +.article-cta:focus-visible .article-cta-beta { + background: rgba(13, 115, 119, 0.12); +} + .article-cta { align-items: center; background: var(--accent); @@ -541,15 +540,6 @@ .tool-cards { grid-template-columns: repeat(2, minmax(0, 1fr)); } - - .timeline-phases { - grid-template-columns: repeat(2, minmax(0, 1fr)); - } - - .timeline-phase::before, - .timeline-phase::after { - display: none; - } } @media (max-width: 600px) { @@ -573,10 +563,6 @@ display: block; } - .timeline-phases { - grid-template-columns: 1fr; - } - .worked-result { padding: 24px 18px; } @@ -632,19 +618,6 @@ color: #fff; } -/* The worked-example timeline advertises clickability it does not have. */ -.timeline-phase-header:hover { - transform: none; -} - -.timeline-phase-header:hover .timeline-phase-num { - box-shadow: 0 4px 15px #0d73774d; -} - -.timeline-phase-card:hover { - border-color: var(--border); -} - /* Anthony asked for the tool names in the question panel to be labelled, so they read as an illustration of what a question might call rather than as a fixed list. Own border/spacing instead of borrowing .tool-name-list's. */ @@ -676,3 +649,58 @@ padding-left: 20px; padding-right: 20px; } + +/* The tool-call trace reuses the legacy .example-* vocabulary, which was + written for the sticky panel of the older article. Two things differ here: + the calls are 
 so the line breaks survive, and .example-section's 2px
+   gap is too tight once the blocks sit inside body-width prose. */
+.call-trace pre.example-code {
+  margin: 0;
+}
+
+.call-trace .example-section + .example-section {
+  margin-top: 18px;
+}
+
+.call-trace .example-file {
+  margin-bottom: 0;
+}
+
+/* The tool cards and the stage narrative borrow the same vocabulary, at a
+   smaller size: only one example is on screen at a time, and neither panel may
+   grow taller than what sits above it. */
+.tool-example,
+.stage-example {
+  margin-top: 16px;
+}
+
+/* Call beside return while there is room, so the tool panel stays shorter than
+   the family cards above it; the narrative column is too narrow to split, so
+   its examples stack at any width. */
+.tool-example {
+  display: grid;
+  gap: 16px 20px;
+  grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
+}
+
+.stage-example .example-section + .example-section {
+  margin-top: 12px;
+}
+
+.tool-example pre.example-code,
+.stage-example pre.example-code {
+  font-size: 0.74rem;
+  line-height: 1.6;
+  margin: 0;
+  padding: 12px 14px;
+}
+
+/* The generated JSON keys and elided values run past the panel on a phone;
+   the code block scrolls rather than the page. */
+@media (max-width: 600px) {
+  .call-trace pre.example-code,
+  .tool-example pre.example-code,
+  .stage-example pre.example-code {
+    font-size: 0.72rem;
+  }
+}