Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 18 additions & 16 deletions content/current-article.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,25 +10,25 @@ PolicyEngine therefore built UK Chat, an AI interface to PolicyEngine's determin

A language model generates text by predicting continuations, not by applying tax and benefit rules. Ask one what Universal Credit (UC) a lone parent receives and it may return a number without factoring in UC's income-based taper rate, work allowance, or the rates that apply to the household's circumstances.

Prompting does not make a model apply tax and benefit rules. Telling it to state only “real” figures changes the wording, not the source of the calculation. The answer reads the same whether it is right or wrong, so prose alone cannot verify it. The UK Parliament's Parliamentary Office of Science and Technology has noted that confidently presented AI output can make errors difficult to challenge.
Prompting does not make a language model apply tax and benefit rules. Telling it to state only “real” figures changes the wording, not the source of the calculation. The answer reads the same whether it is right or wrong, so prose alone cannot verify it. The UK Parliament's Parliamentary Office of Science and Technology has noted that confidently presented AI output can make errors difficult to challenge.

This matters when results inform decisions, as the National Audit Office's review of AI in government illustrates. Anyone comparing two reforms needs figures from a fixed model, not an AI's recollection of figures it has seen. The same applies to a household working out how a reform affects its income: the figure has to come from clear rules, with assumptions written down, so that it can be checked.
This matters when results inform decisions, as the National Audit Office's review of AI in government illustrates. Anyone comparing two reforms needs figures from a fixed microsimulation, not an AI's recollection of figures it has seen. The same applies to a household working out how a reform affects its income: the figure has to come from clear rules, with assumptions written down, so that it can be checked.

We have measured this directly. In PolicyBench, an evaluation built on the US tax and benefit system, we scored how accurately AI models compute taxes and benefits from household prompts with no tools and no lookups, against deterministic PolicyEngine outputs. In the launch results the top model matched PolicyEngine exactly on 80.3% of its scored outputs. Computed amounts scored lowest: income tax before credits scored far below eligibility flags, because getting it right means sequencing income concepts, thresholds, exclusions, and credits in the correct order.
We have measured this directly. In PolicyBench, an evaluation built on the US tax and benefit system, we scored how accurately AI models compute taxes and benefits from household prompts with no tools and no lookups, against deterministic PolicyEngine outputs. In the launch results the top language model matched PolicyEngine exactly on 80.3% of its scored outputs. Computed amounts scored lowest: income tax before credits scored far below eligibility flags, because getting it right means sequencing income concepts, thresholds, exclusions, and credits in the correct order.

## The boundary

The central design decision is where the predictive layer stops and the deterministic layer starts.

The predictive parts are interpreting wording, deciding which kind of analysis is being requested, drafting the explanation, and suggesting a follow-up. Those tasks are open-ended and language-heavy.

The deterministic parts are the ones users need to be able to check: validating the request, looking up parameters, constructing household inputs, running simulations, computing weighted outputs, and producing chart data. None of these depends on model memory. The runtime enforces the boundary rather than asking the model to respect it in a prompt.
The deterministic parts are the ones users need to be able to check: validating the request, looking up parameters, constructing household inputs, running simulations, computing weighted outputs, and producing chart data. None of these depends on the language model's memory. The runtime enforces the boundary rather than asking the language model to respect it in a prompt.

When a question is ready for computation, the model receives the calculation tools plus a reference generated from the installed engine: the engine's capabilities and parameter schema. What the model can be asked to compute is therefore tied to the deployed version of PolicyEngine, not to a hand-written prompt that drifts out of date.
When a question is ready for computation, the language model receives the calculation tools plus a machine-readable description of what the deployed engine can compute: its capabilities and parameter schema. What the language model can be asked to compute is therefore tied to the deployed version of PolicyEngine, not to a hand-written prompt that drifts out of date.

The gateway sits on that line. It takes the plan the model proposes and decides whether it may cross into the deterministic side, which is why the system has three parts rather than two: the model, the gateway, and the tools a plan reaches once the gateway admits it.
The gateway sits on that line. It takes the plan the language model proposes and decides whether it may cross into the deterministic side, which is why the system has three parts rather than two: the language model, the gateway, and the tools a plan reaches once the gateway admits it.

## The model proposes a plan
## The language model proposes a plan

UK Chat breaks the user pathway into three segments: the AI model, the gateway, and supporting tools.

Expand All @@ -38,13 +38,13 @@ Next, UK Chat's gateway verifies and sometimes constrains that plan to ensure Po

## Tools make the calculation deterministic

At this point, UK Chat has a gateway-verified plan, and every figure in the answer comes from the tools that plan calls rather than from the model.
At this point, UK Chat has a gateway-verified plan, and every figure in the answer comes from the tools that plan calls rather than from the language model.

The tools UK Chat exposes to the model are limited in scope. They fall into five types, each keeping the model within supported operations while it creates, verifies, or executes a plan.
The tools UK Chat exposes to the language model are limited in scope. They fall into five types, each keeping it within supported operations while it creates, verifies, or executes a plan.

### Discovery (9 tools)

Find exact variables, parameters, entities, reform targets, and supported outputs. Discovery keeps the model from guessing the names of things in PolicyEngine.
Find exact variables, parameters, entities, reform targets, and supported outputs. Discovery keeps the language model from guessing the names of things in PolicyEngine.

`list_entities`, `search_variables`, `get_variable`, `search_parameters`, `get_parameter`, `list_reform_targets`, `list_household_input_variables`, `list_society_output_variables`, `list_supported_outputs`

Expand Down Expand Up @@ -80,7 +80,7 @@ Here's what happens inside a single request. UK Chat's core runtime is a FastAPI

_AI model_

A fast model reads the user's words and proposes a structured description of the request. It separates inputs the user supplied from the output they asked for and suggests which tools may be needed. This is an interpretation, not yet permission to calculate.
A fast language model reads the user's words and proposes a structured description of the request. It separates inputs the user supplied from the output they asked for and suggests which tools may be needed. This is an interpretation, not yet permission to calculate.

### 2. Resolve — Current catalogue evidence

Expand Down Expand Up @@ -112,17 +112,17 @@ A society-wide reform receives a second bounded check. The resolver searches the

_Supporting tools_

A model selected for the request's type and size executes the verified plan through one or more of 21 public tools. Simulation tools apply the tax and benefit rules; analysis tools derive budget, programme, distributional, poverty, inequality, or gain-and-loss results from a stored simulation.
A language model selected for the request's type and size executes the verified plan through one or more of 21 public tools. Simulation tools apply the tax and benefit rules; analysis tools derive budget, programme, distributional, poverty, inequality, or gain-and-loss results from a stored simulation.

The model chooses and sequences tools, but it does not supply the figures. Each number comes back as tool data produced by the PolicyEngine UK model.
The language model chooses and sequences tools, but it does not supply the figures. Each number comes back as tool data produced by the PolicyEngine UK model.

### 6. Stream — An inspectable response

_Gateway_

The gateway streams text chunks, tool starts, tool inputs, completion summaries, and the final answer as server-sent events. The interface can therefore show what the system tried and which computations completed while the answer is being produced.

The tool loop remains bounded by a 30-round cap, repeated-call detection, result-size limits, and explicit terminal events. Tool errors return as data so the model can recover without inventing a result.
The tool loop remains bounded by a 30-round cap, repeated-call detection, result-size limits, and explicit terminal events. Tool errors return as data so the language model can recover without inventing a result.

## One reform, end to end

Expand Down Expand Up @@ -161,7 +161,7 @@ The Child Benefit reform above is one of four kinds of questions supported today

> A single parent, two children, earning £28,000 — what do they take home?

The household is constructed as a synthetic input, validated, and run through the model to produce taxes, benefits, and net income under stated assumptions. One household containing one benefit unit per call; more complex arrangements need separate calls.
The household is constructed as a synthetic input, validated, and run through the PolicyEngine UK model to produce taxes, benefits, and net income under stated assumptions. One household containing one benefit unit per call; more complex arrangements need separate calls.

`list_household_input_variables`, `validate_household`, `run_household_simulation`

Expand Down Expand Up @@ -189,7 +189,7 @@ Follow-ups stay in the same thread. The conversation carries the context; the si

## Limitations

The chat is a modelling tool, not advice. It reports what the model calculates under stated assumptions, and it is not a substitute for professional guidance on an individual's circumstances.
The chat is a modelling tool, not advice. It reports what the engine calculates under stated assumptions, and it is not a substitute for professional guidance on an individual's circumstances.

Society results are direct static microsimulation estimates. They do not estimate behavioural responses, employment effects, inflation, GDP, market reactions, or general-equilibrium effects. When a request mixes a supported policy result with one of those effects, the gateway notifies the user that it cannot calculate these effects, then proceeds with what it can.

Expand All @@ -202,3 +202,5 @@ We want to widen the range of reforms covered by typed tools, improve the speed
## Try it yourself

UK Chat's answers can be cited and reproduced because the figures come from the same open engine that powers the rest of PolicyEngine. Try it with a reform, then inspect the stated year, dataset, comparator, and method alongside the answer.

[Try PolicyEngine UK Chat](https://policyengine.org/uk/chat)
Loading
Loading