diff --git a/content/current-article.md b/content/current-article.md index 257bee5..1b1d69e 100644 --- a/content/current-article.md +++ b/content/current-article.md @@ -8,7 +8,7 @@ People are increasingly turning to AI language models to answer complex question PolicyEngine therefore built UK Chat, an AI interface to PolicyEngine's deterministic rules and simulation engine. A language model interprets an open-ended request and proposes a structured plan. The gateway checks the proposed inputs and output against the current PolicyEngine catalogue; only then do deterministic tools compute quantitative policy results from actual tax and benefit rules. -UK Chat is released as a **beta**. The pipeline described here is in place and running, but the range of reforms it covers, the wording of its answers, and its judgement about what it cannot compute are all still being tested. We are publishing it at this stage to gather feedback on where it is useful and where it falls short, beginning with PolicyEngine's own policy team. +UK Chat is released as a **beta**. The pipeline described here is in place and running, but the range of reforms it covers, the wording of its answers, and its judgement about what it cannot compute are all still being tested. We are publishing it at this stage to gather feedback on where it is useful and where it falls short. [Try PolicyEngine UK Chat (beta)](https://policyengine.org/uk/chat) @@ -40,14 +40,6 @@ When a question is ready for computation, the language model receives the calcul The gateway sits on that line. It takes the plan the language model proposes and decides whether it may cross into the deterministic side. That is why the system has three parts rather than two: the language model, the gateway, and the tools a plan reaches once the gateway admits it. -## The language model proposes a plan - -UK Chat breaks the user pathway into three segments: the language model, the gateway, and supporting tools. - -First, the language model does the predictive work. It takes the user's open-ended prompt and develops a simulation or analysis plan. The plan has a required format and must use one or more of the deterministic tools connected to the PolicyEngine UK tax and benefit simulation engine. - -Next, UK Chat's gateway verifies and sometimes constrains that plan to ensure PolicyEngine provides the tools required to answer the question. The gateway may also ask the user for clarification before finalising a plan. - ## Tools make the calculation deterministic At this point, UK Chat has a gateway-verified plan, and every figure in the answer comes from the tools that plan calls rather than from the language model. @@ -373,7 +365,9 @@ generate_chart( ## Under the hood -Here's what happens inside a single request. UK Chat's core runtime is a FastAPI server-sent-events handler that calls the Anthropic SDK directly. There is no generic agent framework in the request path. The chat owns both the opening gateway and the tool loop, so the rules that matter can be enforced at the point where each decision is made. Each stage below is tagged with the segment that owns it. +Take one request: _"For 2026, set the Child Benefit eldest-child rate to £30 a week and show the annual budgetary impact."_ It names a policy, a value, a year, and an output. It does not name a PolicyEngine parameter path, a reform object, or the sequence of tools needed to answer it. This section and the next follow that request through the system. + +UK Chat's core runtime is a FastAPI server-sent-events handler that calls the Anthropic SDK directly. There is no generic agent framework in the request path. The chat owns both the opening gateway and the tool loop, so the rules that matter can be enforced at the point where each decision is made. Each stage below is tagged with the segment that owns it. ### 1. Ground — A structured opening plan @@ -384,8 +378,7 @@ A fast language model reads the user's words and proposes a structured descripti _The user's message_ ``` -last_user_message="For 2026, set the Child Benefit eldest-child rate - to £30 a week and show the annual budgetary impact" +last_user_message="…the request above, with no conversation history…" ``` _Proposed plan_ @@ -528,9 +521,7 @@ data: { "type": "done", "content": …, "stop_reason": …, "usage": { … } } ## The same reform, call by call -Consider: “For 2026, set the Child Benefit eldest-child rate to £30 a week and show the annual budgetary impact.” The request contains a policy, a final value, a year, and an output. It does not contain a PolicyEngine parameter path, a reform object, or the sequence of tools needed to answer it. - -The six stages above decide what the system does; underneath, the turn is a short sequence of typed calls, each one taking an argument that an earlier call produced. This is that sequence for the request. +The six stages decide what the system does. Underneath, the approved plan is a short sequence of typed calls, each taking an argument an earlier call produced. The loop re-derives what the gateway already approved, and the match between the two is itself the check. ### Tool-call trace — 5 calls, one turn @@ -626,10 +617,6 @@ Computed inside the engine: - ✓ survey weights applied inside the PolicyEngine UK model - Stored: `result_id` → typed handle, turn-local. -Two features of the sequence carry most of the design. The first is that `run_society_simulation` hands back a handle rather than data: the language model receives a `result_id`, a dataset reference, and a year, and never receives household rows, survey weights, or any serialisable simulation object. Weighting happens inside the PolicyEngine UK model, and the seven analysis tools are the only route from a stored simulation to a number. - -The second is provenance. Every parameter path, value, and handle in the sequence was either supplied by the user or produced by a preceding call, so each argument can be traced to its source. Nothing in it was recalled. - ### Computed result: About £0.9 billion a year in additional government cost A production run compares the reform with 2026 current law using PolicyEngine's Enhanced Family Resources Survey dataset for 2024–25, release 1.56.13. It is a direct static microsimulation estimate. Tax revenue rises because a higher Child Benefit rate increases the High Income Child Benefit Charge paid by higher-income families. @@ -642,8 +629,6 @@ A production run compares the reform with 2026 current law using PolicyEngine's A negative net budgetary impact means a cost to government. Figures are rounded to the nearest £0.1 billion. -If the catalogue returned more than one materially plausible interpretation, or the reform resolver could not determine what year or reform the user was requesting, the calculation would pause before the tool loop and ask the user to confirm the intended construction. - ## What you can ask The Child Benefit reform above is one of four kinds of questions supported today, and users can stack these questions within one conversation without starting a new chat. @@ -680,7 +665,7 @@ Follow-ups stay in the same thread. The conversation carries the context; the si ## Limitations -Some of what follows is a deliberate boundary that will not change. The rest is the current state of a beta: reform coverage is incomplete, answers vary in how fully they state their assumptions, and gateway behaviour will move as we act on what users report. +Some of what follows is a deliberate boundary that will not change. The rest is the current state of a beta: reform coverage is incomplete, answers vary in how fully they state their assumptions, and gateway behaviour will move as we act on what users report. Widening the range of reforms the typed tools cover, and the evidence the gateway can resolve before calculation, is where the work goes next. The chat is a modelling tool, not advice. It reports what the engine calculates under stated assumptions, and it is not a substitute for professional guidance on an individual's circumstances. @@ -688,13 +673,9 @@ Society results are direct static microsimulation estimates. They do not estimat Results depend on the dataset, year, and modelling assumptions, and the chat states these dependencies rather than presenting figures as universal. They should still be checked against independent estimates from organisations such as the Institute for Fiscal Studies, Resolution Foundation, or Office for Budget Responsibility. -## What's next - -We want to widen the range of reforms covered by typed tools, improve the speed and inspectability of more specialised analyses, and keep expanding the evidence the gateway can resolve before calculation. UK Chat also works alongside PolicyEngine's wider generative AI integrations and plugin ecosystem for researchers building their own analyses. - ## Try it yourself -UK Chat's answers can be cited and reproduced because the figures come from the same open engine that powers the rest of PolicyEngine. Try it with a reform, then inspect the stated year, dataset, comparator, and method alongside the answer. +UK Chat's answers can be cited and reproduced because the figures come from the same open engine that powers the rest of PolicyEngine, and it sits alongside PolicyEngine's wider generative AI integrations and plugin ecosystem for researchers building their own analyses. Try it with a reform, then inspect the stated year, dataset, comparator, and method alongside the answer. Because this is a beta, the cases where it fails are more useful to us than the cases where it works: a request refused that the tools should support, an answer that omits an assumption it should state, or a figure that does not match what the engine returns for the same reform. PolicyEngine's policy team is working through the same exercise, and what they and other early users report will decide what we change first. diff --git a/src/App.jsx b/src/App.jsx index e285bad..9c4cada 100644 --- a/src/App.jsx +++ b/src/App.jsx @@ -3,6 +3,7 @@ import { motion, useReducedMotion } from 'framer-motion'; import SiteHeader from './SiteHeader'; import anthonyVolkHeadshot from './assets/authors/anthony-volk.webp'; import vahidAhmadiHeadshot from './assets/authors/vahid-ahmadi.webp'; +import ukChatScreenshot from './assets/uk-chat-screenshot.jpg'; const toolFamilies = [ { @@ -622,9 +623,8 @@ const requestSteps = [ ), example: { - callTitle: 'the user’s message', - call: `last_user_message="For 2026, set the Child Benefit eldest-child rate - to £30 a week and show the annual budgetary impact"`, + callTitle: 'the user’s message, as the gateway receives it', + call: `last_user_message="…the request above, with no conversation history…"`, returnsTitle: 'proposed plan', returns: `{ "domain_status": "uk_or_unspecified", "capability_status": "supported", "tool": "compute_budgetary_impact", @@ -905,8 +905,20 @@ function Introduction() { UK Chat is released as a beta. The pipeline described here is in place and running, but the range of reforms it covers, the wording of its answers, and its judgement about what it cannot compute are all still being tested. We are publishing it at this stage to gather feedback on where it - is useful and where it falls short, beginning with PolicyEngine's own policy team. + is useful and where it falls short.

+ + PolicyEngine UK Chat, showing the question box and suggested questions + ); @@ -1299,22 +1311,6 @@ function ToolExplorer() { return ( -

The language model proposes a plan

-

- UK Chat breaks the user pathway into three segments: the language model, the gateway, and supporting - tools. -

-

- First, the language model does the predictive work. It takes the user's open-ended prompt and - develops a simulation or analysis plan. The plan has a required format and must use one or more of the - deterministic tools connected to the PolicyEngine UK tax and benefit simulation engine. -

-

- Next, UK Chat's gateway verifies and sometimes constrains that plan to ensure PolicyEngine - provides the tools required to answer the question. The gateway may also ask the user for - clarification before finalising a plan. -

-

Tools make the calculation deterministic

At this point, UK Chat has a gateway-verified plan, and every figure in the answer comes from the @@ -1538,11 +1534,20 @@ function ArchitectureScrolly() {

Under the hood

- Here's what happens inside a single request. UK Chat's core runtime is a FastAPI - server-sent-events handler that calls the Anthropic SDK directly. There is no generic agent framework - in the request path. The chat owns both the opening gateway and the tool loop, so the rules that - matter can be enforced at the point where each decision is made. Each stage below is tagged with the - segment that owns it. + Take one request:{' '} + + “For 2026, set the Child Benefit eldest-child rate to £30 a week and show the annual budgetary + impact.” + {' '} + It names a policy, a value, a year, and an output. It does not name a PolicyEngine parameter path, a + reform object, or the sequence of tools needed to answer it. This section and the next follow that + request through the system. +

+

+ UK Chat's core runtime is a FastAPI server-sent-events handler that calls the Anthropic SDK + directly. There is no generic agent framework in the request path. The chat owns both the opening + gateway and the tool loop, so the rules that matter can be enforced at the point where each decision + is made. Each stage below is tagged with the segment that owns it.

@@ -1626,28 +1631,11 @@ function WorkedExample() {

The same reform, call by call

- Consider: “For 2026, set the Child Benefit eldest-child rate to £30 a week and show the annual - budgetary impact.” The request contains a policy, a final value, a year, and an output. It does not - contain a PolicyEngine parameter path, a reform object, or the sequence of tools needed to answer it. -

-

- The six stages above decide what the system does; underneath, the turn is a short sequence of typed - calls, each one taking an argument that an earlier call produced. This is that sequence for the - request. + The six stages decide what the system does. Underneath, the approved plan is a short sequence of typed + calls, each taking an argument an earlier call produced. The loop re-derives what the gateway already + approved, and the match between the two is itself the check.

-

- Two features of the sequence carry most of the design. The first is that{' '} - run_society_simulation hands back a handle rather than data: the language model receives - a result_id, a dataset reference, and a year, and never receives household rows, survey - weights, or any serialisable simulation object. Weighting happens inside the PolicyEngine UK model, - and the seven analysis tools are the only route from a stored simulation to a number. -

-

- The second is provenance. Every parameter path, value, and handle in the sequence was either supplied - by the user or produced by a preceding call, so each argument can be traced to its source. Nothing in - it was recalled. -

@@ -1693,9 +1681,9 @@ function WorkedExample() {

- If the catalogue returned more than one materially plausible interpretation, or the reform resolver - could not determine what year or reform the user was requesting, the calculation would pause before - the tool loop and ask the user to confirm the intended construction. + The seven analysis tools are the only route from a stored simulation to that number, and every + parameter path, value, and handle in the sequence was either supplied by the user or produced by a + preceding call. Nothing in it was recalled.

); @@ -1757,7 +1745,8 @@ function Limitations() {

Some of what follows is a deliberate boundary that will not change. The rest is the current state of a beta: reform coverage is incomplete, answers vary in how fully they state their assumptions, and - gateway behaviour will move as we act on what users report. + gateway behaviour will move as we act on what users report. Widening the range of reforms the typed + tools cover, and the evidence the gateway can resolve before calculation, is where the work goes next.

The chat is a modelling tool, not advice. It reports what the engine calculates under @@ -1783,18 +1772,12 @@ function Limitations() { function NextStepsAndTryIt() { return ( -

What's next

-

- We want to widen the range of reforms covered by typed tools, improve the speed and inspectability of - more specialised analyses, and keep expanding the evidence the gateway can resolve before calculation. - UK Chat also works alongside PolicyEngine's wider generative AI integrations and plugin ecosystem - for researchers building their own analyses. -

Try it yourself

UK Chat's answers can be cited and reproduced because the figures come from the same open engine - that powers the rest of PolicyEngine. Try it with a reform, then inspect the stated year, dataset, - comparator, and method alongside the answer. + that powers the rest of PolicyEngine, and it sits alongside PolicyEngine's wider generative AI + integrations and plugin ecosystem for researchers building their own analyses. Try it with a reform, + then inspect the stated year, dataset, comparator, and method alongside the answer.

Because this is a beta, the cases where it fails are more useful to us than the cases where it works: diff --git a/src/assets/uk-chat-screenshot.jpg b/src/assets/uk-chat-screenshot.jpg new file mode 100644 index 0000000..5edc5a0 Binary files /dev/null and b/src/assets/uk-chat-screenshot.jpg differ diff --git a/src/legacy-encode-policy.css b/src/legacy-encode-policy.css index e49c566..021fd9e 100644 --- a/src/legacy-encode-policy.css +++ b/src/legacy-encode-policy.css @@ -274,7 +274,7 @@ a:hover { align-self: flex-start; } .narrative-step { - min-height: 90vh; + min-height: 58vh; padding: 40px 20px 40px 24px; opacity: 1; border-left: 3px solid transparent; diff --git a/src/uk-chat.css b/src/uk-chat.css index 9b17585..eda315c 100644 --- a/src/uk-chat.css +++ b/src/uk-chat.css @@ -704,3 +704,26 @@ font-size: 0.72rem; } } + +/* The product shot sits between the beta note and the call to action, and is + itself a link into the chat. */ +.product-shot { + border-bottom: 0; + display: block; + margin: 24px auto 4px; + max-width: 620px; +} + +.product-shot img { + border: 1px solid var(--border); + border-radius: 10px; + box-shadow: 0 12px 30px rgba(16, 24, 40, 0.1); + display: block; + height: auto; + width: 100%; +} + +.product-shot:hover img, +.product-shot:focus-visible img { + border-color: var(--accent); +}