From 3527e14e9fcfed23b62f947ecb3d11a01bb77c58 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Fri, 14 Aug 2026 16:49:33 +0300 Subject: [PATCH 1/8] Add interactive UK Chat answer walkthrough --- content/current-article.md | 100 +-- src/App.jsx | 794 ++++++++++++++------ src/uk-chat.css | 1456 +++++++++++++++++++++++++++++++++++- 3 files changed, 2032 insertions(+), 318 deletions(-) diff --git a/content/current-article.md b/content/current-article.md index 257bee5..782c115 100644 --- a/content/current-article.md +++ b/content/current-article.md @@ -8,7 +8,7 @@ People are increasingly turning to AI language models to answer complex question PolicyEngine therefore built UK Chat, an AI interface to PolicyEngine's deterministic rules and simulation engine. A language model interprets an open-ended request and proposes a structured plan. The gateway checks the proposed inputs and output against the current PolicyEngine catalogue; only then do deterministic tools compute quantitative policy results from actual tax and benefit rules. -UK Chat is released as a **beta**. The pipeline described here is in place and running, but the range of reforms it covers, the wording of its answers, and its judgement about what it cannot compute are all still being tested. We are publishing it at this stage to gather feedback on where it is useful and where it falls short, beginning with PolicyEngine's own policy team. +UK Chat is released as a **beta**. The pipeline described here is in place and running, but the range of reforms it covers, the wording of its answers, and its judgement about what it cannot compute are all still being tested. We are publishing it at this stage to gather feedback on where it is useful and where it falls short. [Try PolicyEngine UK Chat (beta)](https://policyengine.org/uk/chat) @@ -371,30 +371,23 @@ generate_chart( } ``` -## Under the hood +## See one answer being built -Here's what happens inside a single request. UK Chat's core runtime is a FastAPI server-sent-events handler that calls the Anthropic SDK directly. There is no generic agent framework in the request path. The chat owns both the opening gateway and the tool loop, so the rules that matter can be enforced at the point where each decision is made. Each stage below is tagged with the segment that owns it. +UK Chat owns both the opening gateway and the tool loop in a FastAPI server-sent-events handler that calls the Anthropic SDK directly. There is no generic agent framework in the request path. Follow one request through that boundary, from the language model's proposed plan to a tool-backed answer. -### 1. Ground — A structured opening plan +This guided reconstruction uses the real stages, tools, assumptions, and result from the Child Benefit example; it is not a live chat session. -_Language model_ +### User request -A fast language model reads the user's words and proposes a structured description of the request. It separates inputs the user supplied from the output they asked for and suggests which tools may be needed. This is an interpretation, not yet permission to calculate. +> For 2026, set the Child Benefit eldest-child rate to £30 a week and show the annual budgetary impact. -_The user's message_ +### 1. Ground — A structured opening plan -``` -last_user_message="For 2026, set the Child Benefit eldest-child rate - to £30 a week and show the annual budgetary impact" -``` +_Language model_ -_Proposed plan_ +A fast language model reads the user's words and proposes a structured description of the request. It separates inputs the user supplied from the output they asked for and suggests which tools may be needed. This is an interpretation, not yet permission to calculate. -``` -{ "domain_status": "uk_or_unspecified", "capability_status": "supported", - "tool": "compute_budgetary_impact", - "catalogue_queries": [{ "kind": "reform_target", "query": "eldest-child rate" }] } -``` +For this request, the proposed plan identifies the Child Benefit eldest-child rate, a final value of £30 a week, the 2026 policy year, and annual budgetary impact as the requested output. ### 2. Resolve — Current catalogue evidence @@ -404,22 +397,7 @@ The gateway checks named policies, variables, and requested outputs against Poli It then refines the plan with the relevant tool inputs and declared defaults — for example, using the current year when the user does not name one. Any choice the system cannot resolve safely stays explicit instead of being silently invented. -_Catalogue query_ - -``` -resolve_catalogue_queries([ - { "kind": "reform_target", "query": "eldest-child rate" } -]) -``` - -_Match_ - -``` -{ - "identifier": "gov.hmrc.child_benefit.amount.eldest", - "match_type": "strong_phrase", "score": 0.9, "authoritative": true -} -``` +Here, the prompt wording “Child Benefit eldest-child rate” matches `gov.hmrc.child_benefit.amount.eldest`, budgetary impact is supported, and no user-owned choices remain unresolved. ### 3. Gate — Five deterministic outcomes @@ -435,23 +413,7 @@ The gate is deliberately biased towards computing. If the classifier errors or r - Out of scope: the requested policy output cannot be modelled, so calculation tools stay closed. - Irrelevant: the request is unrelated to UK tax and benefit policy and takes a lightweight response path. -_Gated plan_ - -``` -gate( - in_domain=true, tool="compute_budgetary_impact", - unmodellable_outputs=[] -) -``` - -_Verdict_ - -``` -{ - "outcome": "ready", - "gating_reasons": [] -} -``` +The complete Child Benefit request is ready: its policy, value, year, and output are explicit and supported. ### 4. Verify — Exact reform construction @@ -459,25 +421,7 @@ _Gateway_ A society-wide reform receives a second bounded check. The resolver searches the rules engine's current reform targets, binds the user's wording to an exact parameter path and date, and constructs reform JSON. The validator must accept that construction before a simulation can run. -What the gateway approves is then the only reform that can run. The simulation tool compares the reform it is handed against the approved construction and refuses anything that does not match, so a different reform cannot be substituted after the check has passed. - -_Resolver construction_ - -``` -emit_reform_assessment( - reform={ "gov.hmrc.child_benefit.amount.eldest": 30.0 }, - confidence=95 -) -``` - -_Approved for this turn_ - -``` -{ - "approved_reform": { "gov.hmrc.child_benefit.amount.eldest": 30.0 }, - "require_approved_reform": true -} -``` +For this request, the approved construction sets `gov.hmrc.child_benefit.amount.eldest` to £30 a week for 2026. Rules-engine validation passes, and the tool context will reject a society reform that differs from the approved construction. ### 5. Calculate — A bounded 21-tool runtime @@ -528,9 +472,7 @@ data: { "type": "done", "content": …, "stop_reason": …, "usage": { … } } ## The same reform, call by call -Consider: “For 2026, set the Child Benefit eldest-child rate to £30 a week and show the annual budgetary impact.” The request contains a policy, a final value, a year, and an output. It does not contain a PolicyEngine parameter path, a reform object, or the sequence of tools needed to answer it. - -The six stages above decide what the system does; underneath, the turn is a short sequence of typed calls, each one taking an argument that an earlier call produced. This is that sequence for the request. +The six stages decide what the system does. Underneath, the approved plan is a short sequence of typed calls, each taking an argument an earlier call produced. The loop re-derives what the gateway already approved, and the match between the two is itself the check. ### Tool-call trace — 5 calls, one turn @@ -626,10 +568,6 @@ Computed inside the engine: - ✓ survey weights applied inside the PolicyEngine UK model - Stored: `result_id` → typed handle, turn-local. -Two features of the sequence carry most of the design. The first is that `run_society_simulation` hands back a handle rather than data: the language model receives a `result_id`, a dataset reference, and a year, and never receives household rows, survey weights, or any serialisable simulation object. Weighting happens inside the PolicyEngine UK model, and the seven analysis tools are the only route from a stored simulation to a number. - -The second is provenance. Every parameter path, value, and handle in the sequence was either supplied by the user or produced by a preceding call, so each argument can be traced to its source. Nothing in it was recalled. - ### Computed result: About £0.9 billion a year in additional government cost A production run compares the reform with 2026 current law using PolicyEngine's Enhanced Family Resources Survey dataset for 2024–25, release 1.56.13. It is a direct static microsimulation estimate. Tax revenue rises because a higher Child Benefit rate increases the High Income Child Benefit Charge paid by higher-income families. @@ -642,7 +580,7 @@ A production run compares the reform with 2026 current law using PolicyEngine's A negative net budgetary impact means a cost to government. Figures are rounded to the nearest £0.1 billion. -If the catalogue returned more than one materially plausible interpretation, or the reform resolver could not determine what year or reform the user was requesting, the calculation would pause before the tool loop and ask the user to confirm the intended construction. +The seven analysis tools are the only route from a stored simulation to that number, and every parameter path, value, and handle in the sequence was either supplied by the user or produced by a preceding call. Nothing in it was recalled. ## What you can ask @@ -680,7 +618,7 @@ Follow-ups stay in the same thread. The conversation carries the context; the si ## Limitations -Some of what follows is a deliberate boundary that will not change. The rest is the current state of a beta: reform coverage is incomplete, answers vary in how fully they state their assumptions, and gateway behaviour will move as we act on what users report. +Some of what follows is a deliberate boundary that will not change. The rest is the current state of a beta: reform coverage is incomplete, answers vary in how fully they state their assumptions, and gateway behaviour will move as we act on what users report. Widening the range of reforms the typed tools cover, and the evidence the gateway can resolve before calculation, is where the work goes next. The chat is a modelling tool, not advice. It reports what the engine calculates under stated assumptions, and it is not a substitute for professional guidance on an individual's circumstances. @@ -688,13 +626,9 @@ Society results are direct static microsimulation estimates. They do not estimat Results depend on the dataset, year, and modelling assumptions, and the chat states these dependencies rather than presenting figures as universal. They should still be checked against independent estimates from organisations such as the Institute for Fiscal Studies, Resolution Foundation, or Office for Budget Responsibility. -## What's next - -We want to widen the range of reforms covered by typed tools, improve the speed and inspectability of more specialised analyses, and keep expanding the evidence the gateway can resolve before calculation. UK Chat also works alongside PolicyEngine's wider generative AI integrations and plugin ecosystem for researchers building their own analyses. - ## Try it yourself -UK Chat's answers can be cited and reproduced because the figures come from the same open engine that powers the rest of PolicyEngine. Try it with a reform, then inspect the stated year, dataset, comparator, and method alongside the answer. +UK Chat's answers can be cited and reproduced because the figures come from the same open engine that powers the rest of PolicyEngine, and it sits alongside PolicyEngine's wider generative AI integrations and plugin ecosystem for researchers building their own analyses. Try it with a reform, then inspect the stated year, dataset, comparator, and method alongside the answer. Because this is a beta, the cases where it fails are more useful to us than the cases where it works: a request refused that the tools should support, an answer that omits an assumption it should state, or a figure that does not match what the engine returns for the same reform. PolicyEngine's policy team is working through the same exercise, and what they and other early users report will decide what we change first. diff --git a/src/App.jsx b/src/App.jsx index e285bad..c854f6f 100644 --- a/src/App.jsx +++ b/src/App.jsx @@ -1,5 +1,12 @@ import { useEffect, useRef, useState } from 'react'; import { motion, useReducedMotion } from 'framer-motion'; +import { + IconArrowLeft, + IconArrowRight, + IconArrowUp, + IconChevronDown, + IconPaperclip, +} from '@tabler/icons-react'; import SiteHeader from './SiteHeader'; import anthonyVolkHeadshot from './assets/authors/anthony-volk.webp'; import vahidAhmadiHeadshot from './assets/authors/vahid-ahmadi.webp'; @@ -606,6 +613,92 @@ const authors = [ }, ]; +const walkthroughUserRequest = + 'For 2026, set the Child Benefit eldest-child rate to £30 a week and show the annual budgetary impact.'; + +const groundProposedPlan = { + domain_status: 'uk_or_unspecified', + capability_status: 'supported', + tool: 'compute_budgetary_impact', + catalogue_queries: [ + { + kind: 'reform_target', + query: 'eldest-child rate', + }, + ], +}; + +const resolveCatalogueCall = `resolve_catalogue_queries([ + { "kind": "reform_target", "query": "eldest-child rate" } +])`; + +const resolveCatalogueMatch = { + identifier: 'gov.hmrc.child_benefit.amount.eldest', + match_type: 'strong_phrase', + score: 0.9, + authoritative: true, +}; + +const gateCall = `gate( + in_domain=true, tool="compute_budgetary_impact", + unmodellable_outputs=[] +)`; + +const gateVerdict = { + outcome: 'ready', + gating_reasons: [], +}; + +const verifyReformCall = `emit_reform_assessment( + reform={ "gov.hmrc.child_benefit.amount.eldest": 30.0 }, + confidence=95 +)`; + +const approvedReform = { + approved_reform: { + 'gov.hmrc.child_benefit.amount.eldest': 30.0, + }, + require_approved_reform: true, +}; + +const societySimulationCall = `run_society_simulation( + year=2026, + reform={ + "gov.hmrc.child_benefit.amount.eldest": 30.0 + } +)`; + +const budgetaryImpactCall = `compute_budgetary_impact( + simulation_id= +)`; + +const calculationToolEvents = [ + { + id: 'society-simulation', + name: 'run_society_simulation', + }, + { + id: 'budgetary-impact', + name: 'compute_budgetary_impact', + }, +]; + +const walkthroughAnswerSummary = + 'Setting the Child Benefit eldest-child rate to £30 a week in 2026 would cost the government about £0.9 billion a year compared with current law. Benefit spending rises by about £1.1 billion, while tax revenue rises by about £0.2 billion because the higher rate also increases High Income Child Benefit Charge receipts.'; + +const walkthroughAnswerMethod = + 'This is a direct static microsimulation estimate using Enhanced FRS 2024–25, release 1.56.13.'; + +const streamEventSequence = `tool_start society simulation +tool_use year + approved reform +tool_result success + simulation result_id +tool_start budgetary impact +tool_use simulation_id +tool_result success + computed totals +thinking_done +chunk assistant answer +done turn complete`; + const requestSteps = [ { number: 1, @@ -905,7 +998,7 @@ function Introduction() { UK Chat is released as a beta. The pipeline described here is in place and running, but the range of reforms it covers, the wording of its answers, and its judgement about what it cannot compute are all still being tested. We are publishing it at this stage to gather feedback on where it - is useful and where it falls short, beginning with PolicyEngine's own policy team. + is useful and where it falls short.

@@ -1378,219 +1471,484 @@ function ToolExplorer() { ); } -function RequestFlowDiagram({ activeStep }) { +function AnswerWalkthrough() { return ( - - - - - - - {requestSteps.map((step, index) => { - const y = 18 + index * 92; - const active = activeStep === index; - return ( - - {index > 0 && ( - - )} - - - - {index + 1} - - - {step.title} - - - {step.segment} - - - ); - })} - - - - - non-ready paths - - - clarify · explain - - - +

See one answer being built

+

+ UK Chat owns both the opening gateway and the tool loop in a FastAPI server-sent-events handler that + calls the Anthropic SDK directly. There is no generic agent framework in the request path. Follow one + request through that boundary, from the language model's proposed plan to a tool-backed answer. +

+

+ This guided reconstruction uses the real stages, tools, assumptions, and result from the Child Benefit + example; it is not a live chat session. Use the arrow buttons in the bottom-right of the stage rail to + move through the interactive. +

+ + + ); +} + +function CalculationWorkingDisclosure() { + const [isExpanded, setIsExpanded] = useState(false); + + return ( +
+ + + {isExpanded && ( +
+

+ I'll run the approved reform and calculate its annual budgetary impact. +

+ {calculationToolEvents.map((tool) => ( +
+ {tool.name} + ✓ +
+ ))} +
+ )} +
); } -function ArchitectureScrolly() { - const [activeStep, setActiveStep] = useState(0); - const stepRefs = useRef([]); - const stepRatios = useRef(new Map()); +function CompressedUkChatPreview() { + const prefersReducedMotion = useReducedMotion(); + const [activeStage, setActiveStage] = useState(0); + const [animationStarted, setAnimationStarted] = useState(false); + const [animationPhase, setAnimationPhase] = useState('idle'); + const [typedRequest, setTypedRequest] = useState(''); + const previewRef = useRef(null); useEffect(() => { - const ratios = stepRatios.current; + if (animationStarted) return undefined; + + const preview = previewRef.current; + if (!preview || typeof IntersectionObserver === 'undefined') { + setAnimationStarted(true); + return undefined; + } + const observer = new IntersectionObserver( - (entries) => { - entries.forEach((entry) => { - const index = Number(entry.target.dataset.step); - ratios.set(index, entry.isIntersecting ? entry.intersectionRatio : 0); - }); - - const mostVisible = [...ratios.entries()] - .filter(([, ratio]) => ratio > 0) - .sort(([, firstRatio], [, secondRatio]) => secondRatio - firstRatio)[0]; - - if (mostVisible) { - setActiveStep(mostVisible[0]); - } + ([entry]) => { + if (!entry.isIntersecting) return; + setAnimationStarted(true); + observer.disconnect(); }, - { rootMargin: '-74px 0px -45% 0px', threshold: [0, 0.15, 0.4, 0.7] }, + { threshold: 0.3 }, ); - stepRefs.current.forEach((node) => node && observer.observe(node)); + observer.observe(preview); + return () => observer.disconnect(); + }, [animationStarted]); + + useEffect(() => { + if (!animationStarted) return undefined; + + if (prefersReducedMotion) { + const reducedMotionTimer = window.setTimeout(() => { + setTypedRequest(walkthroughUserRequest); + setAnimationPhase('complete'); + }, 0); + return () => window.clearTimeout(reducedMotionTimer); + } + + let startTimer; + let submitTimer; + let messageTimer; + let loadingTimer; + let completeTimer; + let characterIndex = 0; + + startTimer = window.setTimeout(() => setAnimationPhase('typing'), 0); + const typingInterval = window.setInterval(() => { + characterIndex += 1; + setTypedRequest(walkthroughUserRequest.slice(0, characterIndex)); + + if (characterIndex < walkthroughUserRequest.length) return; + + window.clearInterval(typingInterval); + submitTimer = window.setTimeout(() => { + setAnimationPhase('submitting'); + messageTimer = window.setTimeout(() => { + setAnimationPhase('message'); + loadingTimer = window.setTimeout(() => { + setAnimationPhase('loading'); + completeTimer = window.setTimeout(() => setAnimationPhase('complete'), 1650); + }, 650); + }, 700); + }, 300); + }, 22); + return () => { - observer.disconnect(); - ratios.clear(); + window.clearTimeout(startTimer); + window.clearInterval(typingInterval); + window.clearTimeout(submitTimer); + window.clearTimeout(messageTimer); + window.clearTimeout(loadingTimer); + window.clearTimeout(completeTimer); }; - }, []); + }, [animationStarted, prefersReducedMotion]); return ( - -

Under the hood

-

- Here's what happens inside a single request. UK Chat's core runtime is a FastAPI - server-sent-events handler that calls the Anthropic SDK directly. There is no generic agent framework - in the request path. The chat owns both the opening gateway and the tool loop, so the rules that - matter can be enforced at the point where each decision is made. Each stage below is tagged with the - segment that owns it. -

-
-
- {requestSteps.map((step, index) => ( -
{ - stepRefs.current[index] = node; - }} - > -
- {step.number} - -

- {step.title} -

- {step.subtitle} - - {step.segment} -
-
- {step.body} - +
+
+
+
+

What's on your mind today?

+ +
+
+
+ {typedRequest} +
+
+
+ +
+ +
-
- ))} -
- -
- + + {activeStage === 0 && animationPhase === 'complete' && ( +
+
+
+ Step 1 · AI model + Ground +
+

A structured opening plan

+
{requestSteps[0].body}
+
+ +
+
+
+ The user's message, as the gateway receives it +
+ 
+                    {`last_user_message="${walkthroughUserRequest}"`}
+
+
+
+
Proposed plan
+ 
+                    {JSON.stringify(groundProposedPlan, null, 2)}
+
+
+
+
+ )} + + {activeStage === 1 && animationPhase === 'complete' && ( +
+
+
+ Step 2 · Gateway + Resolve +
+

Current catalogue evidence

+
{requestSteps[1].body}
+
+ +
+
+
Catalogue query
+ 
+                    {resolveCatalogueCall}
+
+
+
+
Match
+ 
+                    {JSON.stringify(resolveCatalogueMatch, null, 2)}
+
+
+
+
+ )} + + {activeStage === 2 && animationPhase === 'complete' && ( +
+
+
+ Step 3 · Gateway + Gate +
+

Five deterministic outcomes

+
+

+ The gateway applies fixed policy to the grounded, catalogue-backed plan and assigns one of + five outcomes. Only a ready request may enter the calculation loop. +

+

+ The gate is deliberately biased towards computing. If the classifier errors or returns + something the server cannot read, the request is treated as ready rather than refused, on + the view that a wrong refusal costs more than a wrong attempt. The guarantees that follow + come from the tools, not from the gate. +

+
    +
  • + Ready: the request is complete and supported, so calculation can begin. +
    • +
  • + Needs plan: a required choice is missing, so the chat asks a clarifying + question. +
    • +
  • + Partial: the request mixes supported and unsupported outputs, so the + chat offers the supported part. +
    • +
  • + Out of scope: the requested policy output cannot be modelled, so + calculation tools stay closed. +
    • +
  • + Irrelevant: the request is unrelated to UK tax and benefit policy and + takes a lightweight response path. +
    • +
+
+
+ +
+
+
Gated plan
+ 
+                    {gateCall}
+
+
+
+
Verdict
+ 
+                    {JSON.stringify(gateVerdict, null, 2)}
+
+
+
+
+ )} + + {activeStage === 3 && animationPhase === 'complete' && ( +
+
+
+ Step 4 · Gateway + Verify +
+

Exact reform construction

+
+

+ A society-wide reform receives a second bounded check. The resolver searches the rules + engine's current reform targets, binds the user's wording to an exact parameter + path and date, and constructs reform JSON. The validator must accept that construction + before a simulation can run. +

+

+ What the gateway approves is then the only reform that can run. The simulation tool + compares the reform it is handed against the approved construction and refuses anything + that does not match, so a different reform cannot be substituted after the check has + passed. +

+
+
+ +
+
+
Resolver construction
+ 
+                    {verifyReformCall}
+
+
+
+
Approved for this turn
+ 
+                    {JSON.stringify(approvedReform, null, 2)}
+
+
+
+
+ )} + + {activeStage === 4 && animationPhase === 'complete' && ( +
+
+
+ Step 5 · Supporting tools + Calculate +
+

A bounded 21-tool runtime

+
{requestSteps[4].body}
+
+ +
+
+
1 · Run the approved reform
+ 
+                    {societySimulationCall}
+
+
+
+
2 · Derive the requested output
+ 
+                    {budgetaryImpactCall}
+
+
+
+
+ )} + + {activeStage === 5 && animationPhase === 'complete' && ( +
+
+
+ Step 6 · Gateway + Stream +
+

An inspectable response

+
{requestSteps[5].body}
+
+ +
+
+
Public server-sent events
+ 
+                    {streamEventSequence}
+
+
+
+
+ )} +
+ + + +
); } @@ -1626,28 +1984,11 @@ function WorkedExample() {

The same reform, call by call

- Consider: “For 2026, set the Child Benefit eldest-child rate to £30 a week and show the annual - budgetary impact.” The request contains a policy, a final value, a year, and an output. It does not - contain a PolicyEngine parameter path, a reform object, or the sequence of tools needed to answer it. -

-

- The six stages above decide what the system does; underneath, the turn is a short sequence of typed - calls, each one taking an argument that an earlier call produced. This is that sequence for the - request. + The six stages decide what the system does. Underneath, the approved plan is a short sequence of typed + calls, each taking an argument an earlier call produced. The loop re-derives what the gateway already + approved, and the match between the two is itself the check.

-

- Two features of the sequence carry most of the design. The first is that{' '} - run_society_simulation hands back a handle rather than data: the language model receives - a result_id, a dataset reference, and a year, and never receives household rows, survey - weights, or any serialisable simulation object. Weighting happens inside the PolicyEngine UK model, - and the seven analysis tools are the only route from a stored simulation to a number. -

-

- The second is provenance. Every parameter path, value, and handle in the sequence was either supplied - by the user or produced by a preceding call, so each argument can be traced to its source. Nothing in - it was recalled. -

@@ -1693,9 +2034,9 @@ function WorkedExample() {

- If the catalogue returned more than one materially plausible interpretation, or the reform resolver - could not determine what year or reform the user was requesting, the calculation would pause before - the tool loop and ask the user to confirm the intended construction. + The seven analysis tools are the only route from a stored simulation to that number, and every + parameter path, value, and handle in the sequence was either supplied by the user or produced by a + preceding call. Nothing in it was recalled.

); @@ -1757,7 +2098,8 @@ function Limitations() {

Some of what follows is a deliberate boundary that will not change. The rest is the current state of a beta: reform coverage is incomplete, answers vary in how fully they state their assumptions, and - gateway behaviour will move as we act on what users report. + gateway behaviour will move as we act on what users report. Widening the range of reforms the typed + tools cover, and the evidence the gateway can resolve before calculation, is where the work goes next.

The chat is a modelling tool, not advice. It reports what the engine calculates under @@ -1783,18 +2125,12 @@ function Limitations() { function NextStepsAndTryIt() { return ( -

What's next

-

- We want to widen the range of reforms covered by typed tools, improve the speed and inspectability of - more specialised analyses, and keep expanding the evidence the gateway can resolve before calculation. - UK Chat also works alongside PolicyEngine's wider generative AI integrations and plugin ecosystem - for researchers building their own analyses. -

Try it yourself

UK Chat's answers can be cited and reproduced because the figures come from the same open engine - that powers the rest of PolicyEngine. Try it with a reform, then inspect the stated year, dataset, - comparator, and method alongside the answer. + that powers the rest of PolicyEngine, and it sits alongside PolicyEngine's wider generative AI + integrations and plugin ecosystem for researchers building their own analyses. Try it with a reform, + then inspect the stated year, dataset, comparator, and method alongside the answer.

Because this is a beta, the cases where it fails are more useful to us than the cases where it works: @@ -1849,7 +2185,7 @@ export default function App() { - + diff --git a/src/uk-chat.css b/src/uk-chat.css index 9b17585..fc6f94d 100644 --- a/src/uk-chat.css +++ b/src/uk-chat.css @@ -148,12 +148,1346 @@ margin-bottom: 10px; } -/* Specificity has to match legacy-encode-policy.css's - `.diagram-container .flow-diagram-svg { max-height: 320px }`, or the request - diagram renders at ~54% and its labels become unreadable. The viewport clamp - keeps the sticky panel inside the window on short screens. */ -.diagram-container .request-flow-svg { - max-height: min(560px, calc(100vh - 240px)); +.walkthrough-disclosure { + border-left: 3px solid var(--accent-mid); + color: var(--text-mid); + padding-left: 18px; +} + +.answer-walkthrough { + margin: 40px 0 56px; +} + +.walkthrough-prompt { + background: var(--accent); + border-radius: 14px; + box-shadow: 0 14px 34px rgba(13, 115, 119, 0.14); + color: #fff; + padding: 24px 30px 28px; +} + +.walkthrough-prompt-label { + color: rgba(255, 255, 255, 0.76); + font-family: 'JetBrains Mono', monospace; + font-size: 0.7rem; + font-weight: 700; + letter-spacing: 0.12em; + margin-bottom: 9px; + text-transform: uppercase; +} + +.article-wrapper .walkthrough-prompt p { + color: #fff; + font-family: 'JetBrains Mono', monospace; + font-size: 1rem; + font-weight: 600; + line-height: 1.6; + margin: 0; +} + +.walkthrough-scrolly { + align-items: start; + display: grid; + gap: 40px; + grid-template-columns: minmax(0, 1fr) 270px; + margin-top: 20px; + position: relative; +} + +.walkthrough-scrolly-without-map { + grid-template-columns: 1fr; +} + +.walkthrough-narrative { + min-width: 0; +} + +.walkthrough-narrative-step { + align-items: center; + display: flex; + min-height: 78vh; + padding: 34px 0; + scroll-margin-top: calc(var(--header-height) + 28px); +} + +.walkthrough-narrative-step:first-child { + padding-top: 16px; +} + +.walkthrough-narrative-step:last-child { + padding-bottom: 80px; +} + +.walkthrough-stage-card { + background: #fff; + border: 1px solid var(--border-accent); + border-left: 4px solid transparent; + border-radius: 14px; + box-shadow: 0 12px 36px rgba(30, 58, 58, 0.06); + display: grid; + grid-template-columns: minmax(0, 0.92fr) minmax(0, 1.08fr); + overflow: hidden; + width: 100%; + transition: + border-color 0.28s ease, + box-shadow 0.28s ease, + transform 0.28s ease; +} + +.walkthrough-narrative-step.active .walkthrough-stage-card { + border-left-color: var(--accent); + box-shadow: 0 18px 46px rgba(13, 115, 119, 0.12); + transform: translateY(-2px); +} + +.walkthrough-stage-map-button:focus-visible, +.walkthrough-branch-button:focus-visible { + outline: 3px solid rgba(13, 115, 119, 0.32); + outline-offset: 2px; +} + +.walkthrough-stage-map { + align-self: start; + padding-top: 34px; + position: sticky; + top: calc(var(--header-height) + 20px); +} + +.walkthrough-stage-map-panel { + background: #fff; + border: 1px solid var(--border-accent); + border-radius: 14px; + box-shadow: 0 14px 40px rgba(30, 58, 58, 0.08); + padding: 18px; +} + +.walkthrough-stage-map-heading { + border-bottom: 1px solid var(--border); + color: var(--text-dark); + font-family: 'JetBrains Mono', monospace; + font-size: 0.7rem; + font-weight: 700; + letter-spacing: 0.08em; + margin-bottom: 14px; + padding-bottom: 12px; + text-transform: uppercase; +} + +.walkthrough-stage-map ol { + list-style: none; + margin: 0; + padding: 0; +} + +.walkthrough-stage-map li { + margin: 0; +} + +.walkthrough-stage-map-button { + align-items: center; + background: #fff; + border: 1px solid var(--border); + border-radius: 9px; + color: var(--text-mid); + cursor: pointer; + display: flex; + gap: 11px; + padding: 9px 10px; + text-align: left; + transition: + background-color 0.2s ease, + border-color 0.2s ease, + color 0.2s ease; + width: 100%; +} + +.walkthrough-stage-map-button:hover { + border-color: var(--accent); + color: var(--accent); +} + +.walkthrough-stage-map-button.active { + background: var(--accent); + border-color: var(--accent); + color: #fff; +} + +.walkthrough-stage-map-number { + align-items: center; + background: var(--accent-light); + border-radius: 50%; + color: var(--accent); + display: inline-flex; + flex: 0 0 28px; + font-family: 'JetBrains Mono', monospace; + font-size: 0.72rem; + font-weight: 700; + height: 28px; + justify-content: center; +} + +.walkthrough-stage-map-button.active .walkthrough-stage-map-number { + background: #fff; + color: var(--accent); +} + +.walkthrough-stage-map-copy { + min-width: 0; +} + +.walkthrough-stage-map-copy strong, +.walkthrough-stage-map-copy small { + display: block; +} + +.walkthrough-stage-map-copy strong { + font-family: 'JetBrains Mono', monospace; + font-size: 0.78rem; +} + +.walkthrough-stage-map-button.active .walkthrough-stage-map-copy strong { + color: #fff; +} + +.walkthrough-stage-map-copy small { + color: var(--text-light); + font-family: 'Source Serif 4', Georgia, serif; + font-size: 0.72rem; + margin-top: 2px; +} + +.walkthrough-stage-map-button.active small { + color: rgba(255, 255, 255, 0.8); +} + +.walkthrough-stage-map-arrow { + color: var(--accent); + display: block; + font-family: 'JetBrains Mono', monospace; + font-size: 0.95rem; + height: 18px; + line-height: 18px; + padding-left: 20px; +} + +.walkthrough-stage-map-progress { + border-top: 1px solid var(--border); + color: var(--text-light); + font-family: 'JetBrains Mono', monospace; + font-size: 0.66rem; + letter-spacing: 0.05em; + margin-top: 14px; + padding-top: 12px; + text-align: center; + text-transform: uppercase; +} + +.walkthrough-copy { + border-right: 1px solid var(--border); + padding: 32px 30px 36px; +} + +.walkthrough-step-meta { + align-items: center; + color: var(--accent); + display: flex; + font-family: 'JetBrains Mono', monospace; + font-size: 0.7rem; + font-weight: 700; + justify-content: space-between; + letter-spacing: 0.08em; + margin-bottom: 14px; + text-transform: uppercase; +} + +.walkthrough-owner { + background: var(--accent-light); + border: 1px solid var(--accent-mid); + border-radius: 999px; + padding: 4px 8px; +} + +.walkthrough-copy h3 { + color: var(--text-dark); + font-family: 'JetBrains Mono', monospace; + font-size: 1.55rem; + margin: 0; +} + +.walkthrough-subtitle { + color: var(--text-light); + font-size: 0.95rem; + margin-top: 5px; +} + +.walkthrough-step-body { + margin-top: 24px; +} + +.walkthrough-step-body > :first-child { + margin-top: 0; +} + +.walkthrough-step-body > :last-child { + margin-bottom: 0; +} + +.walkthrough-trace { + background: linear-gradient(145deg, #f7fbfa, #edf6f5); + min-width: 0; +} + +.walkthrough-trace-header { + align-items: center; + background: #1e3a3a; + color: #fff; + display: flex; + font-family: 'JetBrains Mono', monospace; + font-size: 0.68rem; + justify-content: space-between; + letter-spacing: 0.08em; + padding: 12px 18px; + text-transform: uppercase; +} + +.walkthrough-trace-header span:last-child { + color: #a9d7d4; +} + +.walkthrough-trace-body { + min-height: 360px; + padding: 28px 30px 34px; +} + +.walkthrough-trace-title { + color: var(--text-dark); + font-family: 'JetBrains Mono', monospace; + font-size: 1rem; + font-weight: 700; + margin-bottom: 20px; +} + +.walkthrough-trace-rows { + background: #fff; + border: 1px solid var(--border); + border-radius: 10px; + margin: 0; + overflow: hidden; +} + +.walkthrough-trace-rows > div { + border-bottom: 1px solid var(--border); + display: grid; + gap: 16px; + grid-template-columns: minmax(112px, 0.72fr) minmax(0, 1.28fr); + padding: 13px 15px; +} + +.walkthrough-trace-rows > div:last-child { + border-bottom: 0; +} + +.walkthrough-trace-rows dt { + color: var(--text-light); + font-family: 'JetBrains Mono', monospace; + font-size: 0.68rem; + font-weight: 600; + letter-spacing: 0.04em; + text-transform: uppercase; +} + +.walkthrough-trace-rows dd { + color: var(--text-dark); + font-size: 0.92rem; + margin: 0; + overflow-wrap: anywhere; +} + +.walkthrough-trace-rows code { + background: var(--accent-light); + border-radius: 5px; + color: var(--accent); + font-size: 0.74rem; + padding: 3px 5px; +} + +.article-wrapper .walkthrough-trace-note { + color: var(--text-light); + font-size: 0.86rem; + line-height: 1.55; + margin: 18px 0 0; +} + +.walkthrough-status { + border-radius: 8px; + display: inline-flex; + font-family: 'JetBrains Mono', monospace; + font-size: 0.78rem; + font-weight: 700; + letter-spacing: 0.08em; + margin-bottom: 18px; + padding: 8px 12px; + text-transform: uppercase; +} + +.walkthrough-status-ready { + background: #dff4ea; + border: 1px solid #8bc9aa; + color: #12623e; +} + +.walkthrough-status-needs-plan { + background: #fff3d6; + border: 1px solid #dfbc66; + color: #76520a; +} + +.walkthrough-checklist { + list-style: none; + margin: 0; + padding: 0; +} + +.article-wrapper .walkthrough-checklist li { + border-bottom: 1px solid var(--border); + font-size: 0.95rem; + margin: 0; + padding: 10px 4px 10px 30px; + position: relative; +} + +.walkthrough-checklist li::before { + color: #12623e; + content: '✓'; + font-family: 'JetBrains Mono', monospace; + font-weight: 700; + left: 5px; + position: absolute; +} + +.walkthrough-checklist li:last-child { + border-bottom: 0; +} + +.walkthrough-boundary-test { + border-top: 1px solid var(--border); + margin-top: 26px; + padding-top: 22px; +} + +.walkthrough-boundary-label { + color: var(--accent); + font-family: 'JetBrains Mono', monospace; + font-size: 0.7rem; + font-weight: 700; + letter-spacing: 0.08em; + text-transform: uppercase; +} + +.article-wrapper .walkthrough-boundary-test p { + font-size: 0.92rem; + line-height: 1.55; + margin: 8px 0 14px; +} + +.walkthrough-branch-button { + background: #fff; + border: 1px solid var(--accent); + border-radius: 7px; + color: var(--accent); + cursor: pointer; + font-family: 'JetBrains Mono', monospace; + font-size: 0.72rem; + font-weight: 600; + padding: 9px 11px; +} + +.walkthrough-branch-button:hover, +.walkthrough-branch-button[aria-pressed='true'] { + background: var(--accent); + color: #fff; +} + +.article-wrapper .walkthrough-branch-prompt { + background: #fff; + border-left: 3px solid #dfbc66; + color: var(--text-mid); + font-size: 0.92rem; + font-style: italic; + margin: 0 0 18px; + padding: 13px 15px; +} + +.walkthrough-clarification { + background: #fff; + border: 1px solid var(--border-accent); + border-radius: 4px 12px 12px 12px; + color: var(--text-dark); + line-height: 1.6; + padding: 16px 18px; +} + +.walkthrough-clarification span { + color: var(--accent); + display: block; + font-family: 'JetBrains Mono', monospace; + font-size: 0.68rem; + font-weight: 700; + letter-spacing: 0.08em; + margin-bottom: 6px; + text-transform: uppercase; +} + +.walkthrough-tool-run-list { + margin-bottom: 22px; +} + +.walkthrough-tool-run { + background: #fff; + border: 1px solid var(--border); + border-left: 4px solid var(--accent); + border-radius: 0 9px 9px 0; + padding: 14px 16px; +} + +.walkthrough-tool-run > div { + align-items: flex-start; + display: flex; + gap: 12px; + justify-content: space-between; +} + +.walkthrough-tool-run code { + color: var(--accent); + font-size: 0.72rem; + font-weight: 700; + overflow-wrap: anywhere; +} + +.walkthrough-tool-run > div span { + color: #12623e; + font-family: 'JetBrains Mono', monospace; + font-size: 0.64rem; + font-weight: 700; + letter-spacing: 0.05em; + text-transform: uppercase; +} + +.article-wrapper .walkthrough-tool-run p { + font-size: 0.86rem; + line-height: 1.5; + margin: 9px 0 6px; +} + +.walkthrough-tool-run small { + color: var(--text-light); + display: block; + font-size: 0.76rem; + line-height: 1.45; +} + +.walkthrough-tool-connector { + color: var(--accent); + font-family: 'JetBrains Mono', monospace; + font-size: 1rem; + height: 22px; + line-height: 22px; + padding-left: 20px; +} + +.walkthrough-result-table-wrap { + margin-top: 0; +} + +.walkthrough-result-table { + margin: 0 0 14px; +} + +.walkthrough-event-list { + display: flex; + flex-wrap: wrap; + gap: 7px; + margin-bottom: 20px; +} + +.walkthrough-event-list code { + background: #fff; + border: 1px solid var(--border-accent); + border-radius: 999px; + color: var(--accent); + font-size: 0.66rem; + padding: 5px 8px; +} + +.walkthrough-answer { + background: #fff; + border: 1px solid var(--border-accent); + border-radius: 4px 14px 14px 14px; + box-shadow: 0 10px 28px rgba(30, 58, 58, 0.07); + padding: 18px 20px; +} + +.walkthrough-answer > span { + color: var(--accent); + display: block; + font-family: 'JetBrains Mono', monospace; + font-size: 0.68rem; + font-weight: 700; + letter-spacing: 0.08em; + margin-bottom: 8px; + text-transform: uppercase; +} + +.article-wrapper .walkthrough-answer p { + font-size: 0.94rem; + line-height: 1.6; + margin: 0 0 12px; +} + +.article-wrapper .walkthrough-answer p:last-child { + color: var(--text-light); + font-size: 0.84rem; + margin-bottom: 0; +} + +.walkthrough-method-tags { + display: flex; + flex-wrap: wrap; + gap: 7px; + margin-top: 16px; +} + +.walkthrough-method-tags span { + background: var(--accent-light); + border-radius: 999px; + color: var(--accent); + font-family: 'JetBrains Mono', monospace; + font-size: 0.64rem; + padding: 5px 8px; +} + +.compressed-chat-walkthrough { + align-items: stretch; + display: grid; + gap: 20px; + grid-template-columns: minmax(0, 1fr) 190px; + height: calc(100vh - var(--header-height) - 40px); + height: calc(100svh - var(--header-height) - 40px); + margin: 20px 0 72px -100px; + width: calc(100% + 200px); +} + +.compressed-chat-preview { + --chat-preview-bg: #fff; + --chat-preview-border: #e5e5e5; + --chat-preview-faint: #b4b4b4; + --chat-preview-muted: #8e8e8e; + --chat-preview-surface-hover: #ececec; + --chat-preview-text: #0d0d0d; + --chat-preview-text-3: #5d5d5d; + background: var(--chat-preview-bg); + border: 1px solid var(--chat-preview-border); + border-radius: 12px; + color: var(--chat-preview-text); + font-family: + system-ui, + -apple-system, + BlinkMacSystemFont, + 'Segoe UI', + sans-serif; + height: 100%; + overflow: hidden; + width: 100%; +} + +.compressed-chat-stage-map { + align-self: stretch; + height: 100%; + padding-top: 0; + position: relative; + top: auto; +} + +.compressed-chat-stage-map .walkthrough-stage-map-panel { + box-sizing: border-box; + display: flex; + flex-direction: column; + height: 100%; + padding: 14px; +} + +.compressed-chat-stage-map .walkthrough-stage-map-heading { + font-size: 0.62rem; + margin-bottom: 10px; + padding-bottom: 10px; +} + +.compressed-chat-stage-map ol { + display: flex; + flex: 1; + flex-direction: column; + min-height: 0; +} + +.compressed-chat-stage-map li:not(:last-child) { + display: flex; + flex: 1; + flex-direction: column; +} + +.compressed-chat-stage-map .walkthrough-stage-map-button { + gap: 8px; + padding: 7px 8px; +} + +.compressed-chat-stage-map .walkthrough-stage-map-number { + flex-basis: 24px; + font-size: 0.66rem; + height: 24px; +} + +.compressed-chat-stage-map .walkthrough-stage-map-copy strong { + font-size: 0.7rem; +} + +.compressed-chat-stage-map .walkthrough-stage-map-copy small { + font-size: 0.66rem; +} + +.compressed-chat-stage-map .walkthrough-stage-map-arrow { + align-items: center; + display: flex; + flex: 1; + height: auto; + min-height: 12px; + padding-left: 18px; +} + +.compressed-chat-stage-controls { + align-items: center; + border-top: 1px solid var(--border); + display: grid; + gap: 6px; + grid-template-columns: 30px minmax(0, 1fr) 30px; + margin-top: 10px; + padding-top: 10px; +} + +.compressed-chat-stage-controls button { + align-items: center; + background: #fff; + border: 1px solid var(--border); + border-radius: 7px; + color: var(--accent); + cursor: pointer; + display: inline-flex; + height: 30px; + justify-content: center; + padding: 0; + width: 30px; +} + +.compressed-chat-stage-controls button:hover:not(:disabled), +.compressed-chat-stage-controls button:focus-visible { + border-color: var(--accent); + outline: none; +} + +.compressed-chat-stage-controls button:disabled { + color: var(--text-light); + cursor: not-allowed; + opacity: 0.38; +} + +.compressed-chat-stage-controls svg { + height: 15px; + stroke-width: 1.8; + width: 15px; +} + +.compressed-chat-stage-controls .walkthrough-stage-map-progress { + border-top: 0; + font-size: 0.58rem; + margin-top: 0; + padding-top: 0; +} + +.compressed-chat-main { + align-items: stretch; + background: var(--chat-preview-bg); + display: grid; + gap: 14px; + grid-template-columns: repeat(2, minmax(0, 1fr)); + grid-template-rows: repeat(2, minmax(0, 1fr)); + height: 100%; + min-width: 0; + overflow: hidden; + padding: 7px 21px 22px; +} + +.compressed-chat-empty-state { + align-self: stretch; + align-items: stretch; + box-sizing: border-box; + display: flex; + flex-direction: column; + grid-column: 1 / -1; + grid-row: 1; + height: 100%; + justify-content: flex-start; + justify-self: center; + max-width: 760px; + min-height: 100%; + min-width: 0; + padding-top: 36px; + width: 100%; +} + +.compressed-chat-empty-state h3 { + color: var(--chat-preview-text); + font-family: + system-ui, + -apple-system, + BlinkMacSystemFont, + 'Segoe UI', + sans-serif; + font-size: 25.5px; + font-weight: 500; + flex: 0 0 auto; + letter-spacing: -0.01em; + line-height: 1.2; + margin: 0 0 18px; + max-height: 40px; + overflow: hidden; + text-align: center; + transition: + max-height 480ms ease, + margin 480ms ease, + opacity 280ms ease, + transform 480ms ease; +} + +.compressed-chat-empty-state.has-message h3 { + margin-bottom: 0; + max-height: 0; + opacity: 0; + transform: translateY(-12px); +} + +.compressed-chat-composer-wrap { + flex: 0 0 auto; + max-width: 100%; + min-width: 0; + position: relative; + transition: max-width 560ms cubic-bezier(0.22, 1, 0.36, 1); + width: 100%; +} + +.compressed-chat-empty-state.has-message .compressed-chat-composer-wrap { + margin-left: auto; + max-width: 75%; +} + +.compressed-chat-composer { + background: var(--chat-preview-bg); + border: 1px solid var(--chat-preview-border); + border-radius: 24px; + box-shadow: 0 1px 3px rgba(0, 0, 0, 0.04); + padding: 12px 15px 8px; + position: relative; + transition: + background-color 420ms ease, + border-color 280ms ease, + border-radius 420ms ease, + box-shadow 280ms ease, + padding 420ms ease; +} + +.compressed-chat-empty-state.has-message .compressed-chat-composer { + background: #f4f4f4; + border-color: transparent; + border-radius: 22px; + box-shadow: none; + padding: 10px 16px; +} + +.compressed-chat-user-request { + color: var(--chat-preview-text); + font-size: 14.5px; + line-height: 1.5; + min-height: 44px; + overflow-wrap: anywhere; + padding: 3px 0; + transition: + font-size 420ms ease, + min-height 420ms ease, + padding 420ms ease; +} + +.compressed-chat-empty-state.has-message .compressed-chat-user-request { + font-size: 15px; + min-height: 0; + padding: 0; +} + +.compressed-chat-caret { + animation: compressed-chat-caret-blink 1s step-end infinite; + background: var(--chat-preview-text); + display: inline-block; + height: 1.05em; + margin-left: 2px; + opacity: 1; + transition: opacity 160ms ease; + vertical-align: -2px; + width: 1.5px; +} + +.compressed-chat-empty-state.has-message .compressed-chat-caret { + opacity: 0; +} + +@keyframes compressed-chat-caret-blink { + 50% { + opacity: 0; + } +} + +.compressed-chat-composer-actions { + align-items: center; + display: flex; + gap: 8px; + justify-content: space-between; + margin-top: 5px; + max-height: 32px; + overflow: hidden; + transition: + margin 360ms ease, + max-height 420ms ease, + opacity 220ms ease; +} + +.compressed-chat-empty-state.has-message .compressed-chat-composer-actions { + margin-top: 0; + max-height: 0; + opacity: 0; +} + +.compressed-chat-transcript { + flex: 1 1 auto; + min-height: 0; + min-width: 0; + overflow: auto; + overscroll-behavior: contain; + width: 100%; +} + +.compressed-chat-composer-tools { + align-items: center; + display: inline-flex; + gap: 5px; +} + +.compressed-chat-attach, +.compressed-chat-send { + align-items: center; + border-radius: 999px; + display: inline-flex; + height: 28px; + justify-content: center; + width: 28px; +} + +.compressed-chat-attach { + color: var(--chat-preview-text-3); +} + +.compressed-chat-attach svg { + height: 16px; + stroke-width: 1.8; + width: 16px; +} + +.compressed-chat-send { + background: var(--chat-preview-surface-hover); + color: var(--chat-preview-muted); + position: relative; + transition: + background-color 140ms ease, + color 140ms ease, + transform 140ms ease; +} + +.compressed-chat-send.is-active { + background: var(--chat-preview-text); + color: #fff; +} + +.compressed-chat-send.is-submitting { + animation: compressed-chat-submit 450ms ease-out both; +} + +.compressed-chat-send svg { + height: 14px; + stroke-width: 2; + width: 14px; +} + +@keyframes compressed-chat-submit { + 0% { + box-shadow: 0 0 0 0 rgba(13, 13, 13, 0.28); + transform: scale(1); + } + + 38% { + box-shadow: 0 0 0 7px rgba(13, 13, 13, 0.08); + transform: scale(0.78); + } + + 100% { + box-shadow: 0 0 0 12px rgba(13, 13, 13, 0); + transform: scale(1); + } +} + +.compressed-chat-loading { + align-items: center; + animation: compressed-chat-loading-reveal 180ms ease-out 420ms both; + display: flex; + gap: 5px; + margin: 18px 0 0 14px; + min-height: 10px; +} + +@keyframes compressed-chat-loading-reveal { + from { + opacity: 0; + } + + to { + opacity: 1; + } +} + +.compressed-chat-loading span { + animation: compressed-chat-thinking-dot 1.2s ease-in-out infinite; + background: var(--chat-preview-muted); + border-radius: 50%; + height: 5px; + width: 5px; +} + +.compressed-chat-loading span:nth-child(2) { + animation-delay: 200ms; +} + +.compressed-chat-loading span:nth-child(3) { + animation-delay: 400ms; +} + +@keyframes compressed-chat-thinking-dot { + 0%, + 80%, + 100% { + opacity: 0.2; + transform: scale(0.8); + } + + 40% { + opacity: 1; + transform: scale(1); + } +} + +.compressed-chat-working { + color: var(--chat-preview-text-3); + font-size: 12px; + font-style: italic; + margin-top: 7px; + padding: 2px 0; + width: 100%; +} + +.compressed-chat-working-disclosure { + margin-top: 7px; + width: 100%; +} + +.compressed-chat-working-toggle { + align-items: center; + background: transparent; + border: 0; + color: var(--chat-preview-text-3); + cursor: pointer; + display: flex; + font-size: 12px; + gap: 6px; + padding: 2px 0; +} + +.compressed-chat-working-toggle span { + font-style: italic; +} + +.compressed-chat-working-toggle svg { + height: 12px; + opacity: 0.5; + stroke-width: 2; + transition: transform 150ms ease; + width: 12px; +} + +.compressed-chat-working-toggle svg.is-collapsed { + transform: rotate(-90deg); +} + +.compressed-chat-working-body { + border-left: 2px solid var(--chat-preview-border); + margin: 8px 0 0 4px; + min-width: 0; + padding: 0 0 0 14px; +} + +.article-wrapper .compressed-chat-working-narration { + color: var(--chat-preview-text-3); + font-family: + system-ui, + -apple-system, + BlinkMacSystemFont, + 'Segoe UI', + sans-serif; + font-size: 13px; + font-style: italic; + line-height: 1.5; + margin: 6px 0 14px; + opacity: 0.65; +} + +.compressed-chat-working-tool { + align-items: center; + color: var(--chat-preview-text-3); + display: flex; + font-family: 'JetBrains Mono', monospace; + font-size: 11px; + gap: 5px; + margin: 9px 0; + max-width: 100%; + min-width: 0; + padding: 2px 0; +} + +.compressed-chat-working-tool code { + background: none; + border-radius: 0; + color: var(--chat-preview-text-3); + font-size: inherit; + overflow-wrap: anywhere; + padding: 0; +} + +.compressed-chat-working-tool > span { + color: var(--chat-preview-muted); +} + +.compressed-chat-working-toggle:focus-visible { + border-radius: 4px; + outline: 2px solid var(--accent); + outline-offset: 2px; +} + +.compressed-chat-assistant-answer { + animation: compressed-chat-ground-reveal 360ms ease-out both; + color: var(--chat-preview-text); + margin-top: 12px; + min-width: 0; + width: 100%; +} + +.article-wrapper .compressed-chat-assistant-answer p { + color: var(--chat-preview-text); + font-family: + system-ui, + -apple-system, + BlinkMacSystemFont, + 'Segoe UI', + sans-serif; + font-size: 14px; + line-height: 1.65; + margin: 0 0 8px; +} + +.article-wrapper .compressed-chat-assistant-answer p:last-child { + color: var(--chat-preview-text-3); + margin-bottom: 0; +} + +.compressed-chat-ground-stage { + align-content: stretch; + align-items: stretch; + animation: compressed-chat-ground-reveal 520ms ease-out both; + display: grid; + gap: 14px; + grid-column: 1 / -1; + grid-row: 2; + grid-template-columns: repeat(2, minmax(0, 1fr)); + grid-template-rows: minmax(0, 1fr); + height: 100%; + min-height: 0; + overflow: hidden; + width: 100%; +} + +.compressed-chat-ground-description, +.compressed-chat-ground-technical { + border: 1px solid var(--chat-preview-border); + border-radius: 11px; + height: 100%; + min-height: 0; + min-width: 0; + padding: 14px 16px; +} + +.compressed-chat-ground-description { + background: #fff; + border-left: 3px solid var(--accent); + display: flex; + flex-direction: column; + overflow: hidden; +} + +.compressed-chat-ground-description-body { + flex: 1 1 auto; + min-height: 0; + overflow-y: auto; + overscroll-behavior: contain; +} + +.compressed-chat-ground-meta { + align-items: center; + color: var(--accent); + display: flex; + font-family: 'JetBrains Mono', monospace; + font-size: 0.68rem; + font-weight: 700; + justify-content: space-between; + letter-spacing: 0.08em; + text-transform: uppercase; +} + +.compressed-chat-ground-description h4 { + color: var(--chat-preview-text); + font-family: + system-ui, + -apple-system, + BlinkMacSystemFont, + 'Segoe UI', + sans-serif; + font-size: 1.15rem; + font-weight: 600; + line-height: 1.3; + margin: 8px 0 0; +} + +.article-wrapper .compressed-chat-ground-description p { + color: var(--chat-preview-text-3); + font-family: + system-ui, + -apple-system, + BlinkMacSystemFont, + 'Segoe UI', + sans-serif; + font-size: 1rem; + line-height: 1.6; + margin: 9px 0 0; +} + +.article-wrapper .compressed-chat-gate-description p { + font-size: 0.9rem; + line-height: 1.45; +} + +.compressed-chat-gate-description .gateway-outcomes { + margin: 10px 0 0; + padding-left: 20px; +} + +.article-wrapper .compressed-chat-gate-description .gateway-outcomes li { + font-size: 0.84rem; + line-height: 1.4; + margin-bottom: 5px; +} + +.article-wrapper .compressed-chat-gate-description .gateway-outcomes li:last-child { + margin-bottom: 0; +} + +.compressed-chat-ground-technical { + background: linear-gradient(145deg, #f7fbfa, #f2f6f5); + display: flex; + flex-direction: column; + overflow-y: auto; + overscroll-behavior: contain; +} + +.compressed-chat-ground-technical .example-section { + margin-bottom: 10px; +} + +.compressed-chat-ground-technical .example-section:last-child { + margin-bottom: 0; +} + +.compressed-chat-ground-technical .example-section-title { + font-size: 0.78rem; + margin-bottom: 6px; +} + +.compressed-chat-ground-code { + background: #fff; + border: 1px solid var(--chat-preview-border); + border-radius: 7px; + color: #1e3a3a; + font-family: 'JetBrains Mono', monospace; + font-size: 0.9rem; + line-height: 1.62; + margin: 0; + overflow-x: auto; + padding: 9px 11px; + white-space: pre-wrap; +} + +.compressed-chat-ground-code-plan { + font-size: 12px; +} + +@keyframes compressed-chat-ground-reveal { + from { + opacity: 0; + transform: translateY(18px); + } + + to { + opacity: 1; + transform: translateY(0); + } +} + +@media (prefers-reduced-motion: reduce) { + .compressed-chat-caret, + .compressed-chat-ground-stage, + .compressed-chat-loading, + .compressed-chat-loading span, + .compressed-chat-send.is-submitting { + animation: none; + } + + .compressed-chat-empty-state h3, + .compressed-chat-composer-wrap, + .compressed-chat-composer, + .compressed-chat-user-request, + .compressed-chat-caret, + .compressed-chat-composer-actions { + transition: none; + } } /* The legacy sheet scopes .network-label and .flow-arrow under @@ -540,6 +1874,61 @@ .tool-cards { grid-template-columns: repeat(2, minmax(0, 1fr)); } + .walkthrough-scrolly { + grid-template-columns: 1fr; + } + + .walkthrough-stage-map { + order: -1; + padding-top: 16px; + position: relative; + top: auto; + } + + .walkthrough-stage-map ol { + display: grid; + gap: 8px; + grid-template-columns: repeat(3, minmax(0, 1fr)); + } + + .walkthrough-stage-map-arrow { + display: none; + } + + .walkthrough-narrative-step { + min-height: auto; + padding: 28px 0; + } + + .compressed-chat-walkthrough { + grid-template-columns: minmax(0, 1fr) 180px; + } + + .compressed-chat-stage-map { + order: 0; + } + + .compressed-chat-stage-map ol { + display: flex; + } + + .compressed-chat-stage-map .walkthrough-stage-map-arrow { + display: block; + } +} + +@media (max-width: 1200px) { + .compressed-chat-walkthrough { + margin-left: -40px; + width: calc(100% + 80px); + } +} + +@media (max-width: 768px) { + .compressed-chat-walkthrough { + margin-left: -5px; + width: calc(100% + 10px); + } } @media (max-width: 600px) { @@ -576,6 +1965,61 @@ margin-bottom: 16px; } + .walkthrough-prompt { + padding: 21px 18px 23px; + } + + .walkthrough-stage-map-panel { + padding: 14px; + } + + .walkthrough-stage-map ol { + grid-template-columns: repeat(2, minmax(0, 1fr)); + } + + .walkthrough-stage-map-button { + height: 100%; + } + + .walkthrough-copy, + .walkthrough-trace-body { + padding: 24px 18px 28px; + } + + .walkthrough-stage-card { + grid-template-columns: 1fr; + } + + .walkthrough-copy { + border-bottom: 1px solid var(--border); + border-right: 0; + } + + .walkthrough-step-meta { + align-items: flex-start; + flex-direction: column; + gap: 8px; + } + + .walkthrough-trace-rows > div { + gap: 5px; + grid-template-columns: 1fr; + } + + .compressed-chat-walkthrough { + grid-template-columns: 1fr; + height: auto; + } + + .compressed-chat-preview { + height: calc(100vh - var(--header-height) - 40px); + height: calc(100svh - var(--header-height) - 40px); + } + + .compressed-chat-stage-map { + padding-top: 0; + } + .worked-result-table th, .worked-result-table td { padding: 11px 10px; From dec7ebb8d7a2a834423c534196d190daffad5e1a Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Mon, 17 Aug 2026 16:45:09 +0300 Subject: [PATCH 2/8] Refine copy and consolidate the answer walkthrough --- content/current-article.md | 149 ++++++------------- src/App.jsx | 282 +++++++++++++++--------------------- src/uk-chat.css | 285 +++++++++++++++---------------------- 3 files changed, 274 insertions(+), 442 deletions(-) diff --git a/content/current-article.md b/content/current-article.md index 782c115..80d5b29 100644 --- a/content/current-article.md +++ b/content/current-article.md @@ -14,9 +14,11 @@ UK Chat is released as a **beta**. The pipeline described here is in place and r ## How policy analysis is done today -PolicyEngine already answers questions of this kind, but not in conversation. Someone costing a Child Benefit change either builds the reform by hand in the web app, or writes code against the model: find the parameter path, construct a reform, run baseline and reform over the same microdata, read off the measure. Both routes are precise and fully inspectable, and both call the engine UK Chat calls underneath. Both also ask the same thing of the person using them: knowing which parameter carries the policy, how to express the change, and which output answers the question. +PolicyEngine already provides tools to answer tax and benefit questions, but not in conversational format. For example, someone looking to determine the cost of a change to the Child Benefit can use one of two PolicyEngine features to do so. They can build the reform by hand in the web app, setting each parameter through the interface and reading the results off the charts it returns. Or they can write code against the PolicyEngine country model: find the parameter path in the catalogue, construct a reform object, run baseline and reform policies over the same survey microdata, then take the budgetary or distributional measure from the output. Either way, a first pass usually takes several attempts to settle the parameter, the date, and the measure. -For a policy analyst, a journalist, or a household working out what a reform would mean for them, the barrier is the setup rather than the arithmetic — and prompting a general-purpose language model looks like the way around it. The rest of this article is about why that shortcut does not produce a figure anyone can rely on. UK Chat replaces neither route, but for the questions its tools already cover it removes the setup and leaves the calculation where it was. +Both routes are precise and fully inspectable, and both call the same engine UK Chat calls underneath. Both also ask the same thing of the person using them: knowing which parameter carries the policy, how to express the change, and which output answers the question. For a policy analyst, a journalist, or a household working out what a reform would mean for them, this setup is a real barrier. + +To this user, prompting a general-purpose AI model may feel like a way around the barrier. However, as this article covers later, this shortcut does not necessarily produce reliable figures, given the current limitations of standard AI models. UK Chat overcomes this limitation by applying the strengths of conversational models to the same engine that already powers the web app and country models, reducing the setup friction mentioned earlier by providing a series of typed tools that ensure results are still deterministically calculated and grounded in fact. ## Separating calculation @@ -30,23 +32,13 @@ We have measured this directly. In PolicyBench, an evaluation built on the US ta ## The boundary -The central design decision is where the predictive layer stops and the deterministic layer starts. +UK Chat's central design decision is where the predictive layer stops and the deterministic layer starts. The predictive parts are interpreting wording, deciding which kind of analysis is being requested, drafting the explanation, and suggesting a follow-up. Those tasks are open-ended and language-heavy. The deterministic parts are the ones users need to be able to check: validating the request, looking up parameters, constructing household inputs, running simulations, computing weighted outputs, and producing chart data. None of these depends on the language model's memory. The runtime enforces the boundary rather than asking the language model to respect it in a prompt. -When a question is ready for computation, the language model receives the calculation tools, but not a list of what exists in the model. It has to ask: names are confirmed by calling discovery tools against the deployed package, so a policy the model half-remembers either resolves to a real parameter path or fails to resolve at all. The gateway works the same way from the other side, describing the tools from their own schemas so its vocabulary cannot drift from what the tools actually accept. - -The gateway sits on that line. It takes the plan the language model proposes and decides whether it may cross into the deterministic side. That is why the system has three parts rather than two: the language model, the gateway, and the tools a plan reaches once the gateway admits it. - -## The language model proposes a plan - -UK Chat breaks the user pathway into three segments: the language model, the gateway, and supporting tools. - -First, the language model does the predictive work. It takes the user's open-ended prompt and develops a simulation or analysis plan. The plan has a required format and must use one or more of the deterministic tools connected to the PolicyEngine UK tax and benefit simulation engine. - -Next, UK Chat's gateway verifies and sometimes constrains that plan to ensure PolicyEngine provides the tools required to answer the question. The gateway may also ask the user for clarification before finalising a plan. +This boundary creates a three-part request path: the language model, the gateway, and supporting tools. The language model turns the user's open-ended prompt into a structured simulation or analysis plan. The gateway checks whether that plan is complete and supported, constraining it or asking the user for clarification when necessary, before calculation begins. Once the plan can proceed, the language model receives the calculation tools, but not a static catalogue of PolicyEngine policies and parameters. It must use discovery tools to resolve the user's wording against the deployed PolicyEngine package, so a policy name either maps to a current parameter path or remains unresolved. The gateway likewise derives its description of each tool from that tool's schema, keeping its checks aligned with the inputs those tools actually accept. ## Tools make the calculation deterministic @@ -397,7 +389,7 @@ The gateway checks named policies, variables, and requested outputs against Poli It then refines the plan with the relevant tool inputs and declared defaults — for example, using the current year when the user does not name one. Any choice the system cannot resolve safely stays explicit instead of being silently invented. -Here, the prompt wording “Child Benefit eldest-child rate” matches `gov.hmrc.child_benefit.amount.eldest`, budgetary impact is supported, and no user-owned choices remain unresolved. +Here, the prompt wording “Child Benefit eldest-child rate” matches `gov.hmrc.child_benefit.amount.eldest`, budgetary impact is supported, and no user-owned choices remain unresolved. The identifier comes from the installed UK package rather than the language model's memory, and the wording matches the parameter's catalogue label. ### 3. Gate — Five deterministic outcomes @@ -421,7 +413,7 @@ _Gateway_ A society-wide reform receives a second bounded check. The resolver searches the rules engine's current reform targets, binds the user's wording to an exact parameter path and date, and constructs reform JSON. The validator must accept that construction before a simulation can run. -For this request, the approved construction sets `gov.hmrc.child_benefit.amount.eldest` to £30 a week for 2026. Rules-engine validation passes, and the tool context will reject a society reform that differs from the approved construction. +For this request, the approved construction sets `gov.hmrc.child_benefit.amount.eldest` to £30 a week for 2026. Rules-engine validation confirms that the path exists, `30.0` is valid for a currency-GBP parameter, and the date range falls inside the package's coverage. An unrecognised path would be rejected. The tool context will also reject a society reform that differs from the approved construction. ### 5. Calculate — A bounded 21-tool runtime @@ -431,75 +423,25 @@ A language model selected for the request's type and size executes the verified The language model chooses and sequences tools, but it does not supply the figures. Each number comes back as tool data produced by the PolicyEngine UK model. -_Plan handed to the tool loop_ - -``` -Required tool order: run_society_simulation -> compute_budgetary_impact. -Runtime handoff: pass the result_id returned by run_society_simulation - as simulation_id to the target derivative. -``` - -_Each tool result_ - -``` -{ - "type": "tool_result", "tool_use_id": …, - "content": …the tool’s JSON, capped at 15,000 characters… -} -``` - -### 6. Stream — An inspectable response - -_Gateway_ - -The gateway streams text chunks, tool starts, tool inputs, completion summaries, and the final answer as server-sent events. The interface can therefore show what the system tried and which computations completed while the answer is being produced. - -The tool loop remains bounded by a 30-round cap, repeated-call detection, result-size limits, and explicit terminal events. Tool errors return as data so the language model can recover without inventing a result. - -_Turn events_ - -``` -ToolStarted → ToolUsed → ToolCompleted → TextChunk → TurnCompleted -``` - -_Server-sent frames_ - -``` -data: { "type": "tool_use", "tool_name": "compute_budgetary_impact", - "status": "pending" } -data: { "type": "done", "content": …, "stop_reason": …, "usage": { … } } -``` - -## The same reform, call by call - -The six stages decide what the system does. Underneath, the approved plan is a short sequence of typed calls, each taking an argument an earlier call produced. The loop re-derives what the gateway already approved, and the match between the two is itself the check. - -### Tool-call trace — 5 calls, one turn +For this request, the tool loop makes five public calls. The first three independently reconstruct and validate the reform the gateway approved; the simulation tool then enforces that the reform it receives matches the approved construction. -**1. Find the target** — _Discovery_ +**1. Find the target — `list_reform_targets`** ``` -list_reform_targets( - query="child benefit eldest" -) +list_reform_targets(query="child benefit eldest") +# returns gov.hmrc.child_benefit.amount.eldest ``` -- ✓ gov.hmrc.child_benefit.amount.eldest -- · returned from the installed UK package, not from memory -- · “eldest-child rate” matches the parameter’s catalogue label +The target comes from the installed UK package, and “eldest-child rate” matches its catalogue label. -**2. Read current law** — _Discovery_ +**2. Read current law — `get_parameter`** ``` get_parameter( path="gov.hmrc.child_benefit.amount.eldest", year=2026 ) -``` - -Returned: - -``` +# returns { "unit": "currency-GBP", "period": "week", @@ -507,10 +449,9 @@ Returned: } ``` -- · the comparator is whatever the package holds, not a remembered rate -- · the unit and period fix what “£30 a week” has to be written as +The deployed value supplies the current-law comparator, while the unit and period determine how “£30 a week” must be expressed. -**3. Validate** — _Validation_ +**3. Validate — `validate_reform`** ``` validate_reform( @@ -523,23 +464,16 @@ validate_reform( ) ``` -- ✓ the path exists in the parameter schema -- ✓ 30.0 is the right type for a currency-GBP parameter -- ✓ the date range falls inside the package's coverage -- Note: an unrecognised path is rejected here. +The validator confirms the path, value type, and date range. An unrecognised path is rejected. -**4. Simulate** — _Simulation_ +**4. Simulate — `run_society_simulation`** ``` run_society_simulation( reform={ …validated above… }, year=2026 ) -``` - -Returned to the language model: - -``` +# returns { "result_id": "society_simulation_…", "dataset": { "label": "Enhanced FRS 2024-25", … }, @@ -547,13 +481,9 @@ Returned to the language model: } ``` -Not returned: - -- ✗ household rows -- ✗ survey weights -- ✗ any serialisable simulation object +The language model receives a typed result handle and metadata, not household rows, survey weights, or a serialisable simulation object. -**5. Compute** — _Analysis_ +**5. Compute — `compute_budgetary_impact`** ``` compute_budgetary_impact( @@ -561,26 +491,33 @@ compute_budgetary_impact( ) ``` -Computed inside the engine: +The analysis tool uses the preceding `result_id` as its `simulation_id`, applies survey weights inside the PolicyEngine UK model, and returns baseline and reform totals for tax revenue and benefit spending together with their net budgetary impact. Every argument in the sequence therefore comes from the user or a preceding call. -- ✓ baseline and reform totals for tax revenue and benefit spending -- ✓ the net budgetary impact reported below -- ✓ survey weights applied inside the PolicyEngine UK model -- Stored: `result_id` → typed handle, turn-local. +### 6. Stream — An inspectable response -### Computed result: About £0.9 billion a year in additional government cost +_Gateway_ -A production run compares the reform with 2026 current law using PolicyEngine's Enhanced Family Resources Survey dataset for 2024–25, release 1.56.13. It is a direct static microsimulation estimate. Tax revenue rises because a higher Child Benefit rate increases the High Income Child Benefit Charge paid by higher-income families. +The gateway streams text chunks, tool starts, tool inputs, completion summaries, and the final answer as server-sent events. The interface can therefore show what the system tried and which computations completed while the answer is being produced. + +The tool loop remains bounded by a 30-round cap, repeated-call detection, result-size limits, and explicit terminal events. Tool errors return as data so the language model can recover without inventing a result. + +_Turn events_ + +``` +ToolStarted → ToolUsed → ToolCompleted → TextChunk → TurnCompleted +``` -| Measure | Change | -| -------------------- | ------------- | -| Tax revenue | +£0.2 billion | -| Benefit spending | +£1.1 billion | -| Net budgetary impact | −£0.9 billion | +_Server-sent frames_ -A negative net budgetary impact means a cost to government. Figures are rounded to the nearest £0.1 billion. +``` +data: { "type": "tool_use", "tool_name": "compute_budgetary_impact", + "status": "pending" } +data: { "type": "done", "content": …, "stop_reason": …, "usage": { … } } +``` + +### Computed result: About £0.9 billion a year in additional government cost -The seven analysis tools are the only route from a stored simulation to that number, and every parameter path, value, and handle in the sequence was either supplied by the user or produced by a preceding call. Nothing in it was recalled. +A production run compares the reform with 2026 current law using PolicyEngine's Enhanced Family Resources Survey dataset for 2024–25, release 1.56.13. It is a direct static microsimulation estimate. Tax revenue rises because a higher Child Benefit rate increases the High Income Child Benefit Charge paid by higher-income families. ## What you can ask diff --git a/src/App.jsx b/src/App.jsx index c854f6f..d505bc2 100644 --- a/src/App.jsx +++ b/src/App.jsx @@ -356,13 +356,13 @@ const questionKinds = [ }, ]; -/* The five calls the Child Benefit request in "The same reform, call by call" actually - makes, one per tool family bar Presentation. Identifiers are shortened and the - current-law rate is elided, because both come from whichever PolicyEngine UK - package is deployed rather than from anything fixed at publication. */ -const traceSteps = [ +/* The five public calls made during the Calculate stage. Identifiers are shortened + and the current-law rate is elided, because both come from whichever PolicyEngine + UK package is deployed rather than from anything fixed at publication. */ +const calculationToolSteps = [ { id: 'find-target', + toolName: 'list_reform_targets', stage: 'Find the target', family: 'Discovery', body: ( @@ -398,6 +398,7 @@ const traceSteps = [ }, { id: 'read-current-law', + toolName: 'get_parameter', stage: 'Read current law', family: 'Discovery', body: ( @@ -449,6 +450,7 @@ const traceSteps = [ }, { id: 'validate', + toolName: 'validate_reform', stage: 'Validate', family: 'Validation', body: ( @@ -497,6 +499,7 @@ const traceSteps = [ }, { id: 'simulate', + toolName: 'run_society_simulation', stage: 'Simulate', family: 'Simulation', body: ( @@ -556,6 +559,7 @@ const traceSteps = [ }, { id: 'compute', + toolName: 'compute_budgetary_impact', stage: 'Compute', family: 'Analysis', body: ( @@ -661,28 +665,6 @@ const approvedReform = { require_approved_reform: true, }; -const societySimulationCall = `run_society_simulation( - year=2026, - reform={ - "gov.hmrc.child_benefit.amount.eldest": 30.0 - } -)`; - -const budgetaryImpactCall = `compute_budgetary_impact( - simulation_id= -)`; - -const calculationToolEvents = [ - { - id: 'society-simulation', - name: 'run_society_simulation', - }, - { - id: 'budgetary-impact', - name: 'compute_budgetary_impact', - }, -]; - const walkthroughAnswerSummary = 'Setting the Child Benefit eldest-child rate to £30 a week in 2026 would cost the government about £0.9 billion a year compared with current law. Benefit spending rises by about £1.1 billion, while tax revenue rises by about £0.2 billion because the higher rate also increases High Income Child Benefit Charge receipts.'; @@ -1010,19 +992,28 @@ function TodayWorkflow() {

How policy analysis is done today

- PolicyEngine already answers questions of this kind, but not in conversation. Someone costing a Child - Benefit change either builds the reform by hand in the web app, or writes code against the model: find - the parameter path, construct a reform, run baseline and reform over the same microdata, read off the - measure. Both routes are precise and fully inspectable, and both call the engine UK Chat calls - underneath. Both also ask the same thing of the person using them: knowing which parameter carries the - policy, how to express the change, and which output answers the question. + PolicyEngine already provides tools to answer tax and benefit questions, but not in conversational + format. For example, someone looking to determine the cost of a change to the Child Benefit can use + one of two PolicyEngine features to do so. They can build the reform by hand in the web app, setting + each parameter through the interface and reading the results off the charts it returns. Or they can + write code against the PolicyEngine country model: find the parameter path in the catalogue, construct + a reform object, run baseline and reform policies over the same survey microdata, then take the + budgetary or distributional measure from the output. Either way, a first pass usually takes several + attempts to settle the parameter, the date, and the measure.

- For a policy analyst, a journalist, or a household working out what a reform would mean for them, the - barrier is the setup rather than the arithmetic — and prompting a general-purpose language model looks - like the way around it. The rest of this article is about why that shortcut does not produce a figure - anyone can rely on. UK Chat replaces neither route, but for the questions its tools already cover it - removes the setup and leaves the calculation where it was. + Both routes are precise and fully inspectable, and both call the same engine UK Chat calls underneath. + Both also ask the same thing of the person using them: knowing which parameter carries the policy, how + to express the change, and which output answers the question. For a policy analyst, a journalist, or a + household working out what a reform would mean for them, this setup is a real barrier. +

+

+ To this user, prompting a general-purpose AI model may feel like a way around the barrier. However, as + this article covers later, this shortcut does not necessarily produce reliable figures, given the + current limitations of standard AI models. UK Chat overcomes this limitation by applying the strengths + of conversational models to the same engine that already powers the web app and country models, + reducing the setup friction mentioned earlier by providing a series of typed tools that ensure results + are still deterministically calculated and grounded in fact.

); @@ -1265,7 +1256,8 @@ function Boundary() {

The boundary

- The central design decision is where the predictive layer stops and the deterministic layer starts. + UK Chat's central design decision is where the predictive layer stops and the deterministic layer + starts.

The predictive parts are interpreting wording, deciding which kind of analysis is @@ -1282,16 +1274,15 @@ function Boundary() {

- When a question is ready for computation, the language model receives the calculation tools, but not a - list of what exists in the model. It has to ask: names are confirmed by calling discovery tools - against the deployed package, so a policy the model half-remembers either resolves to a real parameter - path or fails to resolve at all. The gateway works the same way from the other side, describing the - tools from their own schemas so its vocabulary cannot drift from what the tools actually accept. -

-

- The gateway sits on that line. It takes the plan the language model proposes and decides whether it - may cross into the deterministic side. That is why the system has three parts rather than two: the - language model, the gateway, and the tools a plan reaches once the gateway admits it. + This boundary creates a three-part request path: the language model, the gateway, and supporting + tools. The language model turns the user's open-ended prompt into a structured simulation or + analysis plan. The gateway checks whether that plan is complete and supported, constraining it or + asking the user for clarification when necessary, before calculation begins. Once the plan can + proceed, the language model receives the calculation tools, but not a static catalogue of PolicyEngine + policies and parameters. It must use discovery tools to resolve the user's wording against the + deployed PolicyEngine package, so a policy name either maps to a current parameter path or remains + unresolved. The gateway likewise derives its description of each tool from that tool's schema, + keeping its checks aligned with the inputs those tools actually accept.

); @@ -1392,22 +1383,6 @@ function ToolExplorer() { return ( -

The language model proposes a plan

-

- UK Chat breaks the user pathway into three segments: the language model, the gateway, and supporting - tools. -

-

- First, the language model does the predictive work. It takes the user's open-ended prompt and - develops a simulation or analysis plan. The plan has a required format and must use one or more of the - deterministic tools connected to the PolicyEngine UK tax and benefit simulation engine. -

-

- Next, UK Chat's gateway verifies and sometimes constrains that plan to ensure PolicyEngine - provides the tools required to answer the question. The gateway may also ask the user for - clarification before finalising a plan. -

-

Tools make the calculation deterministic

At this point, UK Chat has a gateway-verified plan, and every figure in the answer comes from the @@ -1482,8 +1457,8 @@ function AnswerWalkthrough() {

This guided reconstruction uses the real stages, tools, assumptions, and result from the Child Benefit - example; it is not a live chat session. Use the arrow buttons in the bottom-right of the stage rail to - move through the interactive. + example; it is not a live chat session. Use the arrow buttons in the bottom-right of the stage + selector panel to move through the interactive.

@@ -1508,13 +1483,18 @@ function CalculationWorkingDisclosure() { {isExpanded && (

- I'll run the approved reform and calculate its annual budgetary impact. + I'll confirm the approved reform against the public catalogue, run it, and calculate its + annual budgetary impact.

- {calculationToolEvents.map((tool) => ( -
- {tool.name} - ✓ -
+ {calculationToolSteps.map((tool) => ( +
+ + +
{tool.body}
+
))}
)} @@ -1727,6 +1707,23 @@ function CompressedUkChatPreview() { {JSON.stringify(resolveCatalogueMatch, null, 2)} +
+
Why the match can be used
+
+
+ + gov.hmrc.child_benefit.amount.eldest +
+
+ + Returned from the installed UK package, not from model memory +
+
+ + “Eldest-child rate” matches the parameter's catalogue label +
+
+
)} @@ -1837,6 +1834,29 @@ function CompressedUkChatPreview() { {JSON.stringify(approvedReform, null, 2)} +
+
Validation findings
+
+
+ + The path exists in the parameter schema +
+
+ + + 30.0 is valid for a currency-GBP parameter + +
+
+ + The 2026 date range falls inside the package's coverage +
+
+ + An unrecognised path would be rejected here +
+
+
)} @@ -1857,17 +1877,34 @@ function CompressedUkChatPreview() { aria-label="Calculate stage public tool sequence" >
-
1 · Run the approved reform
+
Five public calls, one turn
-                    {societySimulationCall}
+                    {`list_reform_targets
+  ↓ parameter path
+get_parameter
+  ↓ current value and unit
+validate_reform
+  ↓ validated reform
+run_society_simulation
+  ↓ result_id
+compute_budgetary_impact`}
-
2 · Derive the requested output
+
Typed result handoff
-                    {budgetaryImpactCall}
+                    {`run_society_simulation returns result_id
+  → passed as simulation_id
+  → compute_budgetary_impact returns computed totals`}
+
+
Provenance rule
+

+ Every argument comes from the user or a preceding call. Expand “Working through the + problem” above to inspect each call and result. +

+
)} @@ -1952,96 +1989,6 @@ function CompressedUkChatPreview() { ); } -const traceItems = traceSteps.map((step) => ({ key: step.id, step })); - -function CallTrace() { - const [activeIndex, setActiveIndex] = useState(0); - - return ( - setActiveIndex(index)} - cardsClassName="tool-cards trace-cards" - renderCard={(item, index) => ( - <> - {index + 1} - {item.step.stage} - {item.step.family} - - )} - > -
{traceSteps[activeIndex].body}
- - ); -} - -function WorkedExample() { - return ( - -

The same reform, call by call

-

- The six stages decide what the system does. Underneath, the approved plan is a short sequence of typed - calls, each taking an argument an earlier call produced. The loop re-derives what the gateway already - approved, and the match between the two is itself the check. -

- -
-
-
-
Computed result
-

About £0.9 billion a year in additional government cost

-
- 2026 -
-

- A production run compares the reform with 2026 current law using PolicyEngine's Enhanced Family - Resources Survey dataset for 2024–25, release 1.56.13. It is a direct static microsimulation - estimate. Tax revenue rises because a higher Child Benefit rate increases the High Income Child - Benefit Charge paid by higher-income families. -

-
- - - - - - - - - - - - - - - - - - - - - - -
Annual change relative to current law
MeasureChange
Tax revenue+£0.2 billion
Benefit spending+£1.1 billion
Net budgetary impact−£0.9 billion
-
-

- A negative net budgetary impact means a cost to government. Figures are rounded to the nearest £0.1 - billion. -

-
-

- The seven analysis tools are the only route from a stored simulation to that number, and every - parameter path, value, and handle in the sequence was either supplied by the user or produced by a - preceding call. Nothing in it was recalled. -

- - ); -} - const questionItems = questionKinds.map((kind) => ({ key: kind.id, kind })); function WhatYouCanAsk() { @@ -2186,7 +2133,6 @@ export default function App() { - diff --git a/src/uk-chat.css b/src/uk-chat.css index fc6f94d..d815b7a 100644 --- a/src/uk-chat.css +++ b/src/uk-chat.css @@ -1078,9 +1078,14 @@ min-width: 0; overflow: auto; overscroll-behavior: contain; + padding-top: 8px; width: 100%; } +.compressed-chat-transcript > :first-child { + margin-top: 0; +} + .compressed-chat-composer-tools { align-items: center; display: inline-flex; @@ -1263,19 +1268,42 @@ } .compressed-chat-working-tool { - align-items: center; color: var(--chat-preview-text-3); - display: flex; font-family: 'JetBrains Mono', monospace; font-size: 11px; - gap: 5px; margin: 9px 0; max-width: 100%; min-width: 0; + overflow: hidden; +} + +.compressed-chat-working-tool summary { + align-items: center; + cursor: pointer; + display: flex; + gap: 6px; + list-style: none; padding: 2px 0; } -.compressed-chat-working-tool code { +.compressed-chat-working-tool summary::-webkit-details-marker { + display: none; +} + +.compressed-chat-working-tool summary svg { + height: 11px; + opacity: 0.5; + stroke-width: 2; + transform: rotate(-90deg); + transition: transform 150ms ease; + width: 11px; +} + +.compressed-chat-working-tool[open] summary svg { + transform: none; +} + +.compressed-chat-working-tool summary code { background: none; border-radius: 0; color: var(--chat-preview-text-3); @@ -1284,10 +1312,58 @@ padding: 0; } -.compressed-chat-working-tool > span { +.compressed-chat-working-tool summary > span { color: var(--chat-preview-muted); } +.compressed-chat-working-tool summary:focus-visible { + border-radius: 4px; + outline: 2px solid var(--accent); + outline-offset: 2px; +} + +.compressed-chat-working-tool-body { + border-left: 1px solid var(--chat-preview-border); + margin: 8px 0 14px 5px; + max-width: calc(100% - 5px); + min-width: 0; + overflow-x: auto; + padding: 0 0 0 13px; +} + +.compressed-chat-working-tool-body .example-section + .example-section { + margin-top: 12px; +} + +.compressed-chat-working-tool-body .example-section-title { + font-size: 10px; + margin-bottom: 5px; +} + +.compressed-chat-working-tool-body pre.example-code { + color: var(--chat-preview-text-3); + font-size: 11px; + line-height: 1.55; + margin: 0; + overflow-x: auto; + padding: 10px 12px; + white-space: pre; +} + +.compressed-chat-working-tool-body .example-code .comment, +.compressed-chat-working-tool-body .example-code .keyword, +.compressed-chat-working-tool-body .example-code .string, +.compressed-chat-working-tool-body .example-code .number, +.compressed-chat-working-tool-body .example-code .error, +.compressed-chat-working-tool-body .example-code .success { + color: inherit; +} + +.compressed-chat-working-tool-body .example-output-line, +.compressed-chat-working-tool-body .example-file { + font-size: 11px; +} + .compressed-chat-working-toggle:focus-visible { border-radius: 4px; outline: 2px solid var(--accent); @@ -1441,6 +1517,42 @@ margin-bottom: 6px; } +.compressed-chat-evidence-list { + display: grid; + gap: 7px; +} + +.compressed-chat-evidence-list > div { + align-items: start; + color: var(--chat-preview-text-3); + display: grid; + font-size: 0.78rem; + gap: 7px; + grid-template-columns: 14px minmax(0, 1fr); + line-height: 1.45; +} + +.compressed-chat-evidence-list code { + background: transparent; + color: var(--chat-preview-text-3); + font-size: inherit; + overflow-wrap: anywhere; + padding: 0; +} + +.article-wrapper .compressed-chat-technical-note { + color: var(--chat-preview-text-3); + font-family: + system-ui, + -apple-system, + BlinkMacSystemFont, + 'Segoe UI', + sans-serif; + font-size: 0.78rem; + line-height: 1.5; + margin: 0; +} + .compressed-chat-ground-code { background: #fff; border: 1px solid var(--chat-preview-border); @@ -1622,134 +1734,6 @@ padding: 7px 9px; } -/* Five calls, one row, following .question-cards. Like that rule this has to - stay above the breakpoints below, or the 2- and 1-column fallbacks lose. */ -.trace-cards { - grid-template-columns: repeat(5, minmax(0, 1fr)); -} - -.trace-cards .iteration-card { - min-height: 152px; -} - -.trace-cards .iteration-num { - margin: 0 auto 12px; -} - -.trace-cards .iteration-title, -.trace-cards .iteration-subtitle { - display: block; -} - -.trace-cards .iteration-title { - margin-bottom: 6px; -} - -.worked-result { - background: linear-gradient(145deg, #f7fbfa, #eef7f6); - border: 1px solid var(--border-accent); - border-radius: 14px; - margin: 40px 0 32px; - padding: 30px; -} - -.worked-result-heading { - align-items: flex-start; - display: flex; - gap: 24px; - justify-content: space-between; -} - -.worked-result-eyebrow { - color: var(--accent); - font-family: 'JetBrains Mono', monospace; - font-size: 0.7rem; - font-weight: 700; - letter-spacing: 0.1em; - margin-bottom: 6px; - text-transform: uppercase; -} - -.worked-result h3 { - margin: 0 0 20px; -} - -.worked-result-year { - background: var(--accent); - border-radius: 999px; - color: #fff; - flex: 0 0 auto; - font-family: 'JetBrains Mono', monospace; - font-size: 0.75rem; - font-weight: 700; - padding: 6px 11px; -} - -.worked-result-table-wrap { - overflow-x: auto; -} - -.worked-result-table { - background: #fff; - border: 1px solid var(--border); - border-collapse: collapse; - border-radius: 10px; - margin: 22px 0 14px; - overflow: hidden; - width: 100%; -} - -.worked-result-table caption { - color: var(--text-mid); - font-family: 'JetBrains Mono', monospace; - font-size: 0.72rem; - font-weight: 600; - letter-spacing: 0.04em; - padding-bottom: 8px; - text-align: left; - text-transform: uppercase; -} - -.worked-result-table th, -.worked-result-table td { - border-bottom: 1px solid var(--border); - font-size: 1rem; - padding: 13px 16px; - text-align: left; -} - -.worked-result-table thead th { - background: var(--accent-light); - color: var(--text-dark); - font-family: 'JetBrains Mono', monospace; - font-size: 0.72rem; - letter-spacing: 0.05em; - text-transform: uppercase; -} - -.worked-result-table td, -.worked-result-table thead th:last-child { - font-variant-numeric: tabular-nums; - text-align: right; -} - -.worked-result-table tr:last-child th, -.worked-result-table tr:last-child td { - border-bottom: 0; -} - -.worked-result-total th, -.worked-result-total td { - color: var(--accent); - font-weight: 700; -} - -.worked-result-note { - color: var(--text-light); - font-size: 0.9rem; - margin: 0; -} - /* Both placements centre the button and carry the beta marker, so the offer reads the same in the introduction and at the end. */ .article-cta-wrap { @@ -1952,19 +1936,6 @@ display: block; } - .worked-result { - padding: 24px 18px; - } - - .worked-result-heading { - display: block; - } - - .worked-result-year { - display: inline-block; - margin-bottom: 16px; - } - .walkthrough-prompt { padding: 21px 18px 23px; } @@ -2019,11 +1990,6 @@ .compressed-chat-stage-map { padding-top: 0; } - - .worked-result-table th, - .worked-result-table td { - padding: 11px 10px; - } } @media (prefers-reduced-motion: reduce) { @@ -2094,22 +2060,6 @@ padding-right: 20px; } -/* The tool-call trace reuses the legacy .example-* vocabulary, which was - written for the sticky panel of the older article. Two things differ here: - the calls are 
 so the line breaks survive, and .example-section's 2px
-   gap is too tight once the blocks sit inside body-width prose. */
-.call-trace pre.example-code {
-  margin: 0;
-}
-
-.call-trace .example-section + .example-section {
-  margin-top: 18px;
-}
-
-.call-trace .example-file {
-  margin-bottom: 0;
-}
-
 /* The tool cards and the stage narrative borrow the same vocabulary, at a
    smaller size: only one example is on screen at a time, and neither panel may
    grow taller than what sits above it. */
@@ -2142,7 +2092,6 @@
 /* The generated JSON keys and elided values run past the panel on a phone;
    the code block scrolls rather than the page. */
 @media (max-width: 600px) {
-  .call-trace pre.example-code,
   .tool-example pre.example-code,
   .stage-example pre.example-code {
     font-size: 0.72rem;

From e620100c4152f7b2c496314fbe47f35ef4968796 Mon Sep 17 00:00:00 2001
From: Anthony Volk <14987227+anth-volk@users.noreply.github.com>
Date: Mon, 17 Aug 2026 17:01:10 +0300
Subject: [PATCH 3/8] Make the answer walkthrough scroll-driven

---
 src/App.jsx     | 71 +++++++++++++++++++++++++++++++++++++++++++------
 src/uk-chat.css | 54 +++++++++++++++++++++++++++++++++++++
 2 files changed, 117 insertions(+), 8 deletions(-)

diff --git a/src/App.jsx b/src/App.jsx
index d505bc2..df53bbf 100644
--- a/src/App.jsx
+++ b/src/App.jsx
@@ -1447,6 +1447,46 @@ function ToolExplorer() {
 }
 
 function AnswerWalkthrough() {
+  const prefersReducedMotion = useReducedMotion();
+  const [activeStage, setActiveStage] = useState(0);
+  const stageRefs = useRef([]);
+  const stageRatios = useRef(new Map());
+
+  useEffect(() => {
+    if (typeof IntersectionObserver === 'undefined') return undefined;
+
+    const ratios = stageRatios.current;
+    const observer = new IntersectionObserver(
+      (entries) => {
+        entries.forEach((entry) => {
+          const index = Number(entry.target.dataset.stage);
+          ratios.set(index, entry.isIntersecting ? entry.intersectionRatio : 0);
+        });
+
+        const mostVisible = [...ratios.entries()]
+          .filter(([, ratio]) => ratio > 0)
+          .sort(([, firstRatio], [, secondRatio]) => secondRatio - firstRatio)[0];
+
+        if (mostVisible) setActiveStage(mostVisible[0]);
+      },
+      { rootMargin: '-74px 0px -45% 0px', threshold: [0, 0.15, 0.4, 0.7] },
+    );
+
+    stageRefs.current.forEach((node) => node && observer.observe(node));
+    return () => {
+      observer.disconnect();
+      ratios.clear();
+    };
+  }, []);
+
+  const selectStage = (index) => {
+    setActiveStage(index);
+    stageRefs.current[index]?.scrollIntoView({
+      behavior: prefersReducedMotion ? 'auto' : 'smooth',
+      block: 'center',
+    });
+  };
+
   return (
     
       
See one answer being built

@@ -1457,10 +1497,26 @@ function AnswerWalkthrough() {
       

       

         This guided reconstruction uses the real stages, tools, assumptions, and result from the Child Benefit
-        example; it is not a live chat session. Use the arrow buttons in the bottom-right of the stage
-        selector panel to move through the interactive.
+        example; it is not a live chat session. Scroll to move through the six stages, or use the stage
+        selector and arrow buttons to move directly between them.
       

-      
+      

+        

+          
+        

+        
     
   );
 }
@@ -1502,9 +1558,8 @@ function CalculationWorkingDisclosure() {
   );
 }
 
-function CompressedUkChatPreview() {
+function CompressedUkChatPreview({ activeStage, onStageChange }) {
   const prefersReducedMotion = useReducedMotion();
-  const [activeStage, setActiveStage] = useState(0);
   const [animationStarted, setAnimationStarted] = useState(false);
   const [animationPhase, setAnimationPhase] = useState('idle');
   const [typedRequest, setTypedRequest] = useState('');
@@ -1945,7 +2000,7 @@ compute_budgetary_impact`}
-
-
Typed result handoff
- 
-                    {`run_society_simulation returns result_id
-  → passed as simulation_id
-  → compute_budgetary_impact returns computed totals`}
-
-
Provenance rule

From fa9d1f3615b6d96f63f1c7d69cb98e68dc5463c4 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Mon, 17 Aug 2026 20:22:57 +0300 Subject: [PATCH 8/8] Link current PolicyBench results --- content/current-article.md | 2 +- src/App.jsx | 5 ++--- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/content/current-article.md b/content/current-article.md index b5295ef..ad503c3 100644 --- a/content/current-article.md +++ b/content/current-article.md @@ -28,7 +28,7 @@ Prompting does not make a language model apply tax and benefit rules. Telling it This matters when results inform decisions, as the National Audit Office's review of AI in government illustrates. Anyone comparing two reforms needs figures from a fixed microsimulation, not an AI's recollection of figures it has seen. The same applies to a household working out how a reform affects its income: the figure has to come from clear rules, with assumptions written down, so that it can be checked. -We have measured this directly. In PolicyBench, an evaluation built on the US tax and benefit system, we scored how accurately AI models compute taxes and benefits from household prompts with no tools and no lookups, against deterministic PolicyEngine outputs. As of August 17, 2026, the top language model matches PolicyEngine exactly on 80.3% of its scored outputs. Computed amounts scored lowest: income tax before credits scored far below eligibility flags, because getting it right means sequencing income concepts, thresholds, exclusions, and credits in the correct order. +We have measured this directly. In [PolicyBench](https://policybench.org), an evaluation built on the US tax and benefit system, we scored how accurately AI models compute taxes and benefits from household prompts with no tools and no lookups, against deterministic PolicyEngine outputs. As of August 17, 2026, the top language model matches PolicyEngine exactly on 88.7% of its scored outputs. Computed amounts scored lowest: income tax before credits scored far below eligibility flags, because getting it right means sequencing income concepts, thresholds, exclusions, and credits in the correct order. ## The boundary diff --git a/src/App.jsx b/src/App.jsx index ed6da8a..c2b1e9d 100644 --- a/src/App.jsx +++ b/src/App.jsx @@ -1044,11 +1044,10 @@ function Problem() { checked.

- We have measured this directly. In{' '} - PolicyBench, an evaluation + We have measured this directly. In PolicyBench, an evaluation built on the US tax and benefit system, we scored how accurately AI models compute taxes and benefits from household prompts with no tools and no lookups, against deterministic PolicyEngine outputs. As of - August 17, 2026, the top language model matches PolicyEngine exactly on 80.3% of its + August 17, 2026, the top language model matches PolicyEngine exactly on 88.7% of its scored outputs. Computed amounts scored lowest: income tax before credits scored far below eligibility flags, because getting it right means sequencing income concepts, thresholds, exclusions, and credits in the correct order.