Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,6 @@ materialization behavior, read `docs/engineering/skills/dataset-sources.md`.

For documentation-sensitive changes, read
`docs/engineering/skills/documentation-review.md`.

For release-specific variable or parameter audits against official policy
sources, read `docs/engineering/skills/policy-audit/SKILL.md`.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,9 @@ env.bak/
venv.bak/
.env*

# Local policy audit worktrees and review state
.policy-audit/

# Coverage / testing
htmlcov/
.tox/
Expand Down
3 changes: 3 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ interactions with upstream dataset materializers, read
When changing public behavior, documentation, metadata surfaces, or generated
documentation, read `docs/engineering/skills/documentation-review.md`.

When auditing a released variable or parameter against official policy sources,
read `docs/engineering/skills/policy-audit/SKILL.md`.

## Non-Negotiable PR Requirements

Every pull request must include a Towncrier changelog fragment under
Expand Down
3 changes: 3 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ upstream materialized dataset paths, read
When changing public behavior, metadata, docs, or generated docs, read
`docs/engineering/skills/documentation-review.md`.

When auditing a released variable or parameter against official policy sources,
read `docs/engineering/skills/policy-audit/SKILL.md`.

## Safety Boundaries

Do not fabricate policy sources, validation metrics, or performance claims. If a
Expand Down
1 change: 1 addition & 0 deletions changelog.d/2083.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- Corrected the 2025-26 Pension Credit standard minimum guarantee and Savings Credit thresholds using official DWP and legislation sources.
1 change: 1 addition & 0 deletions changelog.d/policy-audit-tool.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- Added a release-specific policy audit command and AI workflow for researching variables and parameters against cited UK government sources, including a cited priority queue for discrepancies discovered in related audit units.
2 changes: 2 additions & 0 deletions docs/engineering/skills/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ Current skills:
fragments, draft PR expectations, and pre-commit lint/format rules.
- `model-structure.md`: UK variables, parameters, reforms, enums, program
registry metadata, and refactoring recovery lessons.
- `policy-audit/SKILL.md`: release-specific variable and grouped-parameter
audits using official sources, bounded program context, and cited issues.
- `testing.md`: UK test layout, focused test selection, slow microsimulation
boundaries, and expected commands.

Expand Down
8 changes: 5 additions & 3 deletions docs/engineering/skills/github-prs.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,11 @@ Do not knowingly commit unformatted code or known lint failures.

## Test Expectations

Run focused tests that cover the changed behavior. Use `make test` for broad
model changes, shared behavior, or changes where a narrower command would miss
important regressions. See `testing.md` for details.
Run every new or modified test file before committing or opening the pull
request. Do not require a full repository test run locally; rely on CI for
broader regression coverage. Run a broader command only when the user requests
it or when it is needed to diagnose a failure. See `testing.md` for the focused
commands.

When tests are skipped because they are slow, data-dependent, or require
credentials, say so explicitly in the PR description or handoff.
44 changes: 44 additions & 0 deletions docs/engineering/skills/policy-audit/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
name: audit-policyengine-uk
description: Audit a variable or grouped parameter in an exact policyengine-uk release, research its program context against official UK government sources, record the result, and open both a fully cited issue and a corrective draft pull request when the release is incorrect or outdated.
---

# Audit PolicyEngine UK policy rules

Use the `policy-audit` command for release selection, queue state, context
construction, review validation, and GitHub publication. Do not choose a work
item manually when the request is to audit the next item, edit audit state by
hand, or send unvalidated prose directly to GitHub.

An audit conclusion describes the evidence found on the audit date. It never
asserts that a rule will remain current until a later date.

## Workflow

1. Read [select-work-item.md](references/select-work-item.md), check out the
requested release, and claim the next unit.
2. Read [define-program-context.md](references/define-program-context.md), then
generate and inspect the bounded context for the claimed unit.
3. Read [research-official-sources.md](references/research-official-sources.md)
and research both the encoded rule and subsequent changes.
4. If the research identifies a discrepancy in a different audit unit, read
[enqueue-follow-ups.md](references/enqueue-follow-ups.md) and create a cited
follow-up without misclassifying the selected unit.
5. Read [classify-findings.md](references/classify-findings.md) and write the
structured review.
6. If a change is needed, read
[place-code-citations.md](references/place-code-citations.md) and identify
where every supporting source belongs in the model source.
7. Validate and record the review. For a publishable conclusion, read
[publish-github-issue.md](references/publish-github-issue.md) before opening
the issue.
8. After the issue exists, read
[implement-and-publish-fix.md](references/implement-and-publish-fix.md),
implement the cited correction on a feature branch based on the current
default branch, and open a corrective draft pull request.

Keep the exact-release checkout read-only. A request to execute this workflow
includes both publication outputs for `incorrect_in_release` and
`superseded_since_release`: a cited issue and a corrective draft pull request.
Stop after the audit record only when the conclusion is not publishable or the
user explicitly requests report-only work.
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Classify and record findings

Use one conclusion:

- `current`: no discrepancy was found on the audit date.
- `incorrect_in_release`: the selected release conflicts with the rule that
applied to it.
- `superseded_since_release`: a later official change is absent.
- `announced_not_enacted`: official policy was announced but has not taken
legal effect.
- `insufficient_evidence`: available official sources do not support a firm
conclusion.
- `not_official_policy_rule`: the unit is an input, assumption, statistical
construct, or other item that cannot be verified as a policy rule.
- `program_mapping_required`: ownership must be resolved before research can be
reported accurately.

An issue is normally created only for `incorrect_in_release` and
`superseded_since_release`. Publication of `announced_not_enacted` requires an
explicit option.

The structured review must contain:

- `release_version`, `unit_id`, `audited_at`, and `conclusion`;
- a one-paragraph `summary` and `summary_evidence_ids`;
- non-empty `periods_reviewed` and `jurisdictions`;
- structured official `evidence`;
- findings, implementation parts, and code citation targets when a change is
required;
- cited `follow_ups` when research exposes a possible discrepancy in a
different audit unit; and
- an `issue_title` for publishable conclusions.

Validate and record it:

```bash
policy-audit validate review.yaml
policy-audit record review.yaml
```

Validation proves structural completeness and approved source locations. It
does not replace substantive legal review.

## Review shape

```yaml
release_version: 2.106.1
unit_id: parameter:gov.department.program.amount
audited_at: 2026-10-02
conclusion: superseded_since_release
summary: >-
One paragraph explaining the encoded rule, the official change, and why the
selected release requires an update.
summary_evidence_ids: [source-1]
periods_reviewed: [2025-26, 2026-27]
jurisdictions: [United Kingdom]
issue_title: "Update Program amount for 2026-27"
evidence:
- id: source-1
organisation: Department name
authority_type: legislation
title: "Instrument title, regulation 4"
url: https://www.legislation.gov.uk/example
final_url: https://www.legislation.gov.uk/example
accessed_at: 2026-10-02
locator: regulation 4
findings:
- statement: The release omits the amount effective from April 2026.
evidence_ids: [source-1]
implementation_parts:
- description: Add the April 2026 amount and effective date.
evidence_ids: [source-1]
code_reference_targets:
- path: policyengine_uk/parameters/gov/department/program/amount.yaml
location: values.2026-04-01.metadata.reference
evidence_ids: [source-1]
uncertainties: "Optional, precise statement of evidence or scope limitations."
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Define bounded program context

Generate context with:

```bash
policy-audit context UNIT_ID --version VERSION
```

The output separates four concepts:

- **Owning program:** the program whose rule the target primarily encodes.
- **Component:** the narrower subject within that program.
- **Calculation slice:** the target, its direct calculation inputs, and the
shortest source paths to the registered headline result.
- **External consumers:** calculations owned by other programs that use the
target.

Treat `programs.yaml` as the initial registry, not as sufficient evidence for
broad agency-level prefixes. The tool may return `provisional` or
`program_mapping_required`. Resolve that state with a curated override before
making program-wide claims.

Releases created before `programs.yaml` existed have no automatic ownership
mapping. Their inventory is still valid, but context generation returns
`program_mapping_required` until a reviewed override supplies the owner.

## Scope rules

- For a parameter, inspect every direct variable reader and the variables that
select its breakdown keys.
- For a variable, inspect its direct parameters and variable inputs.
- Follow the shortest downstream paths to the headline program variable.
- Read directly relevant tests and nearby program documentation.
- List cross-program readers separately; dependency does not transfer
ownership.
- Do not load an entire program directory merely because the target belongs to
that program.
- Expand the slice only when the official rule couples the target to another
component, and explain the expansion in the review.

Example: the Universal Credit standard-allowance parameter includes all four
claimant-type amounts, `uc_standard_allowance`, the claimant-type selection,
and the path into the final award. It does not automatically include childcare,
housing costs, or every eligibility condition.

Example: the UK income-tax rate scale includes its complete brackets, the
earned-income calculations that consume them, jurisdiction routing between UK
and Scottish rates, and other direct consumers. This does not certify the
complete savings, pension, Gift Aid, or Scottish income-tax implementations.
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Enqueue a related audit follow-up

Do not classify the selected unit as incorrect merely because research exposes
a discrepancy in another parameter or variable. Add a cited follow-up for the
actual target unit instead:

```yaml
follow_ups:
- unit_id: parameter:gov.department.program.threshold
reason_code: suspected_incorrect_in_release
reason: The released threshold differs from the official amount.
observed_model: The release contains £100 from April 2026.
expected_policy: The official rates document specifies £110.
evidence_ids: [rates-2026]
model_locations:
- path: policyengine_uk/parameters/gov/department/program/threshold.yaml
location: values.2026-04-01
```

The target must exist in the same release catalog. Each model path must belong
to that target, and discrepancy-based follow-ups require official evidence from
the originating review. Recording the review creates or augments one pending
follow-up per release and target unit.

For a discrepancy discovered after the originating review was recorded, create
a standalone request with the same fields plus `release_version`,
`origin_review`, and `evidence`:

```yaml
release_version: 2.106.1
unit_id: parameter:gov.department.program.threshold
origin_review:
unit_id: parameter:gov.department.program.rate
audited_at: 2026-10-02
reason_code: suspected_incorrect_in_release
reason: The released threshold differs from the official amount.
observed_model: The release contains £100 from April 2026.
expected_policy: The official rates document specifies £110.
evidence_ids: [rates-2026]
model_locations:
- path: policyengine_uk/parameters/gov/department/program/threshold.yaml
location: values.2026-04-01
evidence:
- id: rates-2026
organisation: Department name
authority_type: official rates
title: Benefit rates 2026 to 2027
url: https://www.gov.uk/example
final_url: https://www.gov.uk/example
accessed_at: 2026-10-02
locator: Program table, threshold row
```

```bash
policy-audit enqueue-follow-up follow-up.yaml
policy-audit follow-ups --version 2.106.1
```

Recording a review of the target resolves its pending follow-up and retains the
origin and resolution history. Close a false or obsolete follow-up only with an
explicit explanation:

```bash
policy-audit close-follow-up UNIT_ID --version 2.106.1 --reason "Explanation"
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Implement and publish the correction

After publishing an issue for `incorrect_in_release` or
`superseded_since_release`, implement the correction and open a draft pull
request. Do not treat publication of the issue as completion of the audit
workflow.

## Separate the audited release from the correction

- Keep the checkout for the audited release unchanged so the recorded evidence
continues to describe that exact version.
- Apply the correction on a feature branch based on the current upstream
default branch. Never push directly to the default branch.
- Before editing, check whether the current default branch already contains the
correction. If it does, search all pages of open and closed pull requests and
link the existing implementation instead of creating an empty pull request.

## Implement the reviewed parts

- Use the issue's implementation parts as the scope. If implementation reveals
another coupled defect, correct the issue description and audit record before
expanding the pull request.
- Add each official source to the relevant parameter value metadata or variable
reference as described in
[place-code-citations.md](place-code-citations.md).
- Add focused regression tests for the affected periods, household types, and
jurisdictions. Run each new or modified test file, but do not require a full
repository test run before publishing the draft pull request; rely on CI for
broader regression coverage.
- Follow the repository's model structure, testing, documentation review, and
pull request guidance. Run formatting and add the required Towncrier
changelog fragment.

## Publish the draft pull request

- Commit and push only the feature branch.
- Open the pull request as a draft. Put `Fixes #ISSUE_NUMBER` on the first line
of its description.
- Summarize the correction, name the official sources embedded in the model,
and list every validation command and result.
- Verify the pull request directly after creation and record or report its URL.
If publication fails, report the exact failure; do not describe the issue-only
state as a completed publishable audit.
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Place citations in model source

Every proposed policy change must identify where its official sources belong
when the issue is implemented.

- A source supporting one dated parameter value belongs in that value's
`metadata.reference`.
- A source supporting every member or date in a parameter belongs in the
closest common parameter metadata.
- A source supporting formula logic, eligibility, or a definition belongs in
the variable's `reference` attribute.
- When different clauses support different pieces of logic, use multiple
references and identify the relevant code or documentation location.
- Prefer a specific regulation, schedule, rates document, or official guidance
section over a general program page.

Each implementation part must include `code_reference_targets`. Every target
contains a parameter or variable source path, a precise YAML or Python
location, and the evidence identifiers to place there. The issue renderer uses
these fields to make citation work an explicit part of implementation.
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Publish a cited GitHub issue

Render the issue before publication:

```bash
policy-audit render-issue review.yaml
```

The issue must contain a cited summary, visible release and unit metadata,
findings, implementation parts, code citation placements, affected periods,
and jurisdictions. Do not add hidden identifiers.

Publish only the validated structured review:

```bash
policy-audit publish-issue review.yaml --repository PolicyEngine/policyengine-uk
```

The publisher checks a previously recorded issue directly and fetches every
page of open and closed issues. It compares the exact title, release version,
and visible audit-unit identifier before creating anything. If the direct and
exhaustive results disagree, stop and reconcile them rather than creating a
second issue.
Loading
Loading