Repository navigation
Conversation
Saved from a build that stopped when the machine rebooted. The next build verifies this work and finishes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add two tests for a dataset that stores latent rates, as Microcosm does. One shows deductions, the applied cap and the award match origin/main under current law and any floor. The other shows a cut in incidence drops stored rates while a rise adds none. The docs now name every condition the flag checks and say what the floor limits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An 85% floor allowance of £720.25 exceeds a £500 award that the cap takes whole, so the floor does not bind and the family keeps no award for deductions to come from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For a claimant the incidence draw assigns, uc_has_deduction now must hold exactly where an award remains after the cap reduction that allocate_uc_reductions applies with no deductions, under any floor. A new example family has a £500 award the cap takes whole, below an 85% floor's £720.25 allowance. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Aggregate impact on certified data: passBase Checks: 0 broken by this change, 1 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds in Notes
Moves flagged for reviewNone. Spending by benefit and service
Caseloads
Standard reform costings (change in gov_balance)
Standard reforms: change in each reform's target
Household net income by decileNo household's net income changes in any year. No change in any year: receipts by tax head; government balance, each component counted once; model totals; household income totals; population; poverty rates; people in poverty; inequality; mean household net income in each side's own deciles. Checks failing on both base and head (1)
Counted-once checkEach of 91 components was raised by £1 per unit in 2026-27, and every head of its totals was read household by household: its own head must rise by exactly £1, no other tax head may move, no other benefit may rise, and the model's own total must move by what its heads moved. A head that breaks a rule is measured again twice: with every other head held at its baseline value, and with every other head held at its value after the first £1 while the component rises again. It fails where either still shows the response. 0 failed; 0 were skipped.
Provenance
Dataset: Receipts by tax head: head levels
Spending by benefit and service: head levels
Caseloads: head levels
Government balance, each component counted once: head levels
Model totals: head levels
Household income totals: head levels
Population: head levels
Poverty rates: head levels
People in poverty: head levels
Inequality: head levels
Mean household net income in each side's own deciles: head levels
Standard reform costings (change in gov_balance): head levels
Standard reforms: change in each reform's target: head levels
|
Fixes #2739
What and why
The bug.
uc_latent_deduction_rate's formula returns zero whereuc_has_deductionis false.uc_deduction_rateand the advance estimate read the latent rate itself, so a rate that a dataset or situation supplies skipped the flag. In the issue's example, an 85% protected floor gave £720.25 of deductions to a family whose flag was false.The fix (option 1). A new helper,
assigned_latent_deduction_ratein_deduction_amounts.py, returns the latent rate whereuc_has_deductionis true and zero elsewhere.uc_deduction_ratereads it for the capped part and for the above-cap excess, and the advance estimate reads it too. A supplied rate now follows the formula's rule. Two explicit inputs still apply whatever the flag says, because each describes a known deduction rather than a draw from the distribution:uc_advance_repayment_amount, whose documented override (zero included) this keeps;uc_deduction_rateitself.The code supports option 1:
packages/microcosm-build/src/microcosm/build/uk_runtime/uc_deduction_attributes.py, Microcosmorigin/mainat98aef7529) stores four columns: the two draws,uc_latent_deduction_rateanduc_deduction_combination. It writes a rate only where the incidence draw falls below the region's incidence (rates = np.where(assigned, rates, 0.0)). It stores nouc_has_deduction, so policyengine-uk computes the flag from the stored draw. The gate changes no deduction on such a dataset, under current law or a floor (differential below).origin/main(e1acdf96) mentions none of the deduction inputs (the two draws, the flag, the latent rate, the combination and the advance amount), so the Enhanced FRS takes the formula path. The gate leaves that path unchanged under current law.The flag under a protected floor: changed, with no change under current law.
uc_has_deductionread the award left after the gross benefit cap. Under a floor,allocate_uc_reductionsmeets deductions first from the award before the cap, up to the floor allowance (uc_maximum_reduction). The cap reduction then takes whatever allowance is left. So a family that the gross cap takes to nil keeps its award less the allowance, where the award exceeds the allowance, and deductions can come from what it keeps. The flag now readsuniversal_credit_pre_benefit_cap − min(uc_benefit_cap_reduction, uc_maximum_reduction).uc_maximum_reductionis infinite, and the minimum is the gross cap. The differential below finds the formula path bit-equal toorigin/main.origin/main's deductions. With the gate alone, a family on a stored-rate dataset would lose its deductions under a floor. That family is one whose draw assigns deductions and whose award the cap takes whole.origin/maintakes them from its stored rate. On the property module's 24-unit stored dataset under an 85% floor, the gate alone (mutation M3 below) drops the deductions of 4 units, by up to £720.25 each. The floor-aware flag keeps them. On a dataset that stores rates as Microcosm does, deductions, the applied cap and the award then equalorigin/main's, and the formula path gives the same.uc_award_carries_deductionsis true for any family with an award before the cap in the baseline.What users see.
uc_latent_deduction_rategets no deductions unless it also setsuc_has_deductionto true, or supplies an incidence draw that assigns them. Ten existing YAML cases (five inuc_deductions.yaml, five inuc_fixed_deductions.yaml) and three Python test files supplied a rate this way. They now set the flag, or intest_uc_tax_free_childcare_choice.pyan incidence draw of 0.2, below London's 0.439. No expected value changed.origin/main, incidence reforms never reached stored rates. A unit that keeps the flag keeps its stored rate, while the formula rescales each draw by the new incidence, so stored rates don't cost an incidence reform in full.docs/book/validation/uc-deductions.mdand the docstrings of the flag, the latent rate and the deduction rate state these rules.Stored data.
uc_has_deductionstays an engine formula over the stored draw, and its award test reads only reformable model outputs. A dataset that stores latent rates must now also store the draws that assign them. Microcosm stores both.Invariants
These hold for every input.
policyengine_uk/tests/test_uc_deduction_flag_properties.pychecks properties 1 to 9 with Hypothesis over generated benefit units plus explicit examples, and property 10 on a fixed 24-unit dataset.uc_has_deductionis false,uc_deductionsis zero, for any supplied latent rate, combination, gross cap reduction and floor. The documented exceptions are an explicitly supplieduc_advance_repayment_amount, which the property covers, and an explicitly supplieduc_deduction_rate, which replaces the formula altogether.where(uc_has_deduction, rate, 0).uc_deductionsequalsallocate_uc_reductionsapplied to the ungated demand, under any floor.uc_deductions + uc_benefit_cap_reduction_after_floor + universal_creditequals the award before the cap.allocate_uc_reductionsapplies with no deductions, under any floor.origin/main's rule, bit for bit.origin/main(differential). On that dataset, deductions, the applied cap and the award equalorigin/main's under current law and under any floor.Properties 6, 7 and 9 compare head with
MAIN_RULE, which adds backorigin/main's flag, rate and advance formulas (as of7ed52038a;origin/mainat7bcf80b09has the same deduction code) as a reform. A separate cross-version differential, below, runs the realorigin/maintree.Test plan
uc_has_deductionanduc_latent_deduction_rateare imputations undergov.simulation, and the protected floor is a reform, zero under current law. The new YAML cases derive their expected values in comments from the 2025-26 standard allowance for a single claimant aged 25 or over, £400.14 a month.uvx ruff@0.17.2 format --checkanduvx ruff@0.17.2 checkon the eight changed Python files,uvx ruff@0.17.2 check --select F811,F821 policyengine_uk, andgit diff --check: all clean.changelog.d/2739.fixed.md.Results at head
e17612ec3, each file or node-id group run alone with-n 0:origin/main(7bcf80b09) with the new teststest_uc_deduction_flag_properties.py(new, at the final head and on02b380c1a)MAIN_RULE, and 3 cover rulesorigin/mainalready follows)uc_deductions.yaml(14 cases, 9 new) anduc_fixed_deductions.yaml(11)uc_deductions.yaml: 3 failed, 11 passed. The 3 are the supplied rate alone, the supplied rate under a floor without the flag, and the floor-flagged family.uc_fixed_deductions.yaml: 11 passed.test_uc_deductions.pytest_uc_advance_repayment_properties.pytest_uc_tax_free_childcare_choice.pytest_data_built_imputations.pytest_uc_rebalancing.py -k deductionstest_dataframe_dataset.py::test_a_frame_over_several_years_exports_the_same_frame_again, which lists the formulas that read the deduction drawsbaseline/finance/benefit/family/universal_credit/integration/uc_disabled_child_addition.yaml,baseline/gov/dwp/benefit_cap/,benefit_cap_by_scheme.yaml, the HB UC passport, CTR pensioners)test_benefit_cap_lha_scheme_split_properties.pyandtest_benefit_cap_and_ctr_pension_age_properties.pyThe
universal_credit/YAML directory ran atd143bcf77. Head's model code is the same, anduc_deductions.yaml, the only YAML file changed since, passes at head.Cross-version differential. One hand-built dataset of 400 single-adult benefit units (seeded, all 13 regions, awards up to £20,000, gross cap shares from 0 to 2, about 85% claiming) ran under the
origin/maintree and under head. It compares 13 variables in six variants:origin/mainuniversal_creditstays the same on all 400 units.Mutations. Seven mutations of the fix, each in a scratch copy, run against
uc_deductions.yamland the property module (-x):uc_deductions.yamluc_deduction_ratereads the ungated latent rateorigin/maintest_a_supplied_deduction_rate_applies_whatever_the_flag_says)draw <= incidencetest_a_draw_equal_to_the_incidence_assigns_no_deductions)M1 to M4 ran against the property module at
d98c5b7f4, before property 5 existed. M5 ran against the module ate17612ec3, and M6 and M7 against the final head's three explicit tests.Overlap.
ab26-autumn-budget-2026) touches none of these files.docs/book/validation/uc-deductions.mdandtest_uc_deductions.py, in different hunks.git merge-treemerges it with this branch cleanly. On the merged tree,uc_deductions.yamlandbaseline/gov/dwp/benefit_cap/pass (138), and so do Report the benefit cap reduction actually applied to Universal Credit #2647'stest_benefit_cap_reduction_applied_properties.pytests: its four example tests pass, and its main property test passes all five of its explicit examples through its owncheck_invariants. That test's 12 generated examples did not finish inside the 25-minute cap. This PR's property module passes there too (7 passed). Report the benefit cap reduction actually applied to Universal Credit #2647'sEXAMPLE_UNITS[5]is the issue's example: the cap takes its whole £1,000 award. Under an 85% floor, Report the benefit cap reduction actually applied to Universal Credit #2647 alone (c98c879a7) gives it £720.25 of deductions with the flag false. The merge gives the same £720.25 with the flag true. Under current law both give £0 of deductions, a false flag and an applied cap of £1,000.Reproducer
Reproducer: policyengine_uk/tests/test_uc_deduction_flag_properties.py::test_the_issue_example_takes_no_deductions_without_the_flag
axiom: n/a:
uc_has_deductionanduc_latent_deduction_rateare imputations undergov.simulation, not law; the protected floor is JRF's proposal, zero under current law.🤖 Generated with Claude Code