From 0acc9f747b635e5140f6016f95abca3db379bfd5 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Tue, 20 Oct 2026 17:11:29 -0400 Subject: [PATCH 01/13] Count Carer's Allowance earnings-limit pence exactly is_gainfully_employed_for_carers_allowance floored the weekly pence in float32, where the 1e-6 tolerance does nothing near 20,000 pence, so float32 rounding decided cases within about 0.002 pence of a boundary. Earnings of 52 x L + 0.52 a year (a penny a week over) counted as within the 2020-21 to 2024-25 limits of 128, 132, 139 and 151. The formula now takes the stored annual earnings to the nearest penny in float64 (an exact half penny disregarded, reg. 5), spreads them over 52 weeks in whole pence with the fraction disregarded (reg. 5), and compares with the limit in pence (reg. 8(1)). Tests: every limit in earnings_limit.yaml in every year it applies, entered as earnings and as pay; a pounds-and-pence reformed limit; property tests for exact whole-pence judgement, agreement with exact rational arithmetic and monotonicity; YAML cases for 2024-25. The property-test helpers now restate the new rule in Fraction arithmetic. Fixes #2541 Co-Authored-By: Claude Opus 5.5 --- changelog.d/2541.fixed.md | 1 + .../dwp/carers_allowance_earnings_limit.yaml | 29 +++ ...ers_allowance_earnings_limit_properties.py | 244 ++++++++++++++++-- ..._contribution_deduction_rate_properties.py | 30 ++- ...gainfully_employed_for_carers_allowance.py | 23 +- 5 files changed, 290 insertions(+), 37 deletions(-) create mode 100644 changelog.d/2541.fixed.md diff --git a/changelog.d/2541.fixed.md b/changelog.d/2541.fixed.md new file mode 100644 index 0000000000..c08b9d21f2 --- /dev/null +++ b/changelog.d/2541.fixed.md @@ -0,0 +1 @@ +- Fixed the Carer's Allowance earnings limit treating earnings a penny a week over the 2020-21 to 2024-25 limits (£128, £132, £139 and £151) as within them. The test now takes annual earnings to the nearest penny before spreading them over 52 weeks, so float32 rounding no longer decides it. diff --git a/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml b/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml index d33549c435..8571bb67dd 100644 --- a/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml +++ b/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml @@ -321,3 +321,32 @@ carers_allowance_earnings: 10_518.20 is_gainfully_employed_for_carers_allowance: false carers_allowance_pre_overlap: 86.45 * 52 + +- name: Case 21, in 2024-25 a penny a week under the next whole penny over £151 is within the limit. + period: 2024 + absolute_error_margin: 0.01 + input: + age: 45 + employment_income: 7_852.51 + care_hours: 40 + country: ENGLAND + output: + # 7,852.51 / 52 = £151.0098, which counts as £151.00 (reg. 5). + carers_allowance_earnings: 7_852.51 + is_gainfully_employed_for_carers_allowance: false + carers_allowance: 81.90 * 52 + +- name: Case 22, in 2024-25 a whole penny over £151 is over the limit. + period: 2024 + absolute_error_margin: 0.01 + input: + age: 45 + employment_income: 7_852.52 + care_hours: 40 + country: ENGLAND + output: + # 7,852.52 / 52 = £151.01. Float32 holds 7,852.52 as 7,852.52001953125; + # counting pence in float32 used to give £151.00 (#2541). + carers_allowance_earnings: 7_852.52 + is_gainfully_employed_for_carers_allowance: true + carers_allowance: 0 diff --git a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py index 1613bc2003..1ce6e4ecaf 100644 --- a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py +++ b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py @@ -8,34 +8,89 @@ (Computation of Earnings Regulations 1996 regs. 10(4), 13 and 14; reg. 14(2)(a), which deducted Class 2, was omitted from 6 April 2024). +A fraction of a penny is disregarded where counting it would not favour the +claimant (reg. 5). The model holds annual earnings in float32, so the formula +first takes them to the nearest penny (an exact half penny disregarded), then +compares the whole pence a week, after spreading over 52 weeks, with the +limit. _over_limit restates that rule in exact rational arithmetic. + Invariants, for a carer in England in 2026 caring 35 hours or more, with pay, trading profit, employee pension contributions and unearned income drawn freely: 1. Bounds: 0 <= carers_allowance_earnings <= pay + max(0, profit). -2. Differential: the test is passed exactly when - carers_allowance_earnings / 52, with a fraction of a penny disregarded - (Computation of Earnings Regulations 1996 reg. 5), is at most 204, and the - allowance is the full rate then and nil otherwise. The pence are counted - in the precision the model stores earnings in (float32), as the formula - counts them. +2. Differential: the test is passed exactly when _over_limit is false for the + stored carers_allowance_earnings, and the allowance is the full rate then + and nil otherwise. 3. Monotonic: raising pay never lowers net earnings, so a person over the limit stays over it. 4. Unearned income is not earnings: savings interest and pension income leave carers_allowance_earnings unchanged. + +Invariants of the test itself, for every limit in earnings_limit.yaml in every +year it applies, with carers_allowance_earnings entered directly: + +5. Whole pence are judged exactly: annual earnings of A pence are over the + limit L exactly when A // 52 > 100 x L, that is A >= 52 x (100 x L + 1), + whatever float32 does to A. So 52 x L + £0.51 a year is within the limit + and 52 x L + £0.52 (a penny a week more) is over it. +6. Exact arithmetic: for any stored amount, whole pence or not, the formula + agrees with _over_limit. +7. Monotonic in earnings: more earnings never take a person from over the + limit to within it. """ +import datetime +import math +from fractions import Fraction +from pathlib import Path + import numpy as np import pytest +import yaml from hypothesis import HealthCheck, example, given, settings from hypothesis import strategies as st +import policyengine_uk from policyengine_uk import Simulation +from policyengine_uk.system import system from policyengine_uk.utils.class_2 import class_2_contribution_weeks LIMIT = 204 RATE = 86.45 WEEKS = 52 +LIMIT_FILE = ( + Path(policyengine_uk.__file__).parent + / "parameters/gov/dwp/carers_allowance/earnings_limit.yaml" +) + + +LIMIT_VALUES = sorted( + (date, entry["value"]) + for date, entry in yaml.safe_load(LIMIT_FILE.read_text())["values"].items() +) +# The last year the parameter tree lists a value for. +HORIZON = max( + int(value.instant_str[:4]) + for value in system.parameters.gov.dwp.carers_allowance.earnings_limit.values_list +) + + +def _limits_by_year(last_year): + """Each year from the first limit in earnings_limit.yaml to last_year, + with the limit the file sets for it: the value in force on 30 April, after + that year's April up-rating.""" + limits = [] + for year in range(LIMIT_VALUES[0][0].year, last_year + 1): + in_force = [v for date, v in LIMIT_VALUES if date <= datetime.date(year, 4, 30)] + limits.append((year, in_force[-1])) + return limits + + +# Every year each limit applies, the last one up to the horizon. +LIMITS_BY_YEAR = _limits_by_year(HORIZON) +# The same, up to the year the last limit took effect. +SET_YEARS = _limits_by_year(LIMIT_VALUES[-1][0].year) def _person(pay, profit, pension, savings=0.0, private_pension=0.0): @@ -57,8 +112,8 @@ def _person(pay, profit, pension, savings=0.0, private_pension=0.0): }, } sim = Simulation(situation=situation) - # Keep each variable's stored dtype: the reg. 5 check must count pence in - # the precision the formula does. Tolerance checks convert with float(). + # Keep each stored value as the formula reads it (float32 for amounts). + # Tolerance checks convert with float(), which is exact. return { name: np.asarray(sim.calculate(name, 2026))[0] for name in ( @@ -69,13 +124,14 @@ def _person(pay, profit, pension, savings=0.0, private_pension=0.0): } -def _over_limit(earnings): - # Regulation 5: a fraction of a penny is disregarded. ``earnings`` is the - # stored value, so this counts the pence as - # is_gainfully_employed_for_carers_allowance does, in float32: promoting - # to float64 first floors 10,608.52 a year, stored as 10,608.51953125, to - # 204.00 a week instead of 204.01. - return np.floor(earnings / WEEKS * 100 + 1e-6) / 100 > LIMIT +def _over_limit(earnings, limit=LIMIT): + # The formula's rule in exact rational arithmetic, on the stored value: + # the nearest penny a year, an exact half penny disregarded, then whole + # pence a week, a fraction of a penny disregarded (reg. 5), compared with + # the limit. £10,608.52 a year, stored as 10,608.51953125, is 1,060,852 + # pence a year and 20,401 pence (£204.01) a week, a penny over £204. + annual_pence = math.ceil(Fraction(float(earnings)) * 100 - Fraction(1, 2)) + return annual_pence // WEEKS > Fraction(str(limit)) * 100 money = st.floats(min_value=0, max_value=60_000, allow_nan=False) @@ -91,8 +147,8 @@ def _over_limit(earnings): @given(pay=money, profit=profit_money, pension_share=st.floats(0, 0.3)) # Exactly at the 2026-27 limit (10,608 / 52 = 204.00): not gainfully employed. @example(pay=10_608.0, profit=0.0, pension_share=0.0) -# One penny a week over it (10,608.52 / 52 = 204.01), on the float32 boundary -# described in _over_limit: gainfully employed. +# One penny a week over it (10,608.52 / 52 = 204.01), stored just below +# £10,608.52 as described in _over_limit: gainfully employed. @example(pay=10_608.52, profit=0.0, pension_share=0.0) # A trading loss, with earnings below the limit: the derandomized draws have # neither. @@ -113,14 +169,164 @@ def test_a_penny_a_week_over_the_2026_limit_ends_entitlement(pay, gainful): """£10,608 a year is £204.00 a week, at the limit; £10,608.52 is £204.01, a penny over it. This pins the outcome for the amount entered, not only agreement with _over_limit. The model stores £10,608.52 as 10,608.51953125 - (£204.00 a week in exact arithmetic) and reaches £204.01 through float32 - rounding; see #2541.""" + (£204.00 a week in exact arithmetic), and the formula recovers the + £10,608.52 by taking the nearest penny; see #2541.""" result = _person(pay, 0, 0) assert bool(result["is_gainfully_employed_for_carers_allowance"]) == gainful expected = 0 if gainful else RATE * WEEKS assert abs(float(result["carers_allowance"]) - expected) < 0.01 +def _carers(variable, amounts_by_year, reform=None): + """One carer in England per amount, aged 45 and caring 40 hours a week, + with ``variable`` set to that amount in each year.""" + years = list(amounts_by_year) + count = len(amounts_by_year[years[0]]) + people = { + f"carer_{i}": { + "age": {year: 45 for year in years}, + "care_hours": {year: 40 for year in years}, + variable: {year: amounts_by_year[year][i] for year in years}, + } + for i in range(count) + } + return Simulation( + situation={ + "people": people, + "benunits": {f"b{i}": {"members": [f"carer_{i}"]} for i in range(count)}, + "households": { + f"h{i}": { + "members": [f"carer_{i}"], + "country": {year: "ENGLAND" for year in years}, + } + for i in range(count) + }, + }, + reform=reform, + ) + + +# A year's earnings at each limit, a penny a week below what is over it, and a +# penny a week over it: 52 x L + £0.51 is L + £0.0098 a week, which reg. 5 +# takes down to L. +PENNY_STEPS = ((0.0, False), (0.51, False), (0.52, True)) + + +def _penny_steps_misjudged(variable, limits_by_year): + amounts = { + year: [WEEKS * limit + step for step, _ in PENNY_STEPS] + for year, limit in limits_by_year + } + simulation = _carers(variable, amounts) + misjudged = [] + for year, limit in limits_by_year: + assert ( + system.parameters(f"{year}-01-01").gov.dwp.carers_allowance.earnings_limit + == limit + ) + earnings = simulation.calculate("carers_allowance_earnings", year) + gainful = simulation.calculate( + "is_gainfully_employed_for_carers_allowance", year + ) + allowance = simulation.calculate("carers_allowance", year) + for (step, over), amount, stored, judged, paid in zip( + PENNY_STEPS, amounts[year], earnings, gainful, allowance + ): + # Earnings are the amount entered, as float32 holds it. + assert stored == np.float32(amount), (year, amount, stored) + if bool(judged) != over or (paid == 0) != over: + misjudged.append((year, limit, amount, bool(judged), float(paid))) + return misjudged + + +def test_a_penny_a_week_over_each_limit_is_over_it_in_every_year(): + """Reg. 8(1), after reg. 5, for every limit in earnings_limit.yaml in every + year it applies, with carers_allowance_earnings entered directly. Before + #2541 the formula counted pence in float32 and treated 52 x L + £0.52 as + within the 2020-21 to 2024-25 limits (£128, £132, £139 and £151).""" + assert {limit for _, limit in LIMITS_BY_YEAR} == {v for _, v in LIMIT_VALUES} + assert _penny_steps_misjudged("carers_allowance_earnings", LIMITS_BY_YEAR) == [] + + +def test_pay_a_penny_a_week_over_each_limit_ends_entitlement(): + """The same steps entered as pay, in each year up to the latest limit's + first. 52 x L + £0.52 is below the personal allowance and the primary + threshold in each of these years, so earnings are the pay itself, as the + test checks. This is #2541's reproducer.""" + assert _penny_steps_misjudged("employment_income", SET_YEARS) == [] + + +def test_a_limit_in_pounds_and_pence_is_read_exactly(): + """A reformed limit of £204.10 is 20,410 pence, not the 20,409.999... + that 204.1 x 100 gives in floating point: £204.10 a week is within it and + £204.11 is over it.""" + reform = { + "gov.dwp.carers_allowance.earnings_limit": {"2026-01-01.2026-12-31": 204.1} + } + amounts = [WEEKS * 204.1, WEEKS * 204.1 + 0.51, WEEKS * 204.1 + 0.52] + simulation = _carers("carers_allowance_earnings", {2026: amounts}, reform) + gainful = simulation.calculate("is_gainfully_employed_for_carers_allowance", 2026) + assert list(gainful) == [False, False, True] + + +@st.composite +def whole_pence_around_a_limit(draw): + year, limit = draw(st.sampled_from(LIMITS_BY_YEAR)) + # The least number of pence a year that is over the limit. + over = WEEKS * (limit * 100 + 1) + near = st.integers(over - 3 * WEEKS, over + 3 * WEEKS) + # Up to £131,071.99, below which float32 holds every whole-pence amount + # to within half a penny. + anywhere = st.integers(0, 2**17 * 100 - 1) + pence = draw(st.lists(st.one_of(near, anywhere), min_size=1, max_size=40)) + return year, limit, pence + + +@settings( + max_examples=10, + deadline=None, + derandomize=True, + suppress_health_check=[HealthCheck.too_slow], +) +@given(case=whole_pence_around_a_limit()) +def test_whole_pence_are_judged_exactly(case): + """Invariant 5.""" + year, limit, pence = case + simulation = _carers("carers_allowance_earnings", {year: [p / 100 for p in pence]}) + gainful = simulation.calculate("is_gainfully_employed_for_carers_allowance", year) + for amount, judged in zip(pence, gainful): + assert bool(judged) == (amount // WEEKS > limit * 100), (year, amount) + + +@st.composite +def amounts_around_a_limit(draw): + year, limit = draw(st.sampled_from(LIMITS_BY_YEAR)) + over = WEEKS * limit + 0.52 + near = st.floats(over - 0.05, over + 0.05) + anywhere = st.floats(0, 100_000, allow_nan=False) + amounts = draw(st.lists(st.one_of(near, anywhere), min_size=2, max_size=40)) + return year, limit, amounts + + +@settings( + max_examples=10, + deadline=None, + derandomize=True, + suppress_health_check=[HealthCheck.too_slow], +) +@given(case=amounts_around_a_limit()) +def test_the_formula_matches_exact_arithmetic_and_rises_with_earnings(case): + """Invariants 6 and 7, for amounts with any fraction of a penny.""" + year, limit, amounts = case + simulation = _carers("carers_allowance_earnings", {year: amounts}) + stored = simulation.calculate("carers_allowance_earnings", year) + gainful = simulation.calculate("is_gainfully_employed_for_carers_allowance", year) + for value, judged in zip(stored, gainful): + assert bool(judged) == _over_limit(value, limit), (year, float(value)) + in_order = gainful[np.argsort(stored, kind="stable")].astype(int) + assert np.all(np.diff(in_order) >= 0), (year, amounts) + + @settings( max_examples=8, deadline=None, diff --git a/policyengine_uk/tests/test_carers_allowance_pension_contribution_deduction_rate_properties.py b/policyengine_uk/tests/test_carers_allowance_pension_contribution_deduction_rate_properties.py index be9c21dea3..c8cd5d8fd5 100644 --- a/policyengine_uk/tests/test_carers_allowance_pension_contribution_deduction_rate_properties.py +++ b/policyengine_uk/tests/test_carers_allowance_pension_contribution_deduction_rate_properties.py @@ -21,13 +21,17 @@ 3. Eligibility follows earnings through the limit: on both sides the person is gainfully employed exactly when weekly earnings, with a fraction of a penny disregarded (reg. 5), exceed the limit, and then has no underlying - entitlement. The pence are counted in the precision the model stores - earnings in (float32), as the formula counts them. A lower share can only end entitlement and a higher share can - only restore it. + entitlement. The pence are counted as the formula counts them: the stored + (float32) annual earnings are taken to the nearest penny, then spread over + 52 weeks in whole pence. A lower share can only end entitlement and a + higher share can only restore it. 4. Current law: the share is 0.5 in every year from 2015 (the first year the model simulates) to 2030. """ +import math +from fractions import Fraction + import numpy as np from hypothesis import HealthCheck, example, given, settings from hypothesis import strategies as st @@ -111,21 +115,23 @@ def situation(people, year): def calculate(people, year, reform=None): sim = Simulation(situation=situation(people, year), reform=reform) - # Keep each variable's stored dtype: the rounding check below must count - # pence in the precision the formula does. + # Keep each stored value as the formula reads it: the rounding check below + # must start from the same float32 amount. return {v: np.asarray(sim.calculate(v, year)) for v in VARIABLES} def over_limit(earnings, limit): - # Reg. 5: a fraction of a penny is disregarded. ``earnings`` is the stored - # array, so this counts the pence as is_gainfully_employed_for_carers_allowance - # does, in float32: promoting to float64 first floors 10,608.52 a year, - # stored as 10,608.51953125, to 204.00 a week instead of 204.01. - return np.floor(earnings / 52 * 100 + 1e-6) / 100 > limit + # Reg. 5, as is_gainfully_employed_for_carers_allowance applies it, in + # exact rational arithmetic: the stored amount to the nearest penny a + # year, an exact half penny disregarded, then whole pence a week, a + # fraction of a penny disregarded. 10,608.52 a year, stored as + # 10,608.51953125, is 1,060,852 pence and 20,401 pence (204.01) a week. + annual_pence = math.ceil(Fraction(float(earnings)) * 100 - Fraction(1, 2)) + return annual_pence // 52 > Fraction(str(limit)) * 100 -# A carer one penny a week over the 2026-27 limit (10,608.52 / 52 = 204.01), on -# the float32 boundary described in over_limit. +# A carer one penny a week over the 2026-27 limit (10,608.52 / 52 = 204.01), +# stored just below 10,608.52 as described in over_limit. PENNY_OVER = dict( country="ENGLAND", age=45, diff --git a/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py b/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py index b6e6b4b981..d80139a7b3 100644 --- a/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py +++ b/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py @@ -12,7 +12,9 @@ class is_gainfully_employed_for_carers_allowance(Variable): "of the preceding week; annual earnings are spread evenly over 52 " "weeks. A fraction of a penny is disregarded, as regulation 5 of the " "Computation of Earnings Regulations requires when that favours the " - "claimant." + "claimant. Annual earnings are first taken to the nearest penny, " + "because the model stores them in single precision, which holds most " + "sums of money only to within a fraction of a penny." ) definition_period = YEAR reference = [ @@ -32,8 +34,17 @@ class is_gainfully_employed_for_carers_allowance(Variable): def formula(person, period, parameters): limit = parameters(period).gov.dwp.carers_allowance.earnings_limit - weekly = person("carers_allowance_earnings", period) / WEEKS_IN_YEAR - # Disregard a fraction of a penny; the tolerance keeps an exact - # number of pence from being knocked down by floating-point error. - weekly_in_pence = np.floor(weekly * 100 + 1e-6) / 100 - return weekly_in_pence > limit + earnings = person("carers_allowance_earnings", period) + # Earnings are a sum of money, a whole number of pence, but the model + # stores them in float32, which holds most such sums only to within a + # fraction of a penny: £10,608.52 is stored as 10,608.51953125. Take + # the nearest penny, in float64, to recover the sum, with an exact + # half penny disregarded (reg. 5). Then count the weekly pence + # exactly, disregarding the fraction of a penny that spreading over + # 52 weeks leaves (reg. 5), so that no floating-point rounding + # decides whether earnings exceed the limit (reg. 8(1)). + annual_pence = np.ceil(earnings.astype(np.float64) * 100 - 0.5) + weekly_pence = annual_pence // WEEKS_IN_YEAR + # Rounding to a millionth of a penny keeps a limit such as £204.10 + # from being read as 20,409.999... pence. + return weekly_pence > np.round(limit * 100, 6) From ab644ac1b25735dd18947306a95fbeb28dd61bef Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Tue, 20 Oct 2026 17:15:07 -0400 Subject: [PATCH 02/13] Pin #2541's boundary cases as explicit examples in the property tests Co-Authored-By: Claude Opus 5.5 --- .../test_carers_allowance_earnings_limit_properties.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py index 1ce6e4ecaf..e66eeb2687 100644 --- a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py +++ b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py @@ -289,6 +289,10 @@ def whole_pence_around_a_limit(draw): suppress_health_check=[HealthCheck.too_slow], ) @given(case=whole_pence_around_a_limit()) +# Either side of the boundary in 2020-21 and 2024-25, where float32 pence +# misjudged the penny over (#2541). +@example(case=(2020, 128, [665_651, 665_652])) +@example(case=(2024, 151, [785_251, 785_252])) def test_whole_pence_are_judged_exactly(case): """Invariant 5.""" year, limit, pence = case @@ -315,6 +319,9 @@ def amounts_around_a_limit(draw): suppress_health_check=[HealthCheck.too_slow], ) @given(case=amounts_around_a_limit()) +# 2022-23: £6,864.51 and £6,864.52 a year, a penny a week either side of the +# boundary of the £132 limit, with amounts a tenth of a penny off them. +@example(case=(2022, 132, [6_864.509, 6_864.51, 6_864.511, 6_864.52, 6_864.521])) def test_the_formula_matches_exact_arithmetic_and_rises_with_earnings(case): """Invariants 6 and 7, for amounts with any fraction of a penny.""" year, limit, amounts = case From 71d35f5305c86080e920b5607e02b7ed84a0cd83 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 21 Oct 2026 05:10:03 -0400 Subject: [PATCH 03/13] Take CA earnings to the nearest half penny before the limit test The #2543 review found that taking stored earnings to the nearest penny misjudges whole-pence pay less half of an odd-pence pension contribution. Net earnings of 52 x L + 0.515 are 52 x L + 0.51 in law, because reg. 5 counts the half penny of the reg. 10(4)(b) deduction as a penny. Float32 holds that value just above the half penny for most limits, so nearest penny counted it up and over the limit, at all ten limits. The formula now takes the nearest half penny (an exact quarter penny down), disregards a half penny, then divides by 52 in whole pence (1996 reg. 8(1)(b)(iii) with reg. 5). np.floor(x / n) replaces // so that infinite earnings stay over the limit. Every whole-pence and half-penny amount below 65,536 pounds is recovered exactly. Tests: odd-pence contributions at every limit in every set year, a Hypothesis property checking pay less half a contribution against the law in integer arithmetic, YAML Cases 23 and 24, float32 values either side of 52L + 0.515 and 52L + 0.5175 as explicit examples, and boundary examples for every #2541 limit. The Fraction references follow the new rule. changelog.d/2532.fixed.md (from #2534) no longer describes this head, so it is removed; the wording fixes the review asked for are included. Co-Authored-By: Claude Opus 5.5 --- changelog.d/2532.fixed.md | 1 - changelog.d/2541.fixed.md | 2 +- .../dwp/carers_allowance_earnings_limit.yaml | 33 +++- ...ers_allowance_earnings_limit_properties.py | 175 ++++++++++++++++-- ..._contribution_deduction_rate_properties.py | 18 +- ...gainfully_employed_for_carers_allowance.py | 42 +++-- 6 files changed, 231 insertions(+), 40 deletions(-) delete mode 100644 changelog.d/2532.fixed.md diff --git a/changelog.d/2532.fixed.md b/changelog.d/2532.fixed.md deleted file mode 100644 index ec250d10d2..0000000000 --- a/changelog.d/2532.fixed.md +++ /dev/null @@ -1 +0,0 @@ -- Fixed the Carer's Allowance earnings-limit property test to count pence in the precision the model stores earnings in (float32), as the formula does, so a draw whose float32 arithmetic rounds across a penny boundary no longer fails at random. diff --git a/changelog.d/2541.fixed.md b/changelog.d/2541.fixed.md index c08b9d21f2..9e9695283e 100644 --- a/changelog.d/2541.fixed.md +++ b/changelog.d/2541.fixed.md @@ -1 +1 @@ -- Fixed the Carer's Allowance earnings limit treating earnings a penny a week over the 2020-21 to 2024-25 limits (£128, £132, £139 and £151) as within them. The test now takes annual earnings to the nearest penny before spreading them over 52 weeks, so float32 rounding no longer decides it. +- Fixed the Carer's Allowance earnings limit treating earnings a penny a week over the 2020-21 to 2024-25 limits (£128, £132, £139 and £151) as within them. The test now takes annual earnings to the nearest half penny, counts the half penny of half an odd-pence pension contribution as a penny off, and divides by 52 in whole pence, so float32 rounding no longer decides it. diff --git a/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml b/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml index 8571bb67dd..6cf4a77fca 100644 --- a/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml +++ b/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml @@ -322,7 +322,7 @@ is_gainfully_employed_for_carers_allowance: false carers_allowance_pre_overlap: 86.45 * 52 -- name: Case 21, in 2024-25 a penny a week under the next whole penny over £151 is within the limit. +- name: Case 21, in 2024-25 a penny a year short of £151.01 a week is within the limit. period: 2024 absolute_error_margin: 0.01 input: @@ -350,3 +350,34 @@ carers_allowance_earnings: 7_852.52 is_gainfully_employed_for_carers_allowance: true carers_allowance: 0 + +- name: Case 23, in 2024-25 the half penny of half an odd-pence pension contribution counts as a penny off earnings. + period: 2024 + absolute_error_margin: 0.01 + input: + age: 45 + employment_income: 7_852.52 + employee_pension_contributions: 0.01 + care_hours: 40 + country: ENGLAND + output: + # 7,852.52 less half of 0.01 is 7,852.515. Counting the half penny of the + # deduction as a penny favours the claimant (reg. 5), so earnings are + # 7,852.51, or £151.0098 a week, which counts as £151.00. + is_gainfully_employed_for_carers_allowance: false + carers_allowance: 81.90 * 52 + +- name: Case 24, in 2024-25 a penny more pay with the same contribution is over the limit. + period: 2024 + absolute_error_margin: 0.01 + input: + age: 45 + employment_income: 7_852.53 + employee_pension_contributions: 0.01 + care_hours: 40 + country: ENGLAND + output: + # 7,852.53 less 0.005 is 7,852.525, which reg. 5 takes to 7,852.52, or + # £151.01 a week. + is_gainfully_employed_for_carers_allowance: true + carers_allowance: 0 diff --git a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py index e66eeb2687..5d50ec8832 100644 --- a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py +++ b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py @@ -9,10 +9,14 @@ which deducted Class 2, was omitted from 6 April 2024). A fraction of a penny is disregarded where counting it would not favour the -claimant (reg. 5). The model holds annual earnings in float32, so the formula -first takes them to the nearest penny (an exact half penny disregarded), then -compares the whole pence a week, after spreading over 52 weeks, with the -limit. _over_limit restates that rule in exact rational arithmetic. +claimant (reg. 5). Net earnings are whole pence less half of pension +contributions (reg. 10(4)(b)), so a whole number of half pence, and the half +penny of an odd-pence contribution counts as a penny off earnings, which +favours the claimant. The model holds annual earnings in float32, so the +formula first takes them to the nearest half penny, disregards a half penny, +then compares the whole pence a week, after dividing by 52 (reg. 8(1)(b)(iii)) +with the fraction disregarded, with the limit. _over_limit restates that rule +in exact rational arithmetic. Invariants, for a carer in England in 2026 caring 35 hours or more, with pay, trading profit, employee pension contributions and unearned income drawn @@ -33,11 +37,20 @@ 5. Whole pence are judged exactly: annual earnings of A pence are over the limit L exactly when A // 52 > 100 x L, that is A >= 52 x (100 x L + 1), whatever float32 does to A. So 52 x L + £0.51 a year is within the limit - and 52 x L + £0.52 (a penny a week more) is over it. + and 52 x L + £0.52 (a penny a year more, and a penny a week over) is over + it. 6. Exact arithmetic: for any stored amount, whole pence or not, the formula agrees with _over_limit. 7. Monotonic in earnings: more earnings never take a person from over the limit to within it. + +And with pay and an employee pension contribution entered, in whole pence, +in each year up to the latest limit's first, where pay near the limit pays no +tax or NI: + +8. Half of a pension contribution comes off pay, its half penny counting as a + penny (regs. 10(4)(b) and 5): pay of P pence with a contribution of C + pence is over the limit L exactly when (P - ceil(C / 2)) // 52 > 100 x L. """ import datetime @@ -126,12 +139,15 @@ def _person(pay, profit, pension, savings=0.0, private_pension=0.0): def _over_limit(earnings, limit=LIMIT): # The formula's rule in exact rational arithmetic, on the stored value: - # the nearest penny a year, an exact half penny disregarded, then whole - # pence a week, a fraction of a penny disregarded (reg. 5), compared with - # the limit. £10,608.52 a year, stored as 10,608.51953125, is 1,060,852 - # pence a year and 20,401 pence (£204.01) a week, a penny over £204. - annual_pence = math.ceil(Fraction(float(earnings)) * 100 - Fraction(1, 2)) - return annual_pence // WEEKS > Fraction(str(limit)) * 100 + # the nearest half penny a year (an exact quarter penny down), a half + # penny disregarded, then whole pence a week, a fraction of a penny + # disregarded (reg. 5), compared with the limit. £10,608.52 a year, stored + # as 10,608.51953125, is 1,060,852 pence a year and 20,401 pence (£204.01) + # a week, a penny over £204. £7,852.515 (pay of £7,852.52 less half of a + # 1p contribution), stored as 7,852.51513671875, is 785,251 pence a year + # and 15,100 pence (£151.00) a week, within £151. + half_pence = math.ceil(Fraction(float(earnings)) * 200 - Fraction(1, 2)) + return half_pence // 2 // WEEKS > Fraction(str(limit)) * 100 money = st.floats(min_value=0, max_value=60_000, allow_nan=False) @@ -206,9 +222,9 @@ def _carers(variable, amounts_by_year, reform=None): ) -# A year's earnings at each limit, a penny a week below what is over it, and a -# penny a week over it: 52 x L + £0.51 is L + £0.0098 a week, which reg. 5 -# takes down to L. +# A year's earnings at each limit, a penny a year short of £0.01 a week over +# it, and £0.01 a week over it: 52 x L + £0.51 is L + £0.0098 a week, which +# reg. 5 takes down to L. PENNY_STEPS = ((0.0, False), (0.51, False), (0.52, True)) @@ -256,6 +272,112 @@ def test_pay_a_penny_a_week_over_each_limit_ends_entitlement(): assert _penny_steps_misjudged("employment_income", SET_YEARS) == [] +def _pay_less_pension(year, pairs): + """Pay and an employee pension contribution for one carer each.""" + people = { + f"carer_{i}": { + "age": {year: 45}, + "care_hours": {year: 40}, + "employment_income": {year: pay}, + "employee_pension_contributions": {year: contribution}, + } + for i, (pay, contribution) in enumerate(pairs) + } + simulation = Simulation( + situation={ + "people": people, + "benunits": { + f"b{i}": {"members": [f"carer_{i}"]} for i in range(len(pairs)) + }, + "households": { + f"h{i}": {"members": [f"carer_{i}"], "country": {year: "ENGLAND"}} + for i in range(len(pairs)) + }, + } + ) + earnings = simulation.calculate("carers_allowance_earnings", year) + gainful = simulation.calculate("is_gainfully_employed_for_carers_allowance", year) + return earnings, gainful + + +# Pay and an odd-pence pension contribution: (extra pay over 52 x L, the +# contribution, over the limit). Half of the contribution comes off pay +# (reg. 10(4)(b)), and its half penny counts as a penny (reg. 5), so the +# first three leave 52 x L + £0.51 and the last two 52 x L + £0.52. +ODD_PENCE_CONTRIBUTIONS = ( + (0.52, 0.01, False), + (0.53, 0.03, False), + (48.00, 94.97, False), + (0.53, 0.01, True), + (0.54, 0.03, True), +) + + +def test_half_an_odd_pence_pension_contribution_counts_as_a_penny_off(): + """Regs. 10(4)(b) and 5, for every limit in earnings_limit.yaml in each + year up to the latest limit's first. Pay of 52 x L + £0.52 less half of a + 1p contribution is 52 x L + £0.515, which reg. 5 takes to 52 x L + £0.51, + within the limit. Float32 holds that half penny just above it for most + limits (£7,852.515 as 7,852.51513671875), so taking the nearest penny + instead of the nearest half penny would put it over.""" + misjudged = [] + for year, limit in SET_YEARS: + pairs = [(WEEKS * limit + extra, c) for extra, c, _ in ODD_PENCE_CONTRIBUTIONS] + earnings, gainful = _pay_less_pension(year, pairs) + for (pay, contribution), (_, _, over), stored, judged in zip( + pairs, ODD_PENCE_CONTRIBUTIONS, earnings, gainful + ): + # Earnings are pay less half of the contribution, to within + # float32's rounding of the inputs and the subtraction. + assert abs(float(stored) - (pay - contribution / 2)) < 0.002, ( + year, + pay, + contribution, + float(stored), + ) + if bool(judged) != over: + misjudged.append((year, limit, pay, contribution, bool(judged))) + assert misjudged == [] + + +@st.composite +def pay_and_contributions_around_a_limit(draw): + year, limit = draw(st.sampled_from(SET_YEARS)) + # The least pence of earnings a year that is over the limit. + over = WEEKS * (limit * 100 + 1) + pairs = [] + for _ in range(draw(st.integers(1, 30))): + contribution = draw(st.integers(0, 20_000)) + # Pay within a few pence of the boundary after the deduction, or + # anywhere within £500 a year of the limit. Either way pay is below + # the personal allowance and the primary threshold, so no tax or NI. + boundary = over + math.ceil(contribution / 2) + near = st.integers(boundary - 3, boundary + 3) + anywhere = st.integers( + WEEKS * limit * 100 - 50_000, WEEKS * limit * 100 + 50_000 + ) + pairs.append((draw(st.one_of(near, anywhere)), contribution)) + return year, limit, pairs + + +@settings( + max_examples=10, + deadline=None, + derandomize=True, + suppress_health_check=[HealthCheck.too_slow], +) +@given(case=pay_and_contributions_around_a_limit()) +# Odd-pence contributions on either side of the boundary in 2024-25. +@example(case=(2024, 151, [(785_252, 1), (785_253, 1), (785_253, 3), (785_254, 3)])) +def test_pay_less_half_a_pension_contribution_is_judged_as_the_law_says(case): + """Invariant 8.""" + year, limit, pairs = case + _, gainful = _pay_less_pension(year, [(p / 100, c / 100) for p, c in pairs]) + for (pay, contribution), judged in zip(pairs, gainful): + net = pay - math.ceil(contribution / 2) + assert bool(judged) == (net // WEEKS > limit * 100), (year, pay, contribution) + + def test_a_limit_in_pounds_and_pence_is_read_exactly(): """A reformed limit of £204.10 is 20,410 pence, not the 20,409.999... that 204.1 x 100 gives in floating point: £204.10 a week is within it and @@ -292,6 +414,8 @@ def whole_pence_around_a_limit(draw): # Either side of the boundary in 2020-21 and 2024-25, where float32 pence # misjudged the penny over (#2541). @example(case=(2020, 128, [665_651, 665_652])) +@example(case=(2022, 132, [686_451, 686_452])) +@example(case=(2023, 139, [722_851, 722_852])) @example(case=(2024, 151, [785_251, 785_252])) def test_whole_pence_are_judged_exactly(case): """Invariant 5.""" @@ -302,6 +426,20 @@ def test_whole_pence_are_judged_exactly(case): assert bool(judged) == (amount // WEEKS > limit * 100), (year, amount) +def _float32_either_side(amount): + nearest = np.float32(amount) + below = np.nextafter(nearest, np.float32(-np.inf)) + above = np.nextafter(nearest, np.float32(np.inf)) + return [float(below), float(nearest), float(above)] + + +EDGES = { + limit: _float32_either_side(WEEKS * limit + 0.515) + + _float32_either_side(WEEKS * limit + 0.5175) + for limit in (139, 204) +} + + @st.composite def amounts_around_a_limit(draw): year, limit = draw(st.sampled_from(LIMITS_BY_YEAR)) @@ -319,9 +457,14 @@ def amounts_around_a_limit(draw): suppress_health_check=[HealthCheck.too_slow], ) @given(case=amounts_around_a_limit()) -# 2022-23: £6,864.51 and £6,864.52 a year, a penny a week either side of the -# boundary of the £132 limit, with amounts a tenth of a penny off them. +# 2022-23: £6,864.51 (within the £132 limit) and £6,864.52 (a penny a week +# over it), with amounts a tenth of a penny off them. @example(case=(2022, 132, [6_864.509, 6_864.51, 6_864.511, 6_864.52, 6_864.521])) +# The stored amounts either side of 52L + £0.515 (a half penny, within the +# limit) and 52L + £0.5175 (where the nearest half penny turns from £0.515 to +# £0.52, over it), for a limit #2541 affected and one it did not. +@example(case=(2023, 139, EDGES[139])) +@example(case=(2026, 204, EDGES[204])) def test_the_formula_matches_exact_arithmetic_and_rises_with_earnings(case): """Invariants 6 and 7, for amounts with any fraction of a penny.""" year, limit, amounts = case diff --git a/policyengine_uk/tests/test_carers_allowance_pension_contribution_deduction_rate_properties.py b/policyengine_uk/tests/test_carers_allowance_pension_contribution_deduction_rate_properties.py index c8cd5d8fd5..e787e1e6c4 100644 --- a/policyengine_uk/tests/test_carers_allowance_pension_contribution_deduction_rate_properties.py +++ b/policyengine_uk/tests/test_carers_allowance_pension_contribution_deduction_rate_properties.py @@ -22,8 +22,9 @@ is gainfully employed exactly when weekly earnings, with a fraction of a penny disregarded (reg. 5), exceed the limit, and then has no underlying entitlement. The pence are counted as the formula counts them: the stored - (float32) annual earnings are taken to the nearest penny, then spread over - 52 weeks in whole pence. A lower share can only end entitlement and a + (float32) annual earnings are taken to the nearest half penny, a half + penny is disregarded, and the pence are spread over 52 weeks in whole + pence. A lower share can only end entitlement and a higher share can only restore it. 4. Current law: the share is 0.5 in every year from 2015 (the first year the model simulates) to 2030. @@ -122,12 +123,13 @@ def calculate(people, year, reform=None): def over_limit(earnings, limit): # Reg. 5, as is_gainfully_employed_for_carers_allowance applies it, in - # exact rational arithmetic: the stored amount to the nearest penny a - # year, an exact half penny disregarded, then whole pence a week, a - # fraction of a penny disregarded. 10,608.52 a year, stored as - # 10,608.51953125, is 1,060,852 pence and 20,401 pence (204.01) a week. - annual_pence = math.ceil(Fraction(float(earnings)) * 100 - Fraction(1, 2)) - return annual_pence // 52 > Fraction(str(limit)) * 100 + # exact rational arithmetic: the stored amount to the nearest half penny + # a year (an exact quarter penny down), a half penny disregarded, then + # whole pence a week, a fraction of a penny disregarded. 10,608.52 a year, + # stored as 10,608.51953125, is 1,060,852 pence and 20,401 pence (204.01) + # a week. + half_pence = math.ceil(Fraction(float(earnings)) * 200 - Fraction(1, 2)) + return half_pence // 2 // 52 > Fraction(str(limit)) * 100 # A carer one penny a week over the 2026-27 limit (10,608.52 / 52 = 204.01), diff --git a/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py b/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py index d80139a7b3..aa60f81854 100644 --- a/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py +++ b/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py @@ -12,9 +12,12 @@ class is_gainfully_employed_for_carers_allowance(Variable): "of the preceding week; annual earnings are spread evenly over 52 " "weeks. A fraction of a penny is disregarded, as regulation 5 of the " "Computation of Earnings Regulations requires when that favours the " - "claimant. Annual earnings are first taken to the nearest penny, " - "because the model stores them in single precision, which holds most " - "sums of money only to within a fraction of a penny." + "claimant. Net earnings are whole pence less half of pension " + "contributions, so the model first takes the annual figure to the " + "nearest half penny, because it stores amounts in single precision, " + "which holds most sums of money only to within a fraction of a penny. " + "The half penny of an odd-pence pension contribution then counts as a " + "penny off earnings, which favours the claimant." ) definition_period = YEAR reference = [ @@ -30,21 +33,34 @@ class is_gainfully_employed_for_carers_allowance(Variable): title="Social Security Benefit (Computation of Earnings) Regulations 1996 reg. 5", href="https://www.legislation.gov.uk/uksi/1996/2745/regulation/5", ), + dict( + title="Social Security Benefit (Computation of Earnings) Regulations 1996 reg. 8(1)(b)(iii)", + href="https://www.legislation.gov.uk/uksi/1996/2745/regulation/8", + ), + dict( + title="Social Security Benefit (Computation of Earnings) Regulations 1996 reg. 10(4)(b)", + href="https://www.legislation.gov.uk/uksi/1996/2745/regulation/10", + ), ] def formula(person, period, parameters): limit = parameters(period).gov.dwp.carers_allowance.earnings_limit earnings = person("carers_allowance_earnings", period) - # Earnings are a sum of money, a whole number of pence, but the model - # stores them in float32, which holds most such sums only to within a - # fraction of a penny: £10,608.52 is stored as 10,608.51953125. Take - # the nearest penny, in float64, to recover the sum, with an exact - # half penny disregarded (reg. 5). Then count the weekly pence - # exactly, disregarding the fraction of a penny that spreading over - # 52 weeks leaves (reg. 5), so that no floating-point rounding - # decides whether earnings exceed the limit (reg. 8(1)). - annual_pence = np.ceil(earnings.astype(np.float64) * 100 - 0.5) - weekly_pence = annual_pence // WEEKS_IN_YEAR + # Net earnings are pay less tax and NI, in whole pence, less half of + # pension contributions (reg. 10(4)(b)): a whole number of half pence. + # The model stores them in float32, which holds most such sums only to + # within a fraction of a penny (£10,608.52 is stored as + # 10,608.51953125), so take the nearest half penny, in float64, to + # recover the sum. Then disregard a half penny: counting the half + # penny of the pension deduction as a penny favours the claimant + # (reg. 5). Spread the pence over 52 weeks (reg. 8(1)(b)(iii)) and + # disregard the fraction of a penny that leaves (reg. 5). Each step + # is exact in float64 for every float32 amount, so no floating-point + # rounding decides whether earnings have exceeded the limit (ICA + # Regulations reg. 8(1)). + annual_half_pence = np.ceil(earnings.astype(np.float64) * 200 - 0.5) + annual_pence = np.floor(annual_half_pence / 2) + weekly_pence = np.floor(annual_pence / WEEKS_IN_YEAR) # Rounding to a millionth of a penny keeps a limit such as £204.10 # from being read as 20,409.999... pence. return weekly_pence > np.round(limit * 100, 6) From a67d5451afa7898e8f1613c3a9a0a73493444554 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 21 Oct 2026 05:14:38 -0400 Subject: [PATCH 04/13] Bound the exactness claim in the CA earnings-limit comment Co-Authored-By: Claude Opus 5.5 --- .../gov/dwp/is_gainfully_employed_for_carers_allowance.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py b/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py index aa60f81854..869dfa379b 100644 --- a/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py +++ b/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py @@ -55,7 +55,7 @@ def formula(person, period, parameters): # penny of the pension deduction as a penny favours the claimant # (reg. 5). Spread the pence over 52 weeks (reg. 8(1)(b)(iii)) and # disregard the fraction of a penny that leaves (reg. 5). Each step - # is exact in float64 for every float32 amount, so no floating-point + # is exact in float64 below 2^53 pence a year, so no floating-point # rounding decides whether earnings have exceeded the limit (ICA # Regulations reg. 8(1)). annual_half_pence = np.ceil(earnings.astype(np.float64) * 200 - 0.5) From 5eb7e5ae9702a1fcc04b2fd5fd0bc640d90ae397 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Tue, 27 Oct 2026 14:40:26 -0700 Subject: [PATCH 05/13] Apply reg. 5 to each step of CA net earnings, in whole pence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit juaristi22's review of #2543 found a trader within the limit whom the nearest-half-penny rule put over it: £14,000.01 of profit and a £6,039.38 premium leave £10,608.5174 unrounded, which the half-penny step took to £10,608.52 (£204.01 a week). Reg. 5 counts a fraction of a penny in each deduction as a penny, giving £10,608.50. carers_allowance_earnings now works in float64 pence from the whole pence each amount was entered as (utils.pence.stored_pence), taking pay and profit down to a whole penny and each deduction (PAYE, Class 1, notional tax, Class 4, Class 2, half of pension contributions) up to one. Net earnings are whole pence, which is_gainfully_employed_for_carers_allowance reads back exactly before dividing by 52. Tests: stored_pence round-trips every whole-pence amount below £131,072 and moves an amount at most half a float32 step; a through-the-model test checks trading and employee net earnings against the law, deduction by deduction, in exact arithmetic, with the review case; YAML Case 25 pins it. The limits test now reads projected limits from the model after the last legislated one. Co-Authored-By: Claude Opus 5.5 --- changelog.d/2541.fixed.md | 2 +- .../dwp/carers_allowance_earnings_limit.yaml | 33 ++- ...ers_allowance_earnings_limit_properties.py | 246 ++++++++++++++---- ..._contribution_deduction_rate_properties.py | 29 ++- policyengine_uk/tests/test_stored_pence.py | 66 +++++ policyengine_uk/utils/pence.py | 30 +++ .../gov/dwp/carers_allowance_earnings.py | 75 ++++-- ...gainfully_employed_for_carers_allowance.py | 39 ++- 8 files changed, 410 insertions(+), 110 deletions(-) create mode 100644 policyengine_uk/tests/test_stored_pence.py create mode 100644 policyengine_uk/utils/pence.py diff --git a/changelog.d/2541.fixed.md b/changelog.d/2541.fixed.md index 9e9695283e..2a315b4ecb 100644 --- a/changelog.d/2541.fixed.md +++ b/changelog.d/2541.fixed.md @@ -1 +1 @@ -- Fixed the Carer's Allowance earnings limit treating earnings a penny a week over the 2020-21 to 2024-25 limits (£128, £132, £139 and £151) as within them. The test now takes annual earnings to the nearest half penny, counts the half penny of half an odd-pence pension contribution as a penny off, and divides by 52 in whole pence, so float32 rounding no longer decides it. +Fixed the Carer's Allowance earnings limit treating earnings a penny a week over the 2020-21 to 2024-25 limits (£128, £132, £139 and £151) as within them. Net earnings now apply the Computation of Earnings Regulations' rounding to each step: pay and profit are taken down to a whole penny and each deduction (tax, National Insurance and half of pension contributions) up to one. They are worked in double precision from the whole pence each amount was entered as, and the limit test reads the whole pence back exactly, so float32 rounding no longer decides it. diff --git a/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml b/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml index 6cf4a77fca..a8f1e24b9a 100644 --- a/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml +++ b/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml @@ -5,7 +5,8 @@ # are gross pay less income tax, primary Class 1 contributions and half of # pension contributions (Computation of Earnings Regulations 1996 reg. # 10(4)), and trading profit less a notional basic-rate tax and main-rate -# Class 4 (regs. 13(4) and 14). A fraction of a penny is disregarded (reg. 5). +# Class 4 (regs. 13(4) and 14). A fraction of a penny counts in the +# claimant's favour (reg. 5): disregarded in earnings, a penny in a deduction. # 2026-27: personal allowance and NI primary threshold £12,570, employee # Class 1 main rate 8%, Class 4 lower profits limit £12,570 at 6%, basic rate # 20%. Carer's Allowance £86.45 a week. @@ -361,9 +362,10 @@ care_hours: 40 country: ENGLAND output: - # 7,852.52 less half of 0.01 is 7,852.515. Counting the half penny of the - # deduction as a penny favours the claimant (reg. 5), so earnings are - # 7,852.51, or £151.0098 a week, which counts as £151.00. + # Half of 0.01 is 0.005. Counting the half penny of the deduction as a + # penny favours the claimant (reg. 5), so earnings are 7,852.51, or + # £151.0098 a week, which counts as £151.00. + carers_allowance_earnings: 7_852.51 is_gainfully_employed_for_carers_allowance: false carers_allowance: 81.90 * 52 @@ -377,7 +379,28 @@ care_hours: 40 country: ENGLAND output: - # 7,852.53 less 0.005 is 7,852.525, which reg. 5 takes to 7,852.52, or + # 7,852.53 less 0.01 (the half penny counted as a penny) is 7,852.52, or # £151.01 a week. + carers_allowance_earnings: 7_852.52 is_gainfully_employed_for_carers_allowance: true carers_allowance: 0 + +- name: Case 25, a fraction of a penny in each self-employed deduction counts as a penny. + period: 2026 + absolute_error_margin: 0.001 + input: + age: 45 + self_employment_income: 14_000.01 + personal_pension_contributions: 6_039.38 + care_hours: 40 + country: ENGLAND + output: + # Notional tax 20% x 1,430.01 = 286.002 and Class 4 6% x 1,430.01 = + # 85.8006 count as 286.01 and 85.81 (reg. 5), and half the premium is + # 3,019.69: 14,000.01 - 286.01 - 85.81 - 3,019.69 = 10,608.50, or £204.00 + # a week, within the limit. Unrounded, earnings are 10,608.5174; the + # nearest half penny of that is 10,608.52, £204.01 a week, which would + # have ended entitlement (juaristi22's review of #2543). + carers_allowance_earnings: 10_608.50 + is_gainfully_employed_for_carers_allowance: false + carers_allowance: 86.45 * 52 diff --git a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py index 5dd1dfbf1f..cc280cd65d 100644 --- a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py +++ b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py @@ -8,15 +8,17 @@ (Computation of Earnings Regulations 1996 regs. 10(4), 13 and 14; reg. 14(2)(a), which deducted Class 2, was omitted from 6 April 2024). -A fraction of a penny is disregarded where counting it would not favour the -claimant (reg. 5). Net earnings are whole pence less half of pension -contributions (reg. 10(4)(b)), so a whole number of half pence, and the half -penny of an odd-pence contribution counts as a penny off earnings, which -favours the claimant. The model holds annual earnings in float32, so the -formula first takes them to the nearest half penny, disregards a half penny, -then compares the whole pence a week, after dividing by 52 (reg. 8(1)(b)(iii)) -with the fraction disregarded, with the limit. _over_limit restates that rule -in exact rational arithmetic. +A fraction of a penny counts in the claimant's favour (reg. 5): it is +disregarded in pay, profit and net earnings, and counts as a penny in each +deduction. So the half penny of half an odd-pence pension contribution (reg. +10(4)(b)) counts as a penny off earnings, as does the fraction of a penny in +a notional tax or Class 4 deduction. carers_allowance_earnings applies that to +each step, in float64 pence from the whole pence each amount was entered as +(stored_pence), so net earnings are whole pence despite float32 storage. The +test then reads back the whole pence, disregarding any fraction in an amount +entered directly, and compares the whole pence a week, after dividing by 52 +(reg. 8(1)(b)(iii)) with the fraction disregarded, with the limit. +_over_limit restates the test in exact rational arithmetic. Invariants, for a carer in England in 2026 caring 35 hours or more, with pay, trading profit, employee pension contributions and unearned income drawn @@ -31,8 +33,8 @@ 4. Unearned income is not earnings: savings interest and pension income leave carers_allowance_earnings unchanged. -Invariants of the test itself, for every limit in earnings_limit.yaml in every -year it applies, with carers_allowance_earnings entered directly: +Invariants of the test itself, for every limit in every year from 2015 to +the parameter horizon, with carers_allowance_earnings entered directly: 5. Whole pence are judged exactly: annual earnings of A pence are over the limit L exactly when A // 52 > 100 x L, that is A >= 52 x (100 x L + 1), @@ -44,13 +46,18 @@ 7. Monotonic in earnings: more earnings never take a person from over the limit to within it. -And with pay and an employee pension contribution entered, in whole pence, -in each year up to the latest limit's first, where pay near the limit pays no -tax or NI: +And through the model, with pay or profit and a pension contribution entered +in whole pence: 8. Half of a pension contribution comes off pay, its half penny counting as a - penny (regs. 10(4)(b) and 5): pay of P pence with a contribution of C - pence is over the limit L exactly when (P - ceil(C / 2)) // 52 > 100 x L. + penny (regs. 10(4)(b) and 5): in each year up to the latest limit's first, + pay of P pence below the personal allowance and the primary threshold, + with a contribution of C pence, is over the limit L exactly when + (P - ceil(C / 2)) // 52 > 100 x L. +9. Each deduction counts a fraction of a penny as a penny (reg. 5): with tax + and Class 1 or Class 4 to pay, from 2024-25 (Class 2 nil, Class 1 at 8%), + net earnings are P less each of the tax, the contributions and half the + pension contribution, rounded up to a whole penny, exactly. """ import datetime @@ -91,18 +98,27 @@ def _limits_by_year(last_year): """Each year from the first limit in earnings_limit.yaml to last_year, - with the limit the file sets for it: the value in force on 30 April, after - that year's April up-rating.""" + with the limit for it: up to the year of the file's last value, the value + in force on 30 April, after that year's April up-rating; after that, the + model's, which may project the limit beyond the last legislated one.""" limits = [] for year in range(LIMIT_VALUES[0][0].year, last_year + 1): - in_force = [v for date, v in LIMIT_VALUES if date <= datetime.date(year, 4, 30)] - limits.append((year, in_force[-1])) + if year <= LIMIT_VALUES[-1][0].year: + in_force = [ + v for date, v in LIMIT_VALUES if date <= datetime.date(year, 4, 30) + ] + limit = in_force[-1] + else: + limit = system.parameters( + f"{year}-01-01" + ).gov.dwp.carers_allowance.earnings_limit + limits.append((year, limit)) return limits -# Every year each limit applies, the last one up to the horizon. +# Every year from the first limit to the horizon. LIMITS_BY_YEAR = _limits_by_year(HORIZON) -# The same, up to the year the last limit took effect. +# The same, up to the year the last legislated limit took effect. SET_YEARS = _limits_by_year(LIMIT_VALUES[-1][0].year) @@ -137,17 +153,27 @@ def _person(pay, profit, pension, savings=0.0, private_pension=0.0): } +def _stored_pence(value): + """A stored float32 amount in exact pence: the whole pence it lies within + half a float32 step of, or else its exact value.""" + pence = Fraction(float(value)) * 100 + whole = round(pence) + if abs(pence - whole) <= Fraction(float(np.spacing(np.float32(value)))) * 50: + return Fraction(whole) + return pence + + def _over_limit(earnings, limit=LIMIT): # The formula's rule in exact rational arithmetic, on the stored value: - # the nearest half penny a year (an exact quarter penny down), a half - # penny disregarded, then whole pence a week, a fraction of a penny - # disregarded (reg. 5), compared with the limit. £10,608.52 a year, stored - # as 10,608.51953125, is 1,060,852 pence a year and 20,401 pence (£204.01) - # a week, a penny over £204. £7,852.515 (pay of £7,852.52 less half of a - # 1p contribution), stored as 7,852.51513671875, is 785,251 pence a year - # and 15,100 pence (£151.00) a week, within £151. - half_pence = math.ceil(Fraction(float(earnings)) * 200 - Fraction(1, 2)) - return half_pence // 2 // WEEKS > Fraction(str(limit)) * 100 + # the whole pence a year it lies within half a float32 step of, or else + # its pence with the fraction disregarded, then whole pence a week, the + # fraction disregarded (reg. 5), compared with the limit. £10,608.52 a + # year, stored as 10,608.51953125, is 1,060,852 pence a year and 20,401 + # pence (£204.01) a week, a penny over £204. 10,608.517578125 (the + # float32 just below it) is more than half a step from any whole penny, + # so 1,060,851 pence and £204.00 a week, within £204. + annual = math.floor(_stored_pence(earnings)) + return annual // WEEKS > Fraction(str(limit)) * 100 money = st.floats(min_value=0, max_value=60_000, allow_nan=False) @@ -186,7 +212,7 @@ def test_a_penny_a_week_over_the_2026_limit_ends_entitlement(pay, gainful): penny over it. This pins the outcome for the amount entered, not only agreement with _over_limit. The model stores £10,608.52 as 10,608.51953125 (£204.00 a week in exact arithmetic), and the formula recovers the - £10,608.52 by taking the nearest penny; see #2541.""" + £10,608.52, the only whole pence within half a float32 step; see #2541.""" result = _person(pay, 0, 0) assert bool(result["is_gainfully_employed_for_carers_allowance"]) == gainful expected = 0 if gainful else RATE * WEEKS @@ -257,10 +283,11 @@ def _penny_steps_misjudged(variable, limits_by_year): def test_a_penny_a_week_over_each_limit_is_over_it_in_every_year(): """Reg. 8(1), after reg. 5, for every limit in earnings_limit.yaml in every - year it applies, with carers_allowance_earnings entered directly. Before - #2541 the formula counted pence in float32 and treated 52 x L + £0.52 as - within the 2020-21 to 2024-25 limits (£128, £132, £139 and £151).""" - assert {limit for _, limit in LIMITS_BY_YEAR} == {v for _, v in LIMIT_VALUES} + year it applies, and the model's limit in each later year to the horizon, + with carers_allowance_earnings entered directly. Before #2541 the formula + counted pence in float32 and treated 52 x L + £0.52 as within the 2020-21 + to 2024-25 limits (£128, £132, £139 and £151).""" + assert {limit for _, limit in SET_YEARS} == {v for _, v in LIMIT_VALUES} assert _penny_steps_misjudged("carers_allowance_earnings", LIMITS_BY_YEAR) == [] @@ -316,10 +343,10 @@ def _pay_less_pension(year, pairs): def test_half_an_odd_pence_pension_contribution_counts_as_a_penny_off(): """Regs. 10(4)(b) and 5, for every limit in earnings_limit.yaml in each year up to the latest limit's first. Pay of 52 x L + £0.52 less half of a - 1p contribution is 52 x L + £0.515, which reg. 5 takes to 52 x L + £0.51, - within the limit. Float32 holds that half penny just above it for most - limits (£7,852.515 as 7,852.51513671875), so taking the nearest penny - instead of the nearest half penny would put it over.""" + 1p contribution is 52 x L + £0.515, and reg. 5 counts the half penny + deducted as a penny, leaving 52 x L + £0.51, within the limit. Float32 + would hold £7,852.515 as 7,852.51513671875, just above it, so earnings + taken to the nearest penny after the deduction would be over.""" misjudged = [] for year, limit in SET_YEARS: pairs = [(WEEKS * limit + extra, c) for extra, c, _ in ODD_PENCE_CONTRIBUTIONS] @@ -327,9 +354,10 @@ def test_half_an_odd_pence_pension_contribution_counts_as_a_penny_off(): for (pay, contribution), (_, _, over), stored, judged in zip( pairs, ODD_PENCE_CONTRIBUTIONS, earnings, gainful ): - # Earnings are pay less half of the contribution, to within - # float32's rounding of the inputs and the subtraction. - assert abs(float(stored) - (pay - contribution / 2)) < 0.002, ( + # Earnings are pay less half of the contribution rounded up to a + # whole penny, as float32 holds them. + net = round(pay * 100) - math.ceil(round(contribution * 100) / 2) + assert stored == np.float32(net / 100), ( year, pay, contribution, @@ -378,6 +406,120 @@ def test_pay_less_half_a_pension_contribution_is_judged_as_the_law_says(case): assert bool(judged) == (net // WEEKS > limit * 100), (year, pay, contribution) +def _with_fractions(route, year): + """Whole-pence pay or profit P and pension contributions C (odd and even + pence) whose net earnings, deductions unrounded, lie from 4p below to 2p + above the threshold B = 52 x (100 L + 1) pence a year. Returns B and + (P, C) pairs in pence. Trading: profit above the personal allowance and + the lower profits limit, so notional basic-rate tax and Class 4 at the + main rate. Employee: pay above the primary threshold, less a contribution + that leaves it below the personal allowance, so Class 1 and no PAYE (an + employee paying PAYE is over every limit, each below the personal + allowance).""" + hmrc = system.parameters(f"{year}-01-01").gov.hmrc + limit = system.parameters(f"{year}-01-01").gov.dwp.carers_allowance.earnings_limit + threshold = WEEKS * (round(limit * 100) + 1) + pa = Fraction(str(hmrc.income_tax.allowances.personal_allowance.amount)) * 100 + class_4 = hmrc.national_insurance.class_4 + lpl = Fraction(str(class_4.thresholds.lower_profits_limit)) * 100 + basic = Fraction(str(hmrc.income_tax.rates.uk.rates[0])) + main = Fraction(str(class_4.rates.main)) + # Class 1 at 8% of pay above £12,570, to choose pay near the threshold. + pt, class_1 = 1_257_000, Fraction(8, 100) + if route == "trading": + keep = 1 - basic - main + first = 2 * (keep * max(pa, lpl) + basic * pa + main * lpl - threshold) + offset = basic * pa + main * lpl + else: + keep = 1 - class_1 + first = 2 * (pt - threshold) + offset = class_1 * pt + pairs = [] + for contribution in range( + math.ceil(first) + 1_001, math.ceil(first) + 401_000, 9_973 + ): + const = offset - Fraction(contribution, 2) + low = math.ceil((threshold - 4 - const) / keep) + high = math.floor((threshold + 2 - const) / keep) + pairs += [(p, contribution) for p in range(low, high + 1)] + return threshold, pairs + + +@pytest.mark.parametrize("route", ["trading", "employee"]) +@pytest.mark.parametrize("year", [2024, 2026]) +def test_each_deduction_counts_a_fraction_of_a_penny_as_a_penny(route, year): + """Invariant 9, through the model: net earnings are P less each + deduction rounded up to a whole penny (reg. 5), exactly, and the person + is over the limit exactly when that reaches the threshold. Class 1 is the + model's ni_class_1_employee, read as stored_pence reads it. Rounding only + the total, or not at all, misjudges some of these pairs: juaristi22's + review of #2543 found a trader within the limit (C1, below) whom the + earlier nearest-half-penny rule put over it.""" + threshold, pairs = _with_fractions(route, year) + if route == "trading" and year == 2026: + pairs.append((1_400_001, 603_938)) + gross, contribution = ( + ("self_employment_income", "personal_pension_contributions") + if route == "trading" + else ("employment_income", "employee_pension_contributions") + ) + people = { + f"carer_{i}": { + "age": {year: 45}, + "care_hours": {year: 40}, + gross: {year: p / 100}, + contribution: {year: c / 100}, + } + for i, (p, c) in enumerate(pairs) + } + simulation = Simulation( + situation={ + "people": people, + "benunits": { + f"b{i}": {"members": [f"carer_{i}"]} for i in range(len(pairs)) + }, + "households": { + f"h{i}": {"members": [f"carer_{i}"], "country": {year: "ENGLAND"}} + for i in range(len(pairs)) + }, + } + ) + earnings = simulation.calculate("carers_allowance_earnings", year) + gainful = simulation.calculate("is_gainfully_employed_for_carers_allowance", year) + class_1 = simulation.calculate("ni_class_1_employee", year) + hmrc = system.parameters(f"{year}-01-01").gov.hmrc + pa = Fraction(str(hmrc.income_tax.allowances.personal_allowance.amount)) * 100 + lpl = ( + Fraction(str(hmrc.national_insurance.class_4.thresholds.lower_profits_limit)) + * 100 + ) + basic = Fraction(str(hmrc.income_tax.rates.uk.rates[0])) + main = Fraction(str(hmrc.national_insurance.class_4.rates.main)) + fractions = 0 + for (p, c), stored, judged, ni in zip(pairs, earnings, gainful, class_1): + if route == "trading": + assert max(pa, lpl) <= p and ni == 0, (p, c) + deductions = [basic * (p - pa), main * (p - lpl)] + else: + assert p - c < pa and ni > 0, (p, c) + deductions = [_stored_pence(ni)] + deductions.append(Fraction(c, 2)) + net = p - sum(math.ceil(d) for d in deductions) + fractions += any(d.denominator > 1 for d in deductions) + assert stored == np.float32(net / 100), (p, c, float(stored), net) + assert bool(judged) == (net >= threshold), (p, c, net) + # Most pairs carry a fraction of a penny in some deduction. + assert fractions > len(pairs) // 2 + if route == "trading" and year == 2026: + # C1: £14,000.01 of profit and a £6,039.38 premium. Tax £286.002 and + # Class 4 £85.8006 round up to £286.01 and £85.81, and half the + # premium is £3,019.69, leaving £10,608.50: £204.00 a week, within the + # limit, so the full £4,495.40 a year is paid. + assert float(earnings[-1]) == 10_608.5 and not gainful[-1] + allowance = simulation.calculate("carers_allowance", year)[-1] + assert abs(float(allowance) - RATE * WEEKS) < 0.01 + + def test_a_limit_in_pounds_and_pence_is_read_exactly(): """A reformed limit of £204.10 is 20,410 pence, not the 20,409.999... that 204.1 x 100 gives in floating point: £204.10 a week is within it and @@ -433,11 +575,10 @@ def _float32_either_side(amount): return [float(below), float(nearest), float(above)] -EDGES = { - limit: _float32_either_side(WEEKS * limit + 0.515) - + _float32_either_side(WEEKS * limit + 0.5175) - for limit in (139, 204) -} +# The stored amounts either side of 52L + £0.52, the least whole pence a year +# over the limit: the float32 nearest it is those pence, and the one below it +# is more than half a step away, so a fraction of a penny short. +EDGES = {limit: _float32_either_side(WEEKS * limit + 0.52) for limit in (139, 204)} @st.composite @@ -460,11 +601,14 @@ def amounts_around_a_limit(draw): # 2022-23: £6,864.51 (within the £132 limit) and £6,864.52 (a penny a week # over it), with amounts a tenth of a penny off them. @example(case=(2022, 132, [6_864.509, 6_864.51, 6_864.511, 6_864.52, 6_864.521])) -# The stored amounts either side of 52L + £0.515 (a half penny, within the -# limit) and 52L + £0.5175 (where the nearest half penny turns from £0.515 to -# £0.52, over it), for a limit #2541 affected and one it did not. +# The stored amounts either side of 52L + £0.52, for a limit #2541 affected +# and one it did not. @example(case=(2023, 139, EDGES[139])) @example(case=(2026, 204, EDGES[204])) +# 2026-27: 10,608.515625 and 10,608.517578125, a fraction of a penny short of +# £10,608.52 (the second is the earnings of juaristi22's review case under +# the earlier nearest-half-penny rule), and 10,608.5205078125, just over it. +@example(case=(2026, 204, [10_608.515625, 10_608.517578125, 10_608.5205078125])) def test_the_formula_matches_exact_arithmetic_and_rises_with_earnings(case): """Invariants 6 and 7, for amounts with any fraction of a penny.""" year, limit, amounts = case diff --git a/policyengine_uk/tests/test_carers_allowance_pension_contribution_deduction_rate_properties.py b/policyengine_uk/tests/test_carers_allowance_pension_contribution_deduction_rate_properties.py index e787e1e6c4..8cad4d2ff6 100644 --- a/policyengine_uk/tests/test_carers_allowance_pension_contribution_deduction_rate_properties.py +++ b/policyengine_uk/tests/test_carers_allowance_pension_contribution_deduction_rate_properties.py @@ -14,7 +14,8 @@ that no floor binds: 1. Exactness: when the share moves from 0.5 to r, carers_allowance_earnings - falls by exactly the person's contributions x (r - 0.5). + falls by exactly the person's contributions x (r - 0.5), give or take the + penny that rounding each deduction up to a whole penny (reg. 5) can add. 2. Locality: the Housing Benefit share is reformed at the same time to a different value, so earnings that read Housing Benefit's share would miss invariant 1. @@ -22,9 +23,9 @@ is gainfully employed exactly when weekly earnings, with a fraction of a penny disregarded (reg. 5), exceed the limit, and then has no underlying entitlement. The pence are counted as the formula counts them: the stored - (float32) annual earnings are taken to the nearest half penny, a half - penny is disregarded, and the pence are spread over 52 weeks in whole - pence. A lower share can only end entitlement and a + (float32) annual earnings are read as the whole pence they lie within + half a float32 step of, or else with the fraction disregarded, and spread + over 52 weeks in whole pence. A lower share can only end entitlement and a higher share can only restore it. 4. Current law: the share is 0.5 in every year from 2015 (the first year the model simulates) to 2030. @@ -50,7 +51,8 @@ SHARES = [i / 20 for i in range(21) if i != 10] # The model stores amounts as float32, whose spacing is about 0.008 at the # largest earnings drawn (up to 120,000), so two stored values can differ from -# the exact difference by more than a penny. +# the exact difference by more than a penny; reg. 5's rounding of the pension +# deduction adds up to a penny more. TOLERANCE = 0.05 PROPERTY_SETTINGS = settings( @@ -123,13 +125,16 @@ def calculate(people, year, reform=None): def over_limit(earnings, limit): # Reg. 5, as is_gainfully_employed_for_carers_allowance applies it, in - # exact rational arithmetic: the stored amount to the nearest half penny - # a year (an exact quarter penny down), a half penny disregarded, then - # whole pence a week, a fraction of a penny disregarded. 10,608.52 a year, - # stored as 10,608.51953125, is 1,060,852 pence and 20,401 pence (204.01) - # a week. - half_pence = math.ceil(Fraction(float(earnings)) * 200 - Fraction(1, 2)) - return half_pence // 2 // 52 > Fraction(str(limit)) * 100 + # exact rational arithmetic: the whole pence a year the stored amount lies + # within half a float32 step of, or else its pence with the fraction + # disregarded, then whole pence a week, the fraction disregarded. + # 10,608.52 a year, stored as 10,608.51953125, is 1,060,852 pence and + # 20,401 pence (204.01) a week. + pence = Fraction(float(earnings)) * 100 + whole = round(pence) + half_step = Fraction(float(np.spacing(np.float32(earnings)))) * 50 + annual = whole if abs(pence - whole) <= half_step else math.floor(pence) + return annual // 52 > Fraction(str(limit)) * 100 # A carer one penny a week over the 2026-27 limit (10,608.52 / 52 = 204.01), diff --git a/policyengine_uk/tests/test_stored_pence.py b/policyengine_uk/tests/test_stored_pence.py new file mode 100644 index 0000000000..32644d54ce --- /dev/null +++ b/policyengine_uk/tests/test_stored_pence.py @@ -0,0 +1,66 @@ +"""stored_pence: an amount of money held in float32, in pence. + +Invariants: + +1. Whole pence round-trip: every whole number of pence below £131,072, + stored in float32, comes back as exactly that number (exhaustive). +2. Bounded: the result is never more than half a float32 step from the + stored amount, so it moves an amount only to the whole pence it is held + as. +3. Fractions kept: a stored amount more than half a step from every whole + penny comes back exactly, to be rounded as the rule requires. +4. Monotonic: a larger stored amount never gives fewer pence. +""" + +from fractions import Fraction + +import numpy as np +from hypothesis import example, given, settings +from hypothesis import strategies as st + +from policyengine_uk.utils.pence import stored_pence + +# Below 2^17 pounds a float32 step is under a penny. +LIMIT_PENCE = 2**17 * 100 + + +def test_every_whole_pence_amount_below_2_to_the_17_pounds_round_trips(): + for start in range(0, LIMIT_PENCE, 2_000_000): + pence = np.arange(start, min(start + 2_000_000, LIMIT_PENCE)) + stored = (pence / 100).astype(np.float32) + recovered = stored_pence(stored) + assert recovered.dtype == np.float64 + wrong = np.nonzero(recovered != pence)[0] + assert len(wrong) == 0, pence[wrong[:5]] + + +float32_amounts = st.floats( + min_value=0, max_value=2.0**17, width=32, allow_nan=False, allow_infinity=False +) + + +@settings(max_examples=300, derandomize=True) +@given(amount=float32_amounts) +# £10,608.52 is stored as 10,608.51953125; the float32 below it is a +# fraction of a penny short of £10,608.52. +@example(amount=10_608.51953125) +@example(amount=10_608.517578125) +def test_moves_an_amount_at_most_half_a_step_and_keeps_fractions(amount): + stored = np.float32(amount) + exact = Fraction(float(stored)) * 100 + half_step = Fraction(float(np.spacing(stored))) * 50 + result = Fraction(float(stored_pence(stored))) + assert abs(result - exact) <= half_step + whole = round(exact) + if abs(exact - whole) > half_step: + assert result == exact + else: + assert result == whole + + +@settings(max_examples=200, derandomize=True) +@given(amounts=st.lists(float32_amounts, min_size=2, max_size=50)) +@example(amounts=[10_608.517578125, 10_608.51953125, 10_608.5205078125]) +def test_never_gives_fewer_pence_for_more_money(amounts): + stored = np.sort(np.array(amounts, dtype=np.float32)) + assert np.all(np.diff(stored_pence(stored)) >= 0) diff --git a/policyengine_uk/utils/pence.py b/policyengine_uk/utils/pence.py new file mode 100644 index 0000000000..ec57413a1b --- /dev/null +++ b/policyengine_uk/utils/pence.py @@ -0,0 +1,30 @@ +"""Sums of money as the model stores them, in pence. + +The model stores amounts of money in float32, which holds most of them only +to within half a float32 step: £10,608.52 is stored as 10,608.51953125. Rules +that turn on a single penny, such as Carer's Allowance's earnings limit, need +the amount that was entered, not the nearest float32. +""" + +import numpy as np + + +def stored_pence(amount): + """``amount``, stored in float32, in pence as float64: the whole number of + pence it was entered as, where it lies within half a float32 step of one, + and otherwise its exact value. + + An amount entered in whole pence is stored within half a step of them. + Below £131,072 a step is under a penny, so no other whole number of pence + lies that close, and the pence are recovered exactly. An amount with a + fraction of a penny keeps it, to be rounded as the rule requires, unless + the fraction is within half a step of a whole penny, which float32 cannot + tell apart from that penny. + """ + amount = np.asarray(amount, dtype=np.float32) + # Float32 has 24 significant bits, so multiplying by 100 in float64 is + # exact, as are the difference and the half step below. + pence = amount.astype(np.float64) * 100 + whole = np.round(pence) + half_step = np.spacing(np.abs(amount)).astype(np.float64) * 50 + return np.where(np.abs(pence - whole) <= half_step, whole, pence) diff --git a/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py b/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py index e3dc980d29..09b4e1db2d 100644 --- a/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py +++ b/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py @@ -1,5 +1,19 @@ from policyengine_uk.model_api import * from policyengine_uk.utils.class_2 import class_2_contribution_weeks, class_2_liable +from policyengine_uk.utils.pence import stored_pence + + +def _earned(amount): + """Pay or profit in whole pence, a fraction of a penny disregarded (reg. 5).""" + return np.floor(stored_pence(amount)) + + +def _deducted(pence): + """A deduction in whole pence, a fraction of a penny counted as a penny + (reg. 5). The float64 arithmetic that produces it is exact to far less + than a millionth of a penny, so a deduction within a millionth of a penny + above a whole penny is that penny.""" + return np.ceil(pence - 1e-6) class carers_allowance_earnings(Variable): @@ -35,6 +49,12 @@ class carers_allowance_earnings(Variable): "exemption after the week State Pension age is reached. Class 4 uses " "ni_class_4_liable: reaching State Pension age during the tax year " "leaves that year's contributions payable.\n\n" + "A fraction of a penny counts in the claimant's favour (reg. 5): pay " + "and profit are taken down to a whole penny and each deduction up to " + "one, so net earnings are whole pence. The pence are worked in double " + "precision from the whole pence each amount was entered as, since the " + "model stores amounts in single precision, which holds most sums of " + "money only to within a fraction of a penny.\n\n" "Not modelled: the deduction for care charges (Schedule 3, up to half " "of net earnings) and the Schedule 1 disregards, so this can overstate " "earnings for a carer who pays for care while at work." @@ -42,6 +62,10 @@ class carers_allowance_earnings(Variable): definition_period = YEAR unit = GBP reference = [ + dict( + title="Social Security Benefit (Computation of Earnings) Regulations 1996 reg. 5", + href="https://www.legislation.gov.uk/uksi/1996/2745/regulation/5", + ), dict( title="Social Security Benefit (Computation of Earnings) Regulations 1996 reg. 10(4)", href="https://www.legislation.gov.uk/uksi/1996/2745/regulation/10", @@ -71,9 +95,17 @@ def formula(person, period, parameters): scottish = person("pays_scottish_income_tax", period) personal_allowance = income_tax.allowances.personal_allowance.amount + # Work in float64 pence from the whole pence each amount was entered + # as (stored_pence), taking pay and profit down to a whole penny and + # each deduction up to one (reg. 5). Scales and thresholds are in + # pounds. + # Employment: PAYE on the pay alone (reg. 10(4)(a)). - pay = person("employment_income", period) - taxable_pay = max_(0, pay - person("employee_pension_contributions", period)) + pay = _earned(person("employment_income", period)) + employee_pension = stored_pence( + person("employee_pension_contributions", period) + ) + taxable_pay = max_(0, pay - employee_pension) / 100 pay_tax = where( scottish, income_tax.rates.scotland.rates.calc( @@ -81,12 +113,16 @@ def formula(person, period, parameters): ), income_tax.rates.uk.calc(max_(0, taxable_pay - personal_allowance)), ) - primary_class_1 = person("ni_class_1_employee", period) + # Computed in float32 by another variable: a whole-penny amount that + # comes back more than half a step out counts as a penny more, in the + # claimant's favour. + primary_class_1 = stored_pence(person("ni_class_1_employee", period)) # Self-employment: notional basic-rate tax and main-rate Class 4 # (reg. 14(1)-(2)). The tax is on the profit as if it were the only # income, less the full personal allowance. - profit = max_(0, person("self_employment_income", period)) + profit = _earned(max_(0, person("self_employment_income", period))) + profit_pounds = profit / 100 # The basic rate is the first band of the rest-of-UK scale (ITA 2007 # s. 6(2)). The Scottish basic rate is the second bracket of the # Scottish scale, after the starter rate; read it from the parameter @@ -99,13 +135,13 @@ def formula(person, period, parameters): scottish_scale.brackets[1].rate(period.start), income_tax.rates.uk.rates[0], ) - trading_tax = basic_rate * max_(0, profit - personal_allowance) + trading_tax = basic_rate * max_(0, profit_pounds - personal_allowance) trading_class_4 = ( person("ni_class_4_liable", period) * class_4.rates.main * max_( 0, - min_(profit, class_4.thresholds.upper_profits_limit) + min_(profit_pounds, class_4.thresholds.upper_profits_limit) - class_4.thresholds.lower_profits_limit, ) ) @@ -116,25 +152,30 @@ def formula(person, period, parameters): class_2 = p.national_insurance.class_2 trading_class_2 = ( person("ni_liable", period) - * class_2_liable(profit, class_2) + * class_2_liable(profit_pounds, class_2) * class_2.flat_rate * class_2_contribution_weeks(period.start.year) ) # Regs. 10(4)(b) and 13(1)(b)(ii), (4)(c) and (10)(b): one half of - # pension contributions and premiums. + # pension contributions and premiums. The contributions are those + # pension_contributions adds, summed here in float64 pence. pension_share = parameters( period ).gov.dwp.carers_allowance.pension_contribution_deduction_rate - pension_deduction = pension_share * person("pension_contributions", period) - return max_( - 0, + contributions = sum( + stored_pence(person(program, period)) + for program in p.pensions.pensions_programs + ) + pension_deduction = pension_share * contributions + net = ( pay - - pay_tax - - primary_class_1 + - _deducted(pay_tax * 100) + - _deducted(primary_class_1) + profit - - trading_tax - - trading_class_4 - - trading_class_2 - - pension_deduction, + - _deducted(trading_tax * 100) + - _deducted(trading_class_4 * 100) + - _deducted(trading_class_2 * 100) + - _deducted(pension_deduction) ) + return max_(0, net) / 100 diff --git a/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py b/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py index 869dfa379b..2443ab44e4 100644 --- a/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py +++ b/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py @@ -1,4 +1,5 @@ from policyengine_uk.model_api import * +from policyengine_uk.utils.pence import stored_pence class is_gainfully_employed_for_carers_allowance(Variable): @@ -12,12 +13,10 @@ class is_gainfully_employed_for_carers_allowance(Variable): "of the preceding week; annual earnings are spread evenly over 52 " "weeks. A fraction of a penny is disregarded, as regulation 5 of the " "Computation of Earnings Regulations requires when that favours the " - "claimant. Net earnings are whole pence less half of pension " - "contributions, so the model first takes the annual figure to the " - "nearest half penny, because it stores amounts in single precision, " - "which holds most sums of money only to within a fraction of a penny. " - "The half penny of an odd-pence pension contribution then counts as a " - "penny off earnings, which favours the claimant." + "claimant. Net earnings are whole pence, since carers_allowance_earnings " + "applies regulation 5 to each step, and the test reads back the whole " + "pence exactly, although the model stores amounts in single precision, " + "which holds most sums of money only to within a fraction of a penny." ) definition_period = YEAR reference = [ @@ -37,29 +36,21 @@ class is_gainfully_employed_for_carers_allowance(Variable): title="Social Security Benefit (Computation of Earnings) Regulations 1996 reg. 8(1)(b)(iii)", href="https://www.legislation.gov.uk/uksi/1996/2745/regulation/8", ), - dict( - title="Social Security Benefit (Computation of Earnings) Regulations 1996 reg. 10(4)(b)", - href="https://www.legislation.gov.uk/uksi/1996/2745/regulation/10", - ), ] def formula(person, period, parameters): limit = parameters(period).gov.dwp.carers_allowance.earnings_limit earnings = person("carers_allowance_earnings", period) - # Net earnings are pay less tax and NI, in whole pence, less half of - # pension contributions (reg. 10(4)(b)): a whole number of half pence. - # The model stores them in float32, which holds most such sums only to - # within a fraction of a penny (£10,608.52 is stored as - # 10,608.51953125), so take the nearest half penny, in float64, to - # recover the sum. Then disregard a half penny: counting the half - # penny of the pension deduction as a penny favours the claimant - # (reg. 5). Spread the pence over 52 weeks (reg. 8(1)(b)(iii)) and - # disregard the fraction of a penny that leaves (reg. 5). Each step - # is exact in float64 below 2^53 pence a year, so no floating-point - # rounding decides whether earnings have exceeded the limit (ICA - # Regulations reg. 8(1)). - annual_half_pence = np.ceil(earnings.astype(np.float64) * 200 - 0.5) - annual_pence = np.floor(annual_half_pence / 2) + # carers_allowance_earnings is whole pence, which float32 holds to + # within half a step (£10,608.52 as 10,608.51953125); stored_pence + # recovers them exactly below £131,072 a year. An amount entered + # directly with a fraction of a penny has the fraction disregarded + # (reg. 5). + annual_pence = np.floor(stored_pence(earnings)) + # Spread over 52 weeks (reg. 8(1)(b)(iii)), disregarding the fraction + # of a penny that leaves (reg. 5). Both steps are exact in float64, so + # no floating-point rounding decides whether earnings have exceeded + # the limit (ICA Regulations reg. 8(1)). weekly_pence = np.floor(annual_pence / WEEKS_IN_YEAR) # Rounding to a millionth of a penny keeps a limit such as £204.10 # from being read as 20,409.999... pence. From d4b8cb2808bc6c5dcbd9f3b934f0816d41dad395 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 28 Oct 2026 06:23:08 -0400 Subject: [PATCH 06/13] Work out Class 1 exactly and read entered pension totals for CA earnings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review of #2543 at 32f163910 (GPT-6.1 Sol lanes closed; Opus reviewer): - Class 1: the model sums twelve float32 months, which can hold a whole-penny total (£175.56) further out than stored_pence recovers, so rounding up deducted a penny too much and put employees exactly a penny a week over the limit within it. carers_allowance_earnings now works the model's Class 1 formula exactly on the year's NI income in whole pence, where it agrees with the stored total to within float32 error, and otherwise reads the stored total. - Pension: read pension_contributions, summing its programmes again in float64 only where the stored total is their float32 sum (stored_sum_pence), so a total entered directly still comes off. - Docs: PAYE and Class 1 rounding is the claimant-favourable convention, reg. 5 governs the calculated deductions; the self-employed weekly amount cites reg. 11(1)(a); CSP reads these earnings too. - Tests: employee-route Class 1 expectation worked exactly, with the review's rows; invariant 3 restated (2p rises never lower net earnings, a 1p rise can, as reg. 5 intends) and checked over every penny across each threshold; entered pension totals; stored_sum_pence; deadline=None on the stored_pence property tests; YAML Cases 26-31 and two CSP cases. Co-Authored-By: Claude Opus 5.5 --- changelog.d/2541.fixed.md | 2 +- .../dwp/carers_allowance_earnings_limit.yaml | 100 ++++++++++ .../carer_support_payment_earnings_limit.yaml | 30 +++ ...ers_allowance_earnings_limit_properties.py | 175 ++++++++++++++---- policyengine_uk/tests/test_stored_pence.py | 57 +++++- policyengine_uk/utils/pence.py | 20 ++ .../gov/dwp/carers_allowance_earnings.py | 109 +++++++++-- ...gainfully_employed_for_carers_allowance.py | 10 +- 8 files changed, 447 insertions(+), 56 deletions(-) diff --git a/changelog.d/2541.fixed.md b/changelog.d/2541.fixed.md index 2a315b4ecb..614137005c 100644 --- a/changelog.d/2541.fixed.md +++ b/changelog.d/2541.fixed.md @@ -1 +1 @@ -Fixed the Carer's Allowance earnings limit treating earnings a penny a week over the 2020-21 to 2024-25 limits (£128, £132, £139 and £151) as within them. Net earnings now apply the Computation of Earnings Regulations' rounding to each step: pay and profit are taken down to a whole penny and each deduction (tax, National Insurance and half of pension contributions) up to one. They are worked in double precision from the whole pence each amount was entered as, and the limit test reads the whole pence back exactly, so float32 rounding no longer decides it. +Fixed the Carer's Allowance earnings limit treating earnings a penny a week over the 2020-21 to 2024-25 limits (£128, £132, £139 and £151) as within them. Net earnings now apply the Computation of Earnings Regulations' rounding to each step: pay and profit are taken down to a whole penny and each deduction (tax, National Insurance and half of pension contributions) up to one. They are worked in double precision from the whole pence each amount was entered as, with Class 1 worked out again from its formula, and the limit test reads the whole pence back exactly, so float32 rounding no longer decides it. Pension contributions entered as a total still reduce earnings. The Scottish Carer Support Payment, which reads the same earnings, is judged exactly at its limit too. diff --git a/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml b/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml index a8f1e24b9a..1c807b3dce 100644 --- a/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml +++ b/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml @@ -404,3 +404,103 @@ carers_allowance_earnings: 10_608.50 is_gainfully_employed_for_carers_allowance: false carers_allowance: 86.45 * 52 + +- name: Case 26, in 2026-27 a whole penny of Class 1 counts as that penny, not two. + period: 2026 + absolute_error_margin: 0.001 + input: + age: 45 + employment_income: 14_764.46 + employee_pension_contributions: 7_960.76 + care_hours: 40 + country: ENGLAND + output: + # Class 1 is 8% x (14,764.46 - 52 x 241.73) = 175.56 exactly; the model's + # float32 sum of twelve months holds it as 175.5600128, which rounded up + # would be 175.57. Pay less the contribution is 6,803.70, under the + # personal allowance, so no PAYE, and half the contribution is 3,980.38: + # 14,764.46 - 175.56 - 3,980.38 = 10,608.52, or £204.01 a week, over the + # limit. + carers_allowance_earnings: 10_608.52 + is_gainfully_employed_for_carers_allowance: true + carers_allowance: 0 + +- name: Case 27, in 2025-26 the same Class 1 of £175.56 leaves a penny a week over £196. + period: 2025 + absolute_error_margin: 0.001 + input: + age: 45 + employment_income: 14_764.46 + employee_pension_contributions: 8_792.76 + care_hours: 40 + country: ENGLAND + output: + # 14,764.46 - 175.56 - 4,396.38 = 10,192.52, or £196.01 a week. + carers_allowance_earnings: 10_192.52 + is_gainfully_employed_for_carers_allowance: true + carers_allowance: 0 + +- name: Case 28, in 2024-25 Class 1 of £43.80 leaves a penny a week over £151. + period: 2024 + absolute_error_margin: 0.001 + input: + age: 45 + employment_income: 13_117.46 + employee_pension_contributions: 10_442.27 + care_hours: 40 + country: ENGLAND + output: + # Class 1 is 8% x (13,117.46 - 12,569.96) = 43.80, stored as 43.800003. + # Half the contribution is 5,221.135, which counts as 5,221.14 (reg. 5): + # 13,117.46 - 43.80 - 5,221.14 = 7,852.52, or £151.01 a week. + carers_allowance_earnings: 7_852.52 + is_gainfully_employed_for_carers_allowance: true + carers_allowance: 0 + +- name: Case 29, pension contributions entered as a total still come off earnings. + period: 2026 + absolute_error_margin: 0.001 + input: + age: 45 + employment_income: 11_000 + pension_contributions: 1_000 + care_hours: 40 + country: ENGLAND + output: + # 11,000 - 1,000 / 2 = 10,500, or £201.92 a week, within the limit. + carers_allowance_earnings: 10_500 + is_gainfully_employed_for_carers_allowance: false + carers_allowance: 86.45 * 52 + +- name: Case 30, in 2026-27 £12,570.50 of profit with a £3,923.70 premium is a penny a week over the limit. + period: 2026 + absolute_error_margin: 0.001 + input: + age: 45 + self_employment_income: 12_570.50 + personal_pension_contributions: 3_923.70 + care_hours: 40 + country: ENGLAND + output: + # Notional tax 20% x 0.50 = 0.10 and Class 4 6% x 0.50 = 0.03: + # 12,570.50 - 0.10 - 0.03 - 1,961.85 = 10,608.52, £204.01 a week. + carers_allowance_earnings: 10_608.52 + is_gainfully_employed_for_carers_allowance: true + carers_allowance: 0 + +- name: Case 31, a penny more profit can leave a penny less in earnings, as reg. 5 intends. + period: 2026 + absolute_error_margin: 0.001 + input: + age: 45 + self_employment_income: 12_570.51 + personal_pension_contributions: 3_923.70 + care_hours: 40 + country: ENGLAND + output: + # Notional tax 0.102 and Class 4 0.0306 each count as a penny more (0.11 + # and 0.04), so the extra penny of profit adds two pennies of deductions: + # 12,570.51 - 0.11 - 0.04 - 1,961.85 = 10,608.51, £204.00 a week, within. + carers_allowance_earnings: 10_608.51 + is_gainfully_employed_for_carers_allowance: false + carers_allowance: 86.45 * 52 diff --git a/policyengine_uk/tests/policy/baseline/gov/social_security_scotland/carer_support_payment_earnings_limit.yaml b/policyengine_uk/tests/policy/baseline/gov/social_security_scotland/carer_support_payment_earnings_limit.yaml index 3e9ab4f08f..e851f2b5db 100644 --- a/policyengine_uk/tests/policy/baseline/gov/social_security_scotland/carer_support_payment_earnings_limit.yaml +++ b/policyengine_uk/tests/policy/baseline/gov/social_security_scotland/carer_support_payment_earnings_limit.yaml @@ -441,3 +441,33 @@ output: carers_allowance_earnings: 6146.6 carer_support_payment_pre_overlap: 0 + +- name: Each self-employed deduction rounded up a penny leaves earnings within the limit + period: 2026 + absolute_error_margin: 0.001 + input: + age: 45 + self_employment_income: 12576.76 + personal_pension_contributions: 3932.96 + care_hours: 40 + country: SCOTLAND + output: + # Sch2para4: notional tax 20% x 6.76 = 1.352 and Class 4 6% x 6.76 = + # 0.4056 count as 1.36 and 0.41; half the premium is 1,966.48: + # 12,576.76 - 1.36 - 0.41 - 1,966.48 = 10,608.51, or 204.00 a week. + carers_allowance_earnings: 10608.51 + carer_support_payment_pre_overlap: 4495.4 + +- name: A penny more profit leaves earnings a penny a week over the limit + period: 2026 + absolute_error_margin: 0.001 + input: + age: 45 + self_employment_income: 12576.77 + personal_pension_contributions: 3932.96 + care_hours: 40 + country: SCOTLAND + output: + # 12,576.77 - 1.36 - 0.41 - 1,966.48 = 10,608.52, or 204.01 a week. + carers_allowance_earnings: 10608.52 + carer_support_payment_pre_overlap: 0 diff --git a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py index cc280cd65d..35fb4cfd34 100644 --- a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py +++ b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py @@ -12,13 +12,15 @@ disregarded in pay, profit and net earnings, and counts as a penny in each deduction. So the half penny of half an odd-pence pension contribution (reg. 10(4)(b)) counts as a penny off earnings, as does the fraction of a penny in -a notional tax or Class 4 deduction. carers_allowance_earnings applies that to -each step, in float64 pence from the whole pence each amount was entered as -(stored_pence), so net earnings are whole pence despite float32 storage. The -test then reads back the whole pence, disregarding any fraction in an amount -entered directly, and compares the whole pence a week, after dividing by 52 -(reg. 8(1)(b)(iii)) with the fraction disregarded, with the limit. -_over_limit restates the test in exact rational arithmetic. +a notional tax or Class 4 deduction. (PAYE and Class 1 are in law the amounts +actually deducted; the model's figures for them are rounded up the same way.) +carers_allowance_earnings applies that to each step, in float64 pence from the +whole pence each amount was entered as (stored_pence), so net earnings are +whole pence despite float32 storage. The test then reads back the whole +pence, disregarding any fraction in an amount entered directly, and compares +the whole pence a week, after dividing by 52 (regs. 8(1)(b)(iii) and +11(1)(a)) with the fraction disregarded, with the limit. _over_limit restates +the test in exact rational arithmetic. Invariants, for a carer in England in 2026 caring 35 hours or more, with pay, trading profit, employee pension contributions and unearned income drawn @@ -28,8 +30,14 @@ 2. Differential: the test is passed exactly when _over_limit is false for the stored carers_allowance_earnings, and the allowance is the full rate then and nil otherwise. -3. Monotonic: raising pay never lowers net earnings, so a person over the - limit stays over it. +3. Monotonic in steps of 2p: raising pay or profit by 2p or more never lowers + net earnings, so a person over the limit stays over it. The deductions + that rise with pay (PAYE and Class 1) or with profit (notional tax and + Class 4) have marginal rates adding to at most a half, and each is rounded + up, so a rise of R pence adds less than R / 2 + 2 pence to them, which is + at most R when R >= 2. A rise of 1p can lower net earnings by 1p, when it + takes both deductions to their next penny at once. That is reg. 5 working + as intended, not an error (YAML Cases 30 and 31 pin one such pair). 4. Unearned income is not earnings: savings interest and pension income leave carers_allowance_earnings unchanged. @@ -57,7 +65,11 @@ 9. Each deduction counts a fraction of a penny as a penny (reg. 5): with tax and Class 1 or Class 4 to pay, from 2024-25 (Class 2 nil, Class 1 at 8%), net earnings are P less each of the tax, the contributions and half the - pension contribution, rounded up to a whole penny, exactly. + pension contribution, rounded up to a whole penny, exactly. Class 1 is 8% + of P less 52 x £241.73, worked exactly here, not read from the model's + float32 ni_class_1_employee, which can lie a hundredth of a penny from it. +10. A pension total entered directly as pension_contributions comes off + earnings as the same amount entered as a programme does. """ import datetime @@ -445,19 +457,38 @@ def _with_fractions(route, year): return threshold, pairs +# Pairs found in review of #2543, in pence. C1 (juaristi22): a trader within +# the limit whom the earlier nearest-half-penny rule put over it. The +# employees' Class 1 is a whole number of pence (£175.56, £235.38, £1.14, +# £1.26, £2.28) that the model's float32 sum of twelve months holds more than +# half a step out, so reading it with stored_pence and rounding up deducted a +# penny too much and put each of them, exactly a penny a week over, within +# the limit. +REVIEW_PAIRS = { + ("trading", 2026): [(1_400_001, 603_938)], + ("employee", 2024): [(1_476_446, 1_347_276), (1_551_221, 1_484_862)], + ("employee", 2026): [ + (1_476_446, 796_076), + (1_551_221, 933_662), + (1_258_421, 394_910), + (1_258_571, 395_186), + (1_259_846, 397_532), + ], +} + + @pytest.mark.parametrize("route", ["trading", "employee"]) @pytest.mark.parametrize("year", [2024, 2026]) def test_each_deduction_counts_a_fraction_of_a_penny_as_a_penny(route, year): """Invariant 9, through the model: net earnings are P less each deduction rounded up to a whole penny (reg. 5), exactly, and the person - is over the limit exactly when that reaches the threshold. Class 1 is the - model's ni_class_1_employee, read as stored_pence reads it. Rounding only - the total, or not at all, misjudges some of these pairs: juaristi22's - review of #2543 found a trader within the limit (C1, below) whom the - earlier nearest-half-penny rule put over it.""" + is over the limit exactly when that reaches the threshold. Class 1 is + worked exactly from its rate and threshold. Rounding only the total, or + not at all, misjudges some of these pairs, and so does rounding up the + model's float32 Class 1 (REVIEW_PAIRS).""" threshold, pairs = _with_fractions(route, year) - if route == "trading" and year == 2026: - pairs.append((1_400_001, 603_938)) + review_pairs = REVIEW_PAIRS.get((route, year), []) + pairs += review_pairs gross, contribution = ( ("self_employment_income", "personal_pension_contributions") if route == "trading" @@ -495,6 +526,12 @@ def test_each_deduction_counts_a_fraction_of_a_penny_as_a_penny(route, year): ) basic = Fraction(str(hmrc.income_tax.rates.uk.rates[0])) main = Fraction(str(hmrc.national_insurance.class_4.rates.main)) + # Class 1 at 8% above 52 weeks of £241.73 (the model's £12,569.96), as + # employee_scale.yaml sets for both years. + employee_scale = hmrc.national_insurance.class_1.employee_scale + assert employee_scale.components[1].rate == 0.08 + assert employee_scale.components[1].threshold == 241.73 + class_1_rate, primary_threshold = Fraction(8, 100), WEEKS * 24_173 fractions = 0 for (p, c), stored, judged, ni in zip(pairs, earnings, gainful, class_1): if route == "trading": @@ -502,7 +539,7 @@ def test_each_deduction_counts_a_fraction_of_a_penny_as_a_penny(route, year): deductions = [basic * (p - pa), main * (p - lpl)] else: assert p - c < pa and ni > 0, (p, c) - deductions = [_stored_pence(ni)] + deductions = [class_1_rate * (p - primary_threshold)] deductions.append(Fraction(c, 2)) net = p - sum(math.ceil(d) for d in deductions) fractions += any(d.denominator > 1 for d in deductions) @@ -510,6 +547,17 @@ def test_each_deduction_counts_a_fraction_of_a_penny_as_a_penny(route, year): assert bool(judged) == (net >= threshold), (p, c, net) # Most pairs carry a fraction of a penny in some deduction. assert fractions > len(pairs) // 2 + if route == "employee": + # Each review pair is exactly the threshold: a penny a week over. + for (p, c), judged in zip( + review_pairs, gainful[len(pairs) - len(review_pairs) :] + ): + net = ( + p + - math.ceil(class_1_rate * (p - primary_threshold)) + - math.ceil(Fraction(c, 2)) + ) + assert net == threshold and judged, (p, c, net) if route == "trading" and year == 2026: # C1: £14,000.01 of profit and a £6,039.38 premium. Tax £286.002 and # Class 4 £85.8006 round up to £286.01 and £85.81, and half the @@ -621,22 +669,83 @@ def test_the_formula_matches_exact_arithmetic_and_rises_with_earnings(case): assert np.all(np.diff(in_order) >= 0), (year, amounts) -@settings( - max_examples=8, - deadline=None, - derandomize=True, - suppress_health_check=[HealthCheck.too_slow], -) -@given(pay=money, rise=st.floats(min_value=1, max_value=20_000)) -def test_net_earnings_never_fall_as_pay_rises(pay, rise): - lower = _person(pay, 0, 0) - higher = _person(pay + rise, 0, 0) - assert ( - float(higher["carers_allowance_earnings"]) - >= float(lower["carers_allowance_earnings"]) - 0.01 +def _net_pence(variable, pence, other=None): + """carers_allowance_earnings in whole pence in 2026, one carer in England + per amount of ``variable`` (in pence), with ``other`` amounts as well.""" + amounts = {2026: [p / 100 for p in pence]} + simulation = _carers(variable, amounts) + for name, value in (other or {}).items(): + simulation.set_input(name, 2026, np.full(len(pence), value)) + earnings = simulation.calculate("carers_allowance_earnings", 2026) + gainful = simulation.calculate("is_gainfully_employed_for_carers_allowance", 2026) + return np.round(earnings.astype(np.float64) * 100).astype(int), gainful + + +# Every penny across the points where a deduction starts or changes rate in +# 2026-27: the personal allowance and Class 1 primary threshold and lower +# profits limit (£12,570), the higher rate, upper earnings limit and upper +# profits limit (£50,270) and the additional rate (£125,140); and, for +# profit, the boundary pair of YAML Cases 30 and 31. +MONOTONIC_RANGES = { + "employment_income": [ + (1_255_000, 1_259_000), + (5_025_000, 5_029_000), + (12_512_000, 12_516_000), + ], + "self_employment_income": [(1_255_000, 1_259_000), (5_025_000, 5_029_000)], +} + + +@pytest.mark.parametrize("variable", list(MONOTONIC_RANGES)) +def test_net_earnings_never_fall_for_a_rise_of_two_pence_or_more(variable): + """Invariant 3, for every penny in each range: net earnings at P + R are + at least those at P for every R >= 2p, and a 1p rise lowers them by at + most 1p. Some 1p rises do, as reg. 5 intends.""" + one_penny_falls = 0 + for start, end in MONOTONIC_RANGES[variable]: + pence = list(range(start, end)) + net, _ = _net_pence(variable, pence) + # The least net earnings at any P + R, R >= 2, for each P. + later_minimum = np.minimum.accumulate(net[::-1])[::-1] + assert np.all(later_minimum[2:] >= net[:-2]), (variable, start) + steps = np.diff(net) + assert steps.min() >= -1, (variable, start) + one_penny_falls += int(np.sum(steps == -1)) + if variable == "self_employment_income": + assert one_penny_falls > 0 + + +def test_a_penny_more_profit_can_lower_net_earnings_by_a_penny(): + """The intended exception to invariant 3, with a premium of £3,923.70: + £12,570.50 of profit takes notional tax of 10p and Class 4 of 3p, leaving + £10,608.52, a penny a week over the limit; £12,570.51 takes 10.2p and + 3.06p, which count as 11p and 4p (reg. 5), leaving £10,608.51, within + it. Pinned also as YAML Cases 30 and 31.""" + net, gainful = _net_pence( + "self_employment_income", + [1_257_050, 1_257_051], + {"personal_pension_contributions": 3_923.70}, ) - if lower["is_gainfully_employed_for_carers_allowance"]: - assert higher["is_gainfully_employed_for_carers_allowance"] + assert list(net) == [1_060_852, 1_060_851] + assert list(gainful) == [True, False] + + +@pytest.mark.parametrize( + "pay, total", + [(1_100_000, 100_000), (1_100_103, 99_201), (785_253, 1), (1_060_853, 1)], +) +def test_a_pension_total_entered_directly_counts_like_a_programme(pay, total): + """Invariant 10: pension_contributions entered as a total comes off + earnings, half of it with the half penny counted as a penny, exactly as + the same amount entered as personal_pension_contributions (which, like the + total, leaves PAYE alone).""" + results = [] + for name in ("pension_contributions", "personal_pension_contributions"): + net, gainful = _net_pence("employment_income", [pay], {name: total / 100}) + results.append((int(net[0]), bool(gainful[0]))) + assert results[0] == results[1] + if pay < 1_257_000: + assert results[0][0] == pay - math.ceil(total / 2) @settings( diff --git a/policyengine_uk/tests/test_stored_pence.py b/policyengine_uk/tests/test_stored_pence.py index 32644d54ce..4fb17d7d08 100644 --- a/policyengine_uk/tests/test_stored_pence.py +++ b/policyengine_uk/tests/test_stored_pence.py @@ -10,15 +10,25 @@ 3. Fractions kept: a stored amount more than half a step from every whole penny comes back exactly, to be rounded as the rule requires. 4. Monotonic: a larger stored amount never gives fewer pence. + +stored_sum_pence: a float32 sum of float32 amounts, in pence. + +5. Sums recovered: a float32 sum of up to five whole-pence amounts, added + one at a time as policyengine-core's ``adds`` does, comes back as exactly + the sum of their pence while it is below £131,072, including sums that + stored_pence alone would not recover. +6. Entered totals kept: a total that is not a float32 sum of the parts, as + when it is entered directly, comes back as stored_pence(total). """ from fractions import Fraction +from functools import reduce import numpy as np from hypothesis import example, given, settings from hypothesis import strategies as st -from policyengine_uk.utils.pence import stored_pence +from policyengine_uk.utils.pence import stored_pence, stored_sum_pence # Below 2^17 pounds a float32 step is under a penny. LIMIT_PENCE = 2**17 * 100 @@ -39,7 +49,7 @@ def test_every_whole_pence_amount_below_2_to_the_17_pounds_round_trips(): ) -@settings(max_examples=300, derandomize=True) +@settings(max_examples=300, deadline=None, derandomize=True) @given(amount=float32_amounts) # £10,608.52 is stored as 10,608.51953125; the float32 below it is a # fraction of a penny short of £10,608.52. @@ -58,9 +68,50 @@ def test_moves_an_amount_at_most_half_a_step_and_keeps_fractions(amount): assert result == whole -@settings(max_examples=200, derandomize=True) +@settings(max_examples=200, deadline=None, derandomize=True) @given(amounts=st.lists(float32_amounts, min_size=2, max_size=50)) @example(amounts=[10_608.517578125, 10_608.51953125, 10_608.5205078125]) def test_never_gives_fewer_pence_for_more_money(amounts): stored = np.sort(np.array(amounts, dtype=np.float32)) assert np.all(np.diff(stored_pence(stored)) >= 0) + + +def _float32_sum(parts): + """Parts in pence, stored and added one at a time in float32.""" + stored = [(np.asarray(p) / 100).astype(np.float32) for p in parts] + return reduce(np.add, stored), stored + + +def test_sums_of_whole_pence_are_recovered(): + rng = np.random.default_rng(2541) + missed_by_stored_pence = 0 + for count in range(1, 6): + # Up to five parts whose sum is below £131,072. + parts = [rng.integers(0, LIMIT_PENCE // count, 400_000) for _ in range(count)] + total, stored = _float32_sum(parts) + exact = sum(parts) + assert np.array_equal(stored_sum_pence(total, stored), exact), count + missed_by_stored_pence += int(np.sum(stored_pence(total) != exact)) + # Summing in float32 leaves some totals too far from their pence for + # stored_pence alone, which is why the parts are summed again. + assert missed_by_stored_pence > 0 + + +@settings(max_examples=200, deadline=None, derandomize=True) +@given( + parts=st.lists(st.integers(0, LIMIT_PENCE // 5 - 1), min_size=1, max_size=5), + entered=st.integers(0, LIMIT_PENCE - 1), +) +# £1,000 entered as a pension total, with no programme amounts. +@example(parts=[0, 0], entered=100_000) +def test_a_total_entered_directly_is_read_as_entered(parts, entered): + total, stored = _float32_sum([[p] for p in parts]) + entered_total = np.array([entered / 100], dtype=np.float32) + result = stored_sum_pence(entered_total, stored) + if abs(entered - sum(parts)) <= len(parts): + # Within float32 rounding of the parts' sum (a penny for each part + # here, far less below £131,072): read as that sum. + assert result[0] in (entered, sum(parts)) + else: + assert result[0] == entered + assert stored_sum_pence(total, stored)[0] == sum(parts) diff --git a/policyengine_uk/utils/pence.py b/policyengine_uk/utils/pence.py index ec57413a1b..de02155716 100644 --- a/policyengine_uk/utils/pence.py +++ b/policyengine_uk/utils/pence.py @@ -28,3 +28,23 @@ def stored_pence(amount): whole = np.round(pence) half_step = np.spacing(np.abs(amount)).astype(np.float64) * 50 return np.where(np.abs(pence - whole) <= half_step, whole, pence) + + +def stored_sum_pence(total, parts): + """``total``, a float32 sum of the float32 amounts ``parts``, in pence as + float64: the sum of the parts' stored_pence, where ``total`` lies within + float32 rounding of it, and otherwise stored_pence(total), for a total + entered directly rather than summed. + + Summing in float32 rounds at each addition, so a sum of whole-pence + amounts can lie more than half a step from its pence even when each part + is recovered exactly; summing the parts' pence in float64 is exact. Each + addition moves a sum of amounts no larger than it by at most half a step + of the total, and each part lies within half a step of its own pence, so + a sum of n parts lies within n steps of the parts' pence. + """ + total = np.asarray(total, dtype=np.float32) + exact = sum(stored_pence(part) for part in parts) + tolerance = np.spacing(np.abs(total)).astype(np.float64) * 100 * len(parts) + summed = np.abs(total.astype(np.float64) * 100 - exact) <= tolerance + return np.where(summed, exact, stored_pence(total)) diff --git a/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py b/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py index f9c87c6d28..a54834cdba 100644 --- a/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py +++ b/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py @@ -1,6 +1,6 @@ from policyengine_uk.model_api import * from policyengine_uk.utils.class_2 import class_2_contribution_weeks, class_2_liable -from policyengine_uk.utils.pence import stored_pence +from policyengine_uk.utils.pence import stored_pence, stored_sum_pence def _earned(amount): @@ -16,6 +16,68 @@ def _deducted(pence): return np.ceil(pence - 1e-6) +def _primary_class_1(person, period, parameters): + """ni_class_1_employee in float64 pence: its formula (the primary and + additional parts, month by month) worked exactly on the year's NI income + in whole pence, where that agrees with the stored total to within the + error of the model's float32 arithmetic; otherwise, as under a reform + that replaces the formula, the stored total. + + The model works out each month's Class 1 in float32 and sums the twelve, + which can leave a whole-penny total further from it than stored_pence + recovers: £175.56 (8% of £14,764.46 less £12,569.96) is stored as + 175.5600128, and rounding that up would deduct a penny too much. + """ + income = stored_sum_pence( + person("ni_class_1_income", period), + [ + person(name, period) + for name in person.simulation.tax_benefit_system.variables[ + "ni_class_1_income" + ].adds + ], + ) + director = person("is_full_year_category_a_director", period) + exact = 0 + for month in period.get_subperiods(MONTH): + class_1 = parameters(month).gov.hmrc.national_insurance.class_1 + scale = class_1.employee_scale + # Weekly thresholds in pounds, as monthly amounts in pence. + primary_threshold = ( + scale.components[1].threshold * WEEKS_IN_YEAR * 100 / MONTHS_IN_YEAR + ) + upper_limit = ( + scale.components[2].threshold * WEEKS_IN_YEAR * 100 / MONTHS_IN_YEAR + ) + monthly = income / MONTHS_IN_YEAR + above_upper = max_(monthly - upper_limit, 0) + contribution = ( + scale.components[1].rate + * (max_(monthly - primary_threshold, 0) - above_upper) + + scale.components[2].rate * above_upper + ) + directors = class_1.directors.full_year_category_a_2022_23 + if directors.in_effect: + # Annual thresholds in pounds. + rates = directors.employee_scale.rates + thresholds = [t * 100 for t in directors.employee_scale.thresholds] + annual = rates[1] * min_( + max_(income - thresholds[1], 0), thresholds[2] - thresholds[1] + ) + rates[2] * max_(income - thresholds[2], 0) + contribution = where(director, annual / MONTHS_IN_YEAR, contribution) + exact = exact + contribution + exact = exact * person("ni_liable", period) + # The twelve float32 months and their sum err by less than one float32 + # step of the year's NI income plus 64 steps of the contribution. + stored = person("ni_class_1_employee", period) + tolerance = 100 * ( + np.spacing(np.abs(person("ni_class_1_income", period))).astype(np.float64) + + 64 * np.spacing(np.abs(stored)).astype(np.float64) + ) + agrees = np.abs(np.asarray(stored, dtype=np.float64) * 100 - exact) <= tolerance + return where(agrees, exact, stored_pence(stored)) + + class carers_allowance_earnings(Variable): value_type = float entity = Person @@ -50,11 +112,23 @@ class carers_allowance_earnings(Variable): "ni_class_4_liable: reaching State Pension age during the tax year " "leaves that year's contributions payable.\n\n" "A fraction of a penny counts in the claimant's favour (reg. 5): pay " - "and profit are taken down to a whole penny and each deduction up to " - "one, so net earnings are whole pence. The pence are worked in double " - "precision from the whole pence each amount was entered as, since the " - "model stores amounts in single precision, which holds most sums of " - "money only to within a fraction of a penny.\n\n" + "and profit are taken down to a whole penny, and the calculated " + "deductions (notional tax, Class 4 and Class 2, and half of pension " + "contributions) up to one. PAYE and Class 1 are in law the amounts " + "actually deducted, which are whole pence; the model's figures for " + "them are rounded up as well, a fraction counting in the claimant's " + "favour. Net earnings are therefore whole pence. They are worked in " + "double precision from the whole pence each amount was entered as, " + "since the model stores amounts in single precision, which holds most " + "sums of money only to within a fraction of a penny. Class 1, which the " + "model sums from twelve single-precision months, is worked out again " + "from its formula. The model holds each pension programme as one " + "amount for the year, so half of the year's total is deducted, rounded " + "up once.\n\n" + "The Scottish Carer Support Payment reads these earnings too " + "(carer_support_payment_pre_couple_limit): SSI 2023/302 Schedule 2 has " + "the same rounding rule (para. 4) and net earnings and profit (paras. 9 " + "and 12).\n\n" "Not modelled: the deduction for care charges (Schedule 3, up to half " "of net earnings) and the Schedule 1 disregards, so this can overstate " "earnings for a carer who pays for care while at work." @@ -97,8 +171,9 @@ def formula(person, period, parameters): # Work in float64 pence from the whole pence each amount was entered # as (stored_pence), taking pay and profit down to a whole penny and - # each deduction up to one (reg. 5). Scales and thresholds are in - # pounds. + # each deduction up to one (reg. 5 for the calculated deductions; for + # PAYE and Class 1, which the law takes as deducted, the same rounding + # in the claimant's favour). Scales and thresholds are in pounds. # Employment: PAYE on the pay alone (reg. 10(4)(a)). pay = _earned(person("employment_income", period)) @@ -113,10 +188,7 @@ def formula(person, period, parameters): ), income_tax.rates.uk.calc(max_(0, taxable_pay - personal_allowance)), ) - # Computed in float32 by another variable: a whole-penny amount that - # comes back more than half a step out counts as a penny more, in the - # claimant's favour. - primary_class_1 = stored_pence(person("ni_class_1_employee", period)) + primary_class_1 = _primary_class_1(person, period, parameters) # Self-employment: notional basic-rate tax and main-rate Class 4 # (reg. 14(1)-(2)). The tax is on the profit as if it were the only @@ -158,14 +230,17 @@ def formula(person, period, parameters): ) # Regs. 10(4)(b) and 13(1)(b)(ii), (4)(c) and (10)(b): one half of - # pension contributions and premiums. The contributions are those - # pension_contributions adds, summed here in float64 pence. + # pension contributions and premiums. The model holds each programme + # as one amount for the year, not the payments the law halves one by + # one, so the half is of the year's total, rounded up once. The total + # is pension_contributions, whether entered or summed from its + # programmes; a sum is redone in float64 pence. pension_share = parameters( period ).gov.dwp.carers_allowance.pension_contribution_deduction_rate - contributions = sum( - stored_pence(person(program, period)) - for program in p.pensions.pensions_programs + contributions = stored_sum_pence( + person("pension_contributions", period), + [person(program, period) for program in p.pensions.pensions_programs], ) pension_deduction = pension_share * contributions net = ( diff --git a/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py b/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py index 2443ab44e4..ab5fdad684 100644 --- a/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py +++ b/policyengine_uk/variables/gov/dwp/is_gainfully_employed_for_carers_allowance.py @@ -11,7 +11,9 @@ class is_gainfully_employed_for_carers_allowance(Variable): "earnings limit, so that they are treated as gainfully employed and " "are not entitled to the allowance. The regulation tests the earnings " "of the preceding week; annual earnings are spread evenly over 52 " - "weeks. A fraction of a penny is disregarded, as regulation 5 of the " + "weeks, for an employed earner's yearly pay (reg. 8(1)(b)(iii)) and a " + "self-employed earner's average weekly earnings over a year (reg. " + "11(1)(a)). A fraction of a penny is disregarded, as regulation 5 of the " "Computation of Earnings Regulations requires when that favours the " "claimant. Net earnings are whole pence, since carers_allowance_earnings " "applies regulation 5 to each step, and the test reads back the whole " @@ -36,6 +38,10 @@ class is_gainfully_employed_for_carers_allowance(Variable): title="Social Security Benefit (Computation of Earnings) Regulations 1996 reg. 8(1)(b)(iii)", href="https://www.legislation.gov.uk/uksi/1996/2745/regulation/8", ), + dict( + title="Social Security Benefit (Computation of Earnings) Regulations 1996 reg. 11(1)(a)", + href="https://www.legislation.gov.uk/uksi/1996/2745/regulation/11", + ), ] def formula(person, period, parameters): @@ -47,7 +53,7 @@ def formula(person, period, parameters): # directly with a fraction of a penny has the fraction disregarded # (reg. 5). annual_pence = np.floor(stored_pence(earnings)) - # Spread over 52 weeks (reg. 8(1)(b)(iii)), disregarding the fraction + # Spread over 52 weeks (regs. 8(1)(b)(iii) and 11(1)(a)), disregarding the fraction # of a penny that leaves (reg. 5). Both steps are exact in float64, so # no floating-point rounding decides whether earnings have exceeded # the limit (ICA Regulations reg. 8(1)). From 8f1df68a740ff8399191c443c488456fc14b1104 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 28 Oct 2026 08:58:14 -0400 Subject: [PATCH 07/13] Pin two pension programmes summed in whole pence (CA Case 32) A float32 sum of the two programmes can lie more than half a step from its pence; read alone, half of it would round up a penny too far. This case fails if stored_sum_pence reads a sum as stored. Co-Authored-By: Claude Opus 5.5 --- .../dwp/carers_allowance_earnings_limit.yaml | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml b/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml index 1c807b3dce..39851bfc19 100644 --- a/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml +++ b/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml @@ -504,3 +504,24 @@ carers_allowance_earnings: 10_608.51 is_gainfully_employed_for_carers_allowance: false carers_allowance: 86.45 * 52 + +- name: Case 32, two pension programmes are summed in whole pence, not as float32 adds them. + period: 2026 + absolute_error_margin: 0.001 + input: + age: 45 + employment_income: 12_229.84 + employee_pension_contributions: 1_807.49 + personal_pension_contributions: 1_435.15 + care_hours: 40 + country: ENGLAND + output: + # The contributions total 3,242.64, which pension_contributions holds as + # 3,242.64013671875: more than half a float32 step out, so read alone it + # would be 3,242.6401367 and half of it would count as 1,621.33. Summed + # in whole pence, half is 1,621.32. Pay is under the primary threshold, + # and pay less the employee contribution under the personal allowance: + # 12,229.84 - 1,621.32 = 10,608.52, or £204.01 a week, over the limit. + carers_allowance_earnings: 10_608.52 + is_gainfully_employed_for_carers_allowance: true + carers_allowance: 0 From 6e8a53fa5cabc1f839632d6761793cb5e2eef27a Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 28 Oct 2026 10:39:16 -0400 Subject: [PATCH 08/13] Read model sums in whole pence; deduct entered Class 1 as stored Fixes the three findings of the review of 8f1df68a7, and a fourth of the same kind found while fixing them: - Class 1 entered as an input (nil, or any amount near the formula's) was replaced by the formula worked exactly whenever the two lay within float32 error. Class 1 is now worked out again only where the model calculated it with its own formula: no reform changed how it or its parts are calculated, and no month was supplied or stored by a scenario. Otherwise the stored amount is the one deducted (reg. 10(4)(a)). - Sums nested in sums (employee_pension_contributions inside pension_contributions, and employment_income itself) kept float32 noise. amount_pence now recurses through adds/subtracts of amounts the simulation derived, reading each as stored_pence does, and takes their exact sum where the float32 total lies within the rounding of the additions that formed it. Pay is read this way too: 10,276.42 + 1,415.69 is stored as 11,692.109375 and was floored to 11,692.10. - A formula in place of a sum (adds=None) crashed the reader; a variable with a formula, defined_for, another period or another entity's part is now read as stored. YAML Cases 33-36 pin the four executed cases, each a penny a week over the 2026-27 limit; model-level tests pin entered and reformed Class 1, a parameter reform, a formula reform and 1,203 sums the model forms (600 with Class 1, 603 without NI), and the every-year test checks Class 1 against an independent Fraction restatement from 2015 to the horizon. Co-Authored-By: Claude Opus 5.5 --- changelog.d/2541.fixed.md | 2 +- .../dwp/carers_allowance_earnings_limit.yaml | 78 +++++ ...ers_allowance_earnings_limit_properties.py | 329 ++++++++++++++++++ policyengine_uk/tests/test_stored_pence.py | 118 +++++-- policyengine_uk/utils/pence.py | 129 ++++++- .../gov/dwp/carers_allowance_earnings.py | 132 +++++-- 6 files changed, 705 insertions(+), 83 deletions(-) diff --git a/changelog.d/2541.fixed.md b/changelog.d/2541.fixed.md index 614137005c..da2c6a70b3 100644 --- a/changelog.d/2541.fixed.md +++ b/changelog.d/2541.fixed.md @@ -1 +1 @@ -Fixed the Carer's Allowance earnings limit treating earnings a penny a week over the 2020-21 to 2024-25 limits (£128, £132, £139 and £151) as within them. Net earnings now apply the Computation of Earnings Regulations' rounding to each step: pay and profit are taken down to a whole penny and each deduction (tax, National Insurance and half of pension contributions) up to one. They are worked in double precision from the whole pence each amount was entered as, with Class 1 worked out again from its formula, and the limit test reads the whole pence back exactly, so float32 rounding no longer decides it. Pension contributions entered as a total still reduce earnings. The Scottish Carer Support Payment, which reads the same earnings, is judged exactly at its limit too. +Fixed the Carer's Allowance earnings limit treating earnings a penny a week over the 2020-21 to 2024-25 limits (£128, £132, £139 and £151) as within them. Net earnings now apply the Computation of Earnings Regulations' rounding to each step: pay and profit are taken down to a whole penny and each calculated deduction (notional tax, Class 4 and half of pension contributions) up to one; the model's PAYE and Class 1, which in law are the amounts actually deducted, are rounded up too. Earnings are worked in double precision from the whole pence each amount was entered as, and a sum the model forms (pay, pension contributions, NI income) is the exact sum of its parts. Class 1 is worked out again from its formula where the model calculated it; entered as an input, or calculated under a reform of its formula, it is deducted as stored. The limit test reads the whole pence back exactly, so float32 rounding no longer decides it. Pension contributions entered as a total still reduce earnings. The Scottish Carer Support Payment, which reads the same earnings, is judged exactly at its limit too. diff --git a/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml b/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml index 39851bfc19..f0fec1061c 100644 --- a/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml +++ b/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml @@ -525,3 +525,81 @@ carers_allowance_earnings: 10_608.52 is_gainfully_employed_for_carers_allowance: true carers_allowance: 0 + +- name: Case 33, Class 1 entered as nil is deducted as nil, not as the formula would work it. + period: 2026 + absolute_error_margin: 0.001 + input: + age: 45 + employment_income: 12_569.97 + employee_pension_contributions: 3_922.90 + ni_class_1_employee: 0 + care_hours: 40 + country: ENGLAND + output: + # The model's formula would charge 8% of the penny over its £12,569.96 + # primary threshold, 0.0008, counted as a penny. Entered as nil, nil is + # what was deducted (reg. 10(4)(a)): 12,569.97 - 3,922.90 / 2 = 10,608.52, + # or £204.01 a week, over the limit. + carers_allowance_earnings: 10_608.52 + is_gainfully_employed_for_carers_allowance: true + carers_allowance: 0 + +- name: Case 34, a pension total the model sums from two inputs is read in whole pence. + period: 2026 + absolute_error_margin: 0.001 + input: + age: 45 + employment_income_before_lsr: 10_608.96 + salary_sacrifice_returned_to_income: 0.20 + employee_pension_contributions_reported: 1.08 + care_hours: 40 + country: ENGLAND + output: + # Pay is 10,608.96 + 0.20 = 10,609.16; employee contributions are + # 1.08 + 0.20 = 1.28, which float32 adds to 1.28000009, so that half of + # it read alone would count as 0.65. Summed in whole pence, half is 0.64: + # 10,609.16 - 0.64 = 10,608.52, or £204.01 a week, over the limit. + carers_allowance_earnings: 10_608.52 + is_gainfully_employed_for_carers_allowance: true + carers_allowance: 0 + +- name: Case 35, pay the model sums from two inputs is read in whole pence. + period: 2026 + absolute_error_margin: 0.001 + input: + age: 45 + employment_income_before_lsr: 10_276.42 + salary_sacrifice_returned_to_income: 1_415.69 + employee_pension_contributions_reported: 751.49 + care_hours: 40 + country: ENGLAND + output: + # Pay is 10,276.42 + 1,415.69 = 11,692.11, which float32 adds to + # 11,692.109375: more than half a float32 step out, so read alone it would + # count as 11,692.10. Contributions are 751.49 + 1,415.69 = 2,167.18, half + # 1,083.59. Pay is under the primary threshold and pay less contributions + # under the personal allowance: 11,692.11 - 1,083.59 = 10,608.52, or + # £204.01 a week, over the limit. + carers_allowance_earnings: 10_608.52 + is_gainfully_employed_for_carers_allowance: true + carers_allowance: 0 + +- name: Case 36, pay and pension both summed by the model are read in whole pence. + period: 2026 + absolute_error_margin: 0.001 + input: + age: 45 + employment_income_before_lsr: 10_794.69 + salary_sacrifice_returned_to_income: 1_435.15 + employee_pension_contributions_reported: 1_807.49 + care_hours: 40 + country: ENGLAND + output: + # Pay is 10,794.69 + 1,435.15 = 12,229.84 (float32: 12,229.8408203125) + # and contributions 1,807.49 + 1,435.15 = 3,242.64 (3,242.64013671875), + # each more than half a float32 step out of its pence. Summed in whole + # pence: 12,229.84 - 3,242.64 / 2 = 10,608.52, or £204.01 a week, over. + carers_allowance_earnings: 10_608.52 + is_gainfully_employed_for_carers_allowance: true + carers_allowance: 0 diff --git a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py index 35fb4cfd34..65d7110407 100644 --- a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py +++ b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py @@ -70,6 +70,22 @@ float32 ni_class_1_employee, which can lie a hundredth of a penny from it. 10. A pension total entered directly as pension_contributions comes off earnings as the same amount entered as a programme does. + +And for the amounts the deductions start from: + +11. Class 1 is the amount the model calculates: where it calculated + ni_class_1_employee with its own formula, as it does in every year from + 2015 to the horizon, including under a parameter reform, the deduction + is that formula worked exactly. Where Class 1 (for the year or month by + month) or its primary part was supplied as an input, or a reform + neutralized or replaced the formula, the deduction is the stored amount, + as the amount deducted (reg. 10(4)(a)), even within a fraction of a penny + of what the formula would give. +12. Sums the model forms are read in whole pence: pay, NI income and pension + contributions, which the model adds from other amounts in float32, are + the exact sums of their whole-penny parts, and net earnings follow + exact arithmetic on them. +13. An amount calculated by a formula in place of a sum is read as stored. """ import datetime @@ -85,6 +101,7 @@ import policyengine_uk from policyengine_uk import Simulation +from policyengine_uk.model_api import GBP, MONTH, YEAR, Person, Reform, Variable from policyengine_uk.system import system from policyengine_uk.utils.class_2 import class_2_contribution_weeks @@ -815,3 +832,315 @@ def test_trading_profit_takes_the_basic_rate_in_every_year(year, country): assert simulation.calculate("carers_allowance_earnings", year)[0] == pytest.approx( expected, abs=0.01 ) + + +def _earnings_of(rows, reform=None, year=2026): + """Net earnings in whole pence, and whether each is over the limit, for + one carer in England per dict in ``rows``: inputs by variable, each an + amount for ``year`` or a dict of amounts by period.""" + people = {} + for i, row in enumerate(rows): + person = {"age": {year: 45}, "care_hours": {year: 40}} + for name, value in row.items(): + person[name] = value if isinstance(value, dict) else {year: value} + people[f"carer_{i}"] = person + simulation = Simulation( + situation={ + "people": people, + "benunits": { + f"b{i}": {"members": [f"carer_{i}"]} for i in range(len(rows)) + }, + "households": { + f"h{i}": {"members": [f"carer_{i}"], "country": {year: "ENGLAND"}} + for i in range(len(rows)) + }, + }, + reform=reform, + ) + earnings = simulation.calculate("carers_allowance_earnings", year) + gainful = simulation.calculate("is_gainfully_employed_for_carers_allowance", year) + return [int(round(float(e) * 100)) for e in earnings], [bool(g) for g in gainful] + + +# The 2026-27 threshold: 52 x (£204 + 1p) a year, a penny a week over. +THRESHOLD_2026 = WEEKS * (LIMIT * 100 + 1) +MONTHS_2026 = [f"2026-{month:02d}" for month in range(1, 13)] + + +def _class_1(pence): + """Class 1 at 8% of pay above the model's £12,569.96 in 2026, exactly.""" + return Fraction(8, 100) * max(0, pence - WEEKS * 24_173) + + +@pytest.mark.parametrize( + "class_1_input", + [ + {"ni_class_1_employee": 0}, + {"ni_class_1_employee": {month: 0 for month in MONTHS_2026}}, + {"ni_class_1_employee_primary": {month: 0 for month in MONTHS_2026}}, + ], + ids=["year", "months", "primary part"], +) +def test_class_1_supplied_as_an_input_is_deducted_as_supplied(class_1_input): + """Invariant 11, for an input. £12,569.97 of pay is a penny above the + model's primary threshold, so its formula gives Class 1 of 0.08p, which + counts as a penny. Entered as nil, nil is deducted (reg. 10(4)(a)): with + a £3,922.90 contribution, £12,569.97 less £1,961.45 leaves £10,608.52, a + penny a week over the limit.""" + pay, contribution = 1_256_997, 392_290 + row = { + "employment_income": pay / 100, + "employee_pension_contributions": contribution / 100, + } + # Separate simulations: an input set for one person is set for each. + calculated, calculated_over = _earnings_of([row]) + supplied, supplied_over = _earnings_of([{**row, **class_1_input}]) + assert calculated == [pay - math.ceil(_class_1(pay)) - contribution // 2] + assert supplied == [pay - contribution // 2] + assert calculated + supplied == [1_060_851, 1_060_852] + assert calculated_over + supplied_over == [False, True] + + +def test_class_1_supplied_near_its_formula_is_deducted_as_supplied(): + """Invariant 11: £594.40 of Class 1 entered with £20,000 of pay, within a + third of a penny of what the formula gives (£594.4032), is the amount + deducted, not the formula's rounded up.""" + pay = 2_000_000 + paye = math.ceil(Fraction(20, 100) * (pay - 1_257_000)) + calculated, _ = _earnings_of([{"employment_income": pay / 100}]) + supplied, _ = _earnings_of( + [{"employment_income": pay / 100, "ni_class_1_employee": 594.40}] + ) + assert calculated == [pay - paye - math.ceil(_class_1(pay))] + assert supplied == [pay - paye - 59_440] + + +class _NoClass1(Reform): + def apply(self): + self.neutralize_variable("ni_class_1_employee") + + +class _NoPrimaryClass1(Reform): + """The primary part replaced by a formula giving nil.""" + + def apply(self): + class ni_class_1_employee_primary(Variable): + label = "NI Class 1 employee-side primary contributions" + entity = Person + definition_period = MONTH + value_type = float + unit = GBP + + def formula(person, period, parameters): + return person("ni_class_1_income", period) * 0 + + self.replace_variable(ni_class_1_employee_primary) + + +@pytest.mark.parametrize("reform", [_NoClass1, _NoPrimaryClass1]) +def test_class_1_under_a_reform_of_its_formula_is_the_amount_calculated(reform): + """Invariant 11, for a reform: with Class 1 neutralized, or its primary + part replaced by a nil formula, nothing is deducted, not the 0.08p (a + penny) the model's own formula gives at £12,569.97 of pay.""" + row = {"employment_income": 12_569.97, "employee_pension_contributions": 3_922.90} + net, gainful = _earnings_of([row], reform=reform) + assert net == [1_060_852] and gainful == [True] + + +@pytest.mark.parametrize( + "reform, rate", + [ + # The upper earnings limit moved far above this pay: Class 1 is still + # £175.56, which the model's float32 sum holds as 175.5600128. + ( + { + "gov.hmrc.national_insurance.class_1.employee_scale[2].threshold": { + "2026-01-01.2100-12-31": 1_000 + } + }, + Fraction(8, 100), + ), + # A main rate of 10%: £219.45 exactly. + ( + { + "gov.hmrc.national_insurance.class_1.employee_scale[1].rate": { + "2026-01-01.2100-12-31": 0.1 + } + }, + Fraction(10, 100), + ), + ], + ids=["upper earnings limit", "main rate"], +) +def test_class_1_under_a_parameter_reform_is_worked_exactly(reform, rate): + """Invariant 11: a parameter reform changes the formula's parameters, not + the formula, so Class 1 is still worked exactly, with them.""" + pay = 1_476_446 + paye = math.ceil(Fraction(20, 100) * (pay - 1_257_000)) + class_1 = math.ceil(rate * (pay - WEEKS * 24_173)) + net, _ = _earnings_of([{"employment_income": pay / 100}], reform=reform) + assert net == [pay - paye - class_1] + + +class _NIIncomeFormula(Reform): + """ni_class_1_income calculated by a formula rather than as a sum.""" + + def apply(self): + class ni_class_1_income(Variable): + label = "NI Class 1 income" + entity = Person + definition_period = YEAR + value_type = float + unit = GBP + + def formula(person, period, parameters): + return person("employment_income", period) + + self.replace_variable(ni_class_1_income) + + +def test_a_formula_in_place_of_a_sum_is_read_as_stored(): + """Invariant 13 (found in review): with ni_class_1_income calculated by a + formula giving the same amount, £14,764.46 of pay still leaves + £14,764.46 less PAYE of £438.90 and Class 1 of £175.56.""" + net, _ = _earnings_of([{"employment_income": 14_764.46}], reform=_NIIncomeFormula) + assert net == [1_415_000] + + +def test_the_model_calculates_class_1_with_its_own_formula_in_every_year(): + """Invariant 11, for the baseline: in every year from 2015 to the + horizon, no reform the model applies by default changes how Class 1 is + calculated, so the deduction is its formula worked exactly. The formula + is restated here in exact arithmetic, month by month, from the same + parameters: the main rate between the primary threshold and the upper + earnings limit and the additional rate above it, weekly thresholds as + 52 / 12 of a week a month. NI income is the pay entered plus the + haircut the model calculates from 2029 (salary sacrifice cap costs + spread over all pay), read as stored.""" + from policyengine_core.periods import MONTH as month_unit + from policyengine_core.periods import period as make_period + + from policyengine_uk.variables.gov.dwp.carers_allowance_earnings import ( + _class_1_calculated, + _primary_class_1, + ) + + pays = [1_476_446, 6_000_000, 15_000_017] + for year in range(2015, HORIZON + 1): + simulation = _carers("employment_income", {year: [p / 100 for p in pays]}) + person = simulation.populations["person"] + period = make_period(year) + simulation.calculate("ni_class_1_employee", year) + assert _class_1_calculated(person, period), year + worked = _primary_class_1( + person, period, simulation.tax_benefit_system.parameters + ) + haircuts = simulation.calculate("salary_sacrifice_broad_base_haircut", year) + for pay, haircut, value in zip(pays, haircuts, worked): + income = pay + Fraction(float(haircut)) * 100 + exact = Fraction(0) + for month in period.get_subperiods(month_unit): + scale = simulation.tax_benefit_system.parameters( + month + ).gov.hmrc.national_insurance.class_1.employee_scale + monthly = income / 12 + pt, uel = ( + Fraction(str(scale.components[i].threshold)) * 100 * WEEKS / 12 + for i in (1, 2) + ) + main, additional = ( + Fraction(str(scale.components[i].rate)) for i in (1, 2) + ) + exact += main * (min(max(monthly, pt), uel) - pt) + additional * max( + monthly - uel, 0 + ) + # Within 10^-5p: a haircut within half a float32 step of a whole + # penny is read as that penny, which moves Class 1 by far less. + assert abs(Fraction(float(value)) - exact) < Fraction(1, 10**5), ( + year, + pay, + float(value), + float(exact), + ) + + +def _natural_rows(rng, count, above_threshold): + """Inputs the model sums, in whole pence, chosen so that the legal net + earnings lie within 3p of the 2026-27 threshold: pay before labour supply + responses a, salary sacrifice returned to pay b (also a member + contribution), reported employee contributions r, personal contributions + q and statutory sick pay s (NI income, not pay). Below the primary + threshold there is no PAYE or Class 1; above it, Class 1 on a + b + s.""" + rows, nets = [], [] + while len(rows) < count: + b, s = int(rng.integers(0, 200_000)), int(rng.integers(0, 50_000)) + r, q = int(rng.integers(0, 300_000)), int(rng.integers(0, 100_000)) + target = THRESHOLD_2026 + int(rng.integers(-3, 4)) + # Pay that leaves net earnings at the target, given the deductions. + half = math.ceil(Fraction(r + b + q, 2)) + pay = target + half + if above_threshold: + # Net = pay - ceil(8% of (pay + s - PT)) - half, with no PAYE + # while pay less member contributions stays below the allowance. + pay = math.ceil( + (target + half - Fraction(8, 100) * (WEEKS * 24_173 - s)) + / Fraction(92, 100) + ) + a = pay - b + if a <= 0: + continue + employee = r + b + if above_threshold and not ( + pay + s > WEEKS * 24_173 and pay - employee < 1_257_000 + ): + continue + if not above_threshold and (pay + s > WEEKS * 24_173 or pay > 1_257_000): + continue + net = pay - math.ceil(_class_1(pay + s)) - half + rows.append( + { + "employment_income_before_lsr": a / 100, + "salary_sacrifice_returned_to_income": b / 100, + "employee_pension_contributions_reported": r / 100, + "personal_pension_contributions": q / 100, + "statutory_sick_pay": s / 100, + } + ) + nets.append(net) + return rows, nets + + +@pytest.mark.parametrize("above_threshold", [False, True], ids=["no NI", "Class 1"]) +def test_sums_the_model_forms_are_read_in_whole_pence(above_threshold): + """Invariant 12: pay (employment_income, which adds pay before labour + supply responses and returned salary sacrifice), NI income (which adds + statutory sick pay), employee pension contributions (reported plus + returned salary sacrifice) and pension_contributions (the programmes) are + the exact sums of their whole-penny parts, however float32 rounds the + model's sums, and the limit test follows net earnings exactly.""" + rows, nets = _natural_rows(np.random.default_rng(2543), 600, above_threshold) + if not above_threshold: + # From review: returned pay and contributions of 20p, and of £1,435.15. + rows += [ + { + "employment_income_before_lsr": 10_608.96, + "salary_sacrifice_returned_to_income": 0.20, + "employee_pension_contributions_reported": 1.08, + }, + { + "employment_income_before_lsr": 10_794.69, + "salary_sacrifice_returned_to_income": 1_435.15, + "employee_pension_contributions_reported": 1_807.49, + }, + # Pay of £11,692.11, which float32 holds as 11,692.109375. + { + "employment_income_before_lsr": 10_276.42, + "salary_sacrifice_returned_to_income": 1_415.69, + "employee_pension_contributions_reported": 751.49, + }, + ] + nets += [THRESHOLD_2026] * 3 + net, gainful = _earnings_of(rows) + wrong = [(row, n, e) for row, n, e in zip(rows, net, nets) if n != e] + assert not wrong, wrong[:3] + assert gainful == [n >= THRESHOLD_2026 for n in nets] diff --git a/policyengine_uk/tests/test_stored_pence.py b/policyengine_uk/tests/test_stored_pence.py index 4fb17d7d08..57abe7195b 100644 --- a/policyengine_uk/tests/test_stored_pence.py +++ b/policyengine_uk/tests/test_stored_pence.py @@ -11,14 +11,18 @@ penny comes back exactly, to be rounded as the rule requires. 4. Monotonic: a larger stored amount never gives fewer pence. -stored_sum_pence: a float32 sum of float32 amounts, in pence. - -5. Sums recovered: a float32 sum of up to five whole-pence amounts, added - one at a time as policyengine-core's ``adds`` does, comes back as exactly - the sum of their pence while it is below £131,072, including sums that +summed: a float32 sum of float32 amounts, in pence. + +5. Rounding bounded: a float32 sum of up to eight float32 amounts of either + sign, nested in sums of sums and added one at a time as policyengine-core + adds them, lies within summed's tolerance of the exact sum of the stored + amounts, so summed returns that sum. +6. Sums recovered: such a sum of whole-pence amounts below £131,072 each + comes back as exactly the sum of their pence, including sums that stored_pence alone would not recover. -6. Entered totals kept: a total that is not a float32 sum of the parts, as - when it is entered directly, comes back as stored_pence(total). +7. Not a sum: a total further from the parts' sum than that rounding, as when + it is entered directly, comes back as stored_pence(total). For amounts + under £10,000 in all, any total a whole penny or more from their sum is. """ from fractions import Fraction @@ -28,7 +32,7 @@ from hypothesis import example, given, settings from hypothesis import strategies as st -from policyengine_uk.utils.pence import stored_pence, stored_sum_pence +from policyengine_uk.utils.pence import Pence, stored, stored_pence, summed # Below 2^17 pounds a float32 step is under a penny. LIMIT_PENCE = 2**17 * 100 @@ -76,21 +80,75 @@ def test_never_gives_fewer_pence_for_more_money(amounts): assert np.all(np.diff(stored_pence(stored)) >= 0) -def _float32_sum(parts): - """Parts in pence, stored and added one at a time in float32.""" - stored = [(np.asarray(p) / 100).astype(np.float32) for p in parts] - return reduce(np.add, stored), stored +def _float32_sum(tree): + """A tree of float32 amounts (a list is a sum of its items), added one at + a time in float32 as policyengine-core adds them, and read with summed, + each amount as its exact value: (the float32 total, the Pence).""" + if not isinstance(tree, list): + amount = np.float32(tree) + return amount, Pence( + np.float64(amount) * 100, np.float64(abs(amount)), np.float64(1) + ) + totals, parts = zip(*(_float32_sum(item) for item in tree)) + total = reduce(np.add, totals) + return total, summed(total, parts) + + +def _exact(tree): + """The exact sum of a tree's float32 amounts.""" + if isinstance(tree, list): + return sum((_exact(item) for item in tree), Fraction(0)) + return Fraction(float(np.float32(tree))) + + +def _leaves(tree): + return sum(_leaves(item) for item in tree) if isinstance(tree, list) else 1 + + +# Float32 amounts of either sign below £131,072 on a grid of 2^-20, so that +# float64 sums of up to eight of them, in pence, are exact. +grid_amounts = st.integers(-(2**37) + 1, 2**37 - 1).map( + lambda units: float(np.float32(units / 2**20)) +) +sums_of_amounts = st.recursive( + grid_amounts, + lambda items: st.lists(items, min_size=1, max_size=4), + max_leaves=8, +).filter(lambda tree: isinstance(tree, list)) + + +@settings(max_examples=500, deadline=None, derandomize=True) +@given(tree=sums_of_amounts) +# Cancellation: the amounts are far larger than the total. +@example(tree=[65_536.01, -65_536.0, 0.01]) +@example(tree=[[131_071.99, 0.01], [-131_071.99, [-0.01, 0.03]]]) +@example(tree=[[1.0, 2.0**-20, 2.0**-20], [-1.0, 2.0**-20]]) +def test_float32_sums_lie_within_the_tolerance_of_the_exact_sum(tree): + total, read = _float32_sum(tree) + # Every level was accepted as a sum, so the exact sum is returned. + assert read.count == _leaves(tree), (tree, float(total)) + assert Fraction(float(read.pence)) == _exact(tree) * 100, (tree, float(total)) def test_sums_of_whole_pence_are_recovered(): rng = np.random.default_rng(2541) missed_by_stored_pence = 0 - for count in range(1, 6): - # Up to five parts whose sum is below £131,072. - parts = [rng.integers(0, LIMIT_PENCE // count, 400_000) for _ in range(count)] - total, stored = _float32_sum(parts) - exact = sum(parts) - assert np.array_equal(stored_sum_pence(total, stored), exact), count + for shape in ([1], [2], [5], [2, 3], [1, 4, 2]): + # Whole-pence amounts of either sign below £131,072, in groups of the + # sizes ``shape`` lists, each group summed and the groups' sums added. + groups = [ + [rng.integers(1 - LIMIT_PENCE, LIMIT_PENCE, 200_000) for _ in range(size)] + for size in shape + ] + group_totals, group_sums = [], [] + for group in groups: + amounts = [(pence / 100).astype(np.float32) for pence in group] + group_total = reduce(np.add, amounts) + group_totals.append(group_total) + group_sums.append(summed(group_total, [stored(a) for a in amounts])) + total = reduce(np.add, group_totals) + exact = sum(sum(group) for group in groups) + assert np.array_equal(summed(total, group_sums).pence, exact), shape missed_by_stored_pence += int(np.sum(stored_pence(total) != exact)) # Summing in float32 leaves some totals too far from their pence for # stored_pence alone, which is why the parts are summed again. @@ -99,19 +157,17 @@ def test_sums_of_whole_pence_are_recovered(): @settings(max_examples=200, deadline=None, derandomize=True) @given( - parts=st.lists(st.integers(0, LIMIT_PENCE // 5 - 1), min_size=1, max_size=5), - entered=st.integers(0, LIMIT_PENCE - 1), + parts=st.lists(st.integers(0, 199_999), min_size=1, max_size=5), + entered=st.integers(0, 999_999), ) # £1,000 entered as a pension total, with no programme amounts. @example(parts=[0, 0], entered=100_000) -def test_a_total_entered_directly_is_read_as_entered(parts, entered): - total, stored = _float32_sum([[p] for p in parts]) - entered_total = np.array([entered / 100], dtype=np.float32) - result = stored_sum_pence(entered_total, stored) - if abs(entered - sum(parts)) <= len(parts): - # Within float32 rounding of the parts' sum (a penny for each part - # here, far less below £131,072): read as that sum. - assert result[0] in (entered, sum(parts)) - else: - assert result[0] == entered - assert stored_sum_pence(total, stored)[0] == sum(parts) +# A penny from the sum of the amounts. +@example(parts=[180_749, 143_515], entered=324_265) +def test_a_total_that_is_not_the_sum_is_read_as_stored(parts, entered): + """Amounts under £2,000, five at most: the rounding of their sum is under + half a penny, so a total a whole penny or more from it is not the sum.""" + amounts = [np.float32(p / 100) for p in parts] + read = [stored(amount) for amount in amounts] + assert summed(np.float32(entered / 100), read).pence == entered + assert summed(reduce(np.add, amounts), read).pence == sum(parts) diff --git a/policyengine_uk/utils/pence.py b/policyengine_uk/utils/pence.py index de02155716..51dec2944e 100644 --- a/policyengine_uk/utils/pence.py +++ b/policyengine_uk/utils/pence.py @@ -1,12 +1,16 @@ """Sums of money as the model stores them, in pence. The model stores amounts of money in float32, which holds most of them only -to within half a float32 step: £10,608.52 is stored as 10,608.51953125. Rules -that turn on a single penny, such as Carer's Allowance's earnings limit, need -the amount that was entered, not the nearest float32. +to within half a float32 step: £10,608.52 is stored as 10,608.51953125, and +sums it forms in float32 can lie further from their pence. Rules that turn on +a single penny, such as Carer's Allowance's earnings limit, need the amounts +that were entered, and sums of them worked exactly, not the nearest float32. """ +from typing import NamedTuple + import numpy as np +from policyengine_core.parameters import get_parameter def stored_pence(amount): @@ -30,21 +34,116 @@ def stored_pence(amount): return np.where(np.abs(pence - whole) <= half_step, whole, pence) -def stored_sum_pence(total, parts): - """``total``, a float32 sum of the float32 amounts ``parts``, in pence as - float64: the sum of the parts' stored_pence, where ``total`` lies within - float32 rounding of it, and otherwise stored_pence(total), for a total - entered directly rather than summed. +class Pence(NamedTuple): + """An amount in pence as float64, with what bounds the float32 rounding + of the stored amount it was read from: ``bound``, the sum of the + magnitudes of the stored amounts added to form it, in pounds, and + ``count``, how many there were.""" + + pence: np.ndarray + bound: np.ndarray + count: np.ndarray + + +def stored(amount) -> Pence: + """``amount``, stored in float32, read with stored_pence.""" + amount = np.asarray(amount, dtype=np.float32) + return Pence( + stored_pence(amount), + np.abs(amount).astype(np.float64), + np.ones(amount.shape), + ) + + +def summed(total, parts) -> Pence: + """``total``, a float32 sum of the stored amounts ``parts`` (each read as + a Pence, a subtracted one with its pence negated), in pence as float64: + the exact sum of the parts' pence, where ``total`` lies within the + rounding of the float32 additions that formed it, and otherwise + stored(total). Summing in float32 rounds at each addition, so a sum of whole-pence amounts can lie more than half a step from its pence even when each part is recovered exactly; summing the parts' pence in float64 is exact. Each - addition moves a sum of amounts no larger than it by at most half a step - of the total, and each part lies within half a step of its own pence, so - a sum of n parts lies within n steps of the parts' pence. + of the ``count`` stored amounts lies within half a step of its pence, and + each addition rounds to within half a step of its result. Every stored + amount and partial sum is at most ``bound``, up to that rounding, so the + total lies within ``count`` steps of ``bound`` of the exact sum. """ total = np.asarray(total, dtype=np.float32) - exact = sum(stored_pence(part) for part in parts) - tolerance = np.spacing(np.abs(total)).astype(np.float64) * 100 * len(parts) - summed = np.abs(total.astype(np.float64) * 100 - exact) <= tolerance - return np.where(summed, exact, stored_pence(total)) + exact = sum(part.pence for part in parts) + bound = sum(part.bound for part in parts) + count = sum(part.count for part in parts) + # Each rounding is relative, at most 2^-24 of its result, so the partial + # sums exceed ``bound`` by under count * 2^-23 of it. Take the step of the + # next float32 up, so that converting to float32 cannot shrink it. + largest = np.nextafter( + (bound * (1 + count * 2.0**-23)).astype(np.float32), np.float32(np.inf) + ) + tolerance = count * np.spacing(largest).astype(np.float64) * 100 + is_sum = np.abs(total.astype(np.float64) * 100 - exact) <= tolerance + entered = stored(total) + return Pence( + np.where(is_sum, exact, entered.pence), + np.where(is_sum, bound, entered.bound), + np.where(is_sum, count, entered.count), + ) + + +def amount_pence(population, variable_name, period): + """``variable_name`` for ``period``, in pence as float64: as stored_pence + reads it, except that an amount the model calculated by adding others + (``adds`` and ``subtracts``) is the exact sum of those amounts, each read + the same way (see summed). + + policyengine-core forms such a sum one addition at a time in float32, so + a sum of whole-pence amounts can lie more than half a step from its pence. + Only an addition the simulation calculated itself is redone: a value + supplied as an input, or stored by a scenario, is read as stored, as is + one calculated by a formula. + """ + return _amount(population, variable_name, period).pence + + +def _amount(population, variable_name, period) -> Pence: + total = population(variable_name, period) + parts = _summands(population, variable_name, period) + if parts is None: + return stored(total) + read = [] + for name, sign in parts: + part = _amount(population, name, period) + read.append(Pence(sign * part.pence, part.bound, part.count)) + return summed(total, read) + + +def _summands(population, variable_name, period): + """The amounts the simulation added to calculate ``variable_name`` for + ``period``, each with its sign, or None if it did not calculate it by + adding amounts of the same entity and period.""" + simulation = population.simulation + variable = simulation.tax_benefit_system.get_variable(variable_name) + if ( + variable.is_neutralized + or variable.defined_for is not None + or variable.definition_period != period.unit + or variable.get_formula(period) is not None + or not population.get_holder(variable_name).is_derived( + period, simulation.branch_name + ) + ): + return None + summands = [] + for names, sign in ((variable.adds, 1), (variable.subtracts, -1)): + if isinstance(names, str): + # A parameter listing the amounts, as policyengine-core reads it. + names = get_parameter(simulation.tax_benefit_system.parameters, names)( + period.start + ) + for name in names or (): + part = simulation.tax_benefit_system.variables.get(name) + if part is None or part.entity.key != variable.entity.key: + # A parameter, or an amount of another entity. + return None + summands.append((name, sign)) + return summands or None diff --git a/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py b/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py index a54834cdba..76a1f2b3d5 100644 --- a/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py +++ b/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py @@ -1,11 +1,28 @@ from policyengine_uk.model_api import * from policyengine_uk.utils.class_2 import class_2_contribution_weeks, class_2_liable -from policyengine_uk.utils.pence import stored_pence, stored_sum_pence +from policyengine_uk.utils.pence import amount_pence, stored_pence +from policyengine_uk.variables.gov.hmrc.national_insurance.class_1.ni_class_1_employee import ( + ni_class_1_employee, +) +from policyengine_uk.variables.gov.hmrc.national_insurance.class_1.ni_class_1_employee_additional import ( + ni_class_1_employee_additional, +) +from policyengine_uk.variables.gov.hmrc.national_insurance.class_1.ni_class_1_employee_primary import ( + ni_class_1_employee_primary, +) +# The model's Class 1 variables, whose calculation _primary_class_1 works out +# exactly. +_CLASS_1 = ( + ni_class_1_employee, + ni_class_1_employee_primary, + ni_class_1_employee_additional, +) -def _earned(amount): + +def _earned(pence): """Pay or profit in whole pence, a fraction of a penny disregarded (reg. 5).""" - return np.floor(stored_pence(amount)) + return np.floor(pence) def _deducted(pence): @@ -16,27 +33,64 @@ def _deducted(pence): return np.ceil(pence - 1e-6) +def _formula_codes(variable): + """The code of each of a variable's formulas, in date order. ``variable`` + is a Variable or a Variable class.""" + if isinstance(variable, type): + formulas = [ + vars(variable)[name] + for name in sorted(vars(variable)) + if name.startswith("formula") + ] + else: + formulas = list(variable.formulas.values()) + return [formula.__code__ for formula in formulas] + + +def _class_1_calculated(person, period): + """Whether the model calculated ni_class_1_employee for each month of + ``period`` with its own formula: no reform changed how it or its primary + and additional parts are calculated, and no month of any of them was + supplied as an input or stored by a scenario.""" + simulation = person.simulation + for model_variable in _CLASS_1: + name = model_variable.__name__ + variable = simulation.tax_benefit_system.get_variable(name) + if ( + variable.is_neutralized + or variable.definition_period != model_variable.definition_period + or variable.adds != model_variable.adds + or variable.subtracts != model_variable.subtracts + or variable.defined_for != model_variable.defined_for + or _formula_codes(variable) != _formula_codes(model_variable) + ): + return False + holder = person.get_holder(name) + if not all( + holder.is_derived(month, simulation.branch_name) + for month in period.get_subperiods(MONTH) + ): + return False + return True + + def _primary_class_1(person, period, parameters): - """ni_class_1_employee in float64 pence: its formula (the primary and - additional parts, month by month) worked exactly on the year's NI income - in whole pence, where that agrees with the stored total to within the - error of the model's float32 arithmetic; otherwise, as under a reform - that replaces the formula, the stored total. + """ni_class_1_employee in float64 pence. The model works out each month's Class 1 in float32 and sums the twelve, which can leave a whole-penny total further from it than stored_pence recovers: £175.56 (8% of £14,764.46 less £12,569.96) is stored as - 175.5600128, and rounding that up would deduct a penny too much. + 175.5600128, and rounding that up would deduct a penny too much. So where + the model calculated it with its own formula, the formula (the primary + and additional parts, month by month) is worked exactly on the year's NI + income. Otherwise, the deduction is the stored amount: one supplied as + an input, or calculated under a reform that changes the formula, is the + amount deducted (reg. 10(4)(a)), whatever the formula would give. """ - income = stored_sum_pence( - person("ni_class_1_income", period), - [ - person(name, period) - for name in person.simulation.tax_benefit_system.variables[ - "ni_class_1_income" - ].adds - ], - ) + stored = person("ni_class_1_employee", period) + if not _class_1_calculated(person, period): + return stored_pence(stored) + income = amount_pence(person, "ni_class_1_income", period) director = person("is_full_year_category_a_director", period) exact = 0 for month in period.get_subperiods(MONTH): @@ -68,8 +122,10 @@ def _primary_class_1(person, period, parameters): exact = exact + contribution exact = exact * person("ni_liable", period) # The twelve float32 months and their sum err by less than one float32 - # step of the year's NI income plus 64 steps of the contribution. - stored = person("ni_class_1_employee", period) + # step of the year's NI income plus 64 steps of the contribution. A + # larger disagreement would mean the formula worked out here no longer + # matches the model's, and then the stored total, the one calculated, is + # deducted. tolerance = 100 * ( np.spacing(np.abs(person("ni_class_1_income", period))).astype(np.float64) + 64 * np.spacing(np.abs(stored)).astype(np.float64) @@ -120,9 +176,13 @@ class carers_allowance_earnings(Variable): "favour. Net earnings are therefore whole pence. They are worked in " "double precision from the whole pence each amount was entered as, " "since the model stores amounts in single precision, which holds most " - "sums of money only to within a fraction of a penny. Class 1, which the " - "model sums from twelve single-precision months, is worked out again " - "from its formula. The model holds each pension programme as one " + "sums of money only to within a fraction of a penny; an amount the " + "model sums from others (pay, pension contributions, NI income) is the " + "exact sum of its parts, read the same way. Class 1, which the model " + "sums from twelve single-precision months, is worked out again from " + "its formula where the model calculated it with that formula; Class 1 " + "entered as an input, or calculated under a reform of its formula, is " + "deducted as stored. The model holds each pension programme as one " "amount for the year, so half of the year's total is deducted, rounded " "up once.\n\n" "The Scottish Carer Support Payment reads these earnings too " @@ -170,15 +230,16 @@ def formula(person, period, parameters): personal_allowance = income_tax.allowances.personal_allowance.amount # Work in float64 pence from the whole pence each amount was entered - # as (stored_pence), taking pay and profit down to a whole penny and - # each deduction up to one (reg. 5 for the calculated deductions; for - # PAYE and Class 1, which the law takes as deducted, the same rounding - # in the claimant's favour). Scales and thresholds are in pounds. + # as, and sums of them worked exactly (amount_pence), taking pay and + # profit down to a whole penny and each deduction up to one (reg. 5 + # for the calculated deductions; for PAYE and Class 1, which the law + # takes as deducted, the same rounding in the claimant's favour). + # Scales and thresholds are in pounds. # Employment: PAYE on the pay alone (reg. 10(4)(a)). - pay = _earned(person("employment_income", period)) - employee_pension = stored_pence( - person("employee_pension_contributions", period) + pay = _earned(amount_pence(person, "employment_income", period)) + employee_pension = amount_pence( + person, "employee_pension_contributions", period ) taxable_pay = max_(0, pay - employee_pension) / 100 pay_tax = where( @@ -193,7 +254,9 @@ def formula(person, period, parameters): # Self-employment: notional basic-rate tax and main-rate Class 4 # (reg. 14(1)-(2)). The tax is on the profit as if it were the only # income, less the full personal allowance. - profit = _earned(max_(0, person("self_employment_income", period))) + profit = _earned( + stored_pence(max_(0, person("self_employment_income", period))) + ) profit_pounds = profit / 100 # The basic rate is the first band of the rest-of-UK scale (ITA 2007 # s. 6(2)). The Scottish basic rate is the second bracket of the @@ -234,14 +297,11 @@ def formula(person, period, parameters): # as one amount for the year, not the payments the law halves one by # one, so the half is of the year's total, rounded up once. The total # is pension_contributions, whether entered or summed from its - # programmes; a sum is redone in float64 pence. + # programmes (and theirs). pension_share = parameters( period ).gov.dwp.carers_allowance.pension_contribution_deduction_rate - contributions = stored_sum_pence( - person("pension_contributions", period), - [person(program, period) for program in p.pensions.pensions_programs], - ) + contributions = amount_pence(person, "pension_contributions", period) pension_deduction = pension_share * contributions net = ( pay From 9cf7fd7beab5ce3a1359371c838818245ec0b1a1 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 28 Oct 2026 12:35:36 -0400 Subject: [PATCH 09/13] Test the sums reader where wrong-rule swaps survived MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three of the 22 wrong rules swapped in at 6e8a53fa5 failed no test: - Employee pension contributions read as stored for PAYE. test_sums_the_model_forms_are_read_in_whole_pence gains a PAYE regime: 600 sets of inputs with pay less member contributions over the personal allowance, mostly by a multiple of 5p, so that 20% of it is whole pence and contributions misread by a fraction of a penny change the tax. - The reader's input check removed. A pension total entered as £3,242.6401 is the very float32 that programmes of £1,807.49 and £1,435.15 sum to; only the input check tells them apart, and the entered total is read as entered (£10,608.51, within) where the programmes alone give £10,608.52 (over, CA Case 32). - The reader's formula check removed. policyengine-core refuses a variable with both a formula and adds, so the check could never be true; it is dropped, and a test pins the core behaviour it relied on. The fourth survivor, the Class 1 tolerance backstop, is intended: the exact path restates the model's own formula, so the two can only disagree if one is edited without the other. Co-Authored-By: Claude Opus 5.5 --- ...ers_allowance_earnings_limit_properties.py | 110 ++++++++++++++---- policyengine_uk/utils/pence.py | 3 +- 2 files changed, 91 insertions(+), 22 deletions(-) diff --git a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py index 65d7110407..559c008e07 100644 --- a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py +++ b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py @@ -81,11 +81,14 @@ neutralized or replaced the formula, the deduction is the stored amount, as the amount deducted (reg. 10(4)(a)), even within a fraction of a penny of what the formula would give. -12. Sums the model forms are read in whole pence: pay, NI income and pension - contributions, which the model adds from other amounts in float32, are - the exact sums of their whole-penny parts, and net earnings follow - exact arithmetic on them. -13. An amount calculated by a formula in place of a sum is read as stored. +12. Sums the model forms are read in whole pence: pay, NI income, and + employee and all pension contributions, which the model adds from other + amounts in float32, are the exact sums of their whole-penny parts, and + net earnings follow exact arithmetic on them. Only sums the simulation + formed are summed again: a total entered directly is read as entered, + even when it is the very float32 its parts would sum to. +13. An amount calculated by a formula in place of a sum is read as stored + (policyengine-core allows no formula beside a sum). """ import datetime @@ -101,7 +104,7 @@ import policyengine_uk from policyengine_uk import Simulation -from policyengine_uk.model_api import GBP, MONTH, YEAR, Person, Reform, Variable +from policyengine_uk.model_api import GBP, MONTH, YEAR, Person, Reform, Variable, add from policyengine_uk.system import system from policyengine_uk.utils.class_2 import class_2_contribution_weeks @@ -1007,6 +1010,26 @@ def test_a_formula_in_place_of_a_sum_is_read_as_stored(): assert net == [1_415_000] +def test_a_formula_cannot_sit_beside_a_sum(): + """Invariant 13 relies on this: policyengine-core refuses a variable with + both a formula and adds, so a variable calculated by a formula lists no + amounts to add, and amount_pence reads it as stored.""" + + class ni_class_1_income(Variable): + label = "NI Class 1 income" + entity = Person + definition_period = YEAR + value_type = float + unit = GBP + adds = list(system.variables["ni_class_1_income"].adds) + + def formula(person, period, parameters): + return add(person, period, ["employment_income"]) + + with pytest.raises(ValueError, match="mixes computation modes"): + ni_class_1_income() + + def test_the_model_calculates_class_1_with_its_own_formula_in_every_year(): """Invariant 11, for the baseline: in every year from 2015 to the horizon, no reform the model applies by default changes how Class 1 is @@ -1064,22 +1087,65 @@ def test_the_model_calculates_class_1_with_its_own_formula_in_every_year(): ) -def _natural_rows(rng, count, above_threshold): +def test_an_entered_total_is_read_as_entered_even_as_its_parts_float32_sum(): + """Invariant 12: only sums the simulation formed are summed again. A + pension total entered as £3,242.6401, beside programmes of £1,807.49 and + £1,435.15, is stored as 3,242.64013671875, the very float32 the + programmes sum to (CA Case 32). Summed, half would be £1,621.32; entered, + it is read as stored, a fraction of a penny over £3,242.64, so half + counts as £1,621.33 and £12,229.84 of pay leaves £10,608.51, within.""" + row = { + "employment_income": 12_229.84, + "employee_pension_contributions": 1_807.49, + "personal_pension_contributions": 1_435.15, + } + summed_net, summed_over = _earnings_of([row]) + entered_net, entered_over = _earnings_of( + [{**row, "pension_contributions": 3_242.6401}] + ) + assert np.float32(3_242.6401) == np.float32(1_807.49) + np.float32(1_435.15) + assert summed_net + entered_net == [1_222_984 - 162_132, 1_222_984 - 162_133] + assert summed_over + entered_over == [True, False] + + +def _natural_rows(rng, count, regime): """Inputs the model sums, in whole pence, chosen so that the legal net earnings lie within 3p of the 2026-27 threshold: pay before labour supply responses a, salary sacrifice returned to pay b (also a member contribution), reported employee contributions r, personal contributions - q and statutory sick pay s (NI income, not pay). Below the primary - threshold there is no PAYE or Class 1; above it, Class 1 on a + b + s.""" + q and statutory sick pay s (NI income, not pay). + + - "no NI": below the primary threshold, so no PAYE or Class 1. + - "Class 1": Class 1 on a + b + s, with no PAYE while pay less member + contributions stays below the personal allowance. + - "PAYE": pay less member contributions x pence over the personal + allowance, with PAYE of 20% of x and Class 1. x is mostly a multiple of + 5p, so that the tax is whole pence and member contributions read a + fraction of a penny low would add a penny; q puts net earnings on the + target. + """ rows, nets = [], [] while len(rows) < count: b, s = int(rng.integers(0, 200_000)), int(rng.integers(0, 50_000)) r, q = int(rng.integers(0, 300_000)), int(rng.integers(0, 100_000)) target = THRESHOLD_2026 + int(rng.integers(-3, 4)) + paye = 0 + if regime == "PAYE": + x = int(rng.integers(0, 300_000)) + if rng.random() < 0.8: + x -= x % 5 + pay = 1_257_000 + r + b + x + paye = math.ceil(Fraction(20, 100) * x) + # Half the contributions that leaves net earnings at the target. + half = pay - paye - math.ceil(_class_1(pay + s)) - target + q = 2 * half - r - b + if q < 0: + continue # Pay that leaves net earnings at the target, given the deductions. half = math.ceil(Fraction(r + b + q, 2)) - pay = target + half - if above_threshold: + if regime == "no NI": + pay = target + half + elif regime == "Class 1": # Net = pay - ceil(8% of (pay + s - PT)) - half, with no PAYE # while pay less member contributions stays below the allowance. pay = math.ceil( @@ -1090,13 +1156,13 @@ def _natural_rows(rng, count, above_threshold): if a <= 0: continue employee = r + b - if above_threshold and not ( + if regime == "Class 1" and not ( pay + s > WEEKS * 24_173 and pay - employee < 1_257_000 ): continue - if not above_threshold and (pay + s > WEEKS * 24_173 or pay > 1_257_000): + if regime == "no NI" and (pay + s > WEEKS * 24_173 or pay > 1_257_000): continue - net = pay - math.ceil(_class_1(pay + s)) - half + net = pay - paye - math.ceil(_class_1(pay + s)) - half rows.append( { "employment_income_before_lsr": a / 100, @@ -1110,16 +1176,18 @@ def _natural_rows(rng, count, above_threshold): return rows, nets -@pytest.mark.parametrize("above_threshold", [False, True], ids=["no NI", "Class 1"]) -def test_sums_the_model_forms_are_read_in_whole_pence(above_threshold): +@pytest.mark.parametrize("regime", ["no NI", "Class 1", "PAYE"]) +def test_sums_the_model_forms_are_read_in_whole_pence(regime): """Invariant 12: pay (employment_income, which adds pay before labour supply responses and returned salary sacrifice), NI income (which adds statutory sick pay), employee pension contributions (reported plus - returned salary sacrifice) and pension_contributions (the programmes) are - the exact sums of their whole-penny parts, however float32 rounds the - model's sums, and the limit test follows net earnings exactly.""" - rows, nets = _natural_rows(np.random.default_rng(2543), 600, above_threshold) - if not above_threshold: + returned salary sacrifice, which reduce the pay PAYE is charged on) and + pension_contributions (the programmes) are the exact sums of their + whole-penny parts, however float32 rounds the model's sums, and the limit + test follows net earnings exactly, with no NI, with Class 1, and with + PAYE and Class 1.""" + rows, nets = _natural_rows(np.random.default_rng(2543), 600, regime) + if regime == "no NI": # From review: returned pay and contributions of 20p, and of £1,435.15. rows += [ { diff --git a/policyengine_uk/utils/pence.py b/policyengine_uk/utils/pence.py index 51dec2944e..da990292b7 100644 --- a/policyengine_uk/utils/pence.py +++ b/policyengine_uk/utils/pence.py @@ -123,11 +123,12 @@ def _summands(population, variable_name, period): adding amounts of the same entity and period.""" simulation = population.simulation variable = simulation.tax_benefit_system.get_variable(variable_name) + # A variable calculated by a formula lists no amounts to add: + # policyengine-core allows a variable one way of calculating it. if ( variable.is_neutralized or variable.defined_for is not None or variable.definition_period != period.unit - or variable.get_formula(period) is not None or not population.get_holder(variable_name).is_derived( period, simulation.branch_name ) From 6e1179dc2ed5d5846d9c4cb0aae7d8c89f42ea8c Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 28 Oct 2026 13:01:21 -0400 Subject: [PATCH 10/13] Sum Class 1 months exactly; gate the exact path on how core calculates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes the r7 review's findings on PR #2543 (refs #2541): - Class 1 entered for the year (which core divides among the months in float32) or month by month was read from the float32 annual sum, so £1.20 (or 12 x 10p) was deducted as £1.21. amount_pence now sums a monthly flow's months exactly (months_summed), recovering a year's whole pence within the rounding of the division; the Class 1 fallback and backstop read it that way. - The exact Class 1 path accepted a reform that ended the primary formula or dated it from 2027, since it compared formula code alone. It now compares, month by month, the formula core runs (code, the globals it reads, defaults, closure), the variables' other calculation metadata against the model's own classes, abolition by gov.abolitions, and how the formulas read NI income, liability and directorship. - A sum, or Class 1, that the cache policy declines to keep counted as supplied (is_derived is false for an unstored value). Inputs are now told by the holder's input periods. Tests: the reviewer's reproducers, 400-carer sampled property tests of Class 1 entered in whole pence (year and months), reforms that end, re-date, abolish or hold in whole pounds the primary part, run its code with another global value, or change how it reads liability or NI income, cache-policy invariance, and months_summed invariants (exhaustive recovery of a year's whole pence below £32,768). Co-Authored-By: Claude Opus 5.5 --- ...ers_allowance_earnings_limit_properties.py | 265 ++++++++++++++++-- policyengine_uk/tests/test_stored_pence.py | 79 +++++- policyengine_uk/utils/pence.py | 103 ++++++- .../gov/dwp/carers_allowance_earnings.py | 160 ++++++++--- 4 files changed, 534 insertions(+), 73 deletions(-) diff --git a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py index 559c008e07..09152232ea 100644 --- a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py +++ b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py @@ -75,24 +75,30 @@ 11. Class 1 is the amount the model calculates: where it calculated ni_class_1_employee with its own formula, as it does in every year from - 2015 to the horizon, including under a parameter reform, the deduction - is that formula worked exactly. Where Class 1 (for the year or month by - month) or its primary part was supplied as an input, or a reform - neutralized or replaced the formula, the deduction is the stored amount, - as the amount deducted (reg. 10(4)(a)), even within a fraction of a penny - of what the formula would give. + 2015 to the horizon, including under a parameter reform or with a + variable a reform puts in place of a part but calculates the same way, + the deduction is that formula worked exactly. Where Class 1 (for the + year or month by month) or its primary part was supplied as an input, + or a reform neutralized, abolished, replaced, ended or re-dated the + formula, or changed how it reads liability or NI income, the deduction + is the amount calculated or entered, as the amount deducted (reg. + 10(4)(a)), even within a fraction of a penny of what the formula would + give. Class 1 entered in whole pence, for the year or by month, is + deducted to the penny, though float32 sums the months to more or less. 12. Sums the model forms are read in whole pence: pay, NI income, and employee and all pension contributions, which the model adds from other amounts in float32, are the exact sums of their whole-penny parts, and net earnings follow exact arithmetic on them. Only sums the simulation formed are summed again: a total entered directly is read as entered, - even when it is the very float32 its parts would sum to. + even when it is the very float32 its parts would sum to. Whether the + simulation's cache policy keeps a sum, or Class 1, changes nothing. 13. An amount calculated by a formula in place of a sum is read as stored (policyengine-core allows no formula beside a sum). """ import datetime import math +import types from fractions import Fraction from pathlib import Path @@ -105,8 +111,12 @@ import policyengine_uk from policyengine_uk import Simulation from policyengine_uk.model_api import GBP, MONTH, YEAR, Person, Reform, Variable, add +from policyengine_core.parameters import ParameterNode from policyengine_uk.system import system from policyengine_uk.utils.class_2 import class_2_contribution_weeks +from policyengine_uk.variables.gov.hmrc.national_insurance.class_1.ni_class_1_employee_primary import ( + ni_class_1_employee_primary, +) LIMIT = 204 RATE = 86.45 @@ -837,10 +847,11 @@ def test_trading_profit_takes_the_basic_rate_in_every_year(year, country): ) -def _earnings_of(rows, reform=None, year=2026): +def _earnings_of(rows, reform=None, year=2026, uncached=()): """Net earnings in whole pence, and whether each is over the limit, for one carer in England per dict in ``rows``: inputs by variable, each an - amount for ``year`` or a dict of amounts by period.""" + amount for ``year`` or a dict of amounts by period. The simulation keeps + no value of the variables named in ``uncached``.""" people = {} for i, row in enumerate(rows): person = {"age": {year: 45}, "care_hours": {year: 40}} @@ -860,6 +871,9 @@ def _earnings_of(rows, reform=None, year=2026): }, reform=reform, ) + if uncached: + simulation.opt_out_cache = True + simulation.tax_benefit_system.cache_blacklist = list(uncached) earnings = simulation.calculate("carers_allowance_earnings", year) gainful = simulation.calculate("is_gainfully_employed_for_carers_allowance", year) return [int(round(float(e) * 100)) for e in earnings], [bool(g) for g in gainful] @@ -918,6 +932,57 @@ def test_class_1_supplied_near_its_formula_is_deducted_as_supplied(): assert supplied == [pay - paye - 59_440] +@pytest.mark.parametrize( + "class_1_input", + [ + {"ni_class_1_employee": 1.20}, + {"ni_class_1_employee": {month: 0.10 for month in MONTHS_2026}}, + ], + ids=["year", "months"], +) +def test_class_1_entered_in_whole_pence_is_deducted_in_whole_pence(class_1_input): + """Invariant 11 (found in review): £1.20 of Class 1, entered for the + year or as 10p a month, is £1.20 deducted. policyengine-core divides the + year's £1.20 among the months, and either way sums twelve float32 months + to 1.2000001669, which rounded up would deduct £1.21: £10,609.72 of pay + leaves £10,608.52, a penny a week over the limit, not £10,608.51.""" + net, over = _earnings_of([{"employment_income": 10_609.72, **class_1_input}]) + assert net == [1_060_852] and over == [True] + + +@pytest.mark.parametrize("entry", ["year", "months"]) +def test_whole_pence_of_class_1_entered_are_deducted_exactly(entry): + """Invariant 11: Class 1 entered in whole pence, for the year (which + policyengine-core divides among the months in float32) or month by + month, is deducted to the penny. With pay below the personal allowance, + so no PAYE, net earnings are the pay less the Class 1 entered, exactly, + for 400 carers with pay and Class 1 drawn at random.""" + rng = np.random.default_rng(2541) + pays = rng.integers(0, 1_257_000, 400) + pays[:4] = [1_060_972, 1_060_972, 1_119_992, 1_257_000 - 1] + if entry == "year": + class_1 = [rng.integers(0, pay + 1, 1) for pay in pays] + class_1[:4] = [[120], [121], [59_440], [1_257_000 - 1]] + else: + class_1 = [rng.integers(0, pay // 12 + 1, 12) for pay in pays] + class_1[:4] = [[10] * 12, [10] * 11 + [11], [4_953] * 11 + [4_957], [0] * 12] + rows = [ + { + "employment_income": pay / 100, + "ni_class_1_employee": ( + pence[0] / 100 + if entry == "year" + else {month: p / 100 for month, p in zip(MONTHS_2026, pence)} + ), + } + for pay, pence in zip(pays, class_1) + ] + net, _ = _earnings_of(rows) + expected = [int(pay) - int(sum(pence)) for pay, pence in zip(pays, class_1)] + wrong = [(row, n, e) for row, n, e in zip(rows, net, expected) if n != e] + assert not wrong, wrong[:5] + + class _NoClass1(Reform): def apply(self): self.neutralize_variable("ni_class_1_employee") @@ -940,14 +1005,142 @@ def formula(person, period, parameters): self.replace_variable(ni_class_1_employee_primary) -@pytest.mark.parametrize("reform", [_NoClass1, _NoPrimaryClass1]) -def test_class_1_under_a_reform_of_its_formula_is_the_amount_calculated(reform): - """Invariant 11, for a reform: with Class 1 neutralized, or its primary - part replaced by a nil formula, nothing is deducted, not the 0.08p (a - penny) the model's own formula gives at £12,569.97 of pay.""" +def _primary_class_1_reform(formula_name="formula", **attributes): + """A reform replacing the primary part with a variable calculated by the + model's own formula, named ``formula_name`` (giving the date it applies + from), with ``attributes`` (a date it ends, say).""" + variable = type( + "ni_class_1_employee_primary", + (Variable,), + { + "label": ni_class_1_employee_primary.label, + "entity": Person, + "definition_period": MONTH, + "value_type": float, + "unit": GBP, + formula_name: ni_class_1_employee_primary.formula, + **attributes, + }, + ) + + class reform(Reform): + def apply(self): + self.replace_variable(variable) + + return reform + + +# The model's primary formula, run with 53 weeks in a year: the same code +# reading another value of a global it names. +_PRIMARY_IN_53_WEEKS = types.FunctionType( + ni_class_1_employee_primary.formula.__code__, + {**ni_class_1_employee_primary.formula.__globals__, "WEEKS_IN_YEAR": 53}, +) + + +class _AbolishedPrimaryClass1(Reform): + """The primary part abolished by a gov.abolitions parameter, which + policyengine-core reads in place of calculating it.""" + + def apply(self): + def abolish(parameters): + parameters.gov.add_child( + "abolitions", + ParameterNode( + "gov.abolitions", + data={ + "ni_class_1_employee_primary": {"values": {"2015-01-01": True}} + }, + ), + ) + return parameters + + self.modify_parameters(abolish) + + +class _NoLiabilityInDecember(Reform): + """Liability month by month, ending in December.""" + + def apply(self): + class ni_liable(Variable): + label = "age-liable for primary Class 1 and Class 2 NI" + entity = Person + definition_period = MONTH + value_type = bool + + def formula(person, period, parameters): + return ( + person("over_16", period) + & ~person("is_SP_age", period) + & (period.start.month < 12) + ) + + self.replace_variable(ni_liable) + + +class _NoNIIncomeInDecember(Reform): + """NI income month by month, nil in December.""" + + def apply(self): + class ni_class_1_income(Variable): + label = "NI Class 1 income" + entity = Person + definition_period = MONTH + value_type = float + unit = GBP + + def formula(person, period, parameters): + return person("employment_income", period) * (period.start.month < 12) + + self.replace_variable(ni_class_1_income) + + +@pytest.mark.parametrize( + "reform, net", + [ + (_NoClass1, 1_060_852), + (_NoPrimaryClass1, 1_060_852), + (_primary_class_1_reform(end="2025-12-31"), 1_060_852), + (_primary_class_1_reform("formula_2027_01_01"), 1_060_852), + (_AbolishedPrimaryClass1, 1_060_852), + (_primary_class_1_reform(value_type=int), 1_060_852), + (_primary_class_1_reform(formula=_PRIMARY_IN_53_WEEKS), 1_060_852), + (_NoLiabilityInDecember, 1_060_851), + (_NoNIIncomeInDecember, 1_060_851), + (_primary_class_1_reform(), 1_060_851), + ], + ids=[ + "neutralized", + "nil formula", + "ended", + "from 2027", + "abolished", + "whole pounds", + "53 weeks", + "no liability in December", + "no NI income in December", + "the model's formula", + ], +) +def test_class_1_under_a_reform_of_its_formula_is_the_amount_calculated(reform, net): + """Invariant 11, for a reform (the dates and abolition found in review). + At £12,569.97 of pay, a penny above the primary threshold, the model's + own formula gives Class 1 of 0.08p, which counts as a penny, so with a + £3,922.90 contribution £10,608.51 is left, within the limit. With Class + 1 neutralized, its primary part replaced by a nil formula, ended before + 2026, applying only from 2027 or abolished, there is none to deduct, and + £10,608.52 is a penny a week over: the formula is not worked in place of + the amount calculated. So it is with the primary part held in whole + pounds (0.08p is nil), or its formula's code run with 53 weeks in a + year (raising the threshold above this pay). With liability or NI income + month by month and ending in December, the model's Class 1 is eleven + months of it, still a penny, though the year's liability (December's) or + a twelfth of the year's NI income would give none. The model's own + formula, in a variable a reform puts in its place, is still worked + exactly.""" row = {"employment_income": 12_569.97, "employee_pension_contributions": 3_922.90} - net, gainful = _earnings_of([row], reform=reform) - assert net == [1_060_852] and gainful == [True] + result, over = _earnings_of([row], reform=reform) + assert result == [net] and over == [net == THRESHOLD_2026] @pytest.mark.parametrize( @@ -1108,6 +1301,46 @@ def test_an_entered_total_is_read_as_entered_even_as_its_parts_float32_sum(): assert summed_over + entered_over == [True, False] +@pytest.mark.parametrize( + "row, uncached, net", + [ + ( + { + "employment_income_before_lsr": 10_608.96, + "salary_sacrifice_returned_to_income": 0.20, + "employee_pension_contributions_reported": 1.08, + }, + ["employee_pension_contributions"], + 1_060_852, + ), + ( + { + "employment_income": 12_569.97, + "employee_pension_contributions": 3_922.90, + }, + [ + "ni_class_1_employee", + "ni_class_1_employee_primary", + "ni_class_1_employee_additional", + ], + 1_060_851, + ), + ], + ids=["employee pension contributions", "Class 1"], +) +def test_what_the_simulation_keeps_does_not_change_earnings(row, uncached, net): + """Invariants 11 and 12 (found in review): a simulation whose cache policy + keeps no value of a sum, or of Class 1, still calculated them, so reads + them as one that keeps them does. Employee contributions of £1.08 and + 20p of salary sacrifice returned to pay are £1.28, though float32 sums + them to 1.2800001, so half of them is 64p off £10,609.16 of pay, leaving + £10,608.52, over (CA Case 34); and Class 1 at £12,569.97 is still worked + exactly, 0.08p counting as a penny.""" + kept = _earnings_of([row]) + not_kept = _earnings_of([row], uncached=uncached) + assert kept == not_kept == ([net], [net == THRESHOLD_2026]) + + def _natural_rows(rng, count, regime): """Inputs the model sums, in whole pence, chosen so that the legal net earnings lie within 3p of the 2026-27 threshold: pay before labour supply diff --git a/policyengine_uk/tests/test_stored_pence.py b/policyengine_uk/tests/test_stored_pence.py index 57abe7195b..c8588f1708 100644 --- a/policyengine_uk/tests/test_stored_pence.py +++ b/policyengine_uk/tests/test_stored_pence.py @@ -23,6 +23,18 @@ 7. Not a sum: a total further from the parts' sum than that rounding, as when it is entered directly, comes back as stored_pence(total). For amounts under £10,000 in all, any total a whole penny or more from their sum is. + +months_summed: a monthly flow summed over a year, in pence. + +8. A year's amount recovered: every whole number of pence below £32,768, + entered for a year and divided among its twelve months in float32 as + policyengine-core divides it, comes back as exactly that number + (exhaustive). Below £32,768 the months' rounding and the year's is under + half a penny, so no other whole penny lies within it. +9. Months recovered: twelve whole-pence monthly amounts below £131,072 in + all come back as exactly the sum of their pence. +10. Bounded: the result is the exact sum of the months as stored, or a whole + penny within a float32 step of each month and of the year of it. """ from fractions import Fraction @@ -32,7 +44,13 @@ from hypothesis import example, given, settings from hypothesis import strategies as st -from policyengine_uk.utils.pence import Pence, stored, stored_pence, summed +from policyengine_uk.utils.pence import ( + Pence, + months_summed, + stored, + stored_pence, + summed, +) # Below 2^17 pounds a float32 step is under a penny. LIMIT_PENCE = 2**17 * 100 @@ -171,3 +189,62 @@ def test_a_total_that_is_not_the_sum_is_read_as_stored(parts, entered): read = [stored(amount) for amount in amounts] assert summed(np.float32(entered / 100), read).pence == entered assert summed(reduce(np.add, amounts), read).pence == sum(parts) + + +def _months(amounts): + """Twelve float32 monthly amounts, summed one month at a time in float32 + as policyengine-core sums a flow over a year, and read with + months_summed.""" + return months_summed(reduce(np.add, amounts), [stored(m) for m in amounts]) + + +def test_a_years_whole_pence_divided_among_its_months_are_recovered(): + for start in range(0, 32_768 * 100, 1_000_000): + pence = np.arange(start, min(start + 1_000_000, 32_768 * 100)) + # policyengine-core's set_input_divide_by_period: the year's float32 + # amount divided by twelve in float32, for each month. + month = (pence / 100).astype(np.float32) / 12 + assert month.dtype == np.float32 + recovered = _months([month] * 12).pence + wrong = np.nonzero(recovered != pence)[0] + assert len(wrong) == 0, pence[wrong[:5]] + + +def test_whole_pence_entered_for_each_month_are_recovered(): + rng = np.random.default_rng(2541) + pence = rng.integers(0, LIMIT_PENCE // 12, (12, 200_000)) + recovered = _months([(p / 100).astype(np.float32) for p in pence]).pence + assert np.array_equal(recovered, pence.sum(axis=0)) + + +# Float32 monthly amounts below £131,072 in all, on a grid of 2^-20 so that +# float64 sums of twelve of them, in pence, are exact. +monthly_amounts = st.integers(0, 2**37 // 12).map( + lambda units: float(np.float32(units / 2**20)) +) + + +def _step(amount): + """The float32 step of the next float32 above ``amount``, in pounds.""" + return Fraction( + float(np.spacing(np.nextafter(np.float32(amount), np.float32(np.inf)))) + ) + + +@settings(max_examples=300, deadline=None, derandomize=True) +@given(months=st.lists(monthly_amounts, min_size=12, max_size=12)) +# Twelve 10p months, which float32 sums to 1.2000002, more than half a step +# from £1.20. +@example(months=[0.1] * 12) +def test_moves_the_months_sum_only_to_a_whole_penny_within_their_rounding(months): + amounts = [np.float32(m) for m in months] + read = _months(amounts) + exact = sum(Fraction(float(a)) for a in amounts) * 100 + result = Fraction(float(read.pence)) + if result != exact: + steps = sum(_step(a) for a in amounts) + _step(sum(float(a) for a in amounts)) + assert result.denominator == 1 and abs(result - exact) <= steps * 100, ( + months, + float(result), + float(exact), + ) diff --git a/policyengine_uk/utils/pence.py b/policyengine_uk/utils/pence.py index da990292b7..edf70fdab9 100644 --- a/policyengine_uk/utils/pence.py +++ b/policyengine_uk/utils/pence.py @@ -11,6 +11,8 @@ import numpy as np from policyengine_core.parameters import get_parameter +from policyengine_core.periods import MONTH, YEAR +from policyengine_core.variables import QuantityType def stored_pence(amount): @@ -90,11 +92,37 @@ def summed(total, parts) -> Pence: ) +def neutralized(simulation, variable, period): + """Whether policyengine-core gives ``variable`` its default for + ``period`` without calculating it: neutralized, or abolished by a + gov.abolitions parameter, read as policyengine-core reads it.""" + try: + return bool( + variable.is_neutralized + or simulation.tax_benefit_system.parameters(period).gov.abolitions[ + variable.name + ] + ) + except Exception: + return False + + +def supplied(population, variable_name, period): + """Whether the value of ``variable_name`` for ``period`` was supplied, as + an input or a value a scenario stored, rather than calculated. A value + the simulation calculates counts as calculated whether or not its cache + policy keeps it.""" + holder = population.get_holder(variable_name) + return period in holder.get_input_periods(population.simulation.branch_name) + + def amount_pence(population, variable_name, period): """``variable_name`` for ``period``, in pence as float64: as stored_pence reads it, except that an amount the model calculated by adding others (``adds`` and ``subtracts``) is the exact sum of those amounts, each read - the same way (see summed). + the same way (see summed), and a monthly flow for a year is the exact + sum of its months, in whole pence where it lies within the float32 + rounding of a whole penny (see months_summed). policyengine-core forms such a sum one addition at a time in float32, so a sum of whole-pence amounts can lie more than half a step from its pence. @@ -105,9 +133,62 @@ def amount_pence(population, variable_name, period): return _amount(population, variable_name, period).pence +def months_summed(total, months) -> Pence: + """``total``, policyengine-core's float32 sum of a monthly flow over a + year, of which ``months`` are the twelve months (each read as a Pence): + their exact sum, as summed reads it, taken to the nearest whole penny + where it lies within a float32 step of each month and of the year of + one. + + policyengine-core divides an amount entered for the year among its months + in float32, so a month need not lie near whole pence. Each month lies + within half a step of its share of the year's amount as stored, and that + within half a step of its pence, so the months' exact sum lies within + half a step of each month and of the year of the pence entered. Amounts + entered for each month in whole pence are recovered month by month, and + their sum is whole pence already. + """ + exact = summed(total, months) + steps = sum(part.count * _step(part.bound) for part in months) + _step(exact.bound) + whole = np.round(exact.pence) + return Pence( + np.where(np.abs(exact.pence - whole) <= steps * 100, whole, exact.pence), + exact.bound, + exact.count, + ) + + +def _step(bound): + """The float32 step, in pounds, of the next float32 above ``bound``, a + magnitude in pounds as float64: at least that of any float32 up to it.""" + largest = np.nextafter(np.asarray(bound, dtype=np.float32), np.float32(np.inf)) + return np.spacing(largest).astype(np.float64) + + def _amount(population, variable_name, period) -> Pence: + simulation = population.simulation + variable = simulation.tax_benefit_system.get_variable(variable_name) total = population(variable_name, period) - parts = _summands(population, variable_name, period) + if neutralized(simulation, variable, period) or supplied( + population, variable_name, period + ): + return stored(total) + if ( + variable.definition_period == MONTH + and period.unit == YEAR + and variable.quantity_type == QuantityType.FLOW + ): + # policyengine-core's amount of a monthly flow for a year is the sum + # of its months'. (Of a stock, it is the last month's, read as + # stored.) + return months_summed( + total, + [ + _amount(population, variable_name, month) + for month in period.get_subperiods(MONTH) + ], + ) + parts = _summands(simulation, variable, period) if parts is None: return stored(total) read = [] @@ -117,22 +198,14 @@ def _amount(population, variable_name, period) -> Pence: return summed(total, read) -def _summands(population, variable_name, period): - """The amounts the simulation added to calculate ``variable_name`` for +def _summands(simulation, variable, period): + """The amounts the simulation added to calculate ``variable`` for ``period``, each with its sign, or None if it did not calculate it by adding amounts of the same entity and period.""" - simulation = population.simulation - variable = simulation.tax_benefit_system.get_variable(variable_name) # A variable calculated by a formula lists no amounts to add: - # policyengine-core allows a variable one way of calculating it. - if ( - variable.is_neutralized - or variable.defined_for is not None - or variable.definition_period != period.unit - or not population.get_holder(variable_name).is_derived( - period, simulation.branch_name - ) - ): + # policyengine-core allows a variable one way of calculating it. One for + # another period, or masked by defined_for, is read as stored. + if variable.defined_for is not None or variable.definition_period != period.unit: return None summands = [] for names, sign in ((variable.adds, 1), (variable.subtracts, -1)): diff --git a/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py b/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py index 76a1f2b3d5..8ae88970a4 100644 --- a/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py +++ b/policyengine_uk/variables/gov/dwp/carers_allowance_earnings.py @@ -1,6 +1,14 @@ +import functools +import types + from policyengine_uk.model_api import * from policyengine_uk.utils.class_2 import class_2_contribution_weeks, class_2_liable -from policyengine_uk.utils.pence import amount_pence, stored_pence +from policyengine_uk.utils.pence import ( + amount_pence, + neutralized, + stored_pence, + supplied, +) from policyengine_uk.variables.gov.hmrc.national_insurance.class_1.ni_class_1_employee import ( ni_class_1_employee, ) @@ -18,6 +26,14 @@ ni_class_1_employee_primary, ni_class_1_employee_additional, ) +# How their formulas read the amounts they depend on, which _primary_class_1 +# reads for the year: NI income for a month is a twelfth of the year's, and +# liability and directorship for a month are the year's. +_CLASS_1_READS = ( + ("ni_class_1_income", YEAR, FLOW), + ("ni_liable", YEAR, STOCK), + ("is_full_year_category_a_director", YEAR, STOCK), +) def _earned(pence): @@ -33,44 +49,102 @@ def _deducted(pence): return np.ceil(pence - 1e-6) -def _formula_codes(variable): - """The code of each of a variable's formulas, in date order. ``variable`` - is a Variable or a Variable class.""" - if isinstance(variable, type): - formulas = [ - vars(variable)[name] - for name in sorted(vars(variable)) - if name.startswith("formula") +@functools.cache +def _model_class_1(): + """The model's Class 1 variables, as the model defines them.""" + return tuple(variable() for variable in _CLASS_1) + + +def _how_calculated(variable): + """What, besides its formulas, decides how policyengine-core calculates + ``variable``.""" + return ( + variable.entity.key, + variable.definition_period, + variable.value_type, + variable.quantity_type, + variable.default_value, + variable.end, + variable.adds, + variable.subtracts, + variable.defined_for, + variable.is_neutralized, + ) + + +def _global_names(code): + """The global names ``code``, and the code nested in it, can read.""" + names = set(code.co_names) + for constant in code.co_consts: + if isinstance(constant, types.CodeType): + names |= _global_names(constant) + return names + + +def _same_formula(formula, model_formula): + """Whether ``formula`` calculates as ``model_formula`` does: the same + code, reading the same values of the globals it names, with the same + defaults and closure. Either may be None, for no formula.""" + if formula is None or model_formula is None: + return formula is model_formula + absent = object() + try: + cells = [ + (cell.cell_contents, model_cell.cell_contents) + for cell, model_cell in zip( + formula.__closure__ or (), model_formula.__closure__ or () + ) ] - else: - formulas = list(variable.formulas.values()) - return [formula.__code__ for formula in formulas] + except ValueError: + # An empty cell. + return False + return ( + formula.__code__ == model_formula.__code__ + and formula.__defaults__ == model_formula.__defaults__ + and formula.__kwdefaults__ == model_formula.__kwdefaults__ + and all(value is model_value for value, model_value in cells) + and all( + formula.__globals__.get(name, absent) + is model_formula.__globals__.get(name, absent) + for name in _global_names(formula.__code__) + ) + ) def _class_1_calculated(person, period): - """Whether the model calculated ni_class_1_employee for each month of - ``period`` with its own formula: no reform changed how it or its primary - and additional parts are calculated, and no month of any of them was - supplied as an input or stored by a scenario.""" + """Whether policyengine-core calculated ni_class_1_employee for each + month of ``period`` as the model does: no reform changed how it or its + primary and additional parts are calculated in those months (their + formulas, the dates the formulas apply from and to, how the parts are + summed and masked, neutralization or abolition) or how the formulas read + the amounts they depend on, and no month of any of them was supplied as + an input or stored by a scenario.""" simulation = person.simulation - for model_variable in _CLASS_1: - name = model_variable.__name__ - variable = simulation.tax_benefit_system.get_variable(name) + variables = simulation.tax_benefit_system.variables + for name, definition_period, quantity_type in _CLASS_1_READS: + variable = variables.get(name) if ( - variable.is_neutralized - or variable.definition_period != model_variable.definition_period - or variable.adds != model_variable.adds - or variable.subtracts != model_variable.subtracts - or variable.defined_for != model_variable.defined_for - or _formula_codes(variable) != _formula_codes(model_variable) + variable is None + or variable.definition_period != definition_period + or variable.quantity_type != quantity_type ): return False - holder = person.get_holder(name) - if not all( - holder.is_derived(month, simulation.branch_name) - for month in period.get_subperiods(MONTH) + for model_variable in _model_class_1(): + name = model_variable.name + variable = variables.get(name) + if variable is None or _how_calculated(variable) != _how_calculated( + model_variable ): return False + for month in period.get_subperiods(MONTH): + if ( + neutralized(simulation, variable, month) + or supplied(person, name, month) + or not _same_formula( + variable.get_formula(month), model_variable.get_formula(month) + ) + ): + return False return True @@ -83,13 +157,15 @@ def _primary_class_1(person, period, parameters): 175.5600128, and rounding that up would deduct a penny too much. So where the model calculated it with its own formula, the formula (the primary and additional parts, month by month) is worked exactly on the year's NI - income. Otherwise, the deduction is the stored amount: one supplied as - an input, or calculated under a reform that changes the formula, is the - amount deducted (reg. 10(4)(a)), whatever the formula would give. + income. Otherwise, the deduction is the amount stored for each month, + summed exactly (amount_pence): one supplied as an input, for the year or + month by month, or calculated under a reform that changes the formula, + is the amount deducted (reg. 10(4)(a)), whatever the formula would give. """ stored = person("ni_class_1_employee", period) + deducted = amount_pence(person, "ni_class_1_employee", period) if not _class_1_calculated(person, period): - return stored_pence(stored) + return deducted income = amount_pence(person, "ni_class_1_income", period) director = person("is_full_year_category_a_director", period) exact = 0 @@ -124,14 +200,13 @@ def _primary_class_1(person, period, parameters): # The twelve float32 months and their sum err by less than one float32 # step of the year's NI income plus 64 steps of the contribution. A # larger disagreement would mean the formula worked out here no longer - # matches the model's, and then the stored total, the one calculated, is - # deducted. + # matches the model's, and then the amount calculated is deducted. tolerance = 100 * ( np.spacing(np.abs(person("ni_class_1_income", period))).astype(np.float64) + 64 * np.spacing(np.abs(stored)).astype(np.float64) ) agrees = np.abs(np.asarray(stored, dtype=np.float64) * 100 - exact) <= tolerance - return where(agrees, exact, stored_pence(stored)) + return where(agrees, exact, deducted) class carers_allowance_earnings(Variable): @@ -178,11 +253,14 @@ class carers_allowance_earnings(Variable): "since the model stores amounts in single precision, which holds most " "sums of money only to within a fraction of a penny; an amount the " "model sums from others (pay, pension contributions, NI income) is the " - "exact sum of its parts, read the same way. Class 1, which the model " - "sums from twelve single-precision months, is worked out again from " - "its formula where the model calculated it with that formula; Class 1 " - "entered as an input, or calculated under a reform of its formula, is " - "deducted as stored. The model holds each pension programme as one " + "exact sum of its parts, read the same way, whether or not the " + "simulation's cache policy keeps it. Class 1, which the model sums " + "from twelve single-precision months, is worked out again from its " + "formula where the model calculated it with that formula; Class 1 " + "entered as an input (for the year or month by month), or calculated " + "under a reform of its formula or of how it reads NI income, is " + "deducted as entered or calculated, its months summed exactly and an " + "amount entered for the year recovered in whole pence. The model holds each pension programme as one " "amount for the year, so half of the year's total is deducted, rounded " "up once.\n\n" "The Scottish Carer Support Payment reads these earnings too " From a51dd97ea246c859b405510fcd72ba2927c07f2e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 28 Oct 2026 13:08:18 -0400 Subject: [PATCH 11/13] Changelog: Class 1 entered is deducted as entered, months summed exactly Co-Authored-By: Claude Opus 5.5 --- changelog.d/2541.fixed.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelog.d/2541.fixed.md b/changelog.d/2541.fixed.md index da2c6a70b3..24f9a72fcc 100644 --- a/changelog.d/2541.fixed.md +++ b/changelog.d/2541.fixed.md @@ -1 +1 @@ -Fixed the Carer's Allowance earnings limit treating earnings a penny a week over the 2020-21 to 2024-25 limits (£128, £132, £139 and £151) as within them. Net earnings now apply the Computation of Earnings Regulations' rounding to each step: pay and profit are taken down to a whole penny and each calculated deduction (notional tax, Class 4 and half of pension contributions) up to one; the model's PAYE and Class 1, which in law are the amounts actually deducted, are rounded up too. Earnings are worked in double precision from the whole pence each amount was entered as, and a sum the model forms (pay, pension contributions, NI income) is the exact sum of its parts. Class 1 is worked out again from its formula where the model calculated it; entered as an input, or calculated under a reform of its formula, it is deducted as stored. The limit test reads the whole pence back exactly, so float32 rounding no longer decides it. Pension contributions entered as a total still reduce earnings. The Scottish Carer Support Payment, which reads the same earnings, is judged exactly at its limit too. +Fixed the Carer's Allowance earnings limit treating earnings a penny a week over the 2020-21 to 2024-25 limits (£128, £132, £139 and £151) as within them. Net earnings now apply the Computation of Earnings Regulations' rounding to each step: pay and profit are taken down to a whole penny and each calculated deduction (notional tax, Class 4 and half of pension contributions) up to one; the model's PAYE and Class 1, which in law are the amounts actually deducted, are rounded up too. Earnings are worked in double precision from the whole pence each amount was entered as, and a sum the model forms (pay, pension contributions, NI income) is the exact sum of its parts. Class 1 is worked out again from its formula where the model calculated it with that formula; entered as an input (for the year or month by month), or calculated under a reform, it is deducted as entered or calculated, its months summed exactly, so £1.20 entered is £1.20 deducted. The limit test reads the whole pence back exactly, so float32 rounding no longer decides it. Pension contributions entered as a total still reduce earnings. The Scottish Carer Support Payment, which reads the same earnings, is judged exactly at its limit too. From ea3f7d3ffaa33d1f37297457b4f59db5bf3b0a7a Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 28 Oct 2026 13:08:49 -0400 Subject: [PATCH 12/13] Pin Class 1 entered for the year in whole pence (CA Case 37) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The r7 review's reproducer: £1.20 entered with £10,609.72 of pay leaves £10,608.52, a penny a week over, not £10,608.51. Co-Authored-By: Claude Opus 5.5 --- .../dwp/carers_allowance_earnings_limit.yaml | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml b/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml index f0fec1061c..4cf1708f2a 100644 --- a/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml +++ b/policyengine_uk/tests/policy/baseline/gov/dwp/carers_allowance_earnings_limit.yaml @@ -603,3 +603,21 @@ carers_allowance_earnings: 10_608.52 is_gainfully_employed_for_carers_allowance: true carers_allowance: 0 + +- name: Case 37, Class 1 entered for the year in whole pence is deducted in whole pence. + period: 2026 + absolute_error_margin: 0.001 + input: + age: 45 + employment_income: 10_609.72 + ni_class_1_employee: 1.20 + care_hours: 40 + country: ENGLAND + output: + # The model divides the year's 1.20 among the months in float32 and + # sums them to 1.2000002, which rounded up would deduct 1.21. The Class + # 1 deducted is the 1.20 entered (reg. 10(4)(a)): 10,609.72 - 1.20 = + # 10,608.52, or £204.01 a week, over the limit. + carers_allowance_earnings: 10_608.52 + is_gainfully_employed_for_carers_allowance: true + carers_allowance: 0 From 2502953bf8683859034fbe4d65addc8d4f215673 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 28 Oct 2026 13:09:40 -0400 Subject: [PATCH 13/13] Say the sums test checks the model's definitions, not sick pay's treatment The r7 review noted that _natural_rows called its reference legal net earnings, though the model counts statutory sick pay in NI income only. Co-Authored-By: Claude Opus 5.5 --- ...est_carers_allowance_earnings_limit_properties.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py index 09152232ea..c7571eaff2 100644 --- a/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py +++ b/policyengine_uk/tests/test_carers_allowance_earnings_limit_properties.py @@ -1342,11 +1342,13 @@ def test_what_the_simulation_keeps_does_not_change_earnings(row, uncached, net): def _natural_rows(rng, count, regime): - """Inputs the model sums, in whole pence, chosen so that the legal net - earnings lie within 3p of the 2026-27 threshold: pay before labour supply - responses a, salary sacrifice returned to pay b (also a member - contribution), reported employee contributions r, personal contributions - q and statutory sick pay s (NI income, not pay). + """Inputs the model sums, in whole pence, chosen so that net earnings, in + exact arithmetic on the model's own definitions, lie within 3p of the + 2026-27 threshold: pay before labour supply responses a, salary sacrifice + returned to pay b (also a member contribution), reported employee + contributions r, personal contributions q and statutory sick pay s. The + model counts statutory sick pay in NI income but not in pay, so these + rows test its sums, not how it treats statutory sick pay. - "no NI": below the primary threshold, so no PAYE or Class 1. - "Class 1": Class 1 on a + b + s, with no PAYE while pay less member