Repository navigation
Add a trading allowance claim status, so allowance reforms reach claimants without turnover data - #2765
Add a trading allowance claim status, so allowance reforms reach claimants without turnover data#2765juaristi22 wants to merge 5 commits into
Conversation
Add trading_allowance_claim_status, a person enum (UNKNOWN, CLAIMANT, NON_CLAIMANT; default UNKNOWN), and gov.simulation.trading_allowance_in_input_profits, the allowance a claimant's input profit already deducts: £1,000 from 2017-18, not moved by reforms to the trading allowance and not uprated. A claimant's receipts are their profit plus that amount, so the deduction at allowance A is min(profit, A - £1,000) less capital allowances, floored at nil, and a cut below £1,000 leaves their profit as reported. A non-claimant gets no allowance at any level, on trade receipts or on income not otherwise charged. Unknown keeps the current rules, so existing datasets are unchanged. Class 4 and the current-loss election read the same deduction. Issue #2764. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Aggregate impact on certified data: passBase Checks: 0 broken by this change, 1 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds in Notes
Moves flagged for reviewNone. Spending by benefit and service
Caseloads
Standard reform costings (change in gov_balance)
Standard reforms: change in each reform's target
Household net income by decileNo household's net income changes in any year. No change in any year: receipts by tax head; government balance, each component counted once; model totals; household income totals; population; poverty rates; people in poverty; inequality; mean household net income in each side's own deciles. Checks failing on both base and head (1)
Counted-once checkEach of 92 components was raised by £1 per unit in 2026-27, and every head of its totals was read household by household: its own head must rise by exactly £1, no other tax head may move, no other benefit may rise, and the model's own total must move by what its heads moved. A head that breaks a rule is measured again twice: with every other head held at its baseline value, and with every other head held at its value after the first £1 while the component rises again. It fails where either still shows the response. 0 failed; 0 were skipped.
Provenance
Dataset: Receipts by tax head: head levels
Spending by benefit and service: head levels
Caseloads: head levels
Government balance, each component counted once: head levels
Model totals: head levels
Household income totals: head levels
Population: head levels
Poverty rates: head levels
People in poverty: head levels
Inequality: head levels
Mean household net income in each side's own deciles: head levels
Standard reform costings (change in gov_balance): head levels
Standard reforms: change in each reform's target: head levels
|
Independent review of #2765 at
|
Non-claimants keep full relief, which needs no election (ITTOIA 2005 s. 783AE), where supplied receipts, or casual income alone with no profit, are within the allowance; partial relief stays elective. One ordinary-treatment mask replaces the two zeroing masks. A claimant's input allowance is now the allowance legislated for the year the data were measured (a dataset's first year, or the year simulated), read from the unreformed baseline parameters, so data from either side of the April 2027 raise read right. This replaces gov.simulation.trading_allowance_in_input_profits. Claimants: capital allowances are not deducted (partial relief replaces them), the status applies only to a positive profit, the deduction is computed from the allowances directly (no float32 round trip), and the claim is their election, so they take the allowance and give up current losses at any allowance above nil. A cut's effect is set by the new gov.simulation.trading_allowance_claimant_expense_share (default 1, the lower bound), which the trading allowance's description points to. Tests: YAML cases for each review point, a property test whose reference works from the Act's options, and dataset tests for the data year and the status carrying forward. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Response to the review at
|
Without a dataset the year simulated stands in for the year the inputs were measured, even for inputs supplied for an earlier year and carried forward, so a claimant given 2025-26 inputs and simulated in 2027-28 is read as net of £3,000 and gets nothing from the raise. Keep that convention, shared with the State Pension variables, and say so in _data_year, the claim status input and the deduction's documentation, with the workaround (build a dataset for the earlier year). A test pins the behaviour so that a change of convention has to update the docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-review of #2765 at
|
| # | Status |
|---|---|
| 1. Non-claimants and full relief | Fixed. The three cases give £3,486, nil on both amounts, and £3,486. Casual income above the allowance still gets nothing, as partial relief needs the election. The s. 783AL election out works in both directions. |
| 2. Claimant capital allowances | Fixed. Taxable profit is £2,500, and the raise gains the full £2,000. Class 4 and relevant UK earnings follow. |
| 3. Property test | Fixed. It works through the Act's options and exercises the min() cap. It fails each of the four re-applied bugs, as does the YAML. See 2 above for what it still copies. |
| 4. Input allowance | Fixed for datasets. Data measured in 2025 get £2,000 in 2027 and 2028, and data measured in 2027 get nothing. Five forms of reform all leave the input allowance alone. Situations: see 3 above. |
| 5. Cut assumption | Fixed. The new share parameter defaults to 1, and the allowance's description points to it. See 4 above. |
| 6. No-profit claimants | Fixed. They are read as unknown. |
| 7. float32 | Fixed. £1,800.40 gives exactly 0, and a sweep of 9,460 penny-valued profits found no non-zero deduction. |
8. uses_trading_allowance |
Fixed at the input allowance and under a raise. Under a cut, see 1 above. |
| 9. Carried-forward losses | Fixed on datasets: the raise lowers tax by exactly £400, with the loss given up both ways. |
| 10-12. Texts and the single mask | Fixed. |
Checked
- The new YAML (37 cases), the data-year file (6 tests) and the property file pass, and ruff is clean.
- On CI at
6b9a638d7, the YAML, lint and smoke-import jobs pass. The Python test shards were still running. - Not covered: the full suite, and the microsimulation tests, which need the dataset token.
🤖 Generated with Claude Code
A claimant takes the allowance only while it is above nil and at least their expenses (the expense share of the input allowance); below them they deduct those expenses, which is ordinary treatment, so a supplied current loss is claimable under any cut, not only under abolition. A situation's data year is now the latest year, up to the one simulated, for which trading_allowance_claim_status was supplied (read through supplied_input_periods; the status joins SUPPLIED_INPUT_VARIABLES), so inputs carried from an earlier year keep that year's allowance. Microdata keep their first year, chosen through built_from_data. This replaces the limitation recorded in 6b9a638. The expense share is clamped to [0, 1]. A dataset still cannot declare the year its incomes were measured; that is now documented. Tests: the property test's reference derives a claimant's choice by comparing the allowance with deducting their expenses, and gains a scenario at a share of 1.5; YAML cases cover a cut with a current loss, full relief exactly at the allowance and a share above 1; the situation data-year tests now check the supplied year. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Response to the re-review at
|
Third review of #2765 at
|
| # | Status |
|---|---|
| 1. Cut below expenses | Fixed at the default share. The round-2 case gives £3,686 at £0, £1, £500 and £999, £3,986 at £1,000, and £3,586 at £3,000. Over 175 allowance levels and 21 profiles, the only wrong-way steps are the loss jump in 3 above. Dataset and situation paths agree on all 2,856 values compared. |
| 2. Property test | Fixed. The reference compares the claimant's options rather than restating the rule, and it fails the old rule. Two YAML cases now sit exactly at the allowance, and < fails them. The situation loss test can now fail. Limit: it still shares the model's conventions (receipts as profit plus the input allowance, the clamp, a claimant's loss left out), so it can't test those. |
| 3. Situation data year | Fixed for situations whose status carries the profit's year. Every UK dataset path reads the dataset's first year: single- and multi-year datasets, h5 files, data frames, hf:// and gs:// cache hits, Microsimulation, and clones, branches, reforms and baselines. But see 1, 2 and 4 above. |
| 4. Share bounds | Fixed: 1.5 matches 1, and −0.5 matches 0. The lower clamp makes no difference for any allowance of £0 or more, which is harmless. |
| 5. Dataset measurement year | Documented, as the reply says. |
The round-1 fixes all still hold. UNKNOWN status still gives the merge base's results, dtypes included: 400 random people × 7 scenarios × 10 variables.
Checked
- Tests: the touched test files pass: the YAML (41 cases), the data-year file (8) and the property file (4).
- Mutations: each mutation the reply lists fails its tests.
- CI on
894450054: the YAML, lint, docs, smoke-import and aggregate impact jobs pass. The Python jobs were still running. - Not covered: the full suite, and the microsimulation tests, which need the dataset token.
🤖 Generated with Claude Code
_data_year checks the dataset's type, so a policyengine-core Dataset, which loads any attribute it is asked for, no longer crashes; data without years take the earliest year the profit was supplied for. In a situation the data year is now the latest year, up to the one simulated, for which self_employment_income was supplied, which replaces the status in SUPPLIED_INPUT_VARIABLES. Claimants make the same election as everyone else, against their own options: the allowance unless its saving over deducting their expenses is negative or smaller than a supplied current loss, so income tax no longer jumps as the allowance or the loss moves. Their expenses are the share of the input allowance in pennies, or their capital allowances if larger, capped at the input allowance, and their figures are computed in float64 so the two options tie exactly where the allowance equals those expenses. Tests: YAML cases for claimants with losses, the switch, capital allowances in the choice and the penny tie; data-year tests for the profit's year and the year filter; the property reference weighs a claimant's loss; a deterministic sweep of ties. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Response to the third review at
|
Fourth review of #2765 at
|
| # | Status |
|---|---|
| 1. Core datasets | Fixed. Every UK data path, and core simulations over data, read 2025 and give the 2027 claimant £2,000. The old lookup fails test_core_simulation_over_data_gets_imputations. The new rule is untested (3). |
| 2. Profit year | Fixed. A status for 2025 with a profit for 2027 gives 0; a status for 2027 with a profit for 2025 gives £2,000; a status with no year key and a 2027 profit gives 0. Side effects: registering the profit changes nothing for UNKNOWN people: 261 arrays against the merge base show no differences and identical dtypes, with labour supply responses on as well. But see 2. |
| 3. Claimants with a loss | Fixed, except where brought-forward losses cover both options (1). Over 5,355 allowance steps and 339,612 loss steps, neither income tax nor Class 4 ever moves the wrong way, and the reply's figures match. Round 1's example gains £100. The only cliffs left are the unknown and non-claimant full-relief cliffs, which the Act has. |
| 4. Filter | Fixed. Removing it fails the test at :150. |
| 5. Small items | Fixed. Penny rounding gives no flips over 121,200 claimants at exact ties. The float64 figures stay internal, so outputs are float32 or bool as before. The capital-allowance floor gives £2,700. The texts are updated, apart from those in 4. |
Checked
- Tests: the touched test files pass: the YAML (46 cases), the property file (7), the data-year file (9) and
test_data_built_imputations.py(9). - Mutations: each fix's old rule fails its tests.
- Performance:
_data_yearcosts under 0.03 ms a call on 50,000 people. - CI on
243515502:- The YAML, lint, docs, smoke-import, changelog and aggregate-impact jobs pass, and so do Python shards 1, 4, 5 and 6.
- Shards 2 and 3 hit the 1h15m job limit and were cancelled, so "Test / Result" fails.
- At
894450054those two shards took 73.0 and 74.1 minutes, and every shard is 1 to 5 minutes slower here. So the added property and tie-sweep tests have likely tipped them over. - A rerun may pass, but the margin is gone. Smaller sweeps, or another shard, would restore it.
🤖 Generated with Claude Code
Fixes #2764
Survey data hold profit, not turnover, so a raise in the trading allowance missed its main base: traders who already claim £1,000 against larger receipts. The Budget 2026 raise to £3,000 (in current law since #2719) is costed on that base; HMRC's policy paper puts it at about 400,000 taxpayers with a lower bill, by about £380 on average. This PR lets a dataset say who claims, so the engine reaches that base at any allowance level.
What changes
trading_allowance_claim_status(person, enum, defaultUNKNOWN): whether the person claimed the allowance in the year their input profit was measured. It describes a positive profit; a claimant with no profit is read as unknown.CLAIMANT: the profit is receipts less the allowance, as under partial relief (ITTOIA 2005 s. 783AI).NON_CLAIMANT: the profit is receipts less actual expenses (capital allowances come off separately), and those deductions exceed the allowance.self_employment_income) was supplied, so a profit carried from an earlier year keeps that year's allowance, whatever year the status was given for.Dataset, it is the earliest year the profit was supplied for.parameters.baseline, which reforms leave alone.gov.simulation.trading_allowance_claimant_expense_share(default 1, bounded to [0, 1]): claimants' expenses and capital allowances as a share of the input allowance. It only matters under a cut below the input allowance.utils/trading_allowance.py, whichtrading_allowance_deduction,miscellaneous_income_trading_allowance_deductionanduses_trading_allowanceshare:uses_trading_allowance, giving up a current loss) unless its saving over their expenses is negative or smaller than a supplied current loss. Income tax doesn't jump as the allowance or the loss moves.self_employment_gross_receiptsis supplied, otherwise the no-receipts rule. Existing datasets and situations give the same results.ni_class_4_profits_before_losses) and relevant UK earnings read the same deduction, and their code is unchanged.Design choices for review
An enum, not a bool with a known flag. One column holds three exclusive states. With a bool and a known flag, the two columns must agree: a dataset that sets the flag but not its partner is silently ignored, and one that marks everyone known turns every unflagged trader into a non-claimant. The default
UNKNOWNis what leaves existing data alone. Datasets store the enum name (CLAIMANT), whichSimulationencodes like other enum columns, and the value carries forward from the year supplied.The input allowance is a fact about the data, not a simulation-year parameter. Reading it for the data year lets one model version serve data from before and after the April 2027 raise, with no separate copy of the allowance's history.
A situation records the periods its inputs were supplied for, so its data year is the latest year, up to the one simulated, for which the profit was supplied.
One gap remains, documented in
_data_year: data can't declare the year their incomes were measured, so a file saved from a later projected year, or a situation saved over several years, is read as measured in its first year.A cut below the input allowance is set by an explicit share. Claimants' expenses are below the input allowance but unknown. Under a cut, a claimant with no current loss keeps claiming while the new allowance covers their expenses, and otherwise deducts those (ordinary treatment). So their taxable profit rises by the input allowance less the larger of the two.
A claimant with a supplied current loss takes ordinary treatment, claiming the loss, until the allowance's saving over their expenses exceeds it. At the input allowance and the default share that saving is nil, so they claim it there: their claim and the loss contradict each other, and the election settles it as it does for everyone.
The trading allowance's own description points to the parameter.
Non-claimants keep full relief, but not partial relief. Partial relief needs an election (s. 783AM), and electing after a raise is behaviour, as HMRC's costings book it. Full relief needs none.
Without supplied receipts, a non-claimant's receipts are unknown but above their profit plus the input allowance. So the model shows full relief only from supplied receipts, or from casual income alone where there is no profit. Elsewhere, under a raise it gives none, which is a lower bound.
Chapter 8 miscellaneous income (#2725, #2726) is handled, not left as is. A claimant's receipts take up the allowance first (trades first, s. 783AK), and income not otherwise charged gets the rest.
A non-claimant gets it only through full relief.
For the data side (PolicyEngine/microcosm#1428)
CLAIMANTis read as receipts above £1,000 and taxed at baseline, whereUNKNOWNrelieves it in full. Traders whose receipts were within £1,000 belong atUNKNOWN.self_employment_gross_receipts, and the input allowance is nominal, so the gap no longer grows with mixed-income uprating.trading_allowance_deductionto zero, as policyengine-uk-data does on SPI records, bypasses the formula and blocks every allowance reform. Flagging claimants and non-claimants avoids that.Tests
policyengine_uk/tests/policy/baseline/gov/hmrc/income_tax/allowances/trading_allowance_claim_status.yaml(46 cases):policyengine_uk/tests/test_trading_allowance_claim_status_properties.py(hypothesis, 30 people per draw, 15 scenarios of allowance and expense share, one at a share of 1.5):uses_trading_allowance, with Chapter 8 and miscellaneous income drawn separately.policyengine_uk/tests/test_trading_allowance_claim_status_data_year.py(small datasets and situations):test_data_built_imputations.py::test_core_simulation_over_data_gets_imputations, which894450054broke, passes again.Mutation checks. Each of these breaks fails both the YAML and the property tests:
Reading the input allowance at the simulation year fails the data-year tests.
Second-round mutation checks:
<for full relief fails the two exact-boundary YAML cases;Third-round mutation checks:
test_core_simulation_over_data_gets_imputations;test_trading_allowance_claim_status_properties.pyalso sweeps 2,000 claimants with no loss at three exact ties of allowance and expenses; all must take the allowance.Commands run in the locked environment (
uv run --frozen --extra dev):policyengine-core testonpolicyengine_uk/tests/policy/baseline/gov/hmrc: 1,303 passed.pyteston the three files above and on the related tests: 572 passed and 10 skipped, then 25 passed on the touched files after the last test changes. Nine skips are the certified-dataset aggregate-impact tests, which need the dataset token and run in CI. The tenth istest_build_metadata.py's bundle-contract test, which needspolicyengine_bundlesand is on CI's allowlist. The related tests are trading allowance, Chapter 8, trading loss, Class 4, income concepts, data-built imputations, aggregate impact, parameter descriptions and metadata, Pension Credit Class 2, Welsh childcare income, director NI, variable units and build metadata.ruff format .andruff check .: clean..github/check_changelog_fragments.py: OK.Not run locally: the full suite, left to CI, and the microsimulation tests, which need the dataset token.
axiom: n/a: adds an input that records how the data's profits were measured and a gov.simulation convention parameter; the Part 6A rules (ss. 783AE, 783AF, 783AI) are unchanged
🤖 Generated with Claude Code