Land six reviewed pull requests together (rollup B)

changelog.d/2253.fixed.md

@@ -0,0 +1 @@
+- Fixed Gift Aid income tax relief by extending rate limits instead of deducting donations from taxable income, and deducting grossed-up gifts from adjusted net income.

changelog.d/2316.changed.md

@@ -0,0 +1 @@
+- Cut the CPU the Python tests spend in garbage collection: each test process raises the first-generation GC threshold and freezes the objects that exist once the tests are collected (#2316).

changelog.d/2330.fixed.md

@@ -0,0 +1,3 @@
+- Stop the Python tests' memory growing with every Simulation under sys.monitoring coverage: each test process compiles each variable file once and every tax-benefit system runs that code (#2330).
+- Release dead tax-benefit systems' variable modules in the Python tests once eight more systems own modules than were alive at the last release, checked as each new system starts adding its variables as well as after each test.
+- Hand freed memory back to the system after each test file (glibc malloc_trim), and stop keeping the unpinned latest dataset cached after its smoke test.

changelog.d/2337.fixed.md

@@ -0,0 +1 @@
+- Counted statutory sick, maternity and paternity pay as Universal Credit earned income (UC Regs 2013 reg. 55(4)). Universal Credit already deducted the National Insurance on all three, and the income tax on sick and maternity pay, so it rose as the payments rose. Statutory paternity pay is now also counted in household benefits, HBAI income and government spending, as sick and maternity pay already were.

changelog.d/2338.fixed.md

@@ -0,0 +1 @@
+- Counted the Scottish Child Payment, Healthy Start vouchers and the Two Child Limit Payment in government spending, so `gov_balance` no longer leaves out benefits that households receive.

changelog.d/2374.fixed.md

@@ -0,0 +1 @@
+- A failed batch of legacy parameter overrides is now rolled back whole, nodes a structural reform adds to the parameters survive later rebuilds and reprocessing, and the scenario docs say that `Scenario.apply` on a built simulation edits the annualised values.

docs/book/usage/scenarios.md

@@ -396,7 +396,9 @@ Each way of passing a reform reads period keys as it always has:
 
 A single-date key in a `reform=` dict overrides the annualised value on that day only. Formulas read annual values on 1 January, so `"2027-01-01"` sets fiscal year 2027 and `"2027-04-06"` changes nothing. The model logs a warning when a key can change nothing. Use `"start.stop"` keys for dated measures.
 
-Changes applied after the data is loaded (the default) leave the data as built, so a changed economic assumption does not re-uprate incomes. They still come before the steps that read parameters while the simulation is built: the Universal Credit rebalancing awards and the structural reforms that parameters switch on. Set `applied_before_data_load=True` on a scenario to project the data with changed economic assumptions too.
+Changes applied after the data is loaded (the default) leave the data as built, so a changed economic assumption does not re-uprate incomes. Dated changes (`parameter_changes` and `"start.stop"` or bare-year keys) still come before the steps that read parameters while the simulation is built: the Universal Credit rebalancing awards and the structural reforms that parameters switch on. Single-day keys and a scenario's own simulation modifier run after those steps, as they always have. Set `applied_before_data_load=True` on a scenario to project the data with changed economic assumptions too.
+
+All of this applies when the scenario or reform is passed to `Simulation(...)`. `Scenario.apply(simulation)` on a simulation that is already built edits its annualised values instead, as it always has: a `"2027-04-06.2028-04-05"` key there overrides those dates of the fiscal-year values, so it does not set fiscal year 2027, and the model logs a warning saying so. Pass the scenario when constructing the simulation for dated measures.
 
 ### Creating complex scenarios with simulation modifiers
 

docs/book/validation/spi-validation.ipynb

@@ -55,7 +55,7 @@
     "1. Cases where an individual appears to transfer a portion of their Personal Allowance to their spouse as part of the Marriage Allowance. This is because the SPI removes the transferred amount from the individual's Personal Allowance, but does not indicate that the subsequent lower amount is driven by the Marriage Allowance. This may represent as much as 37,998 records (4.8% of total).\n",
     "1. Cases where an individual receives less than £100,000 in annual income and is non-resident for taxation purposes. In this situation, the individual receives £0 Personal Allowance, even though our model would project a full £12,500 allowance. The SPI uses a variable that is not publicly available to calculate this status. This represents 2,054 records (0.3% of total).\n",
     "1. Cases where both the SPI and PolicyEngine calculate the same amount of taxable savings income for an individual receiving any form of savings income classified by the SPI as \"Other Investment Income\" (e.g., interest on securities), but the SPI finds that they owe around £1,000 more in income tax than we do. This represents 1,276 records (0.2% of total).\n",
-    "1. Cases where an individual near the Personal Allowance taper value who claims Gift Aid receives a full Personal Allowance of £12,500 within the SPI model. The SPI model's Gift Aid variable signifies the amount an individual deducts due to Gift Aid donations, while PolicyEngine's represents an individual's total Gift Aid-eligible donations. We then calculate the individual's deduction amount from this value. This error represents 738 records (0.1% of total).\n",
+    "1. Cases where an individual near the Personal Allowance taper value who claims Gift Aid receives a full Personal Allowance of £12,500 within the SPI model. This historical validation identified 738 records (0.1% of total) with differences associated with the interpretation of Gift Aid inputs. PolicyEngine's `gift_aid` input is the net eligible donation. Under [ITA 2007 ss. 414–415](https://www.legislation.gov.uk/ukpga/2007/3/section/414), the grossed-up gift extends the income tax rate limits; it does not reduce taxable income. The grossed-up gift also reduces adjusted net income for the Personal Allowance taper under [s. 58](https://www.legislation.gov.uk/ukpga/2007/3/section/58). These figures describe the original validation and have not been recalculated after that treatment was corrected.\n",
     "\n",
     "## Conclusion\n",
     "PolicyEngine's UK model shows a high level of accordance with the SPI's 2020/21 data, with almost 93% of records with the same inputs returning the same total income tax calculation. While some discrepancies exist, they are limited in scope and often attributable to either testability limitations within the SPI's structure or to inputs that the SPI has not made publicly available. This validation exercise builds upon our previous validations, such as our [Autumn 2023 model calibration](https://policyengine.org/uk/research/uk-calibration-2023-q4). We welcome questions and comments on our findings or methodology - please feel free to [get in touch](mailto:contact@policyengine.org)."

docs/engineering/gift-aid.md

@@ -0,0 +1,84 @@
+# Gift Aid and income tax
+
+`gift_aid` is the net eligible donation. [Income Tax Act 2007
+s. 414(2)(a)](https://www.legislation.gov.uk/ukpga/2007/3/section/414)
+treats the gift as made after deduction of basic-rate income tax, and
+[s. 415](https://www.legislation.gov.uk/ukpga/2007/3/section/415) defines its
+grossed-up amount. The model calculates `gift_aid_grossed_up` as the net gift
+divided by one minus the UK basic rate.
+
+Gift Aid does not belong in `allowances` and does not reduce taxable income.
+Under s. 414(2)(b), the gross gift increases both UK rate limits. For Scottish
+non-savings income, it also increases the Scottish basic-rate upper limit and
+every limit above it; the starter-rate upper limit remains unchanged. Savings
+and dividends use the extended UK limits for Scottish taxpayers too.
+
+The model extends limits where band amounts are calculated. For variables that
+use a complete rate scale, `calc_income_tax_with_gift_aid` in
+`policyengine_uk/utils/income_tax.py` applies an equivalent correction to the
+scale's original calculation. At each extended threshold, it subtracts the
+rate increase multiplied by the income between the old and extended limits.
+For a stacked slice, subtracting the corrected tax at its bottom from the
+corrected tax at its top gives the tax on that slice. This preserves the
+original scale calculation when the gift is zero and handles gifts that cross
+several thresholds.
+
+`net_income` holds the income base previously exposed as `adjusted_net_income`.
+Taxable-income calculations and allowance allocation use this base.
+When adjusted net income is supplied explicitly, its gross Gift Aid adjustment
+is added back to obtain the taxable-income base. This preserves direct income
+inputs when the gift is zero without recovering ordinary taxable income from a
+rounded adjusted-net-income calculation. The marriage-neutral income-tax
+reform splits `net_income` before each person's Gift Aid adjustment.
+`adjusted_net_income` deducts the gross gift once, as [s. 58(1), Step
+2](https://www.legislation.gov.uk/ukpga/2007/3/section/58) requires. The Personal
+Allowance taper and High Income Child Benefit Charge therefore see the lower
+adjusted net income. The Personal Allowance formula does not deduct Gift Aid a
+second time. Capital gains tax uses `net_income`, retains its gross Gift Aid
+band extension, and no longer removes the net gift from `allowances`.
+
+Savings and dividends stack above all non-savings income, property income
+included ([ITA 2007 s. 16](https://www.legislation.gov.uk/ukpga/2007/3/section/16);
+#2352). Property income, savings and dividends therefore fill the bands in that
+order, each against the extended limits. Property income is taxed on top of
+`non_savings_income_after_allowances` less property income after allowances,
+and savings and dividends on top of `non_savings_taxable_income`
+(`earned_taxable_income` plus property income after allowances). Both bases
+derive from `net_income`, so the gift raises the limits without also shrinking
+the income beneath them. They differ only when Marriage Allowance applies,
+which `earned_taxable_income` subtracts (#1947). The Personal Savings Allowance
+compares net income less allowances with the UK basic and higher rate limits,
+both extended by the gross gift ([s. 12B](https://www.legislation.gov.uk/ukpga/2007/3/section/12B),
+including s. 12B(8) for Scottish taxpayers, with s. 414(2)(b)).
+
+## Existing limitations
+
+Zero-gift calculations retain their existing behavior. Several inherited
+limitations remain relevant when interpreting mixed-income results:
+
+- The Scottish top-rate threshold parameter is £112,570 of income after
+  allowances for 2024 onwards. The [HMRC rates and allowances
+  table](https://www.gov.uk/government/publications/rates-and-allowances-income-tax/income-tax-rates-and-allowances-current-and-past)
+  gives £125,140 after allowances. The Gift Aid correction extends the model's
+  existing threshold and does not repair it.
+- Adjusted net income still omits other adjustments in s. 58, including the
+  deduction for pension contributions relieved at source in Step 3.
+- Gifts are assumed to qualify. The model does not charge the donor for a
+  shortfall in income tax and capital gains tax covering the tax attributed to
+  gifts under [ss. 423–424](https://www.legislation.gov.uk/ukpga/2007/3/section/424).
+
+## Relief bounds in tests
+
+For a fixed taxable-income stack and fixed zero-rate allowances, the maximum
+band-extension relief is the gross gift multiplied by the sum of positive
+rate increases at extended limits. For earned income, that cumulative rate
+increase is the top rate minus the basic rate: 25 percentage points for the UK
+schedule and 28 for the current Scottish schedule. A bound using only the
+largest adjacent rate increase would fail even for the £150,000 UK example,
+where both UK limits move.
+
+Restoring Personal Allowance through the adjusted-net-income taper can give
+additional relief. Changing tax band can also increase the Personal Savings
+Allowance discretely under s. 12B, so its relief cannot be bounded by a constant
+times a very small gift. Tests of the continuous band-extension bound must
+allow for that allowance change or hold the zero-rate allowance fixed.

policyengine_uk/reforms/cps/marriage_tax_reforms.py

@@ -1,4 +1,5 @@
 from policyengine_uk.model_api import *
+from policyengine_uk.utils.supplied_inputs import supplied_input
 from typing import Union, Optional
 
 
@@ -153,14 +154,23 @@ def formula(person, period, parameters):
                 COMPONENTS.append("basic_income")
             return max_(0, add(person, period, COMPONENTS))
 
-    class adjusted_net_income(Variable):
-        label = "Optimised adjusted net income"
-        documentation = "Adjusted net income, but split equally between partners if they are married"
+    class net_income(Variable):
+        label = "Optimised net income"
+        documentation = (
+            "Income before Gift Aid adjustments, split equally between married partners"
+        )
         entity = Person
         definition_period = YEAR
         value_type = float
 
         def formula(person, period, parameters):
+            # A supplied adjusted net income already includes the s.58 Gift
+            # Aid step, and it took precedence over this reform's split before
+            # the reform moved to net_income. Keep that: restore the gift for
+            # the taxable base, as the baseline net_income does.
+            supplied_ani = supplied_input(person, "adjusted_net_income", period)
+            if supplied_ani is not None:
+                return supplied_ani + person("gift_aid_grossed_up", period)
             income = person("unadjusted_net_income", period)
             # Split between the spouses or civil partners only, never their
             # children.
@@ -171,9 +181,7 @@ def formula(person, period, parameters):
             originally_split_income_branch = person.simulation.get_branch(
                 "originally_split_income", clone_system=True
             )
-            originally_split_income_branch.set_input(
-                "adjusted_net_income", period, income
-            )
+            originally_split_income_branch.set_input("net_income", period, income)
             originally_split_income_tax = person.benunit.sum(
                 originally_split_income_branch.calculate("income_tax", period)
             )
@@ -186,7 +194,7 @@ def formula(person, period, parameters):
             split_income_branch = person.simulation.get_branch(
                 "split_income", clone_system=True
             )
-            split_income_branch.set_input("adjusted_net_income", period, split_income)
+            split_income_branch.set_input("net_income", period, split_income)
             split_income_tax = person.benunit.sum(
                 split_income_branch.calculate("income_tax", period)
             )
@@ -200,7 +208,7 @@ def formula(person, period, parameters):
     class reform(Reform):
         def apply(self):
             self.add_variable(meets_ma_neutral_tax_conditions)
-            self.update_variable(adjusted_net_income)
+            self.update_variable(net_income)
             self.add_variable(unadjusted_net_income)
 
     return reform