diff --git a/changelog.d/state-pension-age-67-phase-in-dates.fixed.md b/changelog.d/state-pension-age-67-phase-in-dates.fixed.md new file mode 100644 index 0000000000..fc93652d8b --- /dev/null +++ b/changelog.d/state-pension-age-67-phase-in-dates.fixed.md @@ -0,0 +1 @@ +- Removed NumPy's generic-timedelta deprecation warning from State Pension date arithmetic without changing calculated dates. diff --git a/changelog.d/state-pension-age-67-phase-in-household.fixed.md b/changelog.d/state-pension-age-67-phase-in-household.fixed.md new file mode 100644 index 0000000000..b16bd12811 --- /dev/null +++ b/changelog.d/state-pension-age-67-phase-in-household.fixed.md @@ -0,0 +1 @@ +- Household calculations default an unspecified birthday position to six months before 6 October, implying a 6 April birth date and making an entered age of 66 above State Pension age at that check in 2026-27 and below it in 2027-28, while fractional ages or `months_since_last_birthday` can specify a different inferred birth date even when other people supply birthday months. diff --git a/changelog.d/state-pension-age-67-phase-in-migration.fixed.md b/changelog.d/state-pension-age-67-phase-in-migration.fixed.md new file mode 100644 index 0000000000..4eea28eb8e --- /dev/null +++ b/changelog.d/state-pension-age-67-phase-in-migration.fixed.md @@ -0,0 +1 @@ +- Added replacement guidance for reforms, structural reforms, nested scenarios and saved API policies using the removed male and female State Pension age parameters, rejecting obsolete simulation changes before resetting existing policy. diff --git a/changelog.d/state-pension-age-67-phase-in.added.md b/changelog.d/state-pension-age-67-phase-in.added.md new file mode 100644 index 0000000000..633358e0a3 --- /dev/null +++ b/changelog.d/state-pension-age-67-phase-in.added.md @@ -0,0 +1 @@ +Add `months_since_last_birthday`, which places a person's date of birth within their year of age, and `months_since_state_pension_age`. diff --git a/changelog.d/state-pension-age-67-phase-in.fixed.md b/changelog.d/state-pension-age-67-phase-in.fixed.md new file mode 100644 index 0000000000..1bacfc9cb7 --- /dev/null +++ b/changelog.d/state-pension-age-67-phase-in.fixed.md @@ -0,0 +1 @@ +Set State Pension age from each person's date of birth under Pensions Act 1995 Schedule 4 paragraph 1, including the rise from 66 to 67 for people born on or after 6 April 1960 (Pensions Act 2014 s.26): 66-year-olds move below State Pension age through 2026-27 and 2027-28, and are all below it from 2028-29. The Savings Credit age test, the split between basic and new State Pension, and additional State Pension now follow each person's own State Pension age. diff --git a/changelog.d/state-pension-age-67-phase-in.removed.md b/changelog.d/state-pension-age-67-phase-in.removed.md new file mode 100644 index 0000000000..b381fb4703 --- /dev/null +++ b/changelog.d/state-pension-age-67-phase-in.removed.md @@ -0,0 +1 @@ +Remove `gov.dwp.state_pension.age.male` and `gov.dwp.state_pension.age.female`, a single State Pension age for each year. The statutory timetable by date of birth replaces them: `gov.dwp.state_pension.age.age_by_birth_date` (the age, in months) and `day_by_birth_date` (the day), with `male.age` and `male.born_before` for men born before 6 December 1953. A reform to State Pension age now changes those rows, for example the ages for births from a given date; code that read a State Pension age should read `state_pension_age` or `is_SP_age`. diff --git a/docs/book/programs/gov/dwp/state-pension.md b/docs/book/programs/gov/dwp/state-pension.md index e144b6809b..2f068e4f27 100644 --- a/docs/book/programs/gov/dwp/state-pension.md +++ b/docs/book/programs/gov/dwp/state-pension.md @@ -30,7 +30,68 @@ Three variables under `gov/dwp/`: The flag-up split (`state_pension_type`) is settled by [PR #1618](https://github.com/PolicyEngine/policyengine-uk/pull/1618): classification is based on whether the person reaches State Pension age -before or on/after 6 April 2016. +before or on/after 6 April 2016. No component is paid before State +Pension age. + +## State Pension age + +State Pension age depends on date of birth. Pensions Act 1995 Schedule 4 +paragraph 1 sets it for each birth period, either as an age or as the day +on which it is attained, and the Pensions Act 2014 s.26 raises it from 66 +to 67 for people born on or after 6 April 1960: 66 years and 1 to 11 +months for births from 6 April 1960 to 5 March 1961, and 67 from 6 March +1961. The parameters under `gov/dwp/state_pension/age/` encode that +timetable row by row: + +- `age_by_birth_date` — the age, in months, by date of birth (women, and + men born on or after 6 December 1953); +- `day_by_birth_date` — the day, where the statute sets a day instead; +- `male/age` and `male/born_before` — rule (1): men born before 6 + December 1953 attain it at 65. + +A person attains State Pension age on the later of the two. The model +places each person's date of birth with `age` and +`months_since_last_birthday`, which counts months since their last +birthday on 6 October, the middle of the fiscal year: + +- `state_pension_age` is the person's own State Pension age; +- `months_since_state_pension_age` is how long before 6 October they + attained it (negative if later); +- `is_SP_age` is whether they have attained it by 6 October, so are over + it for most of the year. + +Simulations of a household situation take a whole age to be the middle of +the year of age (six months since the birthday); a fractional age is read as +the exact age on 6 October, overriding `months_since_last_birthday` even +when another person supplies birthday months. The default implies a birth +date of 6 April: an entered age of 66 is above State Pension age at the +6 October check in 2026-27 and below it in 2027-28. Set a fractional age +or `months_since_last_birthday` to specify a different inferred birth date. +Survey microdata records whole years only, so in +simulations built from data, including a region or constituency filtered +from it, each single year of age and sex is spread evenly over the year by +weight, and `filter_dataset` carries each person's place into a household it +extracts. The weighted share of each age over State Pension age then matches +the statute: three quarters of 66-year-olds in 2026-27, a quarter in 2027-28 +and none from 2028-29. + +A person attains an age at the start of the anniversary of their birth +(Family Law Reform Act 1969 s.9(1)), and an age of "N years and M months" on +the same day of the month, or the month's last day where that day does not +exist; that also gives the three days rule (7A) sets. A reform can change the +age or day of any row, or where a row starts; a new phase-in that needs extra +rows, such as bringing forward the rise to 68, needs new rows in the +parameter files. + +The removed scalar parameters `gov.dwp.state_pension.age.male` and `.female` +remain discoverable in metadata as migration nodes: explicit root `get_child` +lookups, updates, and scalar value-history reads for saved API policies raise +an error naming the birth-date +scales above. The `male.age` and `male.born_before` children remain available +for the earlier male cohorts. For example, reform +`gov.dwp.state_pension.age.age_by_birth_date[14].amount` to change the age +in months for births from 6 March 1961 until the next bracket; read individual +eligibility from `is_SP_age` or the individual threshold from `state_pension_age`. ## Uprating diff --git a/policyengine_uk/data/filter_dataset.py b/policyengine_uk/data/filter_dataset.py index db0b2bafcc..3b70dd4c0d 100644 --- a/policyengine_uk/data/filter_dataset.py +++ b/policyengine_uk/data/filter_dataset.py @@ -1,5 +1,8 @@ import typing +import numpy as np +import pandas as pd + if typing.TYPE_CHECKING: from policyengine_uk import Microsimulation from policyengine_uk.data import UKSingleYearDataset @@ -33,9 +36,19 @@ def filter_dataset( """ dataset: UKSingleYearDataset = sim.dataset[year] new_dataset = dataset.copy() - new_dataset.person = new_dataset.person[ - new_dataset.person.person_household_id == household_id - ] + person = new_dataset.person + if "months_since_last_birthday" not in person.columns: + # Birthdays are spread over the year across the whole population, so + # carry each person's place across rather than recompute it for one + # household. + months = pd.Series( + np.asarray(sim.calculate("months_since_last_birthday", year)), + index=np.asarray(sim.calculate("person_id", year)), + ) + person = person.assign( + months_since_last_birthday=months.loc[person.person_id].values + ) + new_dataset.person = person[person.person_household_id == household_id] new_dataset.household = new_dataset.household[ new_dataset.household.household_id == household_id ] diff --git a/policyengine_uk/parameters/gov/dwp/pension_credit/savings_credit/cutoff_year.yaml b/policyengine_uk/parameters/gov/dwp/pension_credit/savings_credit/cutoff_year.yaml index 6ade64f304..41325e44dc 100644 --- a/policyengine_uk/parameters/gov/dwp/pension_credit/savings_credit/cutoff_year.yaml +++ b/policyengine_uk/parameters/gov/dwp/pension_credit/savings_credit/cutoff_year.yaml @@ -1,4 +1,4 @@ -description: People reaching State Pension age after this year are not eligible for Savings Credit. +description: People who reach State Pension age on or after 6 April of this year are not eligible for Savings Credit. metadata: label: Savings Credit eligibility cutoff year reference: diff --git a/policyengine_uk/parameters/gov/dwp/pension_credit/savings_credit/minimum_age.yaml b/policyengine_uk/parameters/gov/dwp/pension_credit/savings_credit/minimum_age.yaml new file mode 100644 index 0000000000..5336af15ff --- /dev/null +++ b/policyengine_uk/parameters/gov/dwp/pension_credit/savings_credit/minimum_age.yaml @@ -0,0 +1,9 @@ +description: Savings Credit requires the claimant, or their partner, to have attained this age as well as State Pension age. +values: + 2002-10-01: 65 +metadata: + unit: year + label: Savings Credit minimum age + reference: + - title: State Pension Credit Act 2002, section 3(1)(a) + href: https://www.legislation.gov.uk/ukpga/2002/16/section/3 diff --git a/policyengine_uk/parameters/gov/dwp/state_pension/age/age_by_birth_date.yaml b/policyengine_uk/parameters/gov/dwp/state_pension/age/age_by_birth_date.yaml new file mode 100644 index 0000000000..bfd96fe60c --- /dev/null +++ b/policyengine_uk/parameters/gov/dwp/state_pension/age/age_by_birth_date.yaml @@ -0,0 +1,119 @@ +description: People attain State Pension age when they reach this age, which depends on their date of birth; zero where the statute sets a day instead of an age. +metadata: + type: single_amount + threshold_unit: date + amount_unit: month + label: State Pension age by date of birth + documentation: Applies to women, and to men born on or after 6 December 1953; men born earlier follow the male rule. Where the statute sets the day on which State Pension age is attained, the amount is zero and day_by_birth_date gives the day. A person attains State Pension age on the later of the two. + reference: + - title: Pensions Act 1995, Schedule 4, paragraph 1 (rules for determining pensionable age) + href: https://www.legislation.gov.uk/ukpga/1995/26/schedule/4 + - title: Pensions Act 2014, section 26 (increase in pensionable age to 67) + href: https://www.legislation.gov.uk/ukpga/2014/19/section/26 + - title: Pensions Act 2011, section 1 (equalisation and increase to 66) + href: https://www.legislation.gov.uk/ukpga/2011/19/section/1 + - title: Pensions Act 2007, section 13 (increases to 66, 67 and 68) + href: https://www.legislation.gov.uk/ukpga/2007/22/section/13 + - title: Pensions (Northern Ireland) Order 1995, Schedule 2, paragraph 1 (the same timetable for Northern Ireland, with each rule numbered one higher) + href: https://www.legislation.gov.uk/nisi/1995/3213/schedule/2 + +# The timetable is keyed from the Pensions Act 1995 and reflects its current +# text, last amended by the Pensions Act 2014 (section 26, in force 14 July +# 2014). Earlier versions are not encoded separately: every amendment moved +# only people who had not yet reached pensionable age, so the timetable gives +# the right status for any past date. +# +# Dates of birth and days are written as YYYYMMDD numbers. Each bracket runs +# from its threshold (inclusive) to the next threshold. + +# An age of N years and M months is attained on the same day of the month, or +# on the month's last day where that day does not exist. That gives the three +# days rule (7A) sets for table 3: 30 November 2026 for a birthday on 31 July +# 1960, 30 September 2027 for 31 December 1960 and 30 November 2027 for 31 +# January 1961. +brackets: + # Rule (2): a woman born before 6th April 1950 attains pensionable age when she attains the age of 60. + - threshold: + 1995-07-19: 0 + amount: + 1995-07-19: 720 + # Rules (3) and (5), tables 1 and 2: born 6th April 1950 to 5th October 1954, pensionable age is attained on the day shown in day_by_birth_date. + - threshold: + 1995-07-19: 19500406 + amount: + 1995-07-19: 0 + # Rule (6): a person born after 5th October 1954 but before 6th April 1960 attains pensionable age when the person attains the age of 66. + - threshold: + 1995-07-19: 19541006 + amount: + 1995-07-19: 792 + # Rule (7), table 3: 6th April 1960 to 5th May 1960: 66 years and 1 month. + - threshold: + 1995-07-19: 19600406 + amount: + 1995-07-19: 793 + # Rule (7), table 3: 6th May 1960 to 5th June 1960: 66 years and 2 months. + - threshold: + 1995-07-19: 19600506 + amount: + 1995-07-19: 794 + # Rule (7), table 3: 6th June 1960 to 5th July 1960: 66 years and 3 months. + - threshold: + 1995-07-19: 19600606 + amount: + 1995-07-19: 795 + # Rule (7), table 3: 6th July 1960 to 5th August 1960: 66 years and 4 months. + - threshold: + 1995-07-19: 19600706 + amount: + 1995-07-19: 796 + # Rule (7), table 3: 6th August 1960 to 5th September 1960: 66 years and 5 months. + - threshold: + 1995-07-19: 19600806 + amount: + 1995-07-19: 797 + # Rule (7), table 3: 6th September 1960 to 5th October 1960: 66 years and 6 months. + - threshold: + 1995-07-19: 19600906 + amount: + 1995-07-19: 798 + # Rule (7), table 3: 6th October 1960 to 5th November 1960: 66 years and 7 months. + - threshold: + 1995-07-19: 19601006 + amount: + 1995-07-19: 799 + # Rule (7), table 3: 6th November 1960 to 5th December 1960: 66 years and 8 months. + - threshold: + 1995-07-19: 19601106 + amount: + 1995-07-19: 800 + # Rule (7), table 3: 6th December 1960 to 5th January 1961: 66 years and 9 months. + - threshold: + 1995-07-19: 19601206 + amount: + 1995-07-19: 801 + # Rule (7), table 3: 6th January 1961 to 5th February 1961: 66 years and 10 months. + - threshold: + 1995-07-19: 19610106 + amount: + 1995-07-19: 802 + # Rule (7), table 3: 6th February 1961 to 5th March 1961: 66 years and 11 months. + - threshold: + 1995-07-19: 19610206 + amount: + 1995-07-19: 803 + # Rule (8): a person born after 5th March 1961 but before 6th April 1977 attains pensionable age when the person attains the age of 67. + - threshold: + 1995-07-19: 19610306 + amount: + 1995-07-19: 804 + # Rule (9), table 4: born 6th April 1977 to 5th April 1978, pensionable age is attained on the day shown in day_by_birth_date. + - threshold: + 1995-07-19: 19770406 + amount: + 1995-07-19: 0 + # Rule (10): a person born after 5th April 1978 attains pensionable age when the person attains the age of 68. + - threshold: + 1995-07-19: 19780406 + amount: + 1995-07-19: 816 diff --git a/policyengine_uk/parameters/gov/dwp/state_pension/age/day_by_birth_date.yaml b/policyengine_uk/parameters/gov/dwp/state_pension/age/day_by_birth_date.yaml new file mode 100644 index 0000000000..764a4c1115 --- /dev/null +++ b/policyengine_uk/parameters/gov/dwp/state_pension/age/day_by_birth_date.yaml @@ -0,0 +1,374 @@ +description: People attain State Pension age at the commencement of this day, which depends on their date of birth; zero where the statute sets an age instead of a day. +metadata: + type: single_amount + threshold_unit: date + amount_unit: date + label: State Pension age day by date of birth + documentation: Applies to women, and to men born on or after 6 December 1953; men born earlier follow the male rule. Where the statute sets the age at which State Pension age is attained, the amount is zero and age_by_birth_date gives the age. A person attains State Pension age on the later of the two. + reference: + - title: Pensions Act 1995, Schedule 4, paragraph 1 (rules for determining pensionable age) + href: https://www.legislation.gov.uk/ukpga/1995/26/schedule/4 + - title: Pensions Act 2014, section 26 (increase in pensionable age to 67) + href: https://www.legislation.gov.uk/ukpga/2014/19/section/26 + - title: Pensions Act 2011, section 1 (equalisation and increase to 66) + href: https://www.legislation.gov.uk/ukpga/2011/19/section/1 + - title: Pensions Act 2007, section 13 (increases to 66, 67 and 68) + href: https://www.legislation.gov.uk/ukpga/2007/22/section/13 + - title: Pensions (Northern Ireland) Order 1995, Schedule 2, paragraph 1 (the same timetable for Northern Ireland, with each rule numbered one higher) + href: https://www.legislation.gov.uk/nisi/1995/3213/schedule/2 + +# The timetable is keyed from the Pensions Act 1995 and reflects its current +# text, last amended by the Pensions Act 2014 (section 26, in force 14 July +# 2014). Earlier versions are not encoded separately: every amendment moved +# only people who had not yet reached pensionable age, so the timetable gives +# the right status for any past date. +# +# Dates of birth and days are written as YYYYMMDD numbers. Each bracket runs +# from its threshold (inclusive) to the next threshold. + +brackets: + # Rule (2): born before 6th April 1950, pensionable age is set by age_by_birth_date. + - threshold: + 1995-07-19: 0 + amount: + 1995-07-19: 0 + # Rule (3), table 1: 6th April 1950 to 5th May 1950: 6th May 2010. + - threshold: + 1995-07-19: 19500406 + amount: + 1995-07-19: 20100506 + # Rule (3), table 1: 6th May 1950 to 5th June 1950: 6th July 2010. + - threshold: + 1995-07-19: 19500506 + amount: + 1995-07-19: 20100706 + # Rule (3), table 1: 6th June 1950 to 5th July 1950: 6th September 2010. + - threshold: + 1995-07-19: 19500606 + amount: + 1995-07-19: 20100906 + # Rule (3), table 1: 6th July 1950 to 5th August 1950: 6th November 2010. + - threshold: + 1995-07-19: 19500706 + amount: + 1995-07-19: 20101106 + # Rule (3), table 1: 6th August 1950 to 5th September 1950: 6th January 2011. + - threshold: + 1995-07-19: 19500806 + amount: + 1995-07-19: 20110106 + # Rule (3), table 1: 6th September 1950 to 5th October 1950: 6th March 2011. + - threshold: + 1995-07-19: 19500906 + amount: + 1995-07-19: 20110306 + # Rule (3), table 1: 6th October 1950 to 5th November 1950: 6th May 2011. + - threshold: + 1995-07-19: 19501006 + amount: + 1995-07-19: 20110506 + # Rule (3), table 1: 6th November 1950 to 5th December 1950: 6th July 2011. + - threshold: + 1995-07-19: 19501106 + amount: + 1995-07-19: 20110706 + # Rule (3), table 1: 6th December 1950 to 5th January 1951: 6th September 2011. + - threshold: + 1995-07-19: 19501206 + amount: + 1995-07-19: 20110906 + # Rule (3), table 1: 6th January 1951 to 5th February 1951: 6th November 2011. + - threshold: + 1995-07-19: 19510106 + amount: + 1995-07-19: 20111106 + # Rule (3), table 1: 6th February 1951 to 5th March 1951: 6th January 2012. + - threshold: + 1995-07-19: 19510206 + amount: + 1995-07-19: 20120106 + # Rule (3), table 1: 6th March 1951 to 5th April 1951: 6th March 2012. + - threshold: + 1995-07-19: 19510306 + amount: + 1995-07-19: 20120306 + # Rule (3), table 1: 6th April 1951 to 5th May 1951: 6th May 2012. + - threshold: + 1995-07-19: 19510406 + amount: + 1995-07-19: 20120506 + # Rule (3), table 1: 6th May 1951 to 5th June 1951: 6th July 2012. + - threshold: + 1995-07-19: 19510506 + amount: + 1995-07-19: 20120706 + # Rule (3), table 1: 6th June 1951 to 5th July 1951: 6th September 2012. + - threshold: + 1995-07-19: 19510606 + amount: + 1995-07-19: 20120906 + # Rule (3), table 1: 6th July 1951 to 5th August 1951: 6th November 2012. + - threshold: + 1995-07-19: 19510706 + amount: + 1995-07-19: 20121106 + # Rule (3), table 1: 6th August 1951 to 5th September 1951: 6th January 2013. + - threshold: + 1995-07-19: 19510806 + amount: + 1995-07-19: 20130106 + # Rule (3), table 1: 6th September 1951 to 5th October 1951: 6th March 2013. + - threshold: + 1995-07-19: 19510906 + amount: + 1995-07-19: 20130306 + # Rule (3), table 1: 6th October 1951 to 5th November 1951: 6th May 2013. + - threshold: + 1995-07-19: 19511006 + amount: + 1995-07-19: 20130506 + # Rule (3), table 1: 6th November 1951 to 5th December 1951: 6th July 2013. + - threshold: + 1995-07-19: 19511106 + amount: + 1995-07-19: 20130706 + # Rule (3), table 1: 6th December 1951 to 5th January 1952: 6th September 2013. + - threshold: + 1995-07-19: 19511206 + amount: + 1995-07-19: 20130906 + # Rule (3), table 1: 6th January 1952 to 5th February 1952: 6th November 2013. + - threshold: + 1995-07-19: 19520106 + amount: + 1995-07-19: 20131106 + # Rule (3), table 1: 6th February 1952 to 5th March 1952: 6th January 2014. + - threshold: + 1995-07-19: 19520206 + amount: + 1995-07-19: 20140106 + # Rule (3), table 1: 6th March 1952 to 5th April 1952: 6th March 2014. + - threshold: + 1995-07-19: 19520306 + amount: + 1995-07-19: 20140306 + # Rule (3), table 1: 6th April 1952 to 5th May 1952: 6th May 2014. + - threshold: + 1995-07-19: 19520406 + amount: + 1995-07-19: 20140506 + # Rule (3), table 1: 6th May 1952 to 5th June 1952: 6th July 2014. + - threshold: + 1995-07-19: 19520506 + amount: + 1995-07-19: 20140706 + # Rule (3), table 1: 6th June 1952 to 5th July 1952: 6th September 2014. + - threshold: + 1995-07-19: 19520606 + amount: + 1995-07-19: 20140906 + # Rule (3), table 1: 6th July 1952 to 5th August 1952: 6th November 2014. + - threshold: + 1995-07-19: 19520706 + amount: + 1995-07-19: 20141106 + # Rule (3), table 1: 6th August 1952 to 5th September 1952: 6th January 2015. + - threshold: + 1995-07-19: 19520806 + amount: + 1995-07-19: 20150106 + # Rule (3), table 1: 6th September 1952 to 5th October 1952: 6th March 2015. + - threshold: + 1995-07-19: 19520906 + amount: + 1995-07-19: 20150306 + # Rule (3), table 1: 6th October 1952 to 5th November 1952: 6th May 2015. + - threshold: + 1995-07-19: 19521006 + amount: + 1995-07-19: 20150506 + # Rule (3), table 1: 6th November 1952 to 5th December 1952: 6th July 2015. + - threshold: + 1995-07-19: 19521106 + amount: + 1995-07-19: 20150706 + # Rule (3), table 1: 6th December 1952 to 5th January 1953: 6th September 2015. + - threshold: + 1995-07-19: 19521206 + amount: + 1995-07-19: 20150906 + # Rule (3), table 1: 6th January 1953 to 5th February 1953: 6th November 2015. + - threshold: + 1995-07-19: 19530106 + amount: + 1995-07-19: 20151106 + # Rule (3), table 1: 6th February 1953 to 5th March 1953: 6th January 2016. + - threshold: + 1995-07-19: 19530206 + amount: + 1995-07-19: 20160106 + # Rule (3), table 1: 6th March 1953 to 5th April 1953: 6th March 2016. + - threshold: + 1995-07-19: 19530306 + amount: + 1995-07-19: 20160306 + # Rule (3), table 1: 6th April 1953 to 5th May 1953: 6th July 2016. + - threshold: + 1995-07-19: 19530406 + amount: + 1995-07-19: 20160706 + # Rule (3), table 1: 6th May 1953 to 5th June 1953: 6th November 2016. + - threshold: + 1995-07-19: 19530506 + amount: + 1995-07-19: 20161106 + # Rule (3), table 1: 6th June 1953 to 5th July 1953: 6th March 2017. + - threshold: + 1995-07-19: 19530606 + amount: + 1995-07-19: 20170306 + # Rule (3), table 1: 6th July 1953 to 5th August 1953: 6th July 2017. + - threshold: + 1995-07-19: 19530706 + amount: + 1995-07-19: 20170706 + # Rule (3), table 1: 6th August 1953 to 5th September 1953: 6th November 2017. + - threshold: + 1995-07-19: 19530806 + amount: + 1995-07-19: 20171106 + # Rule (3), table 1: 6th September 1953 to 5th October 1953: 6th March 2018. + - threshold: + 1995-07-19: 19530906 + amount: + 1995-07-19: 20180306 + # Rule (3), table 1: 6th October 1953 to 5th November 1953: 6th July 2018. + - threshold: + 1995-07-19: 19531006 + amount: + 1995-07-19: 20180706 + # Rule (3), table 1: 6th November 1953 to 5th December 1953: 6th November 2018. + - threshold: + 1995-07-19: 19531106 + amount: + 1995-07-19: 20181106 + # Rule (5), table 2: 6th December 1953 to 5th January 1954: 6th March 2019. + - threshold: + 1995-07-19: 19531206 + amount: + 1995-07-19: 20190306 + # Rule (5), table 2: 6th January 1954 to 5th February 1954: 6th May 2019. + - threshold: + 1995-07-19: 19540106 + amount: + 1995-07-19: 20190506 + # Rule (5), table 2: 6th February 1954 to 5th March 1954: 6th July 2019. + - threshold: + 1995-07-19: 19540206 + amount: + 1995-07-19: 20190706 + # Rule (5), table 2: 6th March 1954 to 5th April 1954: 6th September 2019. + - threshold: + 1995-07-19: 19540306 + amount: + 1995-07-19: 20190906 + # Rule (5), table 2: 6th April 1954 to 5th May 1954: 6th November 2019. + - threshold: + 1995-07-19: 19540406 + amount: + 1995-07-19: 20191106 + # Rule (5), table 2: 6th May 1954 to 5th June 1954: 6th January 2020. + - threshold: + 1995-07-19: 19540506 + amount: + 1995-07-19: 20200106 + # Rule (5), table 2: 6th June 1954 to 5th July 1954: 6th March 2020. + - threshold: + 1995-07-19: 19540606 + amount: + 1995-07-19: 20200306 + # Rule (5), table 2: 6th July 1954 to 5th August 1954: 6th May 2020. + - threshold: + 1995-07-19: 19540706 + amount: + 1995-07-19: 20200506 + # Rule (5), table 2: 6th August 1954 to 5th September 1954: 6th July 2020. + - threshold: + 1995-07-19: 19540806 + amount: + 1995-07-19: 20200706 + # Rule (5), table 2: 6th September 1954 to 5th October 1954: 6th September 2020. + - threshold: + 1995-07-19: 19540906 + amount: + 1995-07-19: 20200906 + # Rules (6) to (8): born 6th October 1954 to 5th April 1977, pensionable age is set by age_by_birth_date. + - threshold: + 1995-07-19: 19541006 + amount: + 1995-07-19: 0 + # Rule (9), table 4: 6th April 1977 to 5th May 1977: 6th May 2044. + - threshold: + 1995-07-19: 19770406 + amount: + 1995-07-19: 20440506 + # Rule (9), table 4: 6th May 1977 to 5th June 1977: 6th July 2044. + - threshold: + 1995-07-19: 19770506 + amount: + 1995-07-19: 20440706 + # Rule (9), table 4: 6th June 1977 to 5th July 1977: 6th September 2044. + - threshold: + 1995-07-19: 19770606 + amount: + 1995-07-19: 20440906 + # Rule (9), table 4: 6th July 1977 to 5th August 1977: 6th November 2044. + - threshold: + 1995-07-19: 19770706 + amount: + 1995-07-19: 20441106 + # Rule (9), table 4: 6th August 1977 to 5th September 1977: 6th January 2045. + - threshold: + 1995-07-19: 19770806 + amount: + 1995-07-19: 20450106 + # Rule (9), table 4: 6th September 1977 to 5th October 1977: 6th March 2045. + - threshold: + 1995-07-19: 19770906 + amount: + 1995-07-19: 20450306 + # Rule (9), table 4: 6th October 1977 to 5th November 1977: 6th May 2045. + - threshold: + 1995-07-19: 19771006 + amount: + 1995-07-19: 20450506 + # Rule (9), table 4: 6th November 1977 to 5th December 1977: 6th July 2045. + - threshold: + 1995-07-19: 19771106 + amount: + 1995-07-19: 20450706 + # Rule (9), table 4: 6th December 1977 to 5th January 1978: 6th September 2045. + - threshold: + 1995-07-19: 19771206 + amount: + 1995-07-19: 20450906 + # Rule (9), table 4: 6th January 1978 to 5th February 1978: 6th November 2045. + - threshold: + 1995-07-19: 19780106 + amount: + 1995-07-19: 20451106 + # Rule (9), table 4: 6th February 1978 to 5th March 1978: 6th January 2046. + - threshold: + 1995-07-19: 19780206 + amount: + 1995-07-19: 20460106 + # Rule (9), table 4: 6th March 1978 to 5th April 1978: 6th March 2046. + - threshold: + 1995-07-19: 19780306 + amount: + 1995-07-19: 20460306 + # Rule (10): born after 5th April 1978, pensionable age is set by age_by_birth_date. + - threshold: + 1995-07-19: 19780406 + amount: + 1995-07-19: 0 diff --git a/policyengine_uk/parameters/gov/dwp/state_pension/age/female.yaml b/policyengine_uk/parameters/gov/dwp/state_pension/age/female.yaml deleted file mode 100644 index 84f32f7f9f..0000000000 --- a/policyengine_uk/parameters/gov/dwp/state_pension/age/female.yaml +++ /dev/null @@ -1,12 +0,0 @@ -description: Female State Pension age. -values: - 1950-01-05: 60 - 2010-01-01: 61 - 2012-01-01: 62 - 2014-01-01: 63 - 2016-01-01: 64 - 2019-01-01: 65 - 2020-01-01: 66 -metadata: - unit: year - reference: https://www.iser.essex.ac.uk/research/publications/working-papers/cempa/cempa7-20.pdf#page=88 diff --git a/policyengine_uk/parameters/gov/dwp/state_pension/age/male.yaml b/policyengine_uk/parameters/gov/dwp/state_pension/age/male.yaml deleted file mode 100644 index 6a9c528426..0000000000 --- a/policyengine_uk/parameters/gov/dwp/state_pension/age/male.yaml +++ /dev/null @@ -1,7 +0,0 @@ -description: Male State Pension age. -values: - 1950-01-05: 65 - 2020-01-01: 66 -metadata: - unit: year - reference: https://www.iser.essex.ac.uk/research/publications/working-papers/cempa/cempa7-20.pdf#page=88 diff --git a/policyengine_uk/parameters/gov/dwp/state_pension/age/male/age.yaml b/policyengine_uk/parameters/gov/dwp/state_pension/age/male/age.yaml new file mode 100644 index 0000000000..ddd638b1b4 --- /dev/null +++ b/policyengine_uk/parameters/gov/dwp/state_pension/age/male/age.yaml @@ -0,0 +1,9 @@ +description: Men born before the male cutoff date attain State Pension age when they reach this age. +values: + 1995-07-19: 780 +metadata: + unit: month + label: Male State Pension age before equalisation + reference: + - title: Pensions Act 1995, Schedule 4, paragraph 1, rule (1) + href: https://www.legislation.gov.uk/ukpga/1995/26/schedule/4 diff --git a/policyengine_uk/parameters/gov/dwp/state_pension/age/male/born_before.yaml b/policyengine_uk/parameters/gov/dwp/state_pension/age/male/born_before.yaml new file mode 100644 index 0000000000..a4c075ae88 --- /dev/null +++ b/policyengine_uk/parameters/gov/dwp/state_pension/age/male/born_before.yaml @@ -0,0 +1,9 @@ +description: Men born before this date, written as a YYYYMMDD number, attain State Pension age at the male State Pension age; men born on or after it follow the same timetable as women. +values: + 1995-07-19: 19531206 +metadata: + unit: date + label: Male State Pension age cutoff date of birth + reference: + - title: Pensions Act 1995, Schedule 4, paragraph 1, rule (1) + href: https://www.legislation.gov.uk/ukpga/1995/26/schedule/4 diff --git a/policyengine_uk/simulation.py b/policyengine_uk/simulation.py index 22ddaa61a1..cd39f80a3a 100644 --- a/policyengine_uk/simulation.py +++ b/policyengine_uk/simulation.py @@ -16,7 +16,10 @@ from policyengine_core.tools.hugging_face import download_huggingface_dataset from policyengine_core.tracers import FullTracer, SimpleTracer -from policyengine_uk.utils.parameters import uk_fiscal_year_period +from policyengine_uk.utils.parameters import ( + check_parameter_not_removed, + uk_fiscal_year_period, +) # PolicyEngine UK imports from policyengine_uk.data.dataset_schema import ( @@ -102,6 +105,11 @@ class Simulation(CoreSimulation): calculated_periods: List[str] = [] _variable_dependencies: Dict[str, List[str]] = None dataset = None + # True when built from survey or other microdata rather than a situation + # dictionary. Variables that impute unobserved detail across a population + # (such as months_since_last_birthday) read it; unlike the sum of weights, + # it stays true for a region or constituency filtered from the data. + built_from_dataset: bool = False def __init__( self, @@ -173,6 +181,7 @@ def __init__( self.build_from_dataset_source(get_default_dataset_url()) else: raise ValueError(f"Unsupported dataset type: {dataset.__class__}") + self.built_from_dataset = situation is None # Universal Credit reform (July 2025). Needs closer integration in the baseline, # but adding here for ease of toggling on/off via the 'active' parameter. @@ -232,6 +241,8 @@ def get_known_variables(self): return variables def apply_parameter_changes(self, changes: dict): + for parameter in changes: + check_parameter_not_removed(canonicalize_lsr_parameter_path(parameter)) self.tax_benefit_system.reset_parameters() for parameter in changes: diff --git a/policyengine_uk/tax_benefit_system.py b/policyengine_uk/tax_benefit_system.py index d632be9dd9..c8264232cc 100644 --- a/policyengine_uk/tax_benefit_system.py +++ b/policyengine_uk/tax_benefit_system.py @@ -37,6 +37,7 @@ add_lsr_deprecation_aliases, ) from policyengine_uk.utils.parameters import ( + add_removed_parameter_aliases, backdate_parameters, convert_to_fiscal_year_parameters, ) @@ -80,6 +81,7 @@ def reset_parameters(self) -> None: """Reset parameters by reloading from the parameters directory.""" self._parameters_at_instant_cache = {} self.load_parameters(self.parameters_dir) + self.parameters = add_removed_parameter_aliases(self.parameters) def process_parameters(self) -> None: """Process and transform parameters with UK-specific adjustments. diff --git a/policyengine_uk/tests/fixtures/pensions_act_1995_schedule_4_paragraph_1.txt b/policyengine_uk/tests/fixtures/pensions_act_1995_schedule_4_paragraph_1.txt new file mode 100644 index 0000000000..a6530b4f04 --- /dev/null +++ b/policyengine_uk/tests/fixtures/pensions_act_1995_schedule_4_paragraph_1.txt @@ -0,0 +1,109 @@ +Pensions Act 1995 (c. 26), Schedule 4, Part I, paragraph 1 -- current consolidated text. +Source: https://www.legislation.gov.uk/ukpga/1995/26/schedule/4 (point in time 31 July 2026, fetched 28 September 2026). +Contains public sector information licensed under the Open Government Licence v3.0. + +Rules for determining pensionable age +1 The following rules apply for the purposes of the enactments relating to social security, that is, the following Acts and the instruments made, or having effect as if made, under them: the Social Security Contributions and Benefits Act 1992, the Social Security Administration Act 1992 , the Pension Schemes Act 1993 (c. 48) , the State Pension Credit Act 2002 , Part 1 of the Welfare Reform Act 2007 and the Pensions Act 2014. + +(1) A man born before 6th December 1953 attains pensionable age when he attains the age of 65 years. +(2) A woman born before 6th April 1950 attains pensionable age when she attains the age of 60. +(3) A woman born on any day in a period mentioned in column 1 of table 1 attains pensionable age at the commencement of the day shown against that period in column 2. +(4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . + TABLE 1 + | (1) Period within which woman’s birthday falls | (2) Day pensionable age attained | + | 6th April 1950 to 5th May 1950 | 6th May 2010 | + | 6th May 1950 to 5th June 1950 | 6th July 2010 | + | 6th June 1950 to 5th July 1950 | 6th September 2010 | + | 6th July 1950 to 5th August 1950 | 6th November 2010 | + | 6th August 1950 to 5th September 1950 | 6th January 2011 | + | 6th September 1950 to 5th October 1950 | 6th March 2011 | + | 6th October 1950 to 5th November 1950 | 6th May 2011 | + | 6th November 1950 to 5th December 1950 | 6th July 2011 | + | 6th December 1950 to 5th January 1951 | 6th September 2011 | + | 6th January 1951 to 5th February 1951 | 6th November 2011 | + | 6th February 1951 to 5th March 1951 | 6th January 2012 | + | 6th March 1951 to 5th April 1951 | 6th March 2012 | + | 6th April 1951 to 5th May 1951 | 6th May 2012 | + | 6th May 1951 to 5th June 1951 | 6th July 2012 | + | 6th June 1951 to 5th July 1951 | 6th September 2012 | + | 6th July 1951 to 5th August 1951 | 6th November 2012 | + | 6th August 1951 to 5th September 1951 | 6th January 2013 | + | 6th September 1951 to 5th October 1951 | 6th March 2013 | + | 6th October 1951 to 5th November 1951 | 6th May 2013 | + | 6th November 1951 to 5th December 1951 | 6th July 2013 | + | 6th December 1951 to 5th January 1952 | 6th September 2013 | + | 6th January 1952 to 5th February 1952 | 6th November 2013 | + | 6th February 1952 to 5th March 1952 | 6th January 2014 | + | 6th March 1952 to 5th April 1952 | 6th March 2014 | + | 6th April 1952 to 5th May 1952 | 6th May 2014 | + | 6th May 1952 to 5th June 1952 | 6th July 2014 | + | 6th June 1952 to 5th July 1952 | 6th September 2014 | + | 6th July 1952 to 5th August 1952 | 6th November 2014 | + | 6th August 1952 to 5th September 1952 | 6th January 2015 | + | 6th September 1952 to 5th October 1952 | 6th March 2015 | + | 6th October 1952 to 5th November 1952 | 6th May 2015 | + | 6th November 1952 to 5th December 1952 | 6th July 2015 | + | 6th December 1952 to 5th January 1953 | 6th September 2015 | + | 6th January 1953 to 5th February 1953 | 6th November 2015 | + | 6th February 1953 to 5th March 1953 | 6th January 2016 | + | 6th March 1953 to 5th April 1953 | 6th March 2016 | + | 6th April 1953 to 5th May 1953 | 6th July 2016 | + | 6th May 1953 to 5th June 1953 | 6th November 2016 | + | 6th June 1953 to 5th July 1953 | 6th March 2017 | + | 6th July 1953 to 5th August 1953 | 6th July 2017 | + | 6th August 1953 to 5th September 1953 | 6th November 2017 | + | 6th September 1953 to 5th October 1953 | 6th March 2018 | + | 6th October 1953 to 5th November 1953 | 6th July 2018 | + | 6th November 1953 to 5th December 1953 | 6th November 2018 | +(5) A person born on any day in a period mentioned in column 1 of table 2 attains pensionable age at the commencement of the day shown against that period in column 2. + TABLE 2 + | (1) | (2) | + | Period within which birthday falls | Day pensionable age attained | + | 6th December 1953 to 5th January 1954 | 6th March 2019 | + | 6th January 1954 to 5th February 1954 | 6th May 2019 | + | 6th February 1954 to 5th March 1954 | 6th July 2019 | + | 6th March 1954 to 5th April 1954 | 6th September 2019 | + | 6th April 1954 to 5th May 1954 | 6th November 2019 | + | 6th May 1954 to 5th June 1954 | 6th January 2020 | + | 6th June 1954 to 5th July 1954 | 6th March 2020 | + | 6th July 1954 to 5th August 1954 | 6th May 2020 | + | 6th August 1954 to 5th September 1954 | 6th July 2020 | + | 6th September 1954 to 5th October 1954 | 6th September 2020 | +(6) A person born after 5th October 1954 but before 6th April 1960 attains pensionable age when the person attains the age of 66. +(7) A person born on any day in a period mentioned in column 1 of table 3 attains pensionable age when the person attains the age shown against that period in column 2. + TABLE 3 + | (1) | (2) | + | Period within which birthday falls | Age pensionable age attained | + | 6th April 1960 to 5th May 1960 | 66 years and 1 month | + | 6th May 1960 to 5th June 1960 | 66 years and 2 months | + | 6th June 1960 to 5th July 1960 | 66 years and 3 months | + | 6th July 1960 to 5th August 1960 | 66 years and 4 months | + | 6th August 1960 to 5th September 1960 | 66 years and 5 months | + | 6th September 1960 to 5th October 1960 | 66 years and 6 months | + | 6th October 1960 to 5th November 1960 | 66 years and 7 months | + | 6th November 1960 to 5th December 1960 | 66 years and 8 months | + | 6th December 1960 to 5th January 1961 | 66 years and 9 months | + | 6th January 1961 to 5th February 1961 | 66 years and 10 months | + | 6th February 1961 to 5th March 1961 | 66 years and 11 months | +(7A) For the purposes of table 3— + (a) a person born on 31st July 1960 is to be taken to attain the age of 66 years and 4 months at the commencement of 30th November 2026; + (b) a person born on 31st December 1960 is to be taken to attain the age of 66 years and 9 months at the commencement of 30th September 2027; + (c) a person born on 31st January 1961 is to be taken to attain the age of 66 years and 10 months at the commencement of 30th November 2027. +(8) A person born after 5th March 1961 but before 6th April 1977 attains pensionable age when the person attains the age of 67. +(9) A person born on any day in a period mentioned in column 1 of table 4 attains pensionable age at the commencement of the day shown against that period in column 2. + TABLE 4 + | (1) | (2) | + | Period within which birthday falls | Day pensionable age attained | + | 6th April 1977 to 5th May 1977 | 6th May 2044 | + | 6th May 1977 to 5th June 1977 | 6th July 2044 | + | 6th June 1977 to 5th July 1977 | 6th September 2044 | + | 6th July 1977 to 5th August 1977 | 6th November 2044 | + | 6th August 1977 to 5th September 1977 | 6th January 2045 | + | 6th September 1977 to 5th October 1977 | 6th March 2045 | + | 6th October 1977 to 5th November 1977 | 6th May 2045 | + | 6th November 1977 to 5th December 1977 | 6th July 2045 | + | 6th December 1977 to 5th January 1978 | 6th September 2045 | + | 6th January 1978 to 5th February 1978 | 6th November 2045 | + | 6th February 1978 to 5th March 1978 | 6th January 2046 | + | 6th March 1978 to 5th April 1978 | 6th March 2046 | +(10) A person born after 5th April 1978 attains pensionable age when the person attains the age of 68. diff --git a/policyengine_uk/tests/policy/baseline/gov/dwp/is_SP_age.yaml b/policyengine_uk/tests/policy/baseline/gov/dwp/is_SP_age.yaml new file mode 100644 index 0000000000..6c30919953 --- /dev/null +++ b/policyengine_uk/tests/policy/baseline/gov/dwp/is_SP_age.yaml @@ -0,0 +1,164 @@ +# State Pension age status for whole ages, which single-household simulations +# place in the middle of the year of age (months_since_last_birthday = 6), so a +# 66-year-old on 6 October 2026 was born on 6 April 1960 and one on 6 October +# 2027 on 6 April 1961. Pensions Act 1995 Sch 4 para 1 as amended by the +# Pensions Act 2014 s.26: 66 for people born before 6 April 1960, then 66 and +# 1 to 11 months, then 67 for people born on or after 6 March 1961. + +- name: A 66-year-old is over State Pension age in 2025-26 + period: 2025 + input: + age: 66 + output: + months_since_last_birthday: 6 + state_pension_age: 66 + is_SP_age: true + +- name: A 65-year-old is under State Pension age in 2025-26 + period: 2025 + input: + age: 65 + output: + is_SP_age: false + +- name: A 66-year-old born on 6 April 1960 is over State Pension age of 66 and 1 month in 2026-27 + period: 2026 + absolute_error_margin: 0.001 + input: + age: 66 + output: + state_pension_age: 66.0833 + months_since_state_pension_age: 5 + is_SP_age: true + +- name: A 66-year-old born on 6 April 1961 is under State Pension age of 67 in 2027-28 + period: 2027 + input: + age: 66 + output: + state_pension_age: 67 + months_since_state_pension_age: -6 + is_SP_age: false + is_WA_adult: true + +- name: A 67-year-old is over State Pension age in 2027-28 + period: 2027 + input: + age: 67 + output: + is_SP_age: true + +- name: A 66-year-old is under State Pension age from 2028-29 + period: 2028 + input: + age: 66 + output: + state_pension_age: 67 + is_SP_age: false + +- name: A 67-year-old is over State Pension age in 2028-29 + period: 2028 + input: + age: 67 + output: + state_pension_age: 67 + is_SP_age: true + +- name: A fractional age is the exact age on 6 October (66 years and 3 months, born 6 July 1960) + period: 2026 + absolute_error_margin: 0.001 + input: + age: 66.25 + output: + months_since_last_birthday: 3 + state_pension_age: 66.3333 + is_SP_age: false + +- name: A 66-year-old born in May 1960 is over State Pension age in 2026-27 + period: 2026 + input: + age: 66 + months_since_last_birthday: 5 + output: + is_SP_age: true + +- name: A 66-year-old born in August 1960 is under State Pension age in 2026-27 + period: 2026 + input: + age: 66 + months_since_last_birthday: 2 + output: + is_SP_age: false + +- name: Mixed-age couple in 2027-28 claims Universal Credit, not Pension Credit + period: 2027 + input: + people: + older: + age: 67 + younger: + age: 66 + benunits: + benunit: + members: [older, younger] + households: + household: + members: [older, younger] + output: + is_SP_age: [true, false] + is_WA_adult: [false, true] + is_uc_eligible: true + is_pension_credit_eligible: false + +- name: A single 66-year-old below State Pension age in 2027-28 cannot get Pension Credit or State Pension + period: 2027 + input: + people: + person: + age: 66 + state_pension_reported: 12_000 + benunits: + benunit: + members: [person] + households: + household: + members: [person] + output: + is_pension_credit_eligible: false + is_uc_eligible: true + state_pension_type: NONE + new_state_pension: 0 + additional_state_pension: 0 + state_pension: 0 + +- name: The same 66-year-old over State Pension age in 2025-26 gets State Pension + period: 2025 + input: + people: + person: + age: 66 + state_pension_reported: 12_000 + benunits: + benunit: + members: [person] + households: + household: + members: [person] + output: + is_uc_eligible: false + state_pension_type: NEW + state_pension: 12_000 + +- name: Class 1 employee NI applies to a 66-year-old below State Pension age in 2027-28 + period: 2027 + input: + age: 66 + output: + ni_liable: true + +- name: Class 1 employee NI does not apply to a 66-year-old over State Pension age in 2025-26 + period: 2025 + input: + age: 66 + output: + ni_liable: false diff --git a/policyengine_uk/tests/policy/baseline/gov/dwp/pension_credit/savings_credit/meets_savings_credit_age_requirement.yaml b/policyengine_uk/tests/policy/baseline/gov/dwp/pension_credit/savings_credit/meets_savings_credit_age_requirement.yaml index 136627c125..4b48ff4a6b 100644 --- a/policyengine_uk/tests/policy/baseline/gov/dwp/pension_credit/savings_credit/meets_savings_credit_age_requirement.yaml +++ b/policyengine_uk/tests/policy/baseline/gov/dwp/pension_credit/savings_credit/meets_savings_credit_age_requirement.yaml @@ -1,77 +1,140 @@ -- name: Person born in 1948 reaches SPA before cutoff year +# A whole age places the date of birth in the middle of the year of age: a +# 75-year-old on 6 October 2023 was born on 6 April 1948. + +- name: Man born 6 April 1948 reached State Pension age at 65 in 2013 period: 2023 input: people: person: - age: 75 # Born in 1948, reached SPA around 2013-2014 - state_pension_age: 65 + age: 75 + is_male: true benunits: benunit: members: [person] output: meets_savings_credit_age_requirement: true -- name: Person born in 1952 reaches SPA before cutoff year +- name: Woman born 6 April 1952 reached State Pension age on 6 May 2014 (table 1) period: 2023 input: people: person: - age: 71 # Born in 1952, reached SPA around 2015 - state_pension_age: 63 + age: 71 + is_male: false benunits: benunit: members: [person] output: meets_savings_credit_age_requirement: true -- name: Person born in 1954 reaches SPA after cutoff year +- name: Woman born 6 April 1954 reached State Pension age on 6 November 2019 (table 2) period: 2023 input: people: person: - age: 69 # Born in 1954, reached SPA around 2017-2018 - state_pension_age: 63.5 + age: 69 + is_male: false benunits: benunit: members: [person] output: meets_savings_credit_age_requirement: false -- name: Person born in 1960 reaches SPA after cutoff year +- name: Person born 6 April 1960 has not reached State Pension age period: 2023 input: people: person: - age: 63 # Born in 1960, will reach SPA around 2026-2027 - state_pension_age: 66 + age: 63 benunits: benunit: members: [person] output: meets_savings_credit_age_requirement: false -- name: Test in 2015 - Person born in 1950 reaches SPA before cutoff year +- name: Woman born 6 April 1950 reached State Pension age on 6 May 2010 and is 65 in 2015 period: 2015 input: people: person: - age: 65 # Born in 1950, reached SPA around 2013 - state_pension_age: 63 + age: 65 + is_male: false benunits: benunit: members: [person] output: meets_savings_credit_age_requirement: true -- name: Test in 2017 - Person who reached SPA in 2015 is eligible +- name: Woman born 6 April 1952 reached State Pension age on 6 May 2014 and is 65 in 2017 period: 2017 input: people: person: - age: 65 # Born in 1952, reached SPA in 2015 + age: 65 + is_male: false + benunits: + benunit: + members: [person] + output: + meets_savings_credit_age_requirement: true + +- name: An input State Pension age overrides the timetable (63, so reached in 2015) + period: 2023 + input: + people: + person: + age: 71 + is_male: true state_pension_age: 63 benunits: benunit: members: [person] output: - meets_savings_credit_age_requirement: true \ No newline at end of file + meets_savings_credit_age_requirement: true + +# State Pension Credit Act 2002 s.3(1)(a): attained pensionable age before 6 +# April 2016 and attained the age of 65. Each person's pensionable age comes +# from their date of birth (Pensions Act 1995 Sch 4 para 1). + +- name: Woman born 5 April 1953 reached State Pension age on 6 March 2016, before the cutoff + period: 2023 + input: + age: 70 + months_since_last_birthday: 6.032258 + is_male: false + output: + meets_savings_credit_age_requirement: true + +- name: Woman born 6 April 1953 reached State Pension age on 6 July 2016, after the cutoff + period: 2023 + input: + age: 70 + months_since_last_birthday: 6 + is_male: false + output: + meets_savings_credit_age_requirement: false + +- name: Man born in 1952 reached State Pension age at 65 in 2017, after the cutoff + period: 2023 + input: + age: 71 + is_male: true + output: + meets_savings_credit_age_requirement: false + +- name: Man aged 75 in 2025, born 1950, reached State Pension age at 65 in 2015 + period: 2025 + input: + age: 75 + is_male: true + output: + meets_savings_credit_age_requirement: true + +- name: Woman over State Pension age but under 65 does not yet meet the age requirement + period: 2015 + input: + age: 63 + is_male: false + output: + is_SP_age: true + meets_savings_credit_age_requirement: false diff --git a/policyengine_uk/tests/policy/baseline/gov/dwp/state_pension_age.yaml b/policyengine_uk/tests/policy/baseline/gov/dwp/state_pension_age.yaml new file mode 100644 index 0000000000..36699b8d9e --- /dev/null +++ b/policyengine_uk/tests/policy/baseline/gov/dwp/state_pension_age.yaml @@ -0,0 +1,279 @@ +# State Pension age by date of birth, at each boundary of Pensions Act 1995 +# Schedule 4 paragraph 1. Status is at 6 October of the fiscal year. Age and +# months_since_last_birthday place the date of birth named in each test. +# Expected values come from a day-level reading of the statute. + +- name: Woman born 5 April 1950 attains State Pension age at 60 (rule 2) + period: 2015 + absolute_error_margin: 0.001 + input: + age: 65 + months_since_last_birthday: 6.032258 + is_male: false + output: + state_pension_age: 60.0 # attained 5 April 2010 + is_SP_age: true + +- name: Woman born 6 April 1950 attains it on 6 May 2010 (rule 3, table 1) + period: 2015 + absolute_error_margin: 0.001 + input: + age: 65 + months_since_last_birthday: 6.0 + is_male: false + output: + state_pension_age: 60.0833 # attained 6 May 2010 + is_SP_age: true + +- name: Woman born 6 April 1951 attains it on 6 May 2012 (table 1) + period: 2015 + absolute_error_margin: 0.001 + input: + age: 64 + months_since_last_birthday: 6.0 + is_male: false + output: + state_pension_age: 61.0833 # attained 6 May 2012 + is_SP_age: true + +- name: Man born 6 April 1951 attains it at 65 (rule 1) + period: 2015 + absolute_error_margin: 0.001 + input: + age: 64 + months_since_last_birthday: 6.0 + is_male: true + output: + state_pension_age: 65.0 # attained 6 April 2016 + is_SP_age: false + +- name: Woman born 6 April 1953 attains it on 6 July 2016 (table 1, four-month steps) + period: 2016 + absolute_error_margin: 0.001 + input: + age: 63 + months_since_last_birthday: 6.0 + is_male: false + output: + state_pension_age: 63.25 # attained 6 July 2016 + is_SP_age: true + +- name: Man born 5 December 1953 attains it at 65 on 5 December 2018 (rule 1) + period: 2018 + absolute_error_margin: 0.001 + input: + age: 64 + months_since_last_birthday: 10.033333 + is_male: true + output: + state_pension_age: 65.0 # attained 5 December 2018 + is_SP_age: false + +- name: Man born 6 December 1953 attains it on 6 March 2019 (rule 5, table 2) + period: 2018 + absolute_error_margin: 0.001 + input: + age: 64 + months_since_last_birthday: 10.0 + is_male: true + output: + state_pension_age: 65.25 # attained 6 March 2019 + is_SP_age: false + +- name: Woman born 5 December 1953 attains it on 6 November 2018 (table 1) + period: 2018 + absolute_error_margin: 0.001 + input: + age: 64 + months_since_last_birthday: 10.033333 + is_male: false + output: + state_pension_age: 64.9194 # attained 6 November 2018 + is_SP_age: false + +- name: Woman born 6 December 1953 attains it on 6 March 2019 (table 2) + period: 2018 + absolute_error_margin: 0.001 + input: + age: 64 + months_since_last_birthday: 10.0 + is_male: false + output: + state_pension_age: 65.25 # attained 6 March 2019 + is_SP_age: false + +- name: Man born 5 December 1953 is over State Pension age in 2019-20 + period: 2019 + absolute_error_margin: 0.001 + input: + age: 65 + months_since_last_birthday: 10.033333 + is_male: true + output: + state_pension_age: 65.0 # attained 5 December 2018 + is_SP_age: true + +- name: Person born 5 October 1954 attains it on 6 September 2020 (table 2) + period: 2020 + absolute_error_margin: 0.001 + input: + age: 66 + months_since_last_birthday: 0.033333 + is_male: true + output: + state_pension_age: 65.9194 # attained 6 September 2020 + is_SP_age: true + +- name: Person born 6 October 1954 attains it at 66, on 6 October 2020 (rule 6) + period: 2020 + absolute_error_margin: 0.001 + input: + age: 66 + months_since_last_birthday: 0.0 + is_male: false + output: + state_pension_age: 66.0 # attained 6 October 2020 + is_SP_age: true + +- name: Person born 7 October 1954 attains it at 66, on 7 October 2020 (rule 6) + period: 2020 + absolute_error_margin: 0.001 + input: + age: 65 + months_since_last_birthday: 11.967742 + is_male: false + output: + state_pension_age: 66.0 # attained 7 October 2020 + is_SP_age: false + +- name: Person born 5 April 1960 attains it at 66 (rule 6) + period: 2026 + absolute_error_margin: 0.001 + input: + age: 66 + months_since_last_birthday: 6.032258 + is_male: false + output: + state_pension_age: 66.0 # attained 5 April 2026 + is_SP_age: true + +- name: Person born 6 April 1960 attains it at 66 years and 1 month (rule 7, table 3) + period: 2026 + absolute_error_margin: 0.001 + input: + age: 66 + months_since_last_birthday: 6.0 + is_male: true + output: + state_pension_age: 66.0833 # attained 6 May 2026 + is_SP_age: true + +- name: Person born 5 July 1960 attains it at 66 years and 3 months, on 5 October 2026 + period: 2026 + absolute_error_margin: 0.001 + input: + age: 66 + months_since_last_birthday: 3.033333 + is_male: false + output: + state_pension_age: 66.25 # attained 5 October 2026 + is_SP_age: true + +- name: Person born 6 July 1960 attains it at 66 years and 4 months, on 6 November 2026 + period: 2026 + absolute_error_margin: 0.001 + input: + age: 66 + months_since_last_birthday: 3.0 + is_male: true + output: + state_pension_age: 66.3333 # attained 6 November 2026 + is_SP_age: false + +- name: Person born 5 January 1961 attains it at 66 years and 9 months, on 5 October 2027 + period: 2027 + absolute_error_margin: 0.001 + input: + age: 66 + months_since_last_birthday: 9.032258 + is_male: false + output: + state_pension_age: 66.7499 # attained 5 October 2027 + is_SP_age: true + +- name: Person born 6 January 1961 attains it at 66 years and 10 months, on 6 November 2027 + period: 2027 + absolute_error_margin: 0.001 + input: + age: 66 + months_since_last_birthday: 9.0 + is_male: true + output: + state_pension_age: 66.8333 # attained 6 November 2027 + is_SP_age: false + +- name: Person born 5 March 1961 attains it at 66 years and 11 months (table 3) + period: 2028 + absolute_error_margin: 0.001 + input: + age: 67 + months_since_last_birthday: 7.035714 + is_male: false + output: + state_pension_age: 66.917 # attained 5 February 2028 + is_SP_age: true + +- name: Person born 6 March 1961 attains it at 67 (rule 8) + period: 2028 + absolute_error_margin: 0.001 + input: + age: 67 + months_since_last_birthday: 7.0 + is_male: true + output: + state_pension_age: 67.0 # attained 6 March 2028 + is_SP_age: true + +- name: Person born 5 April 1977 attains it at 67 (rule 8) + period: 2044 + absolute_error_margin: 0.001 + input: + age: 67 + months_since_last_birthday: 6.032258 + is_male: false + output: + state_pension_age: 67.0 # attained 5 April 2044 + is_SP_age: true + +- name: Person born 6 April 1977 attains it on 6 May 2044 (rule 9, table 4) + period: 2044 + absolute_error_margin: 0.001 + input: + age: 67 + months_since_last_birthday: 6.0 + is_male: true + output: + state_pension_age: 67.0833 # attained 6 May 2044 + is_SP_age: true + +- name: Person born 5 April 1978 attains it on 6 March 2046 (table 4) + period: 2046 + absolute_error_margin: 0.001 + input: + age: 68 + months_since_last_birthday: 6.032258 + is_male: false + output: + state_pension_age: 67.9194 # attained 6 March 2046 + is_SP_age: true + +- name: Person born 6 April 1978 attains it at 68 (rule 10) + period: 2046 + absolute_error_margin: 0.001 + input: + age: 68 + months_since_last_birthday: 6.0 + is_male: true + output: + state_pension_age: 68.0 # attained 6 April 2046 + is_SP_age: true diff --git a/policyengine_uk/tests/policy/baseline/gov/dwp/state_pension_type.yaml b/policyengine_uk/tests/policy/baseline/gov/dwp/state_pension_type.yaml index 56d493c79e..c0aea59718 100644 --- a/policyengine_uk/tests/policy/baseline/gov/dwp/state_pension_type.yaml +++ b/policyengine_uk/tests/policy/baseline/gov/dwp/state_pension_type.yaml @@ -37,3 +37,44 @@ is_male: false output: state_pension_type: BASIC + +# Pensions Act 2014 s.1(2): people who reach State Pension age before 6 April +# 2016 are not entitled to the new State Pension. Men born before 6 April 1951 +# reached it at 65 before then; women born 6 March to 5 April 1953 on 6 March +# 2016 and women born 6 April to 5 May 1953 on 6 July 2016 (table 1). + +- name: Man born 5 April 1951 reached State Pension age on 5 April 2016, so gets BASIC + period: 2021 + input: + age: 70 + months_since_last_birthday: 6.032258 + is_male: true + output: + state_pension_type: BASIC + +- name: Man born 6 April 1951 reached State Pension age on 6 April 2016, so gets NEW + period: 2021 + input: + age: 70 + months_since_last_birthday: 6 + is_male: true + output: + state_pension_type: NEW + +- name: Woman born 5 April 1953 reached State Pension age on 6 March 2016, so gets BASIC + period: 2021 + input: + age: 68 + months_since_last_birthday: 6.032258 + is_male: false + output: + state_pension_type: BASIC + +- name: Woman born 6 April 1953 reached State Pension age on 6 July 2016, so gets NEW + period: 2021 + input: + age: 68 + months_since_last_birthday: 6 + is_male: false + output: + state_pension_type: NEW diff --git a/policyengine_uk/tests/test_removed_state_pension_age_parameters.py b/policyengine_uk/tests/test_removed_state_pension_age_parameters.py new file mode 100644 index 0000000000..655dc5a0e3 --- /dev/null +++ b/policyengine_uk/tests/test_removed_state_pension_age_parameters.py @@ -0,0 +1,321 @@ +"""Obsolete State Pension age paths give migration help at every reform entry.""" + +import copy +import sys +from types import SimpleNamespace + +import pytest +from policyengine_core.parameters import ParameterNode, get_parameter +from policyengine_core.reforms import Reform, set_parameter + +from policyengine_uk import Simulation +from policyengine_uk.system import system +from policyengine_uk.utils.parameters import RemovedParameterNode +from policyengine_uk.utils.scenario import Scenario + + +AGE_PREFIX = "gov.dwp.state_pension.age" +REMOVED_PATHS = [f"{AGE_PREFIX}.male", f"{AGE_PREFIX}.female"] +REPLACEMENT_PATH = f"{AGE_PREFIX}.age_by_birth_date[14].amount" +SITUATION = { + "people": {"person": {"age": {"2028": 66}}}, + "benunits": {"benunit": {"members": ["person"]}}, + "households": {"household": {"members": ["person"]}}, +} + + +@pytest.fixture(scope="module", autouse=True) +def reuse_country_model(): + # Exercise the real constructor and reform paths with an independent + # parameter tree, without importing all model variables for every case. + with pytest.MonkeyPatch.context() as patch: + patch.setattr( + "policyengine_uk.simulation.CountryTaxBenefitSystem", system.clone + ) + yield + + +@pytest.fixture(scope="module") +def baseline_simulation(): + return Simulation(situation=SITUATION) + + +@pytest.fixture +def simulation(baseline_simulation): + # Each case has an independent parameter tree without rebuilding entities + # and loading the whole country model for every entry point. + return baseline_simulation.clone() + + +def structural_reform(path, access="get_child", use_modifier=False): + def modifier(parameters): + if access == "attribute": + target = getattr( + parameters.gov.dwp.state_pension.age, path.rsplit(".", 1)[1] + ) + elif access == "children": + target = parameters.gov.dwp.state_pension.age.children[ + path.rsplit(".", 1)[1] + ] + else: + target = parameters.get_child(path) + target.update(period="2028", value=780) + return parameters + + class ChangeAge(Reform): + def apply(self): + if use_modifier: + self.modify_parameters(modifier) + else: + modifier(self.parameters) + + return ChangeAge + + +ENTRY_POINTS = [ + "parameter_lookup", + "simulation_parameter_changes", + "scenario_dict_modifier", + "scenario_apply_scalar", + "scenario_apply_periods", + "scenario_apply_nested", + "scenario_structural_class", + "scenario_structural_tuple", + "simulation_apply_dict", + "simulation_apply_class", + "simulation_apply_tuple", + "system_apply_dict", + "system_apply_class", + "system_apply_tuple", + "system_modify_dict", + "system_modify_callback", + "core_reform_dict", + "structural_attribute", + "structural_children", + "structural_modifier", + "core_set_parameter", + "core_set_parameter_modifier", + "system_imported_reform", + "core_reform_api", + "api_parameter_normalization", + "api_legacy_policy_normalization", +] + + +@pytest.mark.parametrize("path", REMOVED_PATHS) +@pytest.mark.parametrize("entry_point", ENTRY_POINTS) +def test_removed_parameter_entry_points_name_the_replacement( + simulation, monkeypatch, path, entry_point +): + changes = {path: {"2028": 67}} + system = simulation.tax_benefit_system + reform = structural_reform(path) + + with pytest.raises(ValueError) as error: + if entry_point == "parameter_lookup": + system.parameters.get_child(path) + elif entry_point == "simulation_parameter_changes": + simulation.apply_parameter_changes(changes) + elif entry_point == "scenario_dict_modifier": + Scenario.from_reform(changes).simulation_modifier(simulation) + elif entry_point == "scenario_apply_scalar": + Scenario(parameter_changes={path: 67}).apply(simulation) + elif entry_point == "scenario_apply_periods": + Scenario(parameter_changes=changes).apply(simulation) + elif entry_point == "scenario_apply_nested": + Scenario( + parameter_changes={AGE_PREFIX: {path.rsplit(".", 1)[1]: 67}} + ).apply(simulation) + elif entry_point == "scenario_structural_class": + Scenario.from_reform(reform).apply(simulation) + elif entry_point == "scenario_structural_tuple": + Scenario.from_reform((reform,)).apply(simulation) + elif entry_point == "simulation_apply_dict": + simulation.apply_reform(changes) + elif entry_point == "simulation_apply_class": + simulation.apply_reform(reform) + elif entry_point == "simulation_apply_tuple": + simulation.apply_reform((reform,)) + elif entry_point == "system_apply_dict": + system.apply_reform_set(changes) + elif entry_point == "system_apply_class": + system.apply_reform_set(reform) + elif entry_point == "system_apply_tuple": + system.apply_reform_set((reform,)) + elif entry_point == "system_modify_dict": + system.modify_parameters(changes) + elif entry_point == "system_modify_callback": + system.modify_parameters( + set_parameter(path, 67, period="2028", return_modifier=True) + ) + elif entry_point == "core_reform_dict": + Reform.from_dict(changes)(system) + elif entry_point == "structural_attribute": + structural_reform(path, access="attribute")(system) + elif entry_point == "structural_children": + structural_reform(path, access="children")(system) + elif entry_point == "structural_modifier": + structural_reform(path, use_modifier=True)(system) + elif entry_point == "core_set_parameter": + set_parameter(path, 67, period="2028")(system) + elif entry_point == "core_set_parameter_modifier": + set_parameter(path, 67, period="2028", return_modifier=True)( + system.parameters + ) + elif entry_point == "system_imported_reform": + monkeypatch.setattr( + sys.modules[__name__], "ImportedReform", reform, raising=False + ) + system.apply_reform(f"{__name__}.ImportedReform") + elif entry_point == "core_reform_api": + response = SimpleNamespace( + json=lambda: { + "result": { + "policy_json": {path: {"2028-01-01.2028-12-31": 67}}, + "label": "saved obsolete age reform", + } + } + ) + monkeypatch.setattr( + "policyengine_core.reforms.reform.requests.get", + lambda *a, **k: response, + ) + Reform.from_api("obsolete-age", country_id="uk")(system) + elif entry_point == "api_parameter_normalization": + # policyengine-api/country.py::_normalize_reform_values reads the + # final value to infer a type before it calls Parameter.update. + get_parameter(system.parameters, path).values_list[-1].value + elif entry_point == "api_legacy_policy_normalization": + # The same repository's create_policy_reform walks children first, + # then reads values_list before applying its parameter update. + node = system.parameters + for part in path.split("."): + node = node.children[part] + node.values_list[-1].value + else: + raise AssertionError(f"Unknown entry point: {entry_point}") + + message = str(error.value) + assert path in message + assert "has been removed" in message + assert "age_by_birth_date" in message + assert "day_by_birth_date" in message + assert "state_pension_age" in message + + +@pytest.mark.parametrize("path", REMOVED_PATHS) +def test_removed_parameter_rejection_preserves_existing_policy(simulation, path): + parameters = simulation.tax_benefit_system.parameters + parameters.get_child(REPLACEMENT_PATH).update(period="2028", value=780) + with pytest.raises(ValueError, match="age_by_birth_date"): + simulation.apply_parameter_changes({path: {"2028": 67}}) + assert simulation.tax_benefit_system.parameters is parameters + assert parameters.get_child(REPLACEMENT_PATH)("2028") == 780 + + +@pytest.mark.parametrize("path", REMOVED_PATHS) +@pytest.mark.parametrize( + "entry_point", + [ + "reform_dict", + "scenario", + "scenario_before_data", + "structural_class", + "structural_tuple", + ], +) +def test_simulation_constructor_rejects_removed_parameters(path, entry_point): + changes = {path: {"2028": 67}} + if entry_point == "reform_dict": + kwargs = {"reform": changes} + elif entry_point in ("scenario", "scenario_before_data"): + kwargs = { + "scenario": Scenario( + parameter_changes=changes, + applied_before_data_load=entry_point == "scenario_before_data", + ) + } + else: + reform = structural_reform(path) + kwargs = {"reform": (reform,) if entry_point == "structural_tuple" else reform} + + with pytest.raises(ValueError, match="age_by_birth_date") as error: + Simulation(situation=SITUATION, **kwargs) + assert path in str(error.value) + + +@pytest.mark.parametrize( + "entry_point", + [ + "reform_dict", + "scenario_parameter_changes", + "scenario_apply", + "structural_class", + "structural_tuple", + "core_set_parameter", + ], +) +def test_replacement_parameter_can_still_be_reformed(simulation, entry_point): + changes = {REPLACEMENT_PATH: {"2028": 780}} + if entry_point == "reform_dict": + Scenario.from_reform(changes).apply(simulation) + elif entry_point == "scenario_parameter_changes": + simulation.apply_parameter_changes(changes) + elif entry_point == "scenario_apply": + Scenario(parameter_changes={REPLACEMENT_PATH: 780}).apply(simulation) + elif entry_point in ("structural_class", "structural_tuple"): + reform = structural_reform(REPLACEMENT_PATH) + Scenario.from_reform( + (reform,) if entry_point == "structural_tuple" else reform + ).apply(simulation) + else: + simulation.tax_benefit_system = set_parameter( + REPLACEMENT_PATH, 780, period="2028" + )(simulation.tax_benefit_system) + + assert ( + simulation.tax_benefit_system.parameters.get_child(REPLACEMENT_PATH)("2028") + == 780 + ) + assert simulation.calculate("state_pension_age", 2028)[0] == 65 + assert simulation.calculate("is_SP_age", 2028)[0] + + +def test_nested_scenario_can_change_a_valid_child(simulation): + Scenario(parameter_changes={f"{AGE_PREFIX}.male": {"age": 768}}).apply(simulation) + assert ( + simulation.tax_benefit_system.parameters.gov.dwp.state_pension.age.male.age( + "2028" + ) + == 768 + ) + + +@pytest.mark.parametrize("path", REMOVED_PATHS) +@pytest.mark.parametrize("clone_method", ["clone", "deepcopy"]) +def test_removed_parameter_aliases_survive_parameter_tree_copies( + baseline_simulation, path, clone_method +): + original = baseline_simulation.tax_benefit_system.parameters + original_male_age = original.gov.dwp.state_pension.age.male.age("2028") + parameters = ( + original.clone() if clone_method == "clone" else copy.deepcopy(original) + ) + alias = getattr(parameters.gov.dwp.state_pension.age, path.rsplit(".", 1)[1]) + assert isinstance(alias, RemovedParameterNode) + assert isinstance(alias, ParameterNode) + assert alias.metadata["removed"] is True + assert alias.metadata["economy"] is False + assert alias.metadata["household"] is False + assert "age_by_birth_date" in alias.description + with pytest.raises(ValueError, match="age_by_birth_date"): + parameters.get_child(path) + with pytest.raises(ValueError, match="age_by_birth_date"): + alias.update(period="2028", value=67) + + # The valid male children retain their parents and remain reformable. + male = parameters.gov.dwp.state_pension.age.male + assert male.age.parent is male + male.age.update(period="2028", value=768) + assert male.age("2028") == 768 + assert original.gov.dwp.state_pension.age.male.age("2028") == original_male_age diff --git a/policyengine_uk/tests/test_state_pension_age.py b/policyengine_uk/tests/test_state_pension_age.py new file mode 100644 index 0000000000..a8cb7d948c --- /dev/null +++ b/policyengine_uk/tests/test_state_pension_age.py @@ -0,0 +1,543 @@ +"""State Pension age by date of birth (Pensions Act 1995 Schedule 4 para 1). + +The parameters are checked row by row against the statute's text, and the +model is checked against a day-level reading of the same text that shares no +code with the model. +""" + +import calendar +import re +from datetime import date, timedelta +from functools import lru_cache +from pathlib import Path + +import numpy as np +import pandas as pd +import pytest +from hypothesis import HealthCheck, given, settings +from hypothesis import strategies as st + +from policyengine_uk import Simulation +from policyengine_uk.system import system +from policyengine_uk.utils.stochastic import splitmix64_uniform, stratified_uniform + + +@pytest.fixture(scope="module", autouse=True) +def reuse_country_model(): + # Keep the real simulation constructors and independent parameter trees, + # without importing every variable again for each synthetic household. + with pytest.MonkeyPatch.context() as patch: + patch.setattr( + "policyengine_uk.simulation.CountryTaxBenefitSystem", system.clone + ) + yield + + +STATUTE = ( + Path(__file__).parent / "fixtures" / "pensions_act_1995_schedule_4_paragraph_1.txt" +).read_text() +MONTHS = {name: number for number, name in enumerate(calendar.month_name) if name} +DATE = r"\d+(?:st|nd|rd|th) \w+ \d{4}" +NEW_STATE_PENSION_START = date(2016, 4, 6) # Pensions Act 2014 s.1(2) +SAVINGS_CREDIT_CUTOFF = date(2016, 4, 6) # State Pension Credit Act 2002 s.3(1)(a) + + +def parse_date(text: str) -> date: + day, month, year = re.fullmatch(r"(\d+)\w\w (\w+) (\d{4})", text.strip()).groups() + return date(int(year), MONTHS[month], int(day)) + + +def ymd(day: date) -> int: + return day.year * 10000 + day.month * 100 + day.day + + +def table(number: int) -> list: + body = STATUTE.split(f"TABLE {number}\n")[1].split("\n(")[0] + rows = re.findall(rf"\| ({DATE}) to ({DATE}) \| ([^|]+?) \|", body) + return [(parse_date(start), parse_date(end), target) for start, end, target in rows] + + +def rule(number: str, pattern: str) -> tuple: + line = re.search(rf"^\({number}\) (.+)$", STATUTE, flags=re.M).group(1) + return re.fullmatch(pattern, line).groups() + + +TABLE_1 = [(s, e, parse_date(t)) for s, e, t in table(1)] +TABLE_2 = [(s, e, parse_date(t)) for s, e, t in table(2)] +TABLE_3 = [ + (s, e, 12 * int(y) + int(m)) + for s, e, t in table(3) + for y, m in [re.fullmatch(r"(\d+) years and (\d+) months?", t).groups()] +] +TABLE_4 = [(s, e, parse_date(t)) for s, e, t in table(4)] +MEN_BEFORE, MEN_AGE = rule( + "1", + rf"A man born before ({DATE}) attains pensionable age when he attains the age of (\d+) years\.", +) +WOMEN_BEFORE, WOMEN_AGE = rule( + "2", + rf"A woman born before ({DATE}) attains pensionable age when she attains the age of (\d+)\.", +) +AGE_RULES = { + number: rule( + number, + rf"A person born after ({DATE})(?: but before ({DATE}))? attains pensionable age " + r"when the person attains the age of (\d+)\.", + ) + for number in ("6", "8", "10") +} +SPECIAL_DAYS = { + parse_date(born): parse_date(attained) + for born, attained in re.findall( + rf"a person born on ({DATE}) is to be taken to attain the age of 66 years and " + rf"\d+ months at the commencement of ({DATE})", + STATUTE, + ) +} + + +def after(text: str) -> date: + return parse_date(text) + timedelta(days=1) + + +def add_months(day: date, months: int) -> date: + year, month = divmod(day.month - 1 + months, 12) + year += day.year + return date(year, month + 1, min(day.day, calendar.monthrange(year, month + 1)[1])) + + +def reference_attainment_day(birth: date, male: bool) -> date: + """The day a person attains pensionable age, read from the statute text.""" + if male and birth < parse_date(MEN_BEFORE): + return add_months(birth, 12 * int(MEN_AGE)) + if not male and birth < parse_date(WOMEN_BEFORE): + return add_months(birth, 12 * int(WOMEN_AGE)) + for start, end, day in (TABLE_2 if male else TABLE_1 + TABLE_2) + TABLE_4: + if start <= birth <= end: + return day + for start, end, months in TABLE_3: + if start <= birth <= end: + return SPECIAL_DAYS.get(birth, add_months(birth, months)) + for number in ("6", "8", "10"): + born_after, born_before, age = AGE_RULES[number] + if birth > parse_date(born_after) and ( + born_before is None or birth < parse_date(born_before) + ): + return add_months(birth, 12 * int(age)) + raise ValueError(f"No rule covers {birth}") + + +def grid_months(day: date) -> float: + """Months on a grid whose months start on the 6th, written independently of + policyengine_uk.utils.dates.""" + whole = 12 * day.year + day.month - 1 - (day.day < 6) + start = date(whole // 12, whole % 12 + 1, 6) + end = date((whole + 1) // 12, (whole + 1) % 12 + 1, 6) + return whole + (day - start).days / (end - start).days + + +def grid_months_to_date(months: float) -> date: + whole = int(np.floor(months)) + start = date(whole // 12, whole % 12 + 1, 6) + end = date((whole + 1) // 12, (whole + 1) % 12 + 1, 6) + return start + timedelta(days=int(round((months - whole) * (end - start).days))) + + +def situation(people: dict) -> dict: + return { + "people": people, + "benunits": {"benunit": {"members": list(people)}}, + "households": {"household": {"members": list(people)}}, + } + + +@lru_cache(maxsize=1) +def situation_template(): + return Simulation(situation=situation({"template": {}})) + + +def property_simulation(people: dict): + # These properties never reform parameters or variable definitions. Share + # that model and rebuild the populations instead of repeatedly importing + # every variable and retaining thousands of generated modules in sys.modules. + sim = situation_template().clone(clone_tax_benefit_system=False) + sim.build_from_situation(situation(people)) + sim.input_variables = sim.get_known_variables() + sim.calculated_periods = [] + return sim + + +class TestParametersMatchStatute: + """Every parameter row must come from the statute's text.""" + + def setup_method(self): + self.p = system.parameters.gov.dwp.state_pension.age + + def brackets(self, scale) -> list: + at = scale("2026-01-01") + return list(zip(map(int, at.thresholds), map(int, at.amounts))) + + def test_statute_tables_are_complete_and_contiguous(self): + assert [len(TABLE_1), len(TABLE_2), len(TABLE_3), len(TABLE_4)] == [ + 44, + 10, + 11, + 12, + ] + periods = [(s, e) for s, e, _ in TABLE_1 + TABLE_2] + periods += [ + (after(AGE_RULES["6"][0]), parse_date(AGE_RULES["6"][1]) - timedelta(1)) + ] + periods += [(s, e) for s, e, _ in TABLE_3] + periods += [ + (after(AGE_RULES["8"][0]), parse_date(AGE_RULES["8"][1]) - timedelta(1)) + ] + periods += [(s, e) for s, e, _ in TABLE_4] + assert periods[0][0] == parse_date(WOMEN_BEFORE) + for (_, end), (start, _) in zip(periods, periods[1:]): + assert start == end + timedelta(days=1) + assert after(AGE_RULES["10"][0]) == periods[-1][1] + timedelta(days=1) + assert parse_date(MEN_BEFORE) == TABLE_2[0][0] + assert len(SPECIAL_DAYS) == 3 + + def test_age_by_birth_date_matches_statute(self): + expected = [(0, 12 * int(WOMEN_AGE)), (ymd(TABLE_1[0][0]), 0)] + expected += [(ymd(after(AGE_RULES["6"][0])), 12 * int(AGE_RULES["6"][2]))] + expected += [(ymd(start), months) for start, _, months in TABLE_3] + expected += [(ymd(after(AGE_RULES["8"][0])), 12 * int(AGE_RULES["8"][2]))] + expected += [(ymd(TABLE_4[0][0]), 0)] + expected += [(ymd(after(AGE_RULES["10"][0])), 12 * int(AGE_RULES["10"][2]))] + assert self.brackets(self.p.age_by_birth_date) == expected + + def test_day_by_birth_date_matches_statute(self): + expected = [(0, 0)] + expected += [(ymd(start), ymd(day)) for start, _, day in TABLE_1 + TABLE_2] + expected += [(ymd(after(AGE_RULES["6"][0])), 0)] + expected += [(ymd(start), ymd(day)) for start, _, day in TABLE_4] + expected += [(ymd(after(AGE_RULES["10"][0])), 0)] + assert self.brackets(self.p.day_by_birth_date) == expected + + def test_male_rule_matches_statute(self): + assert self.p.male.born_before("2026-01-01") == ymd(parse_date(MEN_BEFORE)) + assert self.p.male.age("2026-01-01") == 12 * int(MEN_AGE) + + +BIRTHS = [date(1945, 1, 1) + timedelta(days=d) for d in range(0, 40 * 366, 3)] +BOUNDARIES = sorted( + { + edge + timedelta(days=shift) + for start, end, _ in TABLE_1 + TABLE_2 + TABLE_3 + TABLE_4 + for edge in (start, end) + for shift in (-1, 0, 1) + } + | set(SPECIAL_DAYS) +) +YEARS = [2015, 2016, 2017, 2018, 2019, 2020, 2025, 2026, 2027, 2028, 2044, 2045, 2046] + + +@lru_cache(maxsize=None) +def differential_simulation() -> tuple: + people, keys = {}, [] + births = sorted(set(BIRTHS) | set(BOUNDARIES)) + for male in (True, False): + for birth in births: + name = f"{'m' if male else 'f'}{birth:%Y%m%d}" + ages, months = {}, {} + for year in YEARS: + age_in_months = grid_months(date(year, 10, 6)) - grid_months(birth) + ages[year] = int(age_in_months // 12) + months[year] = age_in_months - 12 * ages[year] + people[name] = { + "age": ages, + "months_since_last_birthday": months, + "is_male": {year: male for year in YEARS}, + } + keys.append((birth, male)) + return Simulation(situation=situation(people)), keys + + +@pytest.mark.parametrize("year", YEARS) +def test_status_matches_statute_for_every_birth_date(year): + """State Pension age status, type, the Savings Credit age test and the day + State Pension age is attained all agree with a day-level reading of the + statute, for births every third day from 1945 to 1984 and on both sides of + every statutory boundary, at both sexes.""" + sim, keys = differential_simulation() + status = sim.calculate("is_SP_age", year) + pension_type = sim.calculate("state_pension_type", year) + savings = sim.calculate("meets_savings_credit_age_requirement", year) + spa = sim.calculate("state_pension_age", year) + mid_year = date(year, 10, 6) + mismatches = [] + for i, (birth, male) in enumerate(keys): + attained = reference_attainment_day(birth, male) + expected_type = ( + "NONE" + if attained > mid_year + else "BASIC" + if attained < NEW_STATE_PENSION_START + else "NEW" + ) + age_in_months = grid_months(mid_year) - grid_months(birth) + expected_savings = attained < SAVINGS_CREDIT_CUTOFF and age_in_months >= 12 * 65 + model_day = grid_months_to_date(grid_months(birth) + 12 * spa[i]) + if ( + status[i] != (attained <= mid_year) + or pension_type[i] != expected_type + or savings[i] != expected_savings + or model_day != attained + ): + mismatches.append((birth, male, attained, status[i], spa[i])) + assert not mismatches, mismatches[:10] + + +def mid_year_share(year: int, age: int, male: bool) -> float: + """Share of people aged `age` on 6 October who have attained pensionable + age by then, with birthdays spread evenly over the year.""" + mid_year = date(year, 10, 6) + first = add_months(mid_year, -12 * (age + 1)) + timedelta(days=1) + days = [ + first + timedelta(days=d) + for d in range((add_months(mid_year, -12 * age) - first).days + 1) + ] + return np.mean([reference_attainment_day(b, male) <= mid_year for b in days]) + + +def microdata_simulation(weights: np.ndarray, years=range(2015, 2031)): + """A simulation built from data: one person per household, ages 60 to 70 + in turn, alternating sex.""" + from policyengine_uk import Microsimulation + from policyengine_uk.data import UKMultiYearDataset, UKSingleYearDataset + + n = len(weights) + ids = np.arange(n) + + def year(y): + person = pd.DataFrame( + { + "person_id": ids * 10 + 7, + "person_benunit_id": ids, + "person_household_id": ids, + "age": 60 + ids % 11, + "gender": np.where(ids % 2 == 1, "MALE", "FEMALE"), + } + ) + return UKSingleYearDataset( + person=person, + benunit=pd.DataFrame({"benunit_id": ids}), + household=pd.DataFrame({"household_id": ids, "household_weight": weights}), + fiscal_year=y, + ) + + dataset = UKMultiYearDataset(datasets=[year(y) for y in years]) + return Microsimulation(dataset=dataset), 60 + ids % 11, ids % 2 == 1 + + +def assert_shares_match_statute(sim, age, male, weights, years): + for year in years: + status = np.asarray(sim.calculate("is_SP_age", year)) + for a in range(60, 71): + for sex in (True, False): + cell = (age == a) & (male == sex) + share = np.average(status[cell], weights=weights[cell]) + # Within the largest weight in the cell, plus a day of births + # for where the month grid and the calendar differ. + tolerance = weights[cell].max() / weights[cell].sum() + 1 / 365 + expected = mid_year_share(year, a, sex) + assert abs(share - expected) <= tolerance, (year, a, sex, share) + + +def test_microdata_share_at_state_pension_age_matches_statute(): + """In data, each single year of age and sex is spread evenly over the year by + weight, so the weighted share over State Pension age is the statutory share: + three quarters of 66-year-olds in 2026-27 and a quarter in 2027-28, as births + from 6 April 1960 reach it at 66 and 1 to 11 months.""" + weights = np.random.default_rng(0).lognormal(7, 0.5, 11 * 2 * 200) + sim, age, male = microdata_simulation(weights) + years = (2015, 2016, 2017, 2018, 2019, 2020, 2026, 2027, 2028) + assert_shares_match_statute(sim, age, male, weights, years) + + +def test_small_region_of_data_still_spreads_birthdays(): + """A constituency or local authority filtered from the data carries well under + a million people of weight; it is still data, not a household situation.""" + weights = np.random.default_rng(1).lognormal(2, 0.5, 11 * 2 * 100) + assert weights.sum() < 1e6 + sim, age, male = microdata_simulation(weights, years=(2026, 2027)) + assert_shares_match_statute(sim, age, male, weights, (2026, 2027)) + + +def test_extracted_household_keeps_its_birthday(): + """filter_dataset carries each person's place in the year into the extract, + so a household keeps the status it had in the full simulation.""" + from policyengine_uk import Microsimulation + from policyengine_uk.data import UKMultiYearDataset, filter_dataset + + weights = np.random.default_rng(2).lognormal(7, 0.5, 11 * 2 * 20) + sim, age, _ = microdata_simulation(weights, years=(2026,)) + months = np.asarray(sim.calculate("months_since_last_birthday", 2026)) + status = np.asarray(sim.calculate("is_SP_age", 2026)) + for household in np.flatnonzero(age == 66)[:10]: + extract = filter_dataset(sim, household_id=int(household), year=2026) + extracted = Microsimulation(dataset=UKMultiYearDataset(datasets=[extract])) + assert np.allclose( + np.asarray(extracted.calculate("months_since_last_birthday", 2026)), + months[household], + ) + assert ( + np.asarray(extracted.calculate("is_SP_age", 2026))[0] == status[household] + ) + + +@pytest.mark.parametrize("year, expected", [(2026, True), (2027, False), (2028, False)]) +def test_single_household_uses_the_middle_of_the_year_of_age(year, expected): + sim = Simulation(situation=situation({"person": {"age": {year: 66}}})) + assert sim.calculate("months_since_last_birthday", year)[0] == 6 + assert sim.calculate("is_SP_age", year)[0] == expected + + +@settings( + max_examples=40, + deadline=None, + suppress_health_check=[HealthCheck.too_slow], +) +@given( + births=st.lists( + st.dates(min_value=date(1935, 1, 1), max_value=date(1995, 12, 31)), + min_size=2, + max_size=40, + unique=True, + ), + male=st.booleans(), + year=st.integers(min_value=2015, max_value=2050), +) +def test_state_pension_age_properties(births, male, year): + """For any dates of birth: State Pension age lies between 60 and 68; the + day it is attained never falls earlier for someone born later; and status + agrees with months_since_state_pension_age.""" + births = sorted(births) + people = {} + for i, birth in enumerate(births): + age_in_months = grid_months(date(year, 10, 6)) - grid_months(birth) + age = int(age_in_months // 12) + people[f"p{i}"] = { + "age": {year: age}, + "months_since_last_birthday": {year: age_in_months - 12 * age}, + "is_male": {year: male}, + } + sim = property_simulation(people) + spa = sim.calculate("state_pension_age", year) + since = sim.calculate("months_since_state_pension_age", year) + status = sim.calculate("is_SP_age", year) + assert np.all((spa >= 60 - 1e-6) & (spa <= 68 + 1e-6)) + attained = np.array([grid_months(b) for b in births]) + 12 * spa + assert np.all(np.diff(attained) >= -1e-3) + assert np.array_equal(status, since >= 0) + + +def birth_day_from_grid(months: float) -> date: + """The calendar day starting at or after an instant on the grid (within + about a minute and a half, as the model treats float error).""" + whole = int(np.floor(months)) + start = date(whole // 12, whole % 12 + 1, 6) + end = date((whole + 1) // 12, (whole + 1) % 12 + 1, 6) + offset = int(np.ceil((months - whole) * (end - start).days - 1e-3)) + return start + timedelta(days=offset) + + +def legal_age(birth: date, on: date) -> int: + return on.year - birth.year - ((on.month, on.day) < (birth.month, birth.day)) + + +@settings( + max_examples=60, + deadline=None, + suppress_health_check=[HealthCheck.too_slow], +) +@given( + cases=st.lists( + st.tuples( + st.integers(min_value=2015, max_value=2050), + st.integers(min_value=55, max_value=75), + st.floats(min_value=0, max_value=12, exclude_max=True), + st.booleans(), + ), + min_size=1, + max_size=30, + ) +) +def test_any_birthday_position_follows_the_statute(cases): + """For any whole age and any position in the year, including instants within + a day: the day of birth the model uses gives the person that legal age on + 6 October (a person attains an age at the start of the anniversary), and + their status is the statute's for that day.""" + people = { + f"p{i}": { + "age": {year: age}, + "months_since_last_birthday": {year: months}, + "is_male": {year: male}, + } + for i, (year, age, months, male) in enumerate(cases) + } + sim = property_simulation(people) + statuses = {year: sim.calculate("is_SP_age", year) for year, *_ in cases} + for i, (year, age, months, male) in enumerate(cases): + mid_year = date(year, 10, 6) + # The model caps months since the last birthday about four minutes + # short of 12, so the exact age never rounds onto the next birthday. + months = min(months, 12 - 1e-4) + birth = birth_day_from_grid(grid_months(mid_year) - 12 * age - months) + assert legal_age(birth, mid_year) == age, (year, age, months, birth) + expected = reference_attainment_day(birth, male) <= mid_year + assert statuses[year][i] == expected, (year, age, months) + + +@settings(max_examples=100, deadline=None) +@given( + n=st.integers(min_value=1, max_value=300), + strata_count=st.integers(min_value=1, max_value=5), + seed=st.integers(min_value=0, max_value=2**32 - 1), +) +def test_stratified_uniform_properties(n, strata_count, seed): + """Positions lie in [0, 1), ignore record order, and within each stratum the + weighted share below any point is that point to within the largest weight.""" + rng = np.random.default_rng(seed) + strata = rng.integers(0, strata_count, n) + ids = rng.permutation(10 * n)[:n] + weights = rng.lognormal(0, 1.5, n) + draws = splitmix64_uniform(ids, salt=2) + positions = stratified_uniform(strata, draws, weights) + assert np.all((positions >= 0) & (positions < 1)) + order = rng.permutation(n) + shuffled = stratified_uniform(strata[order], draws[order], weights[order]) + assert np.allclose(shuffled, positions[order]) + for s in np.unique(strata): + cell = strata == s + w = weights[cell] / weights[cell].sum() + for x in np.linspace(0, 1, 21): + below = w[positions[cell] < x].sum() + assert abs(below - x) <= w.max() / 2 + 1e-9 + + +@pytest.mark.parametrize("fractional_birthday_months", [None, 0]) +def test_people_a_situation_leaves_out_keep_their_place_in_the_year( + fractional_birthday_months, +): + """When a situation sets months_since_last_birthday for one person only, + the others get the middle of the year of age, and a fractional age is + still the exact age.""" + people = { + "set": {"age": {2027: 70}, "months_since_last_birthday": {2027: 3}}, + "whole": {"age": {2027: 66}}, + "fraction": {"age": {2027: 66.9}}, + } + if fractional_birthday_months is not None: + people["fraction"]["months_since_last_birthday"] = { + 2027: fractional_birthday_months + } + sim = Simulation(situation=situation(people)) + months = sim.calculate("months_since_last_birthday", 2027) + assert list(months[:2]) == [3, 6] + # 66.9 on 6 October 2027: born about 12 November 1960, so State Pension + # age is 66 years and 8 months (table 3), attained in July 2027. + assert list(sim.calculate("is_SP_age", 2027)) == [True, False, True] diff --git a/policyengine_uk/utils/dates.py b/policyengine_uk/utils/dates.py new file mode 100644 index 0000000000..2320bd9f59 --- /dev/null +++ b/policyengine_uk/utils/dates.py @@ -0,0 +1,123 @@ +"""Vectorised dates on a monthly grid whose months start on the 6th. + +UK tax years start on 6 April, and the State Pension age timetable (Pensions +Act 1995 Schedule 4 paragraph 1) groups dates of birth into periods running +from the 6th of one month to the 5th of the next. Measuring dates in months on +that grid makes every statutory boundary a whole number: month ``m`` runs from +the 6th of calendar month ``m % 12 + 1`` of year ``m // 12`` up to the 5th of +the following month, and a fraction of a month is spread evenly over its days. +Adding whole months keeps the day of the month, as attaining an age of "N +years and M months" does. + +Dates are passed in and out as YYYYMMDD integers, which is how parameters +store them. +""" + +import numpy as np + +_MONTHS_BEFORE_1970 = 12 * 1970 + + +def grid_month(year: int, month: int) -> int: + """The grid month that starts on the 6th of ``month`` in ``year``.""" + return 12 * year + month - 1 + + +def _sixth_of(grid_months: np.ndarray) -> np.ndarray: + months_since_1970 = (grid_months - _MONTHS_BEFORE_1970).astype("datetime64[M]") + return months_since_1970.astype("datetime64[D]") + np.timedelta64(5, "D") + + +def yyyymmdd_to_grid_months(dates) -> np.ndarray: + """Convert YYYYMMDD integers to (fractional) grid months.""" + dates = np.asarray(dates, dtype=np.int64) + year, month, day = dates // 10000, dates // 100 % 100, dates % 100 + first_of_month = (12 * year + month - 1 - _MONTHS_BEFORE_1970).astype( + "datetime64[M]" + ) + calendar_date = first_of_month.astype("datetime64[D]") + (day - 1).astype( + "timedelta64[D]" + ) + whole = 12 * year + month - 1 - (day < 6) + start = _sixth_of(whole) + length = (_sixth_of(whole + 1) - start).astype(np.int64) + elapsed = (calendar_date - start).astype(np.int64) + return whole + elapsed / length + + +def grid_months_to_yyyymmdd(months) -> np.ndarray: + """Convert (fractional) grid months to YYYYMMDD integers: the calendar day + that starts at or after each instant. + + A person attains an age at the commencement of the anniversary of their + date of birth (Family Law Reform Act 1969 s.9(1)). Taking a birth instant + that falls within a day as the next day keeps the whole number of years + between it and a later midnight equal to the person's legal age there. + """ + months = np.asarray(months, dtype=np.float64) + whole = np.floor(months).astype(np.int64) + start = _sixth_of(whole) + length = (_sixth_of(whole + 1) - start).astype(np.int64) + # Model variables are stored as float32, which can put an instant that is + # exactly midnight a fraction of a second either side of it. A tolerance + # of about a minute and a half keeps it on its day. + elapsed = np.ceil((months - whole) * length - 1e-3).astype(np.int64) + overflow = elapsed >= length + whole = np.where(overflow, whole + 1, whole) + elapsed = np.where(overflow, 0, elapsed) + start = _sixth_of(whole) + calendar_date = start + elapsed.astype("timedelta64[D]") + calendar_month = calendar_date.astype("datetime64[M]") + year = calendar_month.astype(np.int64) // 12 + 1970 + month = calendar_month.astype(np.int64) % 12 + 1 + day = (calendar_date - calendar_month.astype("datetime64[D]")).astype(np.int64) + 1 + return year * 10000 + month * 100 + day + + +def add_months_to_yyyymmdd(dates, months) -> np.ndarray: + """Add whole months to YYYYMMDD dates, keeping the day of the month. + + Where that day does not exist in the target month, the result is the + month's last day, as when a person born on 31 July attains an age of some + years and four months on 30 November. That also puts a 29 February + anniversary on 28 February in other years; the statute does not say, and + State Pension age status, read on the 6th, is the same either way. + """ + dates = np.asarray(dates, dtype=np.int64) + months = np.round(np.asarray(months, dtype=np.float64)).astype(np.int64) + year, month, day = dates // 10000, dates // 100 % 100, dates % 100 + target = 12 * year + month - 1 + months + first_of_month = (target - _MONTHS_BEFORE_1970).astype("datetime64[M]") + days_in_month = ( + (first_of_month + np.timedelta64(1, "M")).astype("datetime64[D]") + - first_of_month.astype("datetime64[D]") + ).astype(np.int64) + return ( + (target // 12) * 10000 + + (target % 12 + 1) * 100 + + np.minimum(day, days_in_month) + ) + + +# Months since the last birthday stop about four minutes short of 12, beyond +# the tolerance above, so an exact age never rounds onto the next birthday. +_LATEST_MONTHS_SINCE_BIRTHDAY = 12 - 1e-4 + + +def exact_age_in_months(age, months_since_last_birthday) -> np.ndarray: + """Exact age in months, in float64: at around 24,000 months, float32 is + only good to an hour. + + A fractional age is the exact age. A whole age adds the months since the + last birthday, which stop about four minutes short of 12 so the exact age + never rounds onto the next birthday. + """ + age = np.asarray(age, dtype=np.float64) + whole_years = np.floor(age) + fraction = age - whole_years + months = np.where( + fraction > 0, + 12 * fraction, + np.asarray(months_since_last_birthday, dtype=np.float64), + ) + return 12 * whole_years + np.clip(months, 0, _LATEST_MONTHS_SINCE_BIRTHDAY) diff --git a/policyengine_uk/utils/parameters.py b/policyengine_uk/utils/parameters.py index 4df923f0b5..fdbe9b663a 100644 --- a/policyengine_uk/utils/parameters.py +++ b/policyengine_uk/utils/parameters.py @@ -12,7 +12,7 @@ def backdate_parameters(root: str = None, first_instant: str = "2021-01-01") -> first_instant = str_to_instant(first_instant) node = root for param in node.get_descendants(): - if hasattr(param, "values_list"): + if isinstance(param, Parameter): earliest = param.values_list[-1] earliest_value = earliest.value earliest_instant = str_to_instant(earliest.instant_str) @@ -118,3 +118,87 @@ def convert_to_fiscal_year_parameters(parameters): value=value, ) return parameters + + +_STATE_PENSION_AGE_REPLACEMENT = ( + "State Pension age now follows the statutory timetable by date of birth " + "(Pensions Act 1995 Sch 4 para 1). Change the rows of " + "gov.dwp.state_pension.age.age_by_birth_date (the age, in months) or " + "gov.dwp.state_pension.age.day_by_birth_date (the day) for the births you " + "want to change, for example gov.dwp.state_pension.age.age_by_birth_date[14]" + ".amount for people born from 6 March 1961 until the next bracket, or " + "gov.dwp.state_pension.age.male.age for men born before 6 December 1953. " + "Read a person's State Pension age from the state_pension_age or is_SP_age " + "variables." +) + +# Parameters removed from the tree, with what replaces them, so a reform or +# saved policy that still names one fails with directions instead of a bare +# lookup error. +REMOVED_PARAMETERS = { + "gov.dwp.state_pension.age.male": _STATE_PENSION_AGE_REPLACEMENT, + "gov.dwp.state_pension.age.female": _STATE_PENSION_AGE_REPLACEMENT, +} + + +def check_parameter_not_removed(path: str) -> None: + """Raise a ValueError naming the replacement if path was removed.""" + if path in REMOVED_PARAMETERS: + raise ValueError( + f"The parameter {path} has been removed. {REMOVED_PARAMETERS[path]}" + ) + + +class RemovedParameterNode(ParameterNode): + """Keep a removed path discoverable without accepting scalar reforms. + + Male State Pension age still has valid children, so the compatibility + node must support normal tree traversal and cloning. API reform code + reads values_list before updating; both operations give migration help. + """ + + @property + def values_list(self): + check_parameter_not_removed(self.name) + + def update(self, *args, **kwargs): + check_parameter_not_removed(self.name) + + +class RemovedParameterRoot(ParameterNode): + """Reject explicit lookups of removed scalars, including empty policies.""" + + def get_child(self, path: str): + check_parameter_not_removed(path) + return super().get_child(path) + + +def add_removed_parameter_aliases(parameters: ParameterNode) -> ParameterNode: + if not isinstance(parameters, RemovedParameterRoot): + root = RemovedParameterRoot(parameters.name, data={}) + root.__dict__ = parameters.__dict__.copy() + for child in root.children.values(): + child.parent = root + parameters = root + age = parameters.gov.dwp.state_pension.age + for path, replacement in REMOVED_PARAMETERS.items(): + name = path.rsplit(".", 1)[1] + previous = age.children.get(name) + if isinstance(previous, RemovedParameterNode): + continue + alias = RemovedParameterNode(path, data={}) + if previous is not None: + alias.metadata.update(previous.metadata) + for child_name, child in previous.children.items(): + alias.add_child(child_name, child) + alias.description = f"The parameter {path} has been removed. {replacement}" + alias.metadata.update( + label=f"removed {name} State Pension age parameter", + removed=True, + economy=False, + household=False, + ) + age.children[name] = alias + setattr(age, name, alias) + alias.parent = age + return parameters diff --git a/policyengine_uk/utils/scenario.py b/policyengine_uk/utils/scenario.py index ec1e02c021..8484578fb8 100644 --- a/policyengine_uk/utils/scenario.py +++ b/policyengine_uk/utils/scenario.py @@ -1,9 +1,12 @@ -from pydantic import BaseModel +from pydantic import BaseModel, ConfigDict from typing import Optional, Callable, Dict, Type, Union from policyengine_core.simulations import Simulation from policyengine_core.reforms import Reform from policyengine_core.periods import period, instant -from policyengine_uk.utils.parameters import uk_fiscal_year_period +from policyengine_uk.utils.parameters import ( + check_parameter_not_removed, + uk_fiscal_year_period, +) def _apply_reform_class(reform: Type[Reform], simulation: Simulation) -> None: @@ -54,10 +57,7 @@ class Scenario(BaseModel): simulation_modifier: Optional[Callable[["Simulation"], None]] = None """A function that modifies the simulation before running it.""" - class Config: - """Pydantic configuration.""" - - arbitrary_types_allowed = True # Allow Callable types + model_config = ConfigDict(arbitrary_types_allowed=True) def __add__(self, other: "Scenario") -> "Scenario": """Combine two scenarios by merging parameter changes and chaining modifiers. @@ -159,6 +159,7 @@ def modifier(simulation: Simulation) -> None: def modifier(sim: Simulation): for parameter in reform: + check_parameter_not_removed(parameter) target = sim.tax_benefit_system.parameters.get_child(parameter) if isinstance(reform[parameter], dict): for period_str, value in reform[parameter].items(): @@ -238,17 +239,29 @@ def apply(self, simulation: Simulation) -> None: for path, value in self.parameter_changes.items(): if isinstance(value, dict): # Handle nested parameter changes + if not value: + check_parameter_not_removed(path) for sub_path, sub_value in value.items(): full_path = f"{path}.{sub_path}" - simulation.tax_benefit_system.parameters.update( - full_path, + check_parameter_not_removed(full_path) + try: + target = simulation.tax_benefit_system.parameters.get_child( + full_path + ) + except ValueError: + # A saved period-valued policy on a removed scalar + # path still needs the migration message; valid + # children such as male.age remain reformable. + check_parameter_not_removed(path) + raise + target.update( period=None, # Apply to all periods value=sub_value, ) else: # Simple parameter change - simulation.tax_benefit_system.parameters.update( - path, + check_parameter_not_removed(path) + simulation.tax_benefit_system.parameters.get_child(path).update( period=None, value=value, # Apply to all periods ) diff --git a/policyengine_uk/utils/stochastic.py b/policyengine_uk/utils/stochastic.py index 4fcdc61744..788b13ca3f 100644 --- a/policyengine_uk/utils/stochastic.py +++ b/policyengine_uk/utils/stochastic.py @@ -25,3 +25,35 @@ def splitmix64_uniform(ids: np.ndarray, salt: int = 0) -> np.ndarray: # which would otherwise round near-1 draws up to exactly 1.0. draws = (z >> np.uint64(11)).astype(np.float64) / 2.0**53 return np.minimum(draws, 1.0 - 2.0**-24) + + +def stratified_uniform( + strata: np.ndarray, draws: np.ndarray, weights: np.ndarray +) -> np.ndarray: + """Spread records evenly over [0, 1) within each stratum, by weight. + + Records in a stratum are ordered by their draw and each is placed at the + midpoint of its slice of the stratum's cumulative weight. The weighted + distribution within every stratum is then uniform to within the largest + single weight, rather than only in expectation as with independent draws. + The result does not depend on the order of the records. A stratum with no + positive weight keeps the draws. + """ + strata = np.asarray(strata) + draws = np.asarray(draws, dtype=np.float64) + weights = np.maximum(np.asarray(weights, dtype=np.float64), 0) + order = np.lexsort((draws, strata)) + _, group = np.unique(strata[order], return_inverse=True) + sorted_weights = weights[order] + totals = np.bincount(group, weights=sorted_weights) + group_starts = np.concatenate([[0.0], np.cumsum(totals)[:-1]]) + within = np.cumsum(sorted_weights) - group_starts[group] - sorted_weights / 2 + group_totals = totals[group] + positions = np.where( + group_totals > 0, + within / np.where(group_totals > 0, group_totals, 1), + draws[order], + ) + result = np.empty_like(positions) + result[order] = positions + return np.clip(result, 0, 1.0 - 2.0**-24) diff --git a/policyengine_uk/variables/gov/dwp/additional_state_pension.py b/policyengine_uk/variables/gov/dwp/additional_state_pension.py index 2e77b4f770..0467ec7d3e 100644 --- a/policyengine_uk/variables/gov/dwp/additional_state_pension.py +++ b/policyengine_uk/variables/gov/dwp/additional_state_pension.py @@ -51,4 +51,7 @@ def formula(person, period, parameters): max_for_type_period / max_for_type_data, 1, ) - return amount_in_data_year * uprating * WEEKS_IN_YEAR + # No State Pension is paid before State Pension age; the data-year + # type above only splits the reported amount into its components. + is_sp_age = person("is_SP_age", period) + return is_sp_age * amount_in_data_year * uprating * WEEKS_IN_YEAR diff --git a/policyengine_uk/variables/gov/dwp/is_SP_age.py b/policyengine_uk/variables/gov/dwp/is_SP_age.py index e7226290d4..f7ab3b3863 100644 --- a/policyengine_uk/variables/gov/dwp/is_SP_age.py +++ b/policyengine_uk/variables/gov/dwp/is_SP_age.py @@ -5,9 +5,12 @@ class is_SP_age(Variable): value_type = bool entity = Person label = "Whether the person is State Pension Age" + documentation = ( + "Whether the person has attained State Pension age by the middle of " + "the fiscal year (6 October), so is over it for most of the year." + ) definition_period = YEAR + reference = "https://www.legislation.gov.uk/ukpga/1995/26/schedule/4" def formula(person, period, parameters): - age = person("age", period) - threshold = person("state_pension_age", period) - return age >= threshold + return person("months_since_state_pension_age", period) >= 0 diff --git a/policyengine_uk/variables/gov/dwp/months_since_state_pension_age.py b/policyengine_uk/variables/gov/dwp/months_since_state_pension_age.py new file mode 100644 index 0000000000..3a1110e892 --- /dev/null +++ b/policyengine_uk/variables/gov/dwp/months_since_state_pension_age.py @@ -0,0 +1,26 @@ +from policyengine_uk.model_api import * +from policyengine_uk.utils.dates import exact_age_in_months + + +class months_since_state_pension_age(Variable): + value_type = float + entity = Person + label = "months since reaching State Pension age" + documentation = ( + "Months between the day this person attains State Pension age and the " + "middle of the fiscal year (6 October); negative if they attain it " + "later. A whole age adds months_since_last_birthday; a fractional age " + "is the exact age and overrides those months." + ) + definition_period = YEAR + unit = "month" + + def formula(person, period, parameters): + age_in_months = exact_age_in_months( + person("age", period), person("months_since_last_birthday", period) + ) + spa = person("state_pension_age", period).astype(np.float64) + months_since = age_in_months - 12 * spa + # Rounding to a thousandth of a month (under an hour) removes float + # error, so reaching State Pension age exactly on 6 October counts. + return np.round(months_since, 3) diff --git a/policyengine_uk/variables/gov/dwp/pension_credit/savings_credit/meets_savings_credit_age_requirement.py b/policyengine_uk/variables/gov/dwp/pension_credit/savings_credit/meets_savings_credit_age_requirement.py index 407296549e..92eb215656 100644 --- a/policyengine_uk/variables/gov/dwp/pension_credit/savings_credit/meets_savings_credit_age_requirement.py +++ b/policyengine_uk/variables/gov/dwp/pension_credit/savings_credit/meets_savings_credit_age_requirement.py @@ -1,22 +1,28 @@ from policyengine_uk.model_api import * +from policyengine_uk.utils.dates import exact_age_in_months class meets_savings_credit_age_requirement(Variable): value_type = bool entity = Person label = "whether the person reached State Pension Age before the Savings Credit cutoff year" + documentation = ( + "Whether the person attained State Pension age before 6 April of the " + "Savings Credit cutoff year and has attained the minimum age." + ) definition_period = YEAR + reference = "https://www.legislation.gov.uk/ukpga/2002/16/section/3" def formula(person, period, parameters): - # https://www.legislation.gov.uk/ukpga/2002/16/section/3 + # State Pension Credit Act 2002 s.3(1)(a): the claimant "has attained + # pensionable age before 6 April 2016 and has attained the age of 65". p = parameters(period).gov.dwp.pension_credit.savings_credit - current_age = person("age", period) - # State pension age - spa = person("state_pension_age", period) - # The year we're evaluating - evaluation_year = period.start.year - # Calculate what year the person reached/will reach SPA - birth_year = evaluation_year - current_age - spa_year = birth_year + spa - # Get the cutoff year from parameters - return spa_year < p.cutoff_year + months_from_cutoff_to_mid_year = 12 * (period.start.year - p.cutoff_year) + 6 + reached_before_cutoff = ( + person("months_since_state_pension_age", period) + > months_from_cutoff_to_mid_year + ) + age_in_months = exact_age_in_months( + person("age", period), person("months_since_last_birthday", period) + ) + return reached_before_cutoff & (age_in_months >= 12 * p.minimum_age) diff --git a/policyengine_uk/variables/gov/dwp/state_pension_age.py b/policyengine_uk/variables/gov/dwp/state_pension_age.py index 641e07e80c..f93a15799e 100644 --- a/policyengine_uk/variables/gov/dwp/state_pension_age.py +++ b/policyengine_uk/variables/gov/dwp/state_pension_age.py @@ -1,13 +1,60 @@ from policyengine_uk.model_api import * +from policyengine_uk.utils.dates import ( + add_months_to_yyyymmdd, + exact_age_in_months, + grid_month, + grid_months_to_yyyymmdd, + yyyymmdd_to_grid_months, +) class state_pension_age(Variable): value_type = float entity = Person label = "State Pension age for this person" + documentation = ( + "The age at which this person attains State Pension age (pensionable " + "age), set by their date of birth. Where the statute sets the day on " + "which it is attained, this is the person's age on that day. The date " + "of birth comes from age and months_since_last_birthday." + ) definition_period = YEAR unit = "year" + reference = ( + "https://www.legislation.gov.uk/ukpga/1995/26/schedule/4", + "https://www.legislation.gov.uk/ukpga/2014/19/section/26", + ) def formula(person, period, parameters): - SP = parameters(period).gov.dwp.state_pension - return where(person("is_male", period), SP.age.male, SP.age.female) + p = parameters(period).gov.dwp.state_pension.age + # Dates are in months on a grid whose months start on the 6th, so the + # middle of the fiscal year (6 October) is a whole month. + mid_year = grid_month(period.start.year, 10) + age_in_months = exact_age_in_months( + person("age", period), person("months_since_last_birthday", period) + ) + birth = mid_year - age_in_months + # The day of birth: the one starting at or after that instant, so the + # person's legal age on 6 October is age. + birth_date = grid_months_to_yyyymmdd(birth) + + # Pensions Act 1995 Sch 4 para 1 rule (1): men born before 6 December + # 1953 attain pensionable age at 65. Everyone else follows the + # timetable in rules (2) to (10), which sets either an age or a day. + male_rule = person("is_male", period) & (birth_date < p.male.born_before) + age_rule = where(male_rule, p.male.age, p.age_by_birth_date.calc(birth_date)) + day_rule = where(male_rule, 0, p.day_by_birth_date.calc(birth_date)) + day_rule_month = where( + day_rule > 0, + yyyymmdd_to_grid_months(np.maximum(day_rule, 10101)), + -np.inf, + ) + # An age of N years and M months is attained at the commencement of + # the anniversary (Family Law Reform Act 1969 s.9(1)): the same day of + # the month, or the month's last day where that day does not exist, + # which also gives the three days in rule (7A). + anniversary = yyyymmdd_to_grid_months( + add_months_to_yyyymmdd(birth_date, age_rule) + ) + attained = np.maximum(anniversary, day_rule_month) + return (attained - birth) / 12 diff --git a/policyengine_uk/variables/gov/dwp/state_pension_type.py b/policyengine_uk/variables/gov/dwp/state_pension_type.py index 8447e8e5c4..67d37f577a 100644 --- a/policyengine_uk/variables/gov/dwp/state_pension_type.py +++ b/policyengine_uk/variables/gov/dwp/state_pension_type.py @@ -17,7 +17,6 @@ class state_pension_type(Variable): def formula(person, period, parameters): sp = parameters.gov.dwp.state_pension - male = person("is_male", period) is_sp_age = person("is_SP_age", period) # Find the instant the New State Pension was first switched on. @@ -37,14 +36,17 @@ def formula(person, period, parameters): is_sp_age, StatePensionType.BASIC, StatePensionType.NONE ) else: - instant = activation_entry.instant_str - years_since_instant = period.start.year - int(instant[:4]) - male_age = sp.age.male(instant) + years_since_instant - female_age = sp.age.female(instant) + years_since_instant - age = person("age", period) - over_age = where(male, age >= male_age, age >= female_age) + # Pensions Act 2014 s.1(2): a person who reaches pensionable age + # before 6 April 2016 is not entitled to the new State Pension, + # and stays on the basic State Pension instead. + first_year = int(activation_entry.instant_str[:4]) + months_from_start_to_mid_year = 12 * (period.start.year - first_year) + 6 + reached_before_start = ( + person("months_since_state_pension_age", period) + > months_from_start_to_mid_year + ) values_if_sp_age = where( - over_age, StatePensionType.BASIC, StatePensionType.NEW + reached_before_start, StatePensionType.BASIC, StatePensionType.NEW ) return where(is_sp_age, values_if_sp_age, StatePensionType.NONE) diff --git a/policyengine_uk/variables/household/demographic/months_since_last_birthday.py b/policyengine_uk/variables/household/demographic/months_since_last_birthday.py new file mode 100644 index 0000000000..d8a037891b --- /dev/null +++ b/policyengine_uk/variables/household/demographic/months_since_last_birthday.py @@ -0,0 +1,41 @@ +from policyengine_uk.model_api import * +from policyengine_uk.utils.stochastic import splitmix64_uniform, stratified_uniform + + +class months_since_last_birthday(Variable): + value_type = float + entity = Person + label = "months since last birthday" + documentation = ( + "Months since the person's most recent birthday, at the middle of the " + "fiscal year (6 October), from 0 up to 12. With a whole age, this places the " + "date of birth: the person was born age years and this many months " + "before 6 October. A fractional age is read as the exact age on 6 " + "October, whatever this is set to. For a whole age, simulations of a " + "household situation default to 6, " + "the middle of the year of age. Microdata records age only in whole " + "years, so in simulations built from data each single year of age and " + "sex is spread evenly over the year: records are ordered by a " + "deterministic hash of the person id and placed by their share of the " + "group's weight. Datasets and situations can set this directly, and " + "filter_dataset carries it into a household extracted from the data." + ) + definition_period = YEAR + unit = "month" + # A situation that sets this for some people gives the rest the default + # rather than the formula, so the default is the middle of the year too. + default_value = 6 + + def formula(person, period, parameters): + age = person("age", period) + whole_years = np.floor(age) + fraction = age - whole_years + if getattr(person.simulation, "built_from_dataset", False): + position = stratified_uniform( + strata=whole_years * 2 + person("is_male", period), + draws=splitmix64_uniform(person("person_id", period), salt=2), + weights=person("person_weight", period), + ) + else: + position = np.full(person.count, 0.5) + return 12 * where(fraction > 0, fraction, position)