Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/was-wealth-components.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Export `private_pension_wealth`, `directly_held_shares`, `unit_and_investment_trusts`, `stocks_and_shares_isa` and `cash_isa` from the Wealth and Assets Survey, and impute debt secured on non-main property and land (`other_residential_property_secured_debt`, `non_residential_property_secured_debt`, `owned_land_secured_debt`).
1 change: 1 addition & 0 deletions changelog.d/was-wealth-split.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Split private pension wealth out of `corporate_wealth`, which now holds only shares, unit and investment trusts and stocks and shares ISAs, and add buy-to-let property to `other_residential_property_value`.
41 changes: 27 additions & 14 deletions docs/imputations.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ The imputations are applied in this order (dependencies noted):

## Wealth Imputation

**Source:** Wealth and Assets Survey (WAS) Round 7 (2018-2020)
**Source:** Wealth and Assets Survey (WAS) Round 8 (April 2020 to March 2022), household file

Imputes household wealth components using demographic and income predictors.
Imputes household wealth components using demographic and income predictors. The targets are imputed as a chain: each is fitted on the predictors and every earlier target.

### Predictors
| Variable | Description |
Expand All @@ -39,18 +39,31 @@ Imputes household wealth components using demographic and income predictors.
| `region` | UK region |

### Outputs
| Variable | Description |
|----------|-------------|
| `owned_land` | Value of owned land |
| `property_wealth` | Total property wealth |
| `corporate_wealth` | Shares, pensions, investment ISAs |
| `gross_financial_wealth` | Total financial assets |
| `net_financial_wealth` | Financial assets minus liabilities |
| `main_residence_value` | Value of main home |
| `other_residential_property_value` | Value of other properties |
| `non_residential_property_value` | Value of non-residential property |
| `savings` | Savings account balances |
| `num_vehicles` | Number of vehicles owned |
| Variable | WAS round 8 source | Description |
|----------|--------------------|-------------|
| `owned_land` | `DVLUKValR8_sum` | Value of land in the UK |
| `property_wealth` | `DVPropertyR8` | Gross value of all property, main residence included |
| `private_pension_wealth` | `totalpenr8_aggr` − `dvvaldbt_scaper8_aggr` | Private pension wealth other than current-employment defined benefit rights: defined contribution pots, retained rights, pensions in payment |
| `directly_held_shares` | `DVFShUKVR8_aggr` + `DVFESHARESR8_aggr` | UK shares and employee shares and options, held outside ISAs and pooled funds |
| `unit_and_investment_trusts` | `DVFCollVR8_aggr` | Unit trusts and investment trusts (one survey question) |
| `stocks_and_shares_isa` | `DVIISAVR8_aggr` | Stocks and shares (investment) ISAs |
| `corporate_wealth` | derived | `directly_held_shares` + `unit_and_investment_trusts` + `stocks_and_shares_isa`, summed after imputation |
| `gross_financial_wealth` | `HFINWR8_SUM` | Total financial assets |
| `net_financial_wealth` | `HFINWNTR8_Sum` | Financial assets minus liabilities |
| `main_residence_value` | `DVHValueR8` | Value of main home |
| `other_residential_property_value` | `DVHseValR8_sum` + `DVBltValR8_sum` | Second homes and buy-to-let property |
| `non_residential_property_value` | `DVBlDValR8_sum` | Other buildings, such as shops and garages |
| `savings` | `DVSaValR8_aggr` | Savings account balances (cash ISAs excluded) |
| `num_vehicles` | `vcarnr8` | Number of vehicles owned |
| `student_loan_balance` | `Tot_LosR8_aggr` − `Tot_los_exc_SLCR8_aggr` | Student Loans Company balance, allocated to people |
| `cash_isa` | `DVCISAVR8_aggr` | Cash ISAs |
| `other_residential_property_secured_debt` | `DVHseDebtR8_sum` + `DVBLtDebtR8_sum` | Mortgages and loans secured on second homes and buy-to-let property |
| `non_residential_property_secured_debt` | `DVBldDebtR8_sum` | Mortgages and loans secured on other buildings |
| `owned_land_secured_debt` | `DVLUKDebtR8_sum` | Loans secured on UK land |

`corporate_wealth` is the exact sum of its three components on every household; private pension wealth is not part of it, because pension rights are disregarded capital in every means test. A secured debt is zero wherever the household holds none of the asset it is secured on. Land and property overseas (`DVLOSValR8_sum`), other real estate (`DVOPrValR8_sum`) and overseas shares (`DVFShOSVR8_aggr`) are not exported separately; `property_wealth` and the financial wealth totals include them.

Each target draws its quantile with its own seed. microimpute otherwise uses one seed for every target, so a household drawn high for one asset is drawn high for every later target conditioned on it, which makes sparse targets such as the debt secured on land near-certain for the households that hold the asset.

---

Expand Down
125 changes: 109 additions & 16 deletions policyengine_uk_data/datasets/imputations/wealth.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,17 @@
"region",
]

# Chain order matters: each target is fitted on the predictors plus every
# earlier target. Private pension wealth takes the slot the old pension-laden
# corporate_wealth held, the share-like components follow it, and each
# secured debt comes after the asset it is secured on.
IMPUTE_VARIABLES = [
"owned_land",
"property_wealth",
"corporate_wealth",
"private_pension_wealth",
"directly_held_shares",
"unit_and_investment_trusts",
"stocks_and_shares_isa",
"gross_financial_wealth",
"net_financial_wealth",
"main_residence_value",
Expand All @@ -57,17 +64,67 @@
"savings",
"num_vehicles",
"student_loan_balance",
"cash_isa",
"other_residential_property_secured_debt",
"non_residential_property_secured_debt",
"owned_land_secured_debt",
]

# corporate_wealth is not imputed: it is the sum of its imputed components,
# so the identity holds on every household.
CORPORATE_WEALTH_COMPONENTS = (
"directly_held_shares",
"unit_and_investment_trusts",
"stocks_and_shares_isa",
)

# Debt secured on each capital asset (UC Regs 2013 reg. 49(1)(b) and its
# legacy equivalents deduct it from that asset's value only).
SECURED_DEBT_ASSETS = {
"other_residential_property_secured_debt": "other_residential_property_value",
"non_residential_property_secured_debt": "non_residential_property_value",
"owned_land_secured_debt": "owned_land",
}

# microimpute draws each target's quantile from the same seed, so a household
# gets the same quantile for every target. Chained on earlier draws, that makes
# sparse targets near-certain for households drawn high for a related asset
# (nearly every imputed land holder had debt secured on the land). Each target
# gets its own seed so the draws are independent.
QUANTILE_DRAW_SEEDS = {
variable: 1_000 + index for index, variable in enumerate(IMPUTE_VARIABLES)
}

# WAS round 8 sources of the targets built from more than one column. Part of
# the model metadata, so a change of definition retrains a cached model.
DERIVED_TARGETS = {
"private_pension_wealth": ("totalpenr8_aggr", "-dvvaldbt_scaper8_aggr"),
"directly_held_shares": ("DVFShUKVR8_aggr", "DVFESHARESR8_aggr"),
"other_residential_property_value": ("DVHseValR8_sum", "DVBltValR8_sum"),
"other_residential_property_secured_debt": (
"DVHseDebtR8_sum",
"DVBLtDebtR8_sum",
),
"student_loan_balance": ("Tot_LosR8_aggr", "-Tot_los_exc_SLCR8_aggr"),
}

WAS_RENAMES = {
"R8xshhwgt": "household_weight",
# Components for estimating land holdings.
"DVLUKValR8_sum": "owned_land", # In the UK.
"DVLUKDebtR8_sum": "owned_land_secured_debt",
"DVPropertyR8": "property_wealth",
# UK shares (listed or not) and employee shares and options, held outside
# ISAs and pooled funds.
"DVFESHARESR8_aggr": "emp_shares_options",
"DVFShUKVR8_aggr": "uk_shares",
"DVIISAVR8_aggr": "investment_isas",
"DVFCollVR8_aggr": "unit_investment_trusts",
# Investment ISAs: the survey's stocks and shares ISA question.
"DVIISAVR8_aggr": "stocks_and_shares_isa",
"DVCISAVR8_aggr": "cash_isa",
# Unit trusts and investment trusts: one survey question.
"DVFCollVR8_aggr": "unit_and_investment_trusts",
# Total private pension wealth and its current-employment defined benefit
# part, both valued on the SCAPE basis.
"totalpenr8_aggr": "pensions",
"dvvaldbt_scaper8_aggr": "db_pensions",
# Predictors for fusing to FRS.
Expand All @@ -88,12 +145,17 @@
# Other columns for reference.
"DVLOSValR8_sum": "non_uk_land",
"HFINWNTR8_Sum": "net_financial_wealth",
"DVLUKDebtR8_sum": "uk_land_debt",
"HFINWR8_SUM": "gross_financial_wealth",
"TotalWlthR8": "wealth",
"DVhvalueR8": "main_residence_value",
"DVHseValR8_sum": "other_residential_property_value",
# Gross values of property other than the main residence, and the
# mortgages and loans secured on each class.
"DVHseValR8_sum": "second_homes_value",
"DVHseDebtR8_sum": "second_homes_debt",
"DVBltValR8_sum": "buy_to_let_value",
"DVBLtDebtR8_sum": "buy_to_let_debt",
"DVBlDValR8_sum": "non_residential_property_value",
"DVBldDebtR8_sum": "non_residential_property_secured_debt",
"DVTotinc_bhcR8": "household_net_income",
"DVSaValR8_aggr": "savings",
"vcarnr8": "num_vehicles",
Expand Down Expand Up @@ -142,16 +204,18 @@ def generate_was_table(was: pd.DataFrame):

was["is_renting"] = was["is_renter"] == 1

was["non_db_pensions"] = was.pensions - was.db_pensions
was["corporate_wealth"] = was[
[
"non_db_pensions",
"emp_shares_options",
"uk_shares",
"investment_isas",
"unit_investment_trusts",
]
].sum(axis=1)
# Private pension wealth other than current-employment defined benefit
# rights: pension rights are disregarded capital in every means test, so it
# is kept out of corporate_wealth.
was["private_pension_wealth"] = was.pensions - was.db_pensions
was["directly_held_shares"] = was.uk_shares + was.emp_shares_options
was["corporate_wealth"] = was[list(CORPORATE_WEALTH_COMPONENTS)].sum(axis=1)
was["other_residential_property_value"] = (
was.second_homes_value + was.buy_to_let_value
)
was["other_residential_property_secured_debt"] = (
was.second_homes_debt + was.buy_to_let_debt
)
was["student_loan_balance"] = was["total_loans"] - was["total_loans_exc_slc"]
was["region"] = was["region"].map(REGIONS)
return was
Expand All @@ -162,6 +226,8 @@ def generate_was_table(was: pd.DataFrame):
"was_household_tab_filename": CURRENT_WAS_RELEASE.household_tab_filename,
"predictor_variables": tuple(PREDICTOR_VARIABLES),
"impute_variables": tuple(IMPUTE_VARIABLES),
"derived_targets": tuple(DERIVED_TARGETS.items()),
"quantile_draw_seeds": tuple(QUANTILE_DRAW_SEEDS.items()),
}


Expand Down Expand Up @@ -269,6 +335,32 @@ def _allocate_student_loan_balance_to_people(
return balances


def derive_wealth_outputs(output_df: pd.DataFrame) -> pd.DataFrame:
"""
Add the outputs derived from imputed targets.

The share-like components are clipped at zero and summed into
corporate_wealth, so the identity holds on every household. A secured debt
is kept only where the household holds the asset it is secured on.
"""
output_df = output_df.copy()
for column in (*CORPORATE_WEALTH_COMPONENTS, *SECURED_DEBT_ASSETS):
output_df[column] = output_df[column].clip(lower=0)
output_df["corporate_wealth"] = output_df[list(CORPORATE_WEALTH_COMPONENTS)].sum(
axis=1
)
for debt, asset in SECURED_DEBT_ASSETS.items():
output_df[debt] = output_df[debt].where(output_df[asset] > 0, 0.0)
return output_df


def use_independent_quantile_draws(model: QRF) -> QRF:
"""Give each imputed target its own quantile-draw seed."""
for variable, seed in QUANTILE_DRAW_SEEDS.items():
model.model.models[variable].seed = seed
return model


def save_imputation_models():
"""
Train and save wealth imputation model.
Expand All @@ -290,6 +382,7 @@ def save_imputation_models():
was[PREDICTOR_VARIABLES],
was[IMPUTE_VARIABLES],
)
use_independent_quantile_draws(wealth)
wealth.save(get_wealth_model_path())
return wealth

Expand Down Expand Up @@ -338,7 +431,7 @@ def impute_wealth(dataset: UKSingleYearDataset) -> UKSingleYearDataset:
input_df["region"] = input_df["region"].replace(
"NORTHERN_IRELAND", "WALES"
) # WAS doesn't sample NI -> put NI households in Wales (closest aggregate)
output_df = model.predict(input_df)
output_df = derive_wealth_outputs(model.predict(input_df))

for column in output_df.columns:
if column == "student_loan_balance":
Expand Down
8 changes: 8 additions & 0 deletions policyengine_uk_data/storage/uprating_factors.csv
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,23 @@ bsp_reported,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1
capital_gains,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
capital_gains_before_response,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
carers_allowance_reported,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
cash_isa,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
child_benefit_reported,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
child_tax_credit_reported,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
childcare_expenses,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
clothing_and_footwear_consumption,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
communication_consumption,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
corporate_wealth,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
diesel_spending,1.0,1.138,1.57,1.539,1.589,1.54,1.507,1.46,1.415,1.355,1.284,1.284,1.284,1.284,1.284
directly_held_shares,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
dividend_income,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
domestic_energy_consumption,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
education_consumption,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
employee_pension_contributions,1.0,1.059,1.127,1.205,1.261,1.308,1.337,1.365,1.396,1.431,1.431,1.431,1.431,1.431,1.431
employer_pension_contributions,1.0,1.059,1.127,1.205,1.261,1.308,1.337,1.365,1.396,1.431,1.431,1.431,1.431,1.431,1.431
non_residential_property_secured_debt,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
other_residential_property_secured_debt,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
owned_land_secured_debt,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
pension_contributions_via_salary_sacrifice,1.0,1.059,1.127,1.205,1.261,1.308,1.337,1.365,1.396,1.431,1.431,1.431,1.431,1.431,1.431
employment_income,1.0,1.059,1.127,1.205,1.261,1.308,1.337,1.365,1.396,1.431,1.431,1.431,1.431,1.431,1.431
employment_income_before_lsr,1.0,1.059,1.127,1.205,1.261,1.308,1.337,1.365,1.396,1.431,1.431,1.431,1.431,1.431,1.431
Expand Down Expand Up @@ -59,6 +64,7 @@ pension_income,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384
personal_pension_contributions,1.0,1.059,1.127,1.205,1.261,1.308,1.337,1.365,1.396,1.431,1.431,1.431,1.431,1.431,1.431
petrol_spending,1.0,1.104,1.635,1.796,1.531,1.483,1.452,1.406,1.363,1.305,1.237,1.237,1.237,1.237,1.237
private_pension_income,1.0,1.003,1.053,1.106,1.161,1.216,1.261,1.288,1.315,1.346,1.346,1.346,1.346,1.346,1.346
private_pension_wealth,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
private_transfer_income,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
property_income,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
recreation_consumption,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
Expand All @@ -72,9 +78,11 @@ state_pension_reported,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1
statutory_maternity_pay,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
statutory_paternity_pay,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
statutory_sick_pay,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
stocks_and_shares_isa,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
student_loan_repayments,1.0,1.059,1.127,1.205,1.261,1.308,1.337,1.365,1.396,1.431,1.431,1.431,1.431,1.431,1.431
sublet_income,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
transport_consumption,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
unit_and_investment_trusts,1.0,1.0,1.092,1.147,1.19,1.223,1.258,1.297,1.34,1.384,1.384,1.384,1.384,1.384,1.384
universal_credit_reported,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
water_and_sewerage_charges,1.0,1.0,1.0,1.092,1.14,1.21,1.283,1.349,1.4,1.46,1.46,1.46,1.46,1.46,1.46
winter_fuel_allowance_reported,1.0,1.04,1.144,1.209,1.237,1.277,1.301,1.327,1.353,1.38,1.38,1.38,1.38,1.38,1.38
Expand Down
Loading
Loading