Skip to content

Fix #1142: Let several conditioned rules compute the same code item in hierarchy - #1146

Draft
javihern98 wants to merge 1 commit into
1.9.Xfrom
cr-1142
Draft

javihern98 wants to merge 1 commit into
1.9.Xfrom
cr-1142

Conversation

@javihern98

Copy link
Copy Markdown
Contributor

Summary

hierarchy rejected hierarchical rulesets that define a code item in several = rules with different when conditions (1-1-10-10). Hierarchical rulesets in the VTL 2.2 Reference Manual allows them and aggregates their outcomes (its CalcCountryLevel and EuropeanUnionAreaCountries1 examples).

  • Check: 1-1-10-10 now only applies when one of the rules for the code item has no condition, since that rule always overlaps the others. When the conditions of two rules hold on the same data point, both engines raise the new runtime error 2-1-10-1. The manual leaves overlaps undefined and only warns that they give "duplicated data".
  • Rule names: each rule's output is kept under its name. Hierarchical rulesets now reject duplicated rule names (1-3-1-5, as datapoint rulesets do) and raise 1-3-1-7 instead of a ValueError for mixed named and unnamed rules. Unnamed rules keep their 1, 2, … names.
  • Order: the rules of a hierarchy run so that every rule computing a code item comes before the rules that read it (HRDAGAnalyzer.sort_hierarchy_rules). The ruleset's own order, and so the rule names, do not change.
  • pandas:
    • The left side of = no longer reads what another rule computed for the same code item.
    • An operand reads everything computed for its code item, matched to the rule's groups on the identifiers. Outputs of conditioned rules used to be matched by row label, so when C_1 = "Y" then TOTAL = A + B; G = TOTAL + A paired TOTAL on Y with A on X.
    • Viral attributes are combined per rule.
    • hierarchy no longer overwrites the ruleset's rules with its = rules, which made a later check_hierarchy with the same ruleset skip the other rules.
  • DuckDB:
    • A conditioned rule only keeps the rows where its condition holds. It used to return a null data point wherever the condition was FALSE, in every mode but non_null.
    • Viral attributes are combined per rule.
    • A QUALIFY guard raises 2-1-10-1.

Tests: GH_1142_1–GH_1142_11 on both engines. GL_397_31 and GL_397_33, the manual's EuropeanUnionAreaCountries1 example, expected 1-1-10-10 and now check hand-derived results.

Checklist

  • Code quality checks pass (ruff format, ruff check, mypy)
  • Tests pass (pytest): full suite with VTL_ENGINE_BACKEND=pandas and duckdb
  • Documentation updated (if applicable): the error messages page is generated from messages.py

Impact / Risk

  • Breaking changes?
    • Rulesets that define a code item in several conditioned rules now run instead of raising 1-1-10-10.
    • Duplicated rule names in a hierarchical ruleset now raise 1-3-1-5. Mixed named and unnamed rules raise 1-3-1-7 instead of a ValueError.
    • On DuckDB, hierarchy no longer returns null data points for rules whose condition is FALSE.
  • Data/SDMX compatibility concerns? None.
  • Notes for release/changelog? hierarchy accepts several conditioned rules for the same code item, as VTL 2.2 allows, with the new runtime error 2-1-10-1 when their conditions overlap. The unused message 1-3-1-8 is removed.

Notes

Closes #1142

The VTL 2.2 Reference Manual allows a hierarchical ruleset to define a
code item in several '=' rules with different conditions and aggregates
their outcomes. hierarchy rejected those rulesets with 1-1-10-10.

- 1-1-10-10 now only applies when one of the rules for the code item has
  no condition, since that rule always overlaps the others. Two rules
  that compute the same data point raise the new runtime error 2-1-10-1
  on both engines.
- The rules of a hierarchy run in an order where every rule computing a
  code item comes before the rules that read it.
- pandas: rule outputs are kept per rule name, the left side of '=' no
  longer reads what another rule computed, an operand reads everything
  computed for its code item matched on the identifiers, viral attributes
  are combined per rule, and hierarchy no longer drops the non-'=' rules
  of the ruleset for a later check_hierarchy.
- DuckDB: a conditioned rule only keeps the rows where its condition
  holds, viral attributes are combined per rule, and a QUALIFY guard
  raises 2-1-10-1.
- Hierarchical rulesets reject duplicated rule names (1-3-1-5) and raise
  1-3-1-7 instead of a ValueError for mixed named and unnamed rules.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hierarchy raises 1-1-10-10 when a code item is defined by several rules with different conditions

1 participant