Repository navigation
Fix #1142: Let several conditioned rules compute the same code item in hierarchy - #1146
Draft
javihern98 wants to merge 1 commit into
Draft
javihern98 wants to merge 1 commit into
javihern98 wants to merge 1 commit into
Conversation
The VTL 2.2 Reference Manual allows a hierarchical ruleset to define a code item in several '=' rules with different conditions and aggregates their outcomes. hierarchy rejected those rulesets with 1-1-10-10. - 1-1-10-10 now only applies when one of the rules for the code item has no condition, since that rule always overlaps the others. Two rules that compute the same data point raise the new runtime error 2-1-10-1 on both engines. - The rules of a hierarchy run in an order where every rule computing a code item comes before the rules that read it. - pandas: rule outputs are kept per rule name, the left side of '=' no longer reads what another rule computed, an operand reads everything computed for its code item matched on the identifiers, viral attributes are combined per rule, and hierarchy no longer drops the non-'=' rules of the ruleset for a later check_hierarchy. - DuckDB: a conditioned rule only keeps the rows where its condition holds, viral attributes are combined per rule, and a QUALIFY guard raises 2-1-10-1. - Hierarchical rulesets reject duplicated rule names (1-3-1-5) and raise 1-3-1-7 instead of a ValueError for mixed named and unnamed rules.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
hierarchyrejected hierarchical rulesets that define a code item in several=rules with differentwhenconditions (1-1-10-10). Hierarchical rulesets in the VTL 2.2 Reference Manual allows them and aggregates their outcomes (itsCalcCountryLevelandEuropeanUnionAreaCountries1examples).ValueErrorfor mixed named and unnamed rules. Unnamed rules keep their1,2, … names.HRDAGAnalyzer.sort_hierarchy_rules). The ruleset's own order, and so the rule names, do not change.=no longer reads what another rule computed for the same code item.when C_1 = "Y" then TOTAL = A + B; G = TOTAL + ApairedTOTALonYwithAonX.hierarchyno longer overwrites the ruleset's rules with its=rules, which made a latercheck_hierarchywith the same ruleset skip the other rules.non_null.QUALIFYguard raises 2-1-10-1.Tests:
GH_1142_1–GH_1142_11on both engines.GL_397_31andGL_397_33, the manual'sEuropeanUnionAreaCountries1example, expected 1-1-10-10 and now check hand-derived results.Checklist
ruff format,ruff check,mypy)pytest): full suite withVTL_ENGINE_BACKEND=pandasandduckdbmessages.pyImpact / Risk
ValueError.hierarchyno longer returns null data points for rules whose condition is FALSE.hierarchyaccepts several conditioned rules for the same code item, as VTL 2.2 allows, with the new runtime error 2-1-10-1 when their conditions overlap. The unused message 1-3-1-8 is removed.Notes
ruleidfor existing rulesets, so it is filed separately as Unnamed hierarchical rules are numbered after the dependency sort, not in the order they are written #1145.hierarchyside of Engines differ on check_hierarchy all for data points where a rule's condition is FALSE #1141. Thecheck_hierarchyside stays there.Dataset.__eq__accepted the old DuckDB result ofGL_397_31, which had 16 extra null rows: its exact-equality rescue usesdrop_duplicates(keep=False), which drops any row that appears twice. The new tests and the full suite also pass with a stricter comparison that requires equal row counts.Closes #1142