Add Case and Coalesce multi-source combinator primitives - #113
Merged
Conversation
Two combinators for "the same quantity arrives in one of several columns, each possibly needing its own conversion": - Case: switch on a selector source; the first branch whose `when` contains the selector value wins. Models data with an authoritative unit/type flag (e.g. a "units: kg/lbs" column). - Coalesce: pick the first branch whose source is non-null. Models "whichever field was filled in". Both address sources BY NAME (the primitive stores its own ordered source-name list and zips it with the positional values transform() receives) rather than by fragile positional index. A branch is composable: either a single source + op-chain, or several `terms` (each source + op-chain) combined with a reduction -- so a branch can e.g. convert feet->inches, take inches as-is, and sum them. Branch op-chains reuse existing primitives (ConvertUnits, Scale, Round, ...) and round-trip through the factory like MapEach's nested ops. 14 new tests (single-source weight kg/lbs flag, multi-term height feet+inches and meters+cm summing, serialization round-trips, null/default edges). Full suite: 196 passing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The multi-source branch sub-computations were called 'terms', which only read naturally when combine='sum'. 'operands' is precise (an operand is what a combining op acts on) and combine-agnostic. Pure rename of the serialization key, helper functions, and docstrings; no behavior change. Also drops the unused _term_sources helper. 196 tests passing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Coverage gap: combine was only exercised via Case (height sum). Add a Coalesce case where one branch is stone+pounds summed (Scale(14) on stone + leftover pounds, combine="sum") and the other is a single pounds field — no flag, so the populated branch wins. Demonstrates combine doing real work in Coalesce and adds a serialization round-trip for a multi-operand Coalesce branch. 198 tests passing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A passthrough operand had an empty operations list ([]); make it [DoNothing()] to match the single-source examples and read clearly as "passes through unchanged" at every no-op site. Functionally identical (empty chain == DoNothing). 16 tests passing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #112.
What
Adds two multi-source combinator primitives for the pattern where the same quantity arrives in one of several source columns, each possibly needing its own conversion:
Case— switch on a selector/flag column; the first branch whosewhencontains the selector value wins.Coalesce— pick the first non-null source; branch order is precedence.Both address sources by name (each stores its own ordered source-name list and zips it with the positional values
transform()receives — no change toharmonize_dataset, no impact on existing primitives). Branches are composable: a branch is a single source + op-chain, or severaloperands(each source + op-chain) combined with a reduction — so a branch can convert feet→inches, take inches as-is, and sum them. Branch op-chains reuse existing primitives and round-trip throughfactory.deserialize_operationlikeMapEach's nested ops.combineis aReduction(the same enumReduceuses), applied asReduce(Reduction(combine)).transform(operand_values). Valid values:"sum","any","none","all","one-hot", or omitted/nullfor a single-operand branch. It only folds the operands together; per-operand transforms live in each operand's ownoperationschain.API & serialization
Serialized form (a single-source branch normalizes to a one-element
operandslist with"combine": null; nestedoperationsserialize recursively likemap_each). The two blocks are the serialized forms of the two Python examples above.Weight
Case:{ "operation": "case", "sources": ["weight_units", "weight_lbs", "weight_kgs"], "selector": "weight_units", "branches": [ { "when": ["2"], "combine": null, "operands": [ {"source": "weight_lbs", "operations": [{"operation": "do_nothing"}]} ] }, { "when": ["1"], "combine": null, "operands": [ { "source": "weight_kgs", "operations": [ {"operation": "convert_units", "source_unit": "kg", "target_unit": "lb"}, {"operation": "round", "precision": 0} ] } ] } ], "default": null }Coalesce (multi-operand, stone+pounds):
{ "operation": "coalesce", "sources": ["weight_lbs", "weight_stone", "weight_stone_lbs"], "branches": [ {"combine": null, "operands": [{"source": "weight_lbs", "operations": [{"operation": "do_nothing"}]}]}, { "combine": "sum", "operands": [ {"source": "weight_stone", "operations": [{"operation": "scale", "scaling_factor": 14}]}, {"source": "weight_stone_lbs", "operations": [{"operation": "do_nothing"}]} ] } ], "default": null }Why
Harmonizing the RADx data dictionaries, RADx-UP collects height/weight under a unit-selector flag (
self_reported_weight_units_2: 1=kg/2=lbs;self_reported_height_coded: 1=feet+inches/2=meters+cm). The existing primitives couldn't express "switch on the flag, convert accordingly" — the list reducers reject nulls (and exactly one unit field is null per row), andMapEachapplies the same op to every element (but kg needs conversion while lbs doesn't). See #112 for the full design discussion.Changes
src/harmonization_framework/primitives/case.py,coalesce.py— new primitivesprimitives/__init__.py,vocabulary.py(CASE,COALESCE),factory.pytests/test_case_coalesce.py— 17 testsTests
tests/test_case_coalesce.py— 17 tests covering: single-source flag (kg/lbs), int/float selector matching, multi-operand height sums (feet+inches, meters+cm), multi-operand Coalesce withcombine(stone+pounds), serialization round-trips, and null-selector / no-match / all-null defaults.Verified on synthetic RADx-UP rows: pounds → as-is; 68 kg → 150 lbs; 5 ft 7 in → 67 in; 1 m 70 cm → 66.9 in.
🤖 Generated with Claude Code