Skip to content

Add Case and Coalesce multi-source combinator primitives - #113

Merged
matthewhorridge merged 4 commits into
mainfrom
feat/case-coalesce-primitives
Jun 30, 2026
Merged

Add Case and Coalesce multi-source combinator primitives#113
matthewhorridge merged 4 commits into
mainfrom
feat/case-coalesce-primitives

Conversation

@matthewhorridge

@matthewhorridge matthewhorridge commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Closes #112.

What

Adds two multi-source combinator primitives for the pattern where the same quantity arrives in one of several source columns, each possibly needing its own conversion:

  • Case — switch on a selector/flag column; the first branch whose when contains the selector value wins.
  • Coalesce — pick the first non-null source; branch order is precedence.

Both address sources by name (each stores its own ordered source-name list and zips it with the positional values transform() receives — no change to harmonize_dataset, no impact on existing primitives). Branches are composable: a branch is a single source + op-chain, or several operands (each source + op-chain) combined with a reduction — so a branch can convert feet→inches, take inches as-is, and sum them. Branch op-chains reuse existing primitives and round-trip through factory.deserialize_operation like MapEach's nested ops.

combine is a Reduction (the same enum Reduce uses), applied as Reduce(Reduction(combine)).transform(operand_values). Valid values: "sum", "any", "none", "all", "one-hot", or omitted/null for a single-operand branch. It only folds the operands together; per-operand transforms live in each operand's own operations chain.

Reading when: each Case branch's when is the list of selector values that activate it — the branch fires when the selector column's value is in that list. It's a list (not a bare value) so one branch can cover several codes (e.g. ["1", "3"]); the RADx examples each match one code, so every when is a single-element list. Codes come from the dictionary: RADx-UP self_reported_weight_units_2 is 1 = Kilograms, 2 = Pounds, so when: ["1"] is the kilograms branch (the one that converts).

Selector typing. when values are matched as strings, and Case normalizes an integer-valued float so a flag read as 2.0 still matches when: ["2"]. This matters because a CSV/pandas flag column with blank cells is read as float (22.0), so without this the branch would silently miss. (Same class of issue as integer-keyed enum_to_enum.)

API & serialization

# Case — switch on a selector source (RADx-UP weight: kg/lbs flag)
Case(
    sources=["weight_units", "weight_lbs", "weight_kgs"],
    selector="weight_units",
    branches=[
        # weight_units == "2" (Pounds): take the pounds field as-is
        {"when": ["2"], "source": "weight_lbs", "operations": [DoNothing()]},
        # weight_units == "1" (Kilograms): convert the kg field to pounds
        {"when": ["1"], "source": "weight_kgs",
         "operations": [ConvertUnits(Unit.KILOGRAM, Unit.POUNDS), Round(0)]},
    ],
    default=None,
)

# Coalesce with a multi-operand branch + combine: weight as a single lbs field,
# OR as stone + leftover pounds summed. No flag, so the populated branch wins.
Coalesce(
    sources=["weight_lbs", "weight_stone", "weight_stone_lbs"],
    branches=[
        {"source": "weight_lbs", "operations": [DoNothing()]},
        {"combine": "sum", "operands": [
            {"source": "weight_stone", "operations": [Scale(14)]},  # stone -> lbs
            {"source": "weight_stone_lbs", "operations": [DoNothing()]},  # leftover pounds, as-is
        ]},
    ],
    default=None,
)
# 10 st 7 lb -> 10*14 + 7 = 147 lb

Serialized form (a single-source branch normalizes to a one-element operands list with "combine": null; nested operations serialize recursively like map_each). The two blocks are the serialized forms of the two Python examples above.

Weight Case:

{
  "operation": "case",
  "sources": ["weight_units", "weight_lbs", "weight_kgs"],
  "selector": "weight_units",
  "branches": [
    {
      "when": ["2"],
      "combine": null,
      "operands": [
        {"source": "weight_lbs", "operations": [{"operation": "do_nothing"}]}
      ]
    },
    {
      "when": ["1"],
      "combine": null,
      "operands": [
        {
          "source": "weight_kgs",
          "operations": [
            {"operation": "convert_units", "source_unit": "kg", "target_unit": "lb"},
            {"operation": "round", "precision": 0}
          ]
        }
      ]
    }
  ],
  "default": null
}

Coalesce (multi-operand, stone+pounds):

{
  "operation": "coalesce",
  "sources": ["weight_lbs", "weight_stone", "weight_stone_lbs"],
  "branches": [
    {"combine": null, "operands": [{"source": "weight_lbs", "operations": [{"operation": "do_nothing"}]}]},
    {
      "combine": "sum",
      "operands": [
        {"source": "weight_stone", "operations": [{"operation": "scale", "scaling_factor": 14}]},
        {"source": "weight_stone_lbs", "operations": [{"operation": "do_nothing"}]}
      ]
    }
  ],
  "default": null
}

Why

Harmonizing the RADx data dictionaries, RADx-UP collects height/weight under a unit-selector flag (self_reported_weight_units_2: 1=kg/2=lbs; self_reported_height_coded: 1=feet+inches/2=meters+cm). The existing primitives couldn't express "switch on the flag, convert accordingly" — the list reducers reject nulls (and exactly one unit field is null per row), and MapEach applies the same op to every element (but kg needs conversion while lbs doesn't). See #112 for the full design discussion.

Changes

  • src/harmonization_framework/primitives/case.py, coalesce.py — new primitives
  • Registered in primitives/__init__.py, vocabulary.py (CASE, COALESCE), factory.py
  • tests/test_case_coalesce.py — 17 tests

Tests

tests/test_case_coalesce.py — 17 tests covering: single-source flag (kg/lbs), int/float selector matching, multi-operand height sums (feet+inches, meters+cm), multi-operand Coalesce with combine (stone+pounds), serialization round-trips, and null-selector / no-match / all-null defaults.

  • New tests: 17 passing
  • Full suite: 199 passing

Verified on synthetic RADx-UP rows: pounds → as-is; 68 kg → 150 lbs; 5 ft 7 in → 67 in; 1 m 70 cm → 66.9 in.

🤖 Generated with Claude Code

matthewhorridge and others added 3 commits June 29, 2026 16:16
Two combinators for "the same quantity arrives in one of several columns,
each possibly needing its own conversion":

- Case: switch on a selector source; the first branch whose `when` contains
  the selector value wins. Models data with an authoritative unit/type flag
  (e.g. a "units: kg/lbs" column).
- Coalesce: pick the first branch whose source is non-null. Models
  "whichever field was filled in".

Both address sources BY NAME (the primitive stores its own ordered source-name
list and zips it with the positional values transform() receives) rather than
by fragile positional index. A branch is composable: either a single source +
op-chain, or several `terms` (each source + op-chain) combined with a reduction
-- so a branch can e.g. convert feet->inches, take inches as-is, and sum them.
Branch op-chains reuse existing primitives (ConvertUnits, Scale, Round, ...)
and round-trip through the factory like MapEach's nested ops.

14 new tests (single-source weight kg/lbs flag, multi-term height feet+inches
and meters+cm summing, serialization round-trips, null/default edges). Full
suite: 196 passing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The multi-source branch sub-computations were called 'terms', which only read
naturally when combine='sum'. 'operands' is precise (an operand is what a
combining op acts on) and combine-agnostic. Pure rename of the serialization
key, helper functions, and docstrings; no behavior change. Also drops the
unused _term_sources helper.

196 tests passing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Coverage gap: combine was only exercised via Case (height sum). Add a Coalesce
case where one branch is stone+pounds summed (Scale(14) on stone + leftover
pounds, combine="sum") and the other is a single pounds field — no flag, so the
populated branch wins. Demonstrates combine doing real work in Coalesce and adds
a serialization round-trip for a multi-operand Coalesce branch.

198 tests passing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A passthrough operand had an empty operations list ([]); make it [DoNothing()]
to match the single-source examples and read clearly as "passes through
unchanged" at every no-op site. Functionally identical (empty chain == DoNothing).
16 tests passing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@matthewhorridge
matthewhorridge merged commit 7845d72 into main Jun 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Case and Coalesce multi-source combinator primitives

1 participant