Add --list-operations flag; rewrite Multi-Source Rules README section

README.md

@@ -46,6 +46,7 @@ Notes:
 - By default only target columns are written. Add `--include-metadata` to include `source dataset` and `original_id`.
 - Restrict outputs with `--targets nih_age,nih_sex`.
 - `--dataset-name` sets the dataset name used for metadata columns (defaults to the input file name).
+- `harmonize --list-operations` prints the available primitive operations with a short description of each.
 
 #### Validating rules files
 
@@ -98,14 +99,18 @@ The same rule in YAML:
 
 ## Multi-Source Rules
 
-A rule with more than one entry in `sources` receives a list of values (one per source, in order) instead of a scalar. Two combinator primitives choose between sources by name:
+Most rules read one column and write one column. Sometimes, though, a single harmonized value has to be assembled from several columns. Two situations come up all the time in real datasets:
 
-- `case` switches on a **selector** column: the first branch whose `when` list contains the selector value wins. Use it when the data carries an authoritative flag (e.g. a units column). Selector values are matched as text, and integer-valued floats match their integer form (a `2.0` read from CSV matches `when: ['2']`).
-- `coalesce` picks the first branch whose primary source is **non-null**. Use it when "whichever field was filled in" is the right rule. Branch order defines precedence.
+1. The same measurement lives in *different columns depending on how it was recorded* — for example, weight was entered in `weight_lbs` **or** `weight_kgs`, with a `weight_units` column saying which one was used for each row.
+2. One measurement is *split across columns that must be combined* — for example, height recorded as `height_ft` plus `height_in`.
 
-Both share the same branch shape: each branch has one or more `operands` (a source plus its own operation chain), and a branch with multiple operands combines them with a reduction named by `combine` (e.g. `sum`). If no branch matches, the rule yields `default`.
+To handle these, list every column the rule needs in `sources`, and use the `case` or `coalesce` operation to say how to pick or combine them.
 
-Example (YAML): a weight rule where a units flag selects pounds-as-is or kilograms converted to pounds, followed by the same logic expressed as a coalesce over whichever column is populated:
+### `case`: pick a column based on a flag
+
+Use `case` when the data has a flag column that says where the real value is. `case` looks at one column (the `selector`) and picks a branch by its value. Each branch reads: *when the flag has one of these values (`when`), take the value from this column (`source`) and run these operations on it.*
+
+Here, `weight_units` is `2` when the weight was entered in pounds and `1` when it was entered in kilograms:
 
 ```yaml
 - sources: [weight_units, weight_lbs, weight_kgs]
@@ -115,42 +120,56 @@ Example (YAML): a weight rule where a units flag selects pounds-as-is or kilogra
     sources: [weight_units, weight_lbs, weight_kgs]
     selector: weight_units
     branches:
-    - when: ['2']
+    - when: ['2']              # flag says pounds...
       operands:
-      - source: weight_lbs
+      - source: weight_lbs     # ...use weight_lbs unchanged
         operations:
         - {operation: do_nothing}
-    - when: ['1']
+    - when: ['1']              # flag says kilograms...
       operands:
-      - source: weight_kgs
+      - source: weight_kgs     # ...convert weight_kgs to pounds
         operations:
         - {operation: convert_units, source_unit: kilogram, target_unit: pound}
         - {operation: round, precision: 0}
-    default: null
+    default: null              # flag missing or unrecognized -> null
+```
+
+Row by row this means: if `weight_units` is 2, `nih_weight` is `weight_lbs` as-is (`do_nothing` marks "use unchanged"). If it is 1, `nih_weight` is `weight_kgs` converted to pounds and rounded. If the flag is blank or has any other value, `nih_weight` is null.
+
+Flag values in `when` are written as text, but numeric flags match anyway: a `2` or `2.0` read from CSV matches `when: ['2']`.
+
+### `coalesce`: use whichever column is filled in
 
+Use `coalesce` when there is no flag column — each row simply has its value in one column or the other. Branches are tried in order and the first one whose column is non-empty wins:
+
+```yaml
 - sources: [weight_lbs, weight_kgs]
   target: nih_weight
   operations:
   - operation: coalesce
     sources: [weight_lbs, weight_kgs]
     branches:
     - operands:
-      - source: weight_lbs
+      - source: weight_lbs     # prefer weight_lbs when present
         operations:
         - {operation: do_nothing}
     - operands:
-      - source: weight_kgs
+      - source: weight_kgs     # otherwise fall back to weight_kgs
         operations:
         - {operation: convert_units, source_unit: kilogram, target_unit: pound}
         - {operation: round, precision: 0}
-    default: null
+    default: null              # both empty -> null
 ```
 
-A multi-operand branch combines several sources — here summing feet and inches into total inches:
+Row by row: if `weight_lbs` has a value, use it. Otherwise, if `weight_kgs` has a value, convert it to pounds. If both are empty, `nih_weight` is null.
+
+### Adding columns together in a branch
+
+A branch can also combine several columns. List each column under `operands` with the operations that prepare it, and set `combine` to how the prepared values are merged (for example `sum`). Here, height was recorded as feet plus inches, and the harmonized value is total inches:
 
 ```yaml
-- when: ['1']
-  combine: sum
+- when: ['1']                  # flag says feet-and-inches
+  combine: sum                 # add the prepared values together
   operands:
   - source: height_ft
     operations:
@@ -160,7 +179,11 @@ A multi-operand branch combines several sources — here summing feet and inches
     - {operation: do_nothing}
 ```
 
-For multi-source rules without branching, `map_each` applies a nested operation chain to every source value (e.g. cast each one-hot flag to int) before a list-consuming step like `reduce`.
+That is: convert `height_ft` to inches, leave `height_in` as it is, and add the two.
+
+### Multi-source rules without branching
+
+If every source column should get the *same* treatment and then be merged — for example, a set of 0/1 checkbox columns collapsing into one code — no branching is needed: use `map_each` to apply an operation chain to every value, followed by `reduce` to merge the results (see the Primitives Reference below).
 
 ## Primitives Reference
 

src/harmonization_framework/cli.py

@@ -1,11 +1,14 @@
 import argparse
+import inspect
 import os
+import textwrap
 from typing import Iterable, List, Sequence
 
 import pandas as pd
 
 from .harmonize import harmonize_dataset
 from .harmonization_rule import HarmonizationRule
+from .primitives.factory import OPERATION_CLASSES
 from .rule_registry import RuleSet, validate_rules_file
 
 
@@ -92,8 +95,8 @@ def build_parser() -> argparse.ArgumentParser:
     )
     parser.add_argument(
         "--rules",
-        required=True,
         action="append",
+        default=[],
         help="Path to a rules file (JSON, or YAML if the path ends in "
         ".yaml/.yml). Can be provided multiple times.",
     )
@@ -127,9 +130,26 @@ def build_parser() -> argparse.ArgumentParser:
         action="store_true",
         help="Include source dataset and original_id columns in output.",
     )
+    parser.add_argument(
+        "--list-operations",
+        action="store_true",
+        help="List the available primitive operations with a short "
+        "description of each, then exit.",
+    )
     return parser
 
 
+def _list_operations() -> None:
+    # Print each operation name with the first paragraph of its class
+    # docstring, so the listing always matches the implemented primitives.
+    for name in sorted(OPERATION_CLASSES):
+        doc = inspect.getdoc(OPERATION_CLASSES[name])
+        summary = " ".join(doc.split("\n\n")[0].split()) if doc else "(no description)"
+        print(name)
+        print(textwrap.fill(summary, width=78, initial_indent="  ", subsequent_indent="  "))
+        print()
+
+
 def _validate_rules(rule_paths: Iterable[str]) -> int:
     # Validate each rules file independently and report every problem found.
     # Returns a process exit code: 0 if all files are valid, 1 otherwise.
@@ -150,6 +170,14 @@ def main(argv: Sequence[str] | None = None) -> None:
     parser = build_parser()
     args = parser.parse_args(argv)
 
+    if args.list_operations:
+        _list_operations()
+        return
+
+    if not args.rules:
+        parser.error("--rules is required unless --list-operations is given.")
+        return
+
     if args.validate:
         raise SystemExit(_validate_rules(args.rules))
 

src/harmonization_framework/primitives/factory.py

@@ -36,59 +36,45 @@
 from .vocabulary import PrimitiveVocabulary
 
 
+# Registry of operation name -> primitive class. This is the single source of
+# truth for which operations exist: deserialization dispatches through it, and
+# the CLI's --list-operations derives its listing (with help text from each
+# class docstring) from it.
+OPERATION_CLASSES: Dict[str, type] = {
+    PrimitiveVocabulary.BIN.value: Bin,
+    PrimitiveVocabulary.CASE.value: Case,
+    PrimitiveVocabulary.CAST.value: Cast,
+    PrimitiveVocabulary.COALESCE.value: Coalesce,
+    PrimitiveVocabulary.CONVERT_DATE.value: ConvertDate,
+    PrimitiveVocabulary.CONVERT_UNITS.value: ConvertUnits,
+    PrimitiveVocabulary.DO_NOTHING.value: DoNothing,
+    PrimitiveVocabulary.ENUM_TO_ENUM.value: EnumToEnum,
+    PrimitiveVocabulary.EXTRACT_REGEX.value: ExtractRegex,
+    PrimitiveVocabulary.FORMAT_NUMBER.value: FormatNumber,
+    PrimitiveVocabulary.MAP_EACH.value: MapEach,
+    PrimitiveVocabulary.MISSING_CODE.value: MissingCode,
+    PrimitiveVocabulary.NORMALIZE_BOOLEAN.value: NormalizeBoolean,
+    PrimitiveVocabulary.NORMALIZE_TEXT.value: NormalizeText,
+    PrimitiveVocabulary.OFFSET.value: Offset,
+    PrimitiveVocabulary.PARSE_ARRAY.value: ParseArray,
+    PrimitiveVocabulary.REDUCE.value: Reduce,
+    PrimitiveVocabulary.ROUND.value: Round,
+    PrimitiveVocabulary.SCALE.value: Scale,
+    PrimitiveVocabulary.SUBSTITUTE.value: Substitute,
+    PrimitiveVocabulary.THRESHOLD.value: Threshold,
+    PrimitiveVocabulary.TRUNCATE.value: Truncate,
+    PrimitiveVocabulary.VALIDATE_PATTERN.value: ValidatePattern,
+}
+
+
 def deserialize_operation(operation: Dict[str, Any]) -> PrimitiveOperation:
     """
     Build a PrimitiveOperation from its serialized dict.
 
     Raises ValueError for unknown operation names.
     """
     name = operation["operation"]
-    match name:
-        case PrimitiveVocabulary.BIN.value:
-            return Bin.from_serialization(operation)
-        case PrimitiveVocabulary.CASE.value:
-            return Case.from_serialization(operation)
-        case PrimitiveVocabulary.CAST.value:
-            return Cast.from_serialization(operation)
-        case PrimitiveVocabulary.COALESCE.value:
-            return Coalesce.from_serialization(operation)
-        case PrimitiveVocabulary.CONVERT_DATE.value:
-            return ConvertDate.from_serialization(operation)
-        case PrimitiveVocabulary.CONVERT_UNITS.value:
-            return ConvertUnits.from_serialization(operation)
-        case PrimitiveVocabulary.DO_NOTHING.value:
-            return DoNothing.from_serialization(operation)
-        case PrimitiveVocabulary.ENUM_TO_ENUM.value:
-            return EnumToEnum.from_serialization(operation)
-        case PrimitiveVocabulary.EXTRACT_REGEX.value:
-            return ExtractRegex.from_serialization(operation)
-        case PrimitiveVocabulary.FORMAT_NUMBER.value:
-            return FormatNumber.from_serialization(operation)
-        case PrimitiveVocabulary.MAP_EACH.value:
-            return MapEach.from_serialization(operation)
-        case PrimitiveVocabulary.MISSING_CODE.value:
-            return MissingCode.from_serialization(operation)
-        case PrimitiveVocabulary.NORMALIZE_BOOLEAN.value:
-            return NormalizeBoolean.from_serialization(operation)
-        case PrimitiveVocabulary.NORMALIZE_TEXT.value:
-            return NormalizeText.from_serialization(operation)
-        case PrimitiveVocabulary.OFFSET.value:
-            return Offset.from_serialization(operation)
-        case PrimitiveVocabulary.PARSE_ARRAY.value:
-            return ParseArray.from_serialization(operation)
-        case PrimitiveVocabulary.REDUCE.value:
-            return Reduce.from_serialization(operation)
-        case PrimitiveVocabulary.ROUND.value:
-            return Round.from_serialization(operation)
-        case PrimitiveVocabulary.SCALE.value:
-            return Scale.from_serialization(operation)
-        case PrimitiveVocabulary.SUBSTITUTE.value:
-            return Substitute.from_serialization(operation)
-        case PrimitiveVocabulary.THRESHOLD.value:
-            return Threshold.from_serialization(operation)
-        case PrimitiveVocabulary.TRUNCATE.value:
-            return Truncate.from_serialization(operation)
-        case PrimitiveVocabulary.VALIDATE_PATTERN.value:
-            return ValidatePattern.from_serialization(operation)
-        case _:
-            raise ValueError(f"Unknown operation: {name}")
+    cls = OPERATION_CLASSES.get(name)
+    if cls is None:
+        raise ValueError(f"Unknown operation: {name}")
+    return cls.from_serialization(operation)

tests/test_list_operations.py

@@ -0,0 +1,43 @@
+import pytest
+
+from harmonization_framework import cli
+from harmonization_framework.primitives.factory import OPERATION_CLASSES
+from harmonization_framework.primitives.vocabulary import PrimitiveVocabulary
+
+
+def test_registry_covers_full_vocabulary():
+    assert set(OPERATION_CLASSES) == {member.value for member in PrimitiveVocabulary}
+
+
+def test_every_operation_has_help_text():
+    for name, cls in OPERATION_CLASSES.items():
+        assert cls.__doc__ and cls.__doc__.strip(), f"{name} has no docstring"
+
+
+def test_cli_list_operations(capsys):
+    cli.main(["--list-operations"])
+    out = capsys.readouterr().out
+    for name in OPERATION_CLASSES:
+        assert f"\n{name}\n" in f"\n{out}"
+    # Spot-check that help text accompanies the names.
+    assert "Convert numeric values between units" in out
+
+
+def test_cli_list_operations_ignores_other_args(capsys):
+    # The list is printed even if other flags are supplied.
+    cli.main(["--list-operations", "--on-missing", "warn"])
+    assert "do_nothing" in capsys.readouterr().out
+
+
+def test_cli_requires_rules_without_list_operations(capsys):
+    with pytest.raises(SystemExit) as excinfo:
+        cli.main([])
+    assert excinfo.value.code == 2
+    assert "--rules is required" in capsys.readouterr().err
+
+
+def test_cli_validate_requires_rules(capsys):
+    with pytest.raises(SystemExit) as excinfo:
+        cli.main(["--validate"])
+    assert excinfo.value.code == 2
+    assert "--rules is required" in capsys.readouterr().err