diff --git a/README.md b/README.md index 4684d3d..9a05aa7 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,7 @@ Notes: - By default only target columns are written. Add `--include-metadata` to include `source dataset` and `original_id`. - Restrict outputs with `--targets nih_age,nih_sex`. - `--dataset-name` sets the dataset name used for metadata columns (defaults to the input file name). +- `harmonize --list-operations` prints the available primitive operations with a short description of each. #### Validating rules files @@ -98,14 +99,18 @@ The same rule in YAML: ## Multi-Source Rules -A rule with more than one entry in `sources` receives a list of values (one per source, in order) instead of a scalar. Two combinator primitives choose between sources by name: +Most rules read one column and write one column. Sometimes, though, a single harmonized value has to be assembled from several columns. Two situations come up all the time in real datasets: -- `case` switches on a **selector** column: the first branch whose `when` list contains the selector value wins. Use it when the data carries an authoritative flag (e.g. a units column). Selector values are matched as text, and integer-valued floats match their integer form (a `2.0` read from CSV matches `when: ['2']`). -- `coalesce` picks the first branch whose primary source is **non-null**. Use it when "whichever field was filled in" is the right rule. Branch order defines precedence. +1. The same measurement lives in *different columns depending on how it was recorded* — for example, weight was entered in `weight_lbs` **or** `weight_kgs`, with a `weight_units` column saying which one was used for each row. +2. One measurement is *split across columns that must be combined* — for example, height recorded as `height_ft` plus `height_in`. -Both share the same branch shape: each branch has one or more `operands` (a source plus its own operation chain), and a branch with multiple operands combines them with a reduction named by `combine` (e.g. `sum`). If no branch matches, the rule yields `default`. +To handle these, list every column the rule needs in `sources`, and use the `case` or `coalesce` operation to say how to pick or combine them. -Example (YAML): a weight rule where a units flag selects pounds-as-is or kilograms converted to pounds, followed by the same logic expressed as a coalesce over whichever column is populated: +### `case`: pick a column based on a flag + +Use `case` when the data has a flag column that says where the real value is. `case` looks at one column (the `selector`) and picks a branch by its value. Each branch reads: *when the flag has one of these values (`when`), take the value from this column (`source`) and run these operations on it.* + +Here, `weight_units` is `2` when the weight was entered in pounds and `1` when it was entered in kilograms: ```yaml - sources: [weight_units, weight_lbs, weight_kgs] @@ -115,19 +120,29 @@ Example (YAML): a weight rule where a units flag selects pounds-as-is or kilogra sources: [weight_units, weight_lbs, weight_kgs] selector: weight_units branches: - - when: ['2'] + - when: ['2'] # flag says pounds... operands: - - source: weight_lbs + - source: weight_lbs # ...use weight_lbs unchanged operations: - {operation: do_nothing} - - when: ['1'] + - when: ['1'] # flag says kilograms... operands: - - source: weight_kgs + - source: weight_kgs # ...convert weight_kgs to pounds operations: - {operation: convert_units, source_unit: kilogram, target_unit: pound} - {operation: round, precision: 0} - default: null + default: null # flag missing or unrecognized -> null +``` + +Row by row this means: if `weight_units` is 2, `nih_weight` is `weight_lbs` as-is (`do_nothing` marks "use unchanged"). If it is 1, `nih_weight` is `weight_kgs` converted to pounds and rounded. If the flag is blank or has any other value, `nih_weight` is null. + +Flag values in `when` are written as text, but numeric flags match anyway: a `2` or `2.0` read from CSV matches `when: ['2']`. + +### `coalesce`: use whichever column is filled in +Use `coalesce` when there is no flag column — each row simply has its value in one column or the other. Branches are tried in order and the first one whose column is non-empty wins: + +```yaml - sources: [weight_lbs, weight_kgs] target: nih_weight operations: @@ -135,22 +150,26 @@ Example (YAML): a weight rule where a units flag selects pounds-as-is or kilogra sources: [weight_lbs, weight_kgs] branches: - operands: - - source: weight_lbs + - source: weight_lbs # prefer weight_lbs when present operations: - {operation: do_nothing} - operands: - - source: weight_kgs + - source: weight_kgs # otherwise fall back to weight_kgs operations: - {operation: convert_units, source_unit: kilogram, target_unit: pound} - {operation: round, precision: 0} - default: null + default: null # both empty -> null ``` -A multi-operand branch combines several sources — here summing feet and inches into total inches: +Row by row: if `weight_lbs` has a value, use it. Otherwise, if `weight_kgs` has a value, convert it to pounds. If both are empty, `nih_weight` is null. + +### Adding columns together in a branch + +A branch can also combine several columns. List each column under `operands` with the operations that prepare it, and set `combine` to how the prepared values are merged (for example `sum`). Here, height was recorded as feet plus inches, and the harmonized value is total inches: ```yaml -- when: ['1'] - combine: sum +- when: ['1'] # flag says feet-and-inches + combine: sum # add the prepared values together operands: - source: height_ft operations: @@ -160,7 +179,11 @@ A multi-operand branch combines several sources — here summing feet and inches - {operation: do_nothing} ``` -For multi-source rules without branching, `map_each` applies a nested operation chain to every source value (e.g. cast each one-hot flag to int) before a list-consuming step like `reduce`. +That is: convert `height_ft` to inches, leave `height_in` as it is, and add the two. + +### Multi-source rules without branching + +If every source column should get the *same* treatment and then be merged — for example, a set of 0/1 checkbox columns collapsing into one code — no branching is needed: use `map_each` to apply an operation chain to every value, followed by `reduce` to merge the results (see the Primitives Reference below). ## Primitives Reference diff --git a/src/harmonization_framework/cli.py b/src/harmonization_framework/cli.py index 08c0599..30754a6 100644 --- a/src/harmonization_framework/cli.py +++ b/src/harmonization_framework/cli.py @@ -1,11 +1,14 @@ import argparse +import inspect import os +import textwrap from typing import Iterable, List, Sequence import pandas as pd from .harmonize import harmonize_dataset from .harmonization_rule import HarmonizationRule +from .primitives.factory import OPERATION_CLASSES from .rule_registry import RuleSet, validate_rules_file @@ -92,8 +95,8 @@ def build_parser() -> argparse.ArgumentParser: ) parser.add_argument( "--rules", - required=True, action="append", + default=[], help="Path to a rules file (JSON, or YAML if the path ends in " ".yaml/.yml). Can be provided multiple times.", ) @@ -127,9 +130,26 @@ def build_parser() -> argparse.ArgumentParser: action="store_true", help="Include source dataset and original_id columns in output.", ) + parser.add_argument( + "--list-operations", + action="store_true", + help="List the available primitive operations with a short " + "description of each, then exit.", + ) return parser +def _list_operations() -> None: + # Print each operation name with the first paragraph of its class + # docstring, so the listing always matches the implemented primitives. + for name in sorted(OPERATION_CLASSES): + doc = inspect.getdoc(OPERATION_CLASSES[name]) + summary = " ".join(doc.split("\n\n")[0].split()) if doc else "(no description)" + print(name) + print(textwrap.fill(summary, width=78, initial_indent=" ", subsequent_indent=" ")) + print() + + def _validate_rules(rule_paths: Iterable[str]) -> int: # Validate each rules file independently and report every problem found. # Returns a process exit code: 0 if all files are valid, 1 otherwise. @@ -150,6 +170,14 @@ def main(argv: Sequence[str] | None = None) -> None: parser = build_parser() args = parser.parse_args(argv) + if args.list_operations: + _list_operations() + return + + if not args.rules: + parser.error("--rules is required unless --list-operations is given.") + return + if args.validate: raise SystemExit(_validate_rules(args.rules)) diff --git a/src/harmonization_framework/primitives/factory.py b/src/harmonization_framework/primitives/factory.py index e2943bc..e78c97e 100644 --- a/src/harmonization_framework/primitives/factory.py +++ b/src/harmonization_framework/primitives/factory.py @@ -36,6 +36,37 @@ from .vocabulary import PrimitiveVocabulary +# Registry of operation name -> primitive class. This is the single source of +# truth for which operations exist: deserialization dispatches through it, and +# the CLI's --list-operations derives its listing (with help text from each +# class docstring) from it. +OPERATION_CLASSES: Dict[str, type] = { + PrimitiveVocabulary.BIN.value: Bin, + PrimitiveVocabulary.CASE.value: Case, + PrimitiveVocabulary.CAST.value: Cast, + PrimitiveVocabulary.COALESCE.value: Coalesce, + PrimitiveVocabulary.CONVERT_DATE.value: ConvertDate, + PrimitiveVocabulary.CONVERT_UNITS.value: ConvertUnits, + PrimitiveVocabulary.DO_NOTHING.value: DoNothing, + PrimitiveVocabulary.ENUM_TO_ENUM.value: EnumToEnum, + PrimitiveVocabulary.EXTRACT_REGEX.value: ExtractRegex, + PrimitiveVocabulary.FORMAT_NUMBER.value: FormatNumber, + PrimitiveVocabulary.MAP_EACH.value: MapEach, + PrimitiveVocabulary.MISSING_CODE.value: MissingCode, + PrimitiveVocabulary.NORMALIZE_BOOLEAN.value: NormalizeBoolean, + PrimitiveVocabulary.NORMALIZE_TEXT.value: NormalizeText, + PrimitiveVocabulary.OFFSET.value: Offset, + PrimitiveVocabulary.PARSE_ARRAY.value: ParseArray, + PrimitiveVocabulary.REDUCE.value: Reduce, + PrimitiveVocabulary.ROUND.value: Round, + PrimitiveVocabulary.SCALE.value: Scale, + PrimitiveVocabulary.SUBSTITUTE.value: Substitute, + PrimitiveVocabulary.THRESHOLD.value: Threshold, + PrimitiveVocabulary.TRUNCATE.value: Truncate, + PrimitiveVocabulary.VALIDATE_PATTERN.value: ValidatePattern, +} + + def deserialize_operation(operation: Dict[str, Any]) -> PrimitiveOperation: """ Build a PrimitiveOperation from its serialized dict. @@ -43,52 +74,7 @@ def deserialize_operation(operation: Dict[str, Any]) -> PrimitiveOperation: Raises ValueError for unknown operation names. """ name = operation["operation"] - match name: - case PrimitiveVocabulary.BIN.value: - return Bin.from_serialization(operation) - case PrimitiveVocabulary.CASE.value: - return Case.from_serialization(operation) - case PrimitiveVocabulary.CAST.value: - return Cast.from_serialization(operation) - case PrimitiveVocabulary.COALESCE.value: - return Coalesce.from_serialization(operation) - case PrimitiveVocabulary.CONVERT_DATE.value: - return ConvertDate.from_serialization(operation) - case PrimitiveVocabulary.CONVERT_UNITS.value: - return ConvertUnits.from_serialization(operation) - case PrimitiveVocabulary.DO_NOTHING.value: - return DoNothing.from_serialization(operation) - case PrimitiveVocabulary.ENUM_TO_ENUM.value: - return EnumToEnum.from_serialization(operation) - case PrimitiveVocabulary.EXTRACT_REGEX.value: - return ExtractRegex.from_serialization(operation) - case PrimitiveVocabulary.FORMAT_NUMBER.value: - return FormatNumber.from_serialization(operation) - case PrimitiveVocabulary.MAP_EACH.value: - return MapEach.from_serialization(operation) - case PrimitiveVocabulary.MISSING_CODE.value: - return MissingCode.from_serialization(operation) - case PrimitiveVocabulary.NORMALIZE_BOOLEAN.value: - return NormalizeBoolean.from_serialization(operation) - case PrimitiveVocabulary.NORMALIZE_TEXT.value: - return NormalizeText.from_serialization(operation) - case PrimitiveVocabulary.OFFSET.value: - return Offset.from_serialization(operation) - case PrimitiveVocabulary.PARSE_ARRAY.value: - return ParseArray.from_serialization(operation) - case PrimitiveVocabulary.REDUCE.value: - return Reduce.from_serialization(operation) - case PrimitiveVocabulary.ROUND.value: - return Round.from_serialization(operation) - case PrimitiveVocabulary.SCALE.value: - return Scale.from_serialization(operation) - case PrimitiveVocabulary.SUBSTITUTE.value: - return Substitute.from_serialization(operation) - case PrimitiveVocabulary.THRESHOLD.value: - return Threshold.from_serialization(operation) - case PrimitiveVocabulary.TRUNCATE.value: - return Truncate.from_serialization(operation) - case PrimitiveVocabulary.VALIDATE_PATTERN.value: - return ValidatePattern.from_serialization(operation) - case _: - raise ValueError(f"Unknown operation: {name}") + cls = OPERATION_CLASSES.get(name) + if cls is None: + raise ValueError(f"Unknown operation: {name}") + return cls.from_serialization(operation) diff --git a/tests/test_list_operations.py b/tests/test_list_operations.py new file mode 100644 index 0000000..3bd3715 --- /dev/null +++ b/tests/test_list_operations.py @@ -0,0 +1,43 @@ +import pytest + +from harmonization_framework import cli +from harmonization_framework.primitives.factory import OPERATION_CLASSES +from harmonization_framework.primitives.vocabulary import PrimitiveVocabulary + + +def test_registry_covers_full_vocabulary(): + assert set(OPERATION_CLASSES) == {member.value for member in PrimitiveVocabulary} + + +def test_every_operation_has_help_text(): + for name, cls in OPERATION_CLASSES.items(): + assert cls.__doc__ and cls.__doc__.strip(), f"{name} has no docstring" + + +def test_cli_list_operations(capsys): + cli.main(["--list-operations"]) + out = capsys.readouterr().out + for name in OPERATION_CLASSES: + assert f"\n{name}\n" in f"\n{out}" + # Spot-check that help text accompanies the names. + assert "Convert numeric values between units" in out + + +def test_cli_list_operations_ignores_other_args(capsys): + # The list is printed even if other flags are supplied. + cli.main(["--list-operations", "--on-missing", "warn"]) + assert "do_nothing" in capsys.readouterr().out + + +def test_cli_requires_rules_without_list_operations(capsys): + with pytest.raises(SystemExit) as excinfo: + cli.main([]) + assert excinfo.value.code == 2 + assert "--rules is required" in capsys.readouterr().err + + +def test_cli_validate_requires_rules(capsys): + with pytest.raises(SystemExit) as excinfo: + cli.main(["--validate"]) + assert excinfo.value.code == 2 + assert "--rules is required" in capsys.readouterr().err