Add MissingCode primitive + lossless mapping serialization - #110
Merged
Conversation
Adds a `missing_code` primitive that turns declared missing-value codes (e.g. numeric -999) into real nulls while passing every other value — including genuine nulls — through unchanged. This is the identity-preserving null map that EnumToEnum cannot express: EnumToEnum sends every unmapped value to its default, so it cannot null one code without nulling the column. Each code carries a label (e.g. "not_measured"). The label is not written to the output column (a null has no room for a reason); instead the harmonize engine reports each nulled cell — value, label, and row index — to the replay log. Codes and labels are also recorded in rules.json via to_dict. - missing_code.py: primitive; accepts a list or dict of codes, normalises to a dict, coerces int/float keys back from JSON strings on deserialization. - vocabulary/factory/__init__: register and export the primitive. - replay_logger.py: add log_missing_code_hits; tag rule events "event":"rule". - harmonize.py: after apply, re-scan the source column for hits and log them. - transformations.py replay: skip non-rule log events (missing key => "rule" for backward compatibility with older logs). - tests: serialization round-trip incl. numeric-key JSON guard, null pass-through, and end-to-end replay-log hit + replay-skip coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
JSON object keys are always strings, so an EnumToEnum keyed by integers (e.g.
the index from Reduce(ONEHOT)) round-tripped through rules.json with string
keys and then missed every lookup, silently falling through to the default.
The previous defence was a heuristic in from_serialization that only restored
int keys when keys AND values were both int-like — which failed for the common
int-key -> string-label case.
Fix the root cause: serialize the mapping as a list of {"from","to"} entries
(and MissingCode.codes as {"code","label"} entries) instead of a JSON object.
Each key then sits in a value position and keeps its native JSON type, so no
coercion is needed. Removed the is_int_like / is_float_like / key-coercion code
from both primitives. This matches Bin's existing list-of-objects style.
No backward compatibility (framework not widely used yet): only the new entry
form is read. Tests updated to the new shape; added json round-trip guards that
an int-keyed EnumToEnum and a numeric-code MissingCode still match integer
inputs after dumps/loads — now guaranteed by the format, not by coercion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related changes to missing-data handling and mapping serialization.
1.
MissingCodeprimitive (+ per-value replay logging)A new
missing_codeprimitive that turns declared missing-value codes (e.g. numeric-999) into real nulls while passing every other value — including genuine nulls — through unchanged. This is the identity-preserving null map thatEnumToEnumcannot express (an unmapped value always returnsdefault, soEnumToEnumcan't null one code without nulling the column)."not_measured"). The label isn't written to the output (a null has no room for a reason); instead the harmonize engine reports each nulled cell — value, label, row index — to the replay log. Codes + labels are also recorded inrules.json.transformations.py::replayskips non-rule log events (per-value hits aren't replayable); rule events are tagged"event": "rule", and a missing key defaults to"rule"for older logs.2. Lossless mapping serialization (fixes a silent int-key bug)
JSON object keys are always strings, so an
EnumToEnumkeyed by integers (e.g. the index fromReduce(ONEHOT)) round-tripped throughrules.jsonwith string keys and then missed every lookup, silently falling through todefault. The old defence was a coercion heuristic that only restored int keys when keys and values were both int-like — which failed for the common int-key → string-label case.Fixed at the root:
EnumToEnum.mappingnow serializes as a list of{"from","to"}entries, andMissingCode.codesas{"code","label"}entries. Each key sits in a value position and keeps its native JSON type, so no coercion is needed (theis_int_like/is_float_likecode is removed). MatchesBin's existing list-of-objects style.No backward compatibility — only the new entry form is read (framework not widely used yet).
Testing
EnumToEnumand a numeric-codeMissingCodestill match integer inputs afterdumps/loads— now guaranteed by the format, not by coercion.🤖 Generated with Claude Code