feat: emit typed provenance-stamped MetricFrame (evaluation-of-record) (#21) - #22
Merged
Merged
Conversation
release: v0.4.0 — EvaluationFrame architecture, metric catalog, Brier variants
#21) Add EvaluationReport.to_metric_frame() and a MetricFrame value object so views-reporting can consume a typed, transportable evaluation artifact instead of scraping WandB at render time (views-reporting C-108/C-48; views-frames ADR-020). This is the emit side only; pipeline-core injects provenance, reporting adds the EvaluationSource adapter (separate repos). - MetricFrame: string-keyed (eval_type, target, metric, group_id, partition, level) value object reusing the views-frames substrate (FrameMetadata + conformance.assert_frame_envelope). Owns its serialization (save/load round-trip). - Provenance split per ADR-020/C-47: generic identity in FrameMetadata; scoring_code_version (defaults to installed package version) + evaluation_timestamp in MetricFrame's own metadata. All identity injected, not inferred. - Row shape matches the consumer: per-group rows plus a group_id="mean" aggregate. - views-frames is an optional 'frames' extra; core stays numpy/sklearn-only and importable without it (find_spec-gated export). - 30 tests (green/beige/red): envelope conformance, round-trip, provenance split, fail-loud construction, and a C-41 canonical-token drift guard. - Update EvaluationReport CIC for the new method + lazy-views-frames deviation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address findings from the PR review: - save(): json.dumps now coerces numpy scalar types via a default handler, so an injected numpy timestamp/seed no longer crashes serialization (latent TypeError). - MetricFrame._validate: reject non-1D identifier arrays (fail-loud on the public constructor; not reachable via to_metric_frame but MetricFrame is exported). - to_metric_frame docstring: clarify the group_id="mean" denominator is the groups that reported each metric (= all groups in the normal uniform-metric-set emit path). - Add regression tests: numpy-typed timestamp/seed round-trip; 2D identifier rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Satisfies the check-branch 'up-to-date with main' gate. main is content-identical to development (no file delta); this only brings in the #19 merge node.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the emit side of #21.
Why
views-reporting scrapes per-model metrics from WandB
get_latest_run().summaryat render time; in production this silently mis-selected runs (their C-48: 22/25 constituents rendered "not calculated"). The durable fix (views-reporting C-108 / ADR-018, views-frames ADR-020) is for views-evaluation to emit the evaluation-of-record as a typed, transportable, provenance-stampedMetricFramethat reporting consumes via an injected adapter — no WandB at render time.What
MetricFrame(new) — a string-keyed value object on the axes(eval_type, target, metric, group_id, partition, level), reusing the views-frames substrate:FrameMetadatafor generic provenance andconformance.assert_frame_envelopeas the structural checker. Owns its serialization (save/loadround-trip; views-frames IO is(time,unit)-only).EvaluationReport.to_metric_frame(...)(new, Level-1 bridge) — flattens the nested results into per-group rows plus agroup_id="mean"cross-group aggregate (the value views-reporting's_canonical_rowmatches). Maps schema names to the consumer-facingeval_typespelling. Purely additive:to_dict()/to_dataframe()unchanged.model_id/run_id/data_version) goes inFrameMetadata;scoring_code_version(defaults to the installed package version — not a git SHA) andevaluation_timestampstay in the MetricFrame's own metadata. All identity is injected, not inferred.views-framesis aframesextra; the core stays numpy/sklearn-only and importable without it (find_spec-gated export).Contract / consumer alignment (verified against the repos)
run_idis intentionally optional/Noneat emit time (the WandB run does not exist yet at the evaluation call site in pipeline-core).Tests
30 tests (green/beige/red): emit for all 4 (task, pred_type) cells,
assert_frame_envelopeconformance, save/load round-trip incl. metadata, provenance split, NaN handling, fully-empty frame, fail-loud construction, and the canonical-token drift guard. Full suite: 279 passed, ruff clean.EvaluationReportCIC updated.Out of scope (separate repos)
model_id/run_id/data_version/partition/levelintoto_metric_frame(...).EvaluationSourceadapter (their ADR-018 Phase 3).schema_versionround-trip contract test (views-frames C-46 open half) beyond the local one here.🤖 Generated with Claude Code