design(v2.10+): govern retraining succession, champion-challenger evaluation, forgetting checks, and controlled model retirement

[dmidlo]

Parents: #605 #698 #713 #714 #716
Related: #657 #681 #693 #699 #710–#714

Problem

The roadmap now governs model training/evaluation, live feature parity, post-deployment drift, score-to-action policy, and effective deployment periods. What remains missing is one canonical contract for retraining succession: when a challenger may be trained, how it is compared to the active champion, how older regimes are retained in qualification, how catastrophic forgetting is detected, and how a successor is promoted without contaminating protected evidence or silently changing the deployed decision stack.

A retrained model is a new scientific artifact even when architecture/hyperparameters are unchanged. Retraining on later data changes weights, training domain, preprocessing fit, calibration, and often the effective hypothesis tested.

Immutable retraining run

Define RetrainingRunV1 carrying at least:

• parent/champion model and deployment IDs;
• training-corpus, feature-schema, preprocessing, label, universe and regime-policy identities;
• authorized training/validation windows and cutoff;
• synthetic/observed/member/scenario composition and evidence weights;
• search/hyperparameter budget ledger under design(v2.10): govern protected model selection, repeated backtests, multiple testing, and synthetic-heavy research overfitting #698;
• random/replay/environment identity;
• challenger model hash;
• challenger decision-policy candidate identity (design(v2.10+): freeze ML score-to-action policy, thresholding, sizing, abstention, and execution handoff #714);
• evaluation matrix and acceptance thresholds;
• retained-regime suite;
• promotion recommendation state;
• lineage root and run hash.

Each retraining run is append-only. Re-running after changing data, cutoff, thresholds or search space creates a successor run identity.

Champion-challenger contract

For frozen evaluation strata r in R with predeclared nonnegative weights alpha_r, sum alpha_r=1, define a reference aggregate utility

U(M) = sum_r alpha_r U_r(M).

Promotion must not rely only on aggregate utility. Require per-stratum guardrails so a challenger cannot obtain one large modern-regime gain by catastrophically degrading older/stress/regime support without explicit policy approval.

Container-vetted fixture: weights (0.5,0.3,0.2), champion stratum scores (0.62,0.58,0.55), challenger (0.64,0.56,0.60) produce U_champion=0.594, U_challenger=0.608, delta +0.014. This demonstrates aggregate improvement but does not by itself authorize promotion because one stratum degraded (0.58 -> 0.56).

Forgetting / retained-regime checks

For retained regime r, define performance before retraining P_r^old and challenger performance P_r^new. One simple forgetting diagnostic is

F_r = P_r^old - P_r^new.

Report at least max positive forgetting

F_max = max_r max(0,F_r)

plus weighted/quantile summaries and support. Container fixture with retained scores (0.70,0.65) before and (0.66,0.59) after gives forgetting (0.04,0.06) and F_max=0.06.

The exact performance metric may differ by target/decision layer, but its sign/orientation must be frozen so forgetting is interpretable.

Retained evaluation matrix

Every challenger must be evaluated across a frozen matrix including, where scientifically available:

• recent/modern target domain;
• older historical eras and structural regimes (research(v2.10): quantify historical-to-modern regime drift and qualify causal transport/reweighting for legacy evidence #710/design(v2.10): freeze causal regime/change-point state versus retrospective segmentation for transport, traders, and ML #711);
• major volatility/session/calendar regimes;
• point-in-time instrument/universe strata (design(v2.10): freeze point-in-time instrument/universe lifecycle and prevent survivorship-biased feature/trader eligibility #695);
• triangle/cross-currency support strata;
• broker/feed observation states;
• stress scenarios (test(v2.8): certify deterministic tail-stress scenarios across reconstruction, traders, execution, margin, and wide ML features #708);
• synthetic versus observed/live domains (test(v2.10): quantify synthetic-to-observed/live domain shift across market, strategy, flow, and broker-style feature planes #699);
• uncertainty/support strata (design(v2.10): propagate reconstruction/context/population uncertainty into wide ML features and counterfactual labels #709);
• abstention/coverage behavior (test(v2.10): govern live feature/model drift monitoring, calibration decay, and inference abstention after deployment #713);
• post-cost/risk-constrained decision performance (design(v2.10+): freeze ML score-to-action policy, thresholding, sizing, abstention, and execution handoff #714).

Do not let the current regime become the only acceptance surface merely because it is newest.

Data-window policy

Retraining policy must explicitly choose one of: expanding window, rolling window, weighted historical retention, regime-balanced replay, fixed anchor set plus recent data, or another versioned policy.

A rolling window that discards older regimes is not automatically wrong, but it must expose the resulting support loss and forgetting risk. Synthetic descendants cannot be counted as additional observed evidence (#607/#656).

Rehearsal/retention data

If older observations or synthetic stress/reconstruction members are replayed during retraining, their inclusion/weights are part of the retraining identity. Do not use protected test data as rehearsal examples and then continue to call it untouched evaluation.

Challenger decision policy

A challenger model may need a separately fitted #714 decision policy. Compare at least:

1. model-to-model with a common reference decision policy where meaningful;
2. full champion stack versus full challenger stack;
3. raw prediction, calibration, decision, cost, risk and fill-realized decomposition.

This avoids attributing a threshold/sizing change to model improvement.

Promotion gates

Promotion from challenger to candidate/shadow/canary under #716 requires all predeclared gates, such as:

• protected statistical/decision improvement or non-inferiority;
• no forbidden retained-regime degradation;
• acceptable domain/support coverage;
• test(v2.10): certify offline-training versus live-inference feature parity and prevent training-serving skew #712 offline/live parity;
• test(v2.10): govern live feature/model drift monitoring, calibration decay, and inference abstention after deployment #713 live-validity compatibility;
• design(v2.10+): freeze ML score-to-action policy, thresholding, sizing, abstention, and execution handoff #714 decision-policy qualification;
• resource/latency envelopes;
• schema/preprocessing compatibility;
• stress and rollback readiness.

A challenger that fails a hard gate remains retained evidence; it is not silently discarded from the search ledger.

Non-inferiority reference

Where a metric U is higher-is-better and challenger minus champion difference is Delta, a predeclared non-inferiority margin delta >= 0 may require

Delta >= -delta

on retained strata while stricter superiority is required on the intended improvement objective. Statistical interval/testing choice must be frozen in #698 governance; this equation only defines the acceptance direction.

Retraining trigger versus promotion evidence

#713 drift can trigger a retraining candidate process, but drift is not itself evidence that the newly trained challenger is superior. Likewise, a scheduled retraining cadence does not imply automatic promotion.

Retirement and rollback

When #716 promotes a challenger, the former champion becomes superseded/rollback-eligible under an explicit retention policy. Preserve exact model, preprocessing, decision policy, environment and deployment artifacts for the rollback horizon.

Retirement after the rollback horizon must still preserve enough immutable metadata/lineage to reproduce historical decisions and scientific reports. Artifact GC follows durability/provenance rules (#642–#646), not an ad-hoc model cleanup script.

Continual/online learning boundary

Automatic online weight updates are not covered by ordinary fixed-model deployment. If allowed later, each update or governed update batch needs successor model identity, causal training evidence, evaluation semantics and rollback state. Do not mutate model parameters in place under one permanent deployment ID.

Negative controls

Certification must fail/flag:

• protected holdout reused as retraining data then reused as untouched test;
• challenger selected from many attempts while failed attempts disappear from search ledger;
• aggregate improvement hides forbidden collapse in an older/stress regime;
• later regime used to retrospectively choose the historical retained-set definition;
• challenger threshold/sizing differs but gain is attributed entirely to model weights;
• drift alarm automatically promotes newly trained model without protected qualification;
• previous champion deleted before rollback/reproducibility horizon;
• online model weights mutate under unchanged model/deployment ID;
• synthetic reconstruction member count inflates retraining sample evidence;
• live canary data used to tune challenger while same canary is still claimed as untouched qualification.

Acceptance

• one versioned retraining-run registry exists;
• champion/challenger comparisons retain all attempted challengers and full evaluation matrices;
• retained-regime/forgetting metrics and hard guardrails are explicit;
• data-window/rehearsal weighting is reproducible and evidence-mass-safe;
• model improvement is separated from decision-policy improvement;
• promotion always flows through ops(v2.10+): freeze model/policy promotion, effective-period registry, rollback, and live decision provenance #716 shadow/canary/effective-period controls;
• previous champions remain reproducible for rollback and historical attribution;
• feat(traceability): implement a canonical machine-readable room/archive-to-issue requirements ledger with orphan detection #691 records the requirement and docs(v2.5): publish an implemented-versus-evidenced capability and certification matrix #525 tracks planned/implemented/executed/scientifically-passed states separately.

[dmidlo]

Editorial ownership under #721/#724

This issue is private execution / public scientific guidance. The champion-challenger registry, retraining runs, protected result matrices, promotion decisions, retained-model state, and retirement/rollback operations belong in the separate private ML repository. Public histdata.com-tools may retain neutral schemas or reference fixtures only where needed for interoperability, provenance, or reproducibility of downstream references. Completion of concrete retraining operations is not a prerequisite for #721. #724 will preserve the requirements and remap their ownership without orphaning #691 traceability.