ops(v2.10+): freeze model/policy promotion, effective-period registry, rollback, and live decision provenance

[dmidlo]

Parents: #605 #712 #713 #714
Related: #632–#646 #681 #693 #697 #698 #710–#714

Problem

The project now has contracts for training, preprocessing, model selection, offline/online parity, live validity, and score-to-action policy. What is still missing is one canonical deployment/promotion/rollback registry proving which exact model-policy-feature stack was active for every live decision and allowing deterministic rollback without rewriting historical deployment identity.

A rollback is not 'changing history back'. It is a new effective-period deployment event selecting a previously qualified artifact stack from that point forward.

Immutable deployment identity

Define LiveDeploymentV1 carrying at least:

• deployment ID/version/hash;
• model artifact/hash;
• ModelDecisionPolicyV1 identity (design(v2.10+): freeze ML score-to-action policy, thresholding, sizing, abstention, and execution handoff #714);
• feature registry/projection/schema hash;
• preprocessing artifact hash (design(v2.10): freeze train-only preprocessing, normalization, encoding, and imputation-fit semantics for model-ready views #697);
• live validity policy identity (test(v2.10): govern live feature/model drift monitoring, calibration decay, and inference abstention after deployment #713);
• broker/plugin/capture compatibility identities;
• execution, financing, margin, jurisdiction/account policy identities;
• environment/runtime manifest identity (milestone(v2.9+): govern computational reproducibility, environment identity, and cross-platform numerical replay #637–test(v2.9+): certify bitwise, semantic, numerical, and statistical replay across supported environments #641);
• start/effective time and optional superseded time;
• promotion evidence/qualification receipts;
• deployment mode (shadow, canary, limited, active, disabled);
• rollback target where applicable;
• lineage root under feat(v2.10): add end-to-end feature lineage DAG from immutable evidence to every final ML column #693.

Any scientifically material change creates a successor deployment identity even if model weights are unchanged.

Effective-period function

For deployment intervals D_i=[a_i,b_i), define active deployment

d(t) = unique i such that a_i <= t < b_i.

Certified live operation requires at most one active decision-authority deployment for each account/universe scope at any timestamp unless an explicit experiment/shadow policy allows multiple non-authoritative deployments.

Container planning fixture with intervals [0,10)->A, [10,20)->B, [20,30)->A_rollback gives deterministic ownership: t=5 -> A, t=10 -> B, t=20 -> A_rollback.

This half-open ownership rule must be consistent with #694/#701 ordering semantics.

Promotion stages

Support separately versioned stages such as:

1. candidate: qualified offline artifact only;
2. shadow: receives live features but cannot create authoritative orders;
3. canary: bounded account/notional/instrument scope;
4. limited: expanded but capped production scope;
5. active: normal authorized scope;
6. disabled: no new decisions;
7. superseded: retained historical identity.

Promotion criteria must be frozen before observing promotion outcomes where those outcomes would otherwise become selection evidence.

Rollback semantics

Rollback must:

• select a previously qualified deployment stack or a separately qualified emergency-safe policy;
• create a new effective-period record;
• never overwrite the interval during which the superseded deployment actually ran;
• preserve existing orders/positions under an explicit transition policy;
• preserve lineage linking triggering evidence, rollback decision, predecessor and successor;
• remain reproducible in replay.

If model B ran from 10:00 to 10:05 and then rollback selects model A, historical replay must still show B as authoritative in [10:00,10:05) and A again only from rollback effective time onward.

Position/order transition policy

Deployment switches must explicitly define handling for:

• open positions created by predecessor;
• pending orders;
• trailing stops/targets;
• account state/checkpoints;
• strategy/decision hysteresis;
• partial fills in flight;
• financing/margin state;
• model-specific state unavailable to successor.

Allowed policies may include inherit_and_manage, cancel_pending_keep_positions, flatten_then_switch, or a separately frozen transition operator. No implicit reset-to-flat.

Promotion evidence

Before authoritative promotion require, as applicable:

• protected offline evaluation (design(v2.10): govern protected model selection, repeated backtests, multiple testing, and synthetic-heavy research overfitting #698);
• schema/preprocessing compatibility;
• test(v2.10): certify offline-training versus live-inference feature parity and prevent training-serving skew #712 batch/live parity;
• test(v2.10): govern live feature/model drift monitoring, calibration decay, and inference abstention after deployment #713 live validity in shadow/canary;
• execution/account/risk compatibility;
• deterministic replay/checkpoint tests;
• broker/plugin capability and capture-health qualification;
• resource/SLO envelope;
• explicit human/operator authorization where operational policy requires it.

An issue being closed is not deployment evidence; retain machine-readable receipts.

Live decision provenance

Every authoritative prediction/action/order must retain or resolve to:

• deployment ID;
• model ID;
• decision-policy ID;
• feature snapshot/lineage root;
• validity state;
• account/risk state;
• execution-policy identity;
• broker/plugin/capture identity;
• exact effective timestamp/order sequence.

Given a historical live order, the system must be able to reconstruct the exact active stack without consulting mutable 'current model' configuration.

Rollback triggers

Triggers may include:

• test(v2.10): govern live feature/model drift monitoring, calibration decay, and inference abstention after deployment #713 validity/refusal state;
• parity regression;
• schema/preprocessing incompatibility;
• broker/plugin failure;
• execution/risk reconciliation failure;
• software/runtime integrity failure;
• deterministic operator/feature defect;
• explicit operator emergency action.

Do not use future realized P&L as if it were contemporaneously known when replaying why a rollback happened; trigger evidence obeys #694 availability semantics.

Canary/traffic allocation

If multiple deployments run simultaneously for experiment/shadow purposes, distinguish decision_authority=false shadow from authoritative allocation.

For authoritative randomized/deterministic allocation across deployment candidates, the routing policy itself requires a stable identity and evidence-unit ownership so outcomes cannot be cherry-picked after the fact. Protected-evaluation/multiple-testing rules from #698 apply.

Rollback correctness metrics

At minimum report:

• time from trigger-available-at to rollback effective time;
• number/notional of actions emitted after trigger but before rollback;
• unresolved pending orders at transition;
• position/exposure discontinuity attributable to switch;
• replay mismatch rate;
• orphan decisions lacking deployment identity.

Acceptance target for orphan authoritative decisions is zero.

Negative controls

Certification must fail/flag:

• mutable current_model config used as sole historical provenance;
• rollback overwrites prior model effective interval;
• model weights unchanged but threshold/policy changed without new deployment identity;
• serving stack has different preprocessing/schema than promoted receipt;
• canary/shadow predictions accidentally route authoritative orders;
• deployment switch silently drops open positions or pending orders;
• rollback trigger replay uses future outcome not known at trigger time;
• two authoritative deployments overlap the same account/scope without explicit routing policy;
• historical order cannot resolve exact model/policy/feature stack;
• prior deployment artifacts garbage-collected despite retained live decisions depending on them.

Acceptance

• one immutable deployment/effective-period registry exists;
• every authoritative live decision resolves to exactly one deployment identity;
• shadow/canary/active authority semantics are explicit;
• rollback creates successor effective periods and preserves historical truth;
• open-position/order transition is deterministic and replayable;
• full promotion/rollback replay passes under test(v2.10): certify offline-training versus live-inference feature parity and prevent training-serving skew #712/test(v2.9+): define numerical tolerance, reduction-order, and reference-kernel policy for reproducible science #640;
• retained artifacts are protected by durable lifecycle/GC rules (milestone(v2.9+): govern durable scientific artifact retention, integrity scrubbing, backup, and disaster recovery #642–test(v2.9+): run disaster-recovery restore drills and measure RPO/RTO for scientific evidence #646);
• feat(traceability): implement a canonical machine-readable room/archive-to-issue requirements ledger with orphan detection #691 records the requirement and docs(v2.5): publish an implemented-versus-evidenced capability and certification matrix #525 tracks planned/implemented/executed/scientifically-passed states separately.

[dmidlo]

Editorial ownership under #721/#724

This issue is private execution with an optional public interoperability schema. histdata.com-tools may retain a neutral deployment/effective-period/provenance contract if downstream artifacts need a stable shared representation, but the actual live deployment registry, promotion state, authoritative account routing, rollback operations, canary traffic, and model-policy effective periods belong in the separate private ML repository. Those operational states must not block the public ML-substrate handoff. Preserve the scientific requirements as downstream design provenance; #724 owns the formal requirement-level split.