design(v2.10+): freeze immutable experiment bundle identity for reproducible training, evaluation, ablation, and promotion evidence

[dmidlo]

Parents: #605 #698 #714 #716 #717
Related: #637–#646 #681 #689 #693 #697 #699 #709 #710 #712 #713

Problem

The roadmap now governs datasets, feature schemas, preprocessing, models, decision policies, evaluation, deployment, rollback and retraining. Those components are individually versioned, but there is still no single canonical experiment envelope binding the exact combination used for a scientific result.

Without one immutable experiment identity, a reported metric such as Sharpe, calibration error, domain distance, or challenger delta can become unreproducible because the dataset version, split policy, preprocessing artifact, feature projection, model weights, decision policy, execution assumptions, environment or code revision changed independently.

Issue comments and ad hoc filenames are not sufficient provenance for a scientific experiment.

Canonical ExperimentBundleV1

Define an immutable, content-addressed ExperimentBundleV1 that binds at least:

• experiment ID/version/hash;
• research question / hypothesis ID;
• repository commit/tree identity and branch only as descriptive metadata;
• execution environment/lock/SBOM identity (milestone(v2.9+): govern computational reproducibility, environment identity, and cross-platform numerical replay #637–test(v2.9+): certify bitwise, semantic, numerical, and statistical replay across supported environments #641);
• dataset/product identities and lineage roots (feat(v2.10): add end-to-end feature lineage DAG from immutable evidence to every final ML column #693);
• chronological/evidence-unit split policy and exact split artifact;
• feature registry/schema/projection identity (feat(v2.10): certify a complete wide-feature coverage manifest across bars, triangle, context, 1,000 strategies, flow, uncertainty, and labels #681/feat(v2.10): define scalable physical layout and projection contracts for the 10k+-column logical training substrate #689);
• preprocessing/imputation artifact identity (design(v2.10): freeze train-only preprocessing, normalization, encoding, and imputation-fit semantics for model-ready views #697);
• target/label identity (design(v2.10): freeze target/label interval semantics, overlap purging, and synthetic-member outcome identity #696);
• uncertainty/member/scenario policy (design(v2.10): propagate reconstruction/context/population uncertainty into wide ML features and counterfactual labels #709);
• historical-to-modern transport policy where used (research(v2.10): quantify historical-to-modern regime drift and qualify causal transport/reweighting for legacy evidence #710);
• model architecture/configuration and trained-weight hash;
• RNG semantic namespace/seeds where stochastic training is allowed;
• model calibration artifact;
• ModelDecisionPolicyV1 identity (design(v2.10+): freeze ML score-to-action policy, thresholding, sizing, abstention, and execution handoff #714) when evaluating trading actions;
• transaction-cost, financing, margin, execution and account-policy identities (design(v2.8/v2.10): freeze FX financing, carry, settlement-value-date, rollover, and swap-point semantics for traders and ML #703–design(v2.8): freeze transaction-cost, slippage, latency, partial-fill, and execution-quality semantics for synthetic traders #705);
• evaluation metric registry and version;
• protected-evaluation status and search-budget lineage (design(v2.10): govern protected model selection, repeated backtests, multiple testing, and synthetic-heavy research overfitting #698);
• domain/regime/support strata used for reporting (test(v2.10): quantify synthetic-to-observed/live domain shift across market, strategy, flow, and broker-style feature planes #699/design(v2.10): freeze causal regime/change-point state versus retrospective segmentation for transport, traders, and ML #711);
• produced metric/result artifacts and hashes;
• promotion/challenger/deployment links (ops(v2.10+): freeze model/policy promotion, effective-period registry, rollback, and live decision provenance #716/design(v2.10+): govern retraining succession, champion-challenger evaluation, forgetting checks, and controlled model retirement #717);
• terminal status: planned/running/failed/completed/invalidated/superseded.

The canonical bundle hash must be a deterministic function of canonical scientific inputs, not mutable labels such as branch name or human display title.

Identity math

Let the canonical ordered scientific component identities be

C = (d, s, f, p, y, u, m, c, q, e, v)

for dataset, split, feature schema, preprocessing, label, uncertainty/scenario policy, model, calibration, decision policy, execution/risk policy, and environment/code identity.

Define

experiment_id = SHA256(canonical(C, metric_registry, hypothesis_id, fit_policy)).

If any scientifically material component changes, the experiment identity must change.

Changing only non-scientific display metadata must not change the scientific experiment identity.

Result identity

For result artifact R produced by experiment E, bind

result_id = SHA256(experiment_id || canonical(metric_id, stratum_id, output_payload_hash)).

A metric value without its experiment/result identity is not sufficient release evidence.

Comparison / ablation contract

An ablation comparing experiment A and B must explicitly state the controlled-difference set

Delta(A,B) = {component fields whose identities differ}.

For a valid one-factor ablation, require the declared manipulated factor(s) to equal the observed component identity differences. Unexpected differences invalidate the causal attribution.

Examples:

• X versus X + strategy.* should differ in feature projection/schema and downstream trained model identity, while split/evaluation policy remains identical;
• same model under different design(v2.10+): freeze ML score-to-action policy, thresholding, sizing, abstention, and execution handoff #714 decision thresholds is a decision-policy ablation, not a model-performance ablation;
• raw historical versus research(v2.10): quantify historical-to-modern regime drift and qualify causal transport/reweighting for legacy evidence #710 transported data changes dataset/transport identity and must not be described as only a preprocessing change.

Container/reference fixture

Reference canonicalization fixture should prove:

1. same scientific components with dictionary/key insertion order changed => identical experiment hash;
2. one changed decision threshold/policy hash => different experiment hash;
3. changed display title only => unchanged scientific hash;
4. exact retry under same stochastic semantic namespace and deterministic replay class => same scientific identity, while separate run-attempt/receipt IDs may differ.

Run attempts versus experiment identity

Separate the scientific experiment from execution attempts.

Define ExperimentAttemptV1 with:

• experiment ID;
• attempt ID;
• start/end times;
• runtime/worker identities;
• logs/checkpoints;
• resource envelope;
• failure/retry reason;
• produced artifact hashes.

Retries do not automatically create a new scientific experiment if every scientific input remains identical. A retry that changes batch size in a numerically material way, code, environment, seed namespace, preprocessing or any scientific component creates a successor experiment identity.

Metric registry

Every durable evaluation metric must have a stable semantic identity including:

• formula/version;
• directionality;
• units;
• weighting/evidence-mass treatment;
• missing/censored handling;
• aggregation across pairs/regimes/members;
• confidence/uncertainty method where applicable.

Do not reuse sharpe, accuracy, drawdown, brier, domain_distance, etc. for materially different formulas under one metric identity.

Protected evaluation and search lineage

#698 search-budget governance must attach every explored candidate to an experiment identity, including failed/unfavorable candidates. A promoted/champion result must resolve to the complete ancestor search graph.

A protected holdout that was inspected and then used to redesign any component creates a successor research cycle; the original experiment receipt remains immutable.

Reproducible report generation

Human reports, tables and plots must be generated from the experiment/result registry rather than manually copied metric values. A report manifest must enumerate the exact result IDs it cites.

Thus a statement such as 'challenger improves post-cost utility by 1.4%' can be traced to:

• the two experiment IDs;
• metric identity;
• evaluation stratum;
• component difference set;
• exact result artifacts.

Deployment linkage

#716 promotion receipts must reference the exact qualifying experiment bundle(s). A live deployment cannot cite a generic model file as qualification evidence without the experiment/split/metric context that justified promotion.

#717 challenger comparisons must themselves be immutable comparison artifacts referencing champion/challenger experiment IDs.

Invalidated / superseded experiments

If a later audit discovers leakage, schema corruption, incorrect units, bad source provenance, or another scientific defect, mark the experiment invalidated with reason and successor link. Do not delete or rewrite the original result.

Downstream reports/deployments that depend on an invalidated experiment must become discoverable through reverse lineage.

Negative controls

Certification must detect:

• same experiment ID after dataset/split/feature/preprocessing/model/decision-policy change;
• different experiment ID caused only by JSON key order or display title;
• manually reported metric not tied to a result artifact;
• ablation whose supposedly controlled components differ unexpectedly;
• protected holdout reused for tuning without successor research-cycle identity;
• failed search candidates omitted from search-budget lineage;
• deployment references model hash but no qualifying experiment;
• retry changes scientific configuration but is logged as same attempt/experiment;
• old result silently overwritten after bug fix;
• metric semantic formula changes under same metric ID.

Acceptance

• one machine-readable ExperimentBundleV1, ExperimentAttemptV1, metric registry and comparison/ablation schema exist;
• deterministic canonicalization/hash fixtures pass independently;
• every major ML/synthetic-first campaign result resolves to an experiment and result ID;
• X-vs-X+Z, member/scenario, transport, strategy-bank, uncertainty and champion/challenger comparisons expose exact component-difference sets;
• protected search lineage includes unfavorable/failed candidates;
• promotion/deployment receipts reference qualifying experiment identities;
• invalidation propagates discoverably without rewriting retained evidence;
• feat(traceability): implement a canonical machine-readable room/archive-to-issue requirements ledger with orphan detection #691 records the requirement and docs(v2.5): publish an implemented-versus-evidenced capability and certification matrix #525 tracks planned/implemented/executed/scientifically-passed states separately.

[dmidlo]

Editorial ownership under #721/#724

This issue is shared public contract / private concrete instances. ExperimentBundleV1, attempt/result identity rules, metric semantic identity, ablation-difference contracts, invalidation/supersession semantics, and canonicalization fixtures are appropriate public interoperability contracts. Concrete proprietary experiment bundles, search ledgers, trained-model hashes, protected metrics, promotion evidence and model-result archives belong to the separate private ML repository. The public handoff should expose enough schema/version information for those private bundles to reference immutable public corpus identities; it must not contain the private experiment registry itself.