design(v2.10+): freeze learned-decision attribution, grouped explainability, and causal-nonclaim semantics

[dmidlo]

Parents: #605 #714 #716 #718
Related: #660 #681 #689 #690 #693 #697 #699 #709 #712 #713 #717

Problem

The project now has strong identities for datasets, features, preprocessing, models, decision policies, live deployments, experiments, and rollback. A remaining gap is scientific explainability of a learned prediction/action.

A model explanation is not automatically a causal explanation of the market. Feature attribution methods can be unstable under correlated features, duplicated strategy proxies, multitimeframe collinearity, preprocessing transformations, and out-of-domain inputs. The project therefore needs an explicit attribution contract that preserves model semantics, feature-group structure, uncertainty, and the distinction between predictive attribution and causal claims.

Canonical attribution object

Define immutable DecisionAttributionV1 carrying at least:

• experiment/model/deployment IDs (design(v2.10+): freeze immutable experiment bundle identity for reproducible training, evaluation, ablation, and promotion evidence #718/ops(v2.10+): freeze model/policy promotion, effective-period registry, rollback, and live decision provenance #716);
• exact feature snapshot/projection/preprocessing identities;
• model raw output and calibrated output;
• explanation method/policy ID and version;
• baseline/background/reference-domain identity;
• explained output dimension/class/horizon;
• per-feature attribution where supported;
• grouped attribution by feature family/namespace;
• interaction attribution where supported;
• approximation/error/convergence diagnostics;
• correlated-feature handling policy;
• unsupported/out-of-domain state;
• downstream design(v2.10+): freeze ML score-to-action policy, thresholding, sizing, abstention, and execution handoff #714 decision-policy transformations and active constraints;
• final action/order reason decomposition;
• lineage root under feat(v2.10): add end-to-end feature lineage DAG from immutable evidence to every final ML column #693.

Changing explanation method, background distribution, grouping, approximation budget, or correlated-feature policy creates a successor attribution identity.

Predictive attribution versus causal explanation

Retain the nonclaim:

model attribution != causal effect in the historical market.

A feature may receive large attribution because the trained model relies on it. This does not prove changing that feature in the real market would cause the predicted outcome.

Do not label SHAP/gradient/permutation/ablation contributions as causal unless a separately identified causal design/estimand justifies that claim.

Additive-reference contract

For additive explanation methods with baseline f_0, require where mathematically applicable

f(x) - f_0 = sum_i phi_i

within frozen numerical tolerance.

Container-vetted two-feature fixture:
f(x1,x2)=3*x1 + 2*x2 + 4*x1*x2, baseline (0,0), explained point (1,1).
Coalition values are v(empty)=0, v({1})=3, v({2})=2, v({1,2})=9.
Two-player Shapley values are

phi_1 = .5[(3-0)+(9-2)] = 5
phi_2 = .5[(2-0)+(9-3)] = 4

so phi_1+phi_2=9=f(1,1)-f(0,0).
Production certification must use the exact admitted explainer equations and background semantics; this fixture validates the additive-accounting pattern only.

Grouped attribution for the wide substrate

The final logical schema can exceed 10k columns, many of which are strongly related. Support hierarchical groups such as:

• raw/market microstructure;
• 1m/5m/15m/30m/1h/4h/1d bars and indicators;
• triangle/currency-graph;
• calendar/macro/forecast/positioning;
• strategy.FXES-* grouped by strategy, family and primitive/dependency class;
• synthetic_flow.* / population state;
• broker-style/transport;
• regime state;
• uncertainty/support;
• missingness/imputation masks.

Where group attribution is derived from feature-level values, preserve the aggregation rule. Where the explanation method operates directly on groups, that is a distinct semantic identity.

A family with many near-duplicate columns must not look more important merely because attribution is summed over more redundant dimensions without concentration/effective-dimension reporting (#690).

Correlated-feature ambiguity

Feature attributions can redistribute credit among correlated substitutes. Require diagnostics such as:

• attribution stability across admissible background/reference samples;
• feature-group rather than only singleton attribution;
• correlated-cluster attribution or conditional explainer policy where scientifically supported;
• effective-dimension/concentration diagnostics;
• sensitivity to removing/permuting one correlated sibling;
• explicit attribution_nonidentifiable state when credit allocation is not stable enough to interpret at singleton level.

Do not force an exact singleton ranking when the model only identifies a correlated group.

Strategy-bank attribution

If a learned model consumes the 1,000 strategy.FXES-* response blocks, support attribution at several levels:

1. exact FXES ID;
2. strategy family;
3. declared input-dependency class;
4. population/ensemble aggregate.

A large contribution from strategy.FXES-123.signal means the learned model relied on that proxy feature; it does not mean FXES-123 historically caused market movement or that the downstream learned trade should be attributed to that fictional trader.

Preprocessing inversion / transformed features

Attribution must state the semantic space in which it is computed:

• raw logical feature space;
• normalized/transformed feature space;
• PCA/latent-component space;
• grouped source-feature space.

If a transform is invertible or has a qualified attribution mapping, expose it explicitly. Do not present a PCA-component attribution as if it were an exact raw-feature attribution without a defined mapping.

Imputed values must retain missingness/imputation state; a large attribution on an imputed value may actually indicate the model is exploiting the imputation/mask mechanism.

Decision decomposition beyond the model

A final live action must distinguish at least:

1. model raw score attribution;
2. calibration transformation;
3. design(v2.10+): freeze ML score-to-action policy, thresholding, sizing, abstention, and execution handoff #714 threshold/dead-band/cost gate;
4. sizing/turnover/hysteresis;
5. portfolio/currency-risk projection;
6. design(v2.8): freeze leverage, margin, maintenance, liquidation, and account-solvency semantics for synthetic traders #704 margin/risk constraints;
7. test(v2.10): govern live feature/model drift monitoring, calibration decay, and inference abstention after deployment #713 validity/abstention;
8. design(v2.8): freeze transaction-cost, slippage, latency, partial-fill, and execution-quality semantics for synthetic traders #705 execution outcome.

Example: model may strongly favor long EURUSD, but final target can be zero because expected costs exceed edge or margin policy rejects the order. The explanation must not say model predicted neutral in that case.

Counterfactual / perturbation explanations

Any counterfactual explanation x -> x' must respect the feature manifold and causal/time semantics. Do not perturb impossible combinations such as:

• inconsistent triangle legs;
• future macro values at an earlier cutoff;
• incompatible session/calendar state;
• impossible universe membership;
• raw feature changed while dependent derived features remain frozen inconsistently.

Counterfactual explanations should preferably operate through approved feature-generation/intervention policies rather than arbitrary independent column edits.

Global explanation reports

Global importance must be stratified by pair, regime, era, observed/synthetic domain, support, session, and live broker state where relevant. One global average can hide sign reversals or importance drift.

Report at minimum:

• mean/median absolute attribution by group;
• signed attribution distribution;
• concentration/effective number of contributing groups;
• stability across eras/regimes;
• observed-versus-synthetic attribution differences;
• live drift of attribution distributions where monitored under test(v2.10): govern live feature/model drift monitoring, calibration decay, and inference abstention after deployment #713.

Attribution concentration

For normalized nonnegative absolute group contributions a_g, report

HHI_attr = sum_g a_g^2

and effective contributing groups

N_eff_attr = 1 / HHI_attr.

This is a concentration diagnostic, not proof of causal independence.

Experiment and deployment linkage

#718 experiment results may include attribution artifacts but the explainer identity must be frozen in the experiment/result lineage when used for scientific claims.

#716 live actions should be able to resolve to the attribution artifact/policy used for post-hoc explanation when such explanation is generated. Generating an explanation later does not change the historical decision identity.

Negative controls

Certification must fail/flag:

• attribution sum does not reconcile to explained output for an additive explainer beyond tolerance;
• explanation background includes future/protected information improperly;
• correlated strategy proxies receive unstable singleton rankings presented as precise fact;
• model attribution described as causal market effect;
• attribution computed in transformed space but labeled as raw-feature contribution;
• imputed value attribution reported without missing/imputation state;
• final zero action attributed to neutral model score when the real cause was design(v2.10+): freeze ML score-to-action policy, thresholding, sizing, abstention, and execution handoff #714/design(v2.8): freeze leverage, margin, maintenance, liquidation, and account-solvency semantics for synthetic traders #704/test(v2.10): govern live feature/model drift monitoring, calibration decay, and inference abstention after deployment #713 gating;
• counterfactual perturbation breaks triangle/unit/time/universe constraints;
• live explanation generated with a different model/preprocessing/schema than the deployment action;
• attribution method/background changes under same attribution ID.

Acceptance

• one versioned DecisionAttributionV1 / explanation-policy registry exists;
• independent additive/Shapley-style reference fixtures pass for admitted explainers;
• grouped attribution covers all major wide-feature planes and v33 family/strategy levels;
• correlated-feature ambiguity is measured and can produce group-level/nonidentified states;
• model-score explanation is separated from decision/risk/execution explanation;
• causal-nonclaim language is enforced in artifacts/reports;
• counterfactual explanations respect point-in-time/unit/triangle/universe constraints;
• experiment/deployment lineage can resolve exact explanation identities;
• feat(traceability): implement a canonical machine-readable room/archive-to-issue requirements ledger with orphan detection #691 records the requirement and docs(v2.5): publish an implemented-versus-evidenced capability and certification matrix #525 tracks planned/implemented/executed/scientifically-passed states separately.

[dmidlo]

Editorial ownership under #721/#724

This issue is shared public contract / private model-specific artifacts. Public histdata.com-tools may define the neutral attribution schema, feature-group taxonomy, causal-nonclaim rules, lineage requirements, correlated-feature ambiguity states, and reference accounting fixtures. Actual model-specific attribution values, backgrounds, explanation runs, production decision explanations and global attribution reports belong to the separate private ML repository. Those private artifacts are not handoff blockers; the public requirement is only that downstream explanations can reference immutable public feature/lineage identities without misrepresenting predictive attribution as causal market evidence.