computational-ml-stem-problem-forge

A laboratory for original computationally intensive ML, statistical, and STEM problems: unambiguous formalization, a Python reference, an independent verifier, detection of subtly wrong but numerically plausible answers, reproducible ground truth, and audit of candidate solutions.

This is working notes for quantitative review of computational arguments, not a product, not a question bank, and not a claim of empirical discovery.

Author: Dr. Pavanam Thomas (GitHub, thomaspavanam@gmail.com).

The distinctive claim is independent verification: a reference solution is not accepted merely because it runs. Each problem states an object, a second calculation of a different invariant, and a third code path. Copying the reference module is not verification.

Research question

When a candidate answer is numerically plausible, what independent object would still detect that it is wrong?

Start here:

1. FLAGSHIP_CASE_STUDY.md — entity-intercept leakage from formulation through an incorrect candidate.
2. problems/ml/entity_group_leakage/ and problems/ml/nested_cv_optimism/ — laboratories with three ground-truth units each.
3. problems/adversarial/ — a leaked AUC that looks like 0.90, and an audit that names the earliest protocol failure.
4. docs/ground_truth_protocol.md and docs/adversarial_failure_taxonomy.md.
5. src/problemforge/ — path-based registry, schema, runner, audit. Twelve problems are complete.
6. tests/ — seed regeneration, schema, independence of verifier files, deliberate-failure detection, tolerance boundaries.

python -m pip install -e .
python -m pytest
python scripts/run_all.py
python -m problemforge list

Python 3.11 or newer.

Design

• Three ground-truth units, not one number. GT1 is the primary object. GT2 is an invariant that is a different computation (split occupancy, majority-classifier identities, translation of log-sum-exp, Gram condition number, Monte Carlo SE). GT3 is a different code path. Copying the reference module is not verification.
• Tolerance is a named policy. Coverage is judged by Monte Carlo SE, not phat == 0.95. KKT is a residual, not solver status. Boundaries are tested just inside and just outside.
• Leakage has sentinels that do not reuse the primary metric. Entity overlap counts. Source timestamps. Feature permutation while labels stay fixed.
• Audit reports the earliest substantive failure. A random split on grouped data is a split failure even if the quoted score is also inner CV.
• The registry is path-based. Adding problems/<domain>/<id>/problem.yaml is enough. No hand-maintained index.

Failure cases

These are required to keep failing in the documented way:

• Naive KFold on an entity-intercept DGP (entity_group_leakage).
• Inner-CV best score as generalisation (nested_cv_optimism).
• Majority accuracy with recall 0 at ~2% prevalence (imbalance_metrics_threshold).
• A leaked score with AUC in 0.85–0.95 (plausible_wrong_auc).
• iid bootstrap under clustering (bootstrap_dependence_trap).
• (X'X)^{-1}X'y on a Hilbert-like design (ill_conditioned_normal_equations).
• Naive log(sum(exp(x))) overflow (logsumexp_stability).

Locked by per-problem tests and docs/failures_and_corrections.md.

Independent verification

Every problem has independent_verifier.py imported from a qualified path so that twelve files named reference_solution.py do not collide. Repo-level tests assert that the two source files are not identical and that a deliberately broken reference fails invariants. SciPy log-sum-exp, Mann–Whitney AUC, analytic cluster-robust variance, NNLS KKT multipliers, and SVD least squares appear as GT3 paths.

Reproducibility

python -m pip install -e .
python -m pytest
python scripts/run_all.py
python -m problemforge validate all
python -m problemforge run entity_group_leakage
python -m problemforge audit ai_nested_cv_audit

Default seed 2026. Figures and outputs/tables/run_summary.csv are regenerable. Provenance: docs/data_policy.md, docs/reproducibility.md.

There is no observational dataset. Every draw is generated in code.

Known limitations

• The twelve DGPs are stylised. They check procedures. They are not models of a labour market, a clinic, or a trading book.
• Candidate audit reads declared protocol YAML. It does not parse arbitrary Python.
• Nested-CV optimism is shown for accuracy on one grid. It is not a universal constant.
• No result here is a causal finding. Predictive metrics are not identifying assumptions.
• Passing CI means the laboratory still runs. It is not a warranty for an applied study.

Remaining corpus bounds

GT2 must remain a different computation from GT1; an assignment gt2 = gt1 would make the independence tests fail. Nested-CV optimism is shown for accuracy on one grid, not for log-loss. The QP laboratory is inequality-only. Candidate audit reads protocol YAML and does not parse arbitrary Python. See ROADMAP.md.

Corpus (12 problems)

id	difficulty	object
ml/entity_group_leakage	EXPERT	naive vs GroupKFold on intercepts
ml/temporal_feature_leakage	EXPERT	rolling window includes t+1
ml/nested_cv_optimism	EXPERT	inner best vs nested outer
ml/imbalance_metrics_threshold	EXPERT	accuracy vs recall/PR-AUC at 2%
ml/calibration_vs_discrimination	EXPERT	AUC vs ECE
numerical/logsumexp_stability	L4	max-subtraction vs overflow
stats/monte_carlo_ci_coverage	L3	coverage judged by MC SE
stem/qp_kkt_verification	EXPERT	KKT residual, not status
adversarial/plausible_wrong_auc	ADVERSARIAL	leaked 0.9 AUC
adversarial/ai_nested_cv_audit	ADVERSARIAL	earliest protocol failure
stats/bootstrap_dependence_trap	L4	iid vs cluster bootstrap
numerical/ill_conditioned_normal_equations	ADVERSARIAL	Gram inverse vs QR

Repository structure

computational-ml-stem-problem-forge/
├── FLAGSHIP_CASE_STUDY.md
├── src/problemforge/
├── problems/<domain>/<id>/
├── docs/
├── scripts/run_all.py
├── tests/
└── .github/workflows/ci.yml

Related repositories

• statistical-reasoning-validation — probability identities and inferential mistakes.
• econometrics-causal-inference-lab — estimands when the question is causal.
• ai-response-evaluation-benchmarks — structured review of quantitative arguments.

Citation

See CITATION.cff. Licence: MIT, Copyright 2026 Dr. Pavanam Thomas.