statistical-reasoning-validation

Probability, statistical inference, Monte Carlo simulation, diagnostics, and quantitative error analysis.

This repository is a statistical laboratory. It checks probability identities against closed forms, estimates sampling behaviour by simulation, and records common inferential mistakes with corrected statements. It is working notes for quantitative review, not a product, a dashboard, or a claim of empirical discovery.

Author: Dr. Pavanam Thomas (GitHub, thomaspavanam@gmail.com).

Reading order

Inspect, in this order:

1. docs/statistical_error_catalogue.md — forty error patterns, each with an incorrect claim, why it fails, a corrected statement, and a small example.
2. docs/failures_and_corrections.md — failures the tests are required to keep visible.
3. ROADMAP.md — remaining bounds (issues #3–#5 are closed).
4. tests/ — property checks (Bayes arithmetic, LLN MSE, interval coverage, bootstrap seeds, omitted-variable bias direction, robust versus classical SEs, accuracy versus recall, power).
5. src/statval/testing.py and src/statval/monte_carlo.py — p-value misuse demonstration; heteroskedasticity and HC1/HC3 coverage simulation.
6. notebooks/01_coverage_and_power.ipynb — coverage of a 95% mean interval and a power curve, both labelled as simulations.

Reproduce from a clean clone:

python -m pip install -e .
python -m pytest
python scripts/run_all.py

Python 3.11 or newer.

What problem this repository addresses

Quantitative claims fail in recurring ways: a p-value is read as a posterior, a wide interval is treated as “no effect,” a high R² is treated as identification, accuracy is quoted on a rare-event task, conventional standard errors are trusted under heteroskedasticity. The laboratory makes those failures visible under known data-generating processes, where the truth is available and the mistake can be named.

What is checked

• Probability: discrete random variables, expectation, variance, covariance, conditional probability, Bayes’ rule.
• Sampling: law of large numbers, central limit theorem, Monte Carlo sampling variability.
• Inference: hypothesis tests, Type I/II error, power, effect size versus significance, confidence-interval coverage.
• Resampling: iid percentile bootstrap of a mean and a median, BCa interval for a scalar statistic, and pairs bootstrap of an OLS slope, with a reproducible generator.
• Regression diagnostics: omitted-variable bias, nonlinear misspecification, influential observations, heteroskedasticity, multicollinearity.
• Classification metrics: confusion-matrix denominators, threshold trade-offs, calibration, why accuracy misleads when classes are unbalanced.
• Research-software practice: tests that check statistical properties, a single reproduction script, GitHub Actions.

Methodology

Identities are checked against closed forms. Sampling behaviour is estimated by Monte Carlo under a stated DGP. Interpretation is restricted to that DGP.

Methods implemented

Area	What is implemented	How it is checked
Probability	Finite-support moments; binary Bayes posterior	Closed form; population simulation
Sampling	MSE of the mean; standardised means from a skewed parent	MSE declines with n; CLT location/scale
Tests	One-sample t p-values; Type I rate; power curve; misuse demo	Type I near nominal; power rises with n and with the mean shift
Intervals	Student-t mean interval	Monte Carlo coverage near 95% under a Normal DGP
Bootstrap	Percentile intervals for the mean and the median; BCa interval for a scalar statistic; pairs bootstrap of an OLS slope	Identical seed ⇒ identical percentile replicates; BCa is not an alias of the percentile interval on a skewed mean
Multiple testing	Bonferroni FWER simulation	Unadjusted family-wise error exceeds Bonferroni on the designed battery
Monte Carlo	Bias/variance/MSE of the mean; OLS under heteroskedasticity	Classical versus HC1/HC3 SE and coverage
Regression	Deliberate OVB, quadratic, leverage, collinear, hetero DGPs	Bias sign; residual–x² correlation; Cook’s distance; VIF
Classification	Accuracy, precision, recall, specificity, F1, thresholds, reliability	Majority classifier: high accuracy, recall 0

All Monte Carlo studies are simulations. They describe sampling behaviour under a known DGP. They are not estimates for a real population.

Data

There is no observational dataset in this repository. Every draw is generated in code. Provenance, licence implications, and regeneration rules are in docs/data_policy.md.

Assumptions that matter

• Default sampling models are iid unless a DGP says otherwise.
• The Student-t mean interval is checked under normality, which is the textbook sampling model of that interval — not a general licence for non-Normal data.
• HC1/HC3 change the variance estimator for the same OLS projection; they do not repair omitted variables or dependence.
• The iid bootstrap mimics iid sampling. It is not offered as a fix for clustered or time-series data.
• Classification illustrations include an in-sample logistic fit used to show threshold geometry, not to claim out-of-sample performance.
• Default seed is 2026 unless a test uses a second seed so that two simulation arms are independent.

Repository structure

statistical-reasoning-validation/
├── README.md
├── LICENSE
├── CITATION.cff
├── pyproject.toml
├── docs/
│   ├── statistical_error_catalogue.md
│   └── data_policy.md
├── src/statval/
├── scripts/run_all.py
├── tests/
├── notebooks/01_coverage_and_power.ipynb
├── outputs/          # regenerable figures and tables
└── .github/workflows/ci.yml

Reproducibility

python -m pip install -e .
python -m pytest
python scripts/run_all.py

scripts/run_all.py writes figures under outputs/figures/ and a numeric summary to outputs/tables/run_summary.csv. Those files are regenerable and are not the source of truth; the source of truth is the code plus the tests.

CI installs the package, runs pytest, and runs scripts/run_all.py with a non-interactive matplotlib backend.

Tests and validation

Tests check properties, not merely that a function returns an object:

• Bayes posterior matches the closed form; a large simulated population tracks it.
• MSE of the sample mean decreases as n increases.
• Monte Carlo coverage of the 95% t-interval lies near the nominal rate, within Monte Carlo error.
• Bootstrap replicates are deterministic given a seed.
• Omitted-variable OLS is biased in the direction implied by Cov(X, Z) and the omitted slope.
• Under a heteroskedastic DGP, HC1/HC3 standard errors differ from the classical SE (HC3 is larger in that design).
• On an imbalanced classification DGP, accuracy is high while recall is poor.
• Power increases with n and with effect size.

Example outputs

Run python scripts/run_all.py for the current numbers on your machine. The following identities do not depend on a Monte Carlo draw:

• Screening example: P(H) = 0.01, P(E|H) = 0.99, P(E|not H) = 0.05 ⇒ P(H|E) = 0.0099 / 0.0594 ≈ 0.1667.
• For iid observations with variance σ², MSE of the sample mean is σ²/n.
• Majority classifier on prevalence 5%: accuracy = 0.95, recall = 0.

Simulation quantities (coverage, power, bias) are printed by the script with their Monte Carlo sample sizes. They should not be copied into a paper as if they were experimental measurements.

Limitations

• The DGPs are stylised. They are tools for checking procedures, not models of a labour market, a clinic, or a trading book.
• Finite-sample coverage of robust standard errors is itself a Monte Carlo fact, not an asymptotic theorem demonstrated here.
• The catalogue is a review aid. It is not a complete logic of statistical mistakes, and it does not replace a design document for a real study.
• No result in this repository is a causal finding about the world. Related identification work lives in a separate laboratory (link below).
• Figures produced from an in-sample classifier illustrate metrics; they are not a model-validation report.

Related repositories

• econometrics-causal-inference-lab — estimands, identifying assumptions, and diagnostics when the question is causal.
• ai-response-evaluation-benchmarks — structured review of quantitative arguments, including statistical failure modes catalogued here.

Citation

See CITATION.cff. Licence: MIT, Copyright 2026 Dr. Pavanam Thomas.