A three-tier evaluation strategy — credible public benchmarks, fast CI smoke datasets, and agentic capability suites — fed by pluggable dataset loaders, with every published number produced by a bundled, runnable harness.
Firefly DataScience separates how we prove the framework is good from how we load data day-to-day. The same DatasetLoaderPort that powers a quick iris smoke test in CI also pulls real OpenML benchmark suites for credibility runs. This page describes the evaluation strategy, shows how to load datasets through the loaders, and lists the real, reproducible results. Every figure here was produced by running a script in benchmarks/ with no manual tuning — see the full table in benchmarks/RESULTS.md.
!!! firefly "The recurring thesis — the LLM proposes; the classical engine decides"
GenAI proposes feature code; a deterministic classical engine measures the cross-validated lift;
and a **cost/benefit gate keeps only what is proven on the data**. That is why the GenAI ablation
below can only improve or stay neutral **on the train-CV metric the gate optimizes** (holdout scores
are reported as measured). The benchmarks measure both the classical
core and the gated accelerator on the same footing.
| Tier | Purpose | Sources | When it runs |
|---|---|---|---|
| Tier 1 — Credibility | Compare against the literature on standard suites | AMLB, OpenML‑CC18, OpenML‑CTR23 | Offline / scheduled (network) |
| Tier 2 — CI smoke | Fast, deterministic, no-network correctness | breast_cancer, iris, wine, digits, diabetes, california_housing |
Every PR |
| Tier 3 — Agentic capability | Measure end-to-end agent problem solving | MLE‑bench, DSBench | Periodic, sandboxed |
Tier 2 is the only tier that runs without network access, which is why it backs the default CI gate. Tiers 1 and 3 are the roadmap — they define how the framework earns external credibility over time.
Two loaders ship today, both implementing DatasetLoaderPort (name, can_load, load).
=== "Tier 2 — scikit-learn (offline, no network)"
`SklearnDatasetLoader` resolves bare names or `sklearn:`-prefixed names against scikit-learn's
bundled datasets. No download, fully deterministic.
```python
from fireflyframework_datascience.datasets.adapters import SklearnDatasetLoader
loader = SklearnDatasetLoader()
loader.can_load("breast_cancer") # True
loader.can_load("sklearn:diabetes") # True (prefix is stripped)
ds = loader.load("breast_cancer")
print(ds.name, ds.task, ds.n_rows, ds.n_features)
# breast_cancer TaskType.BINARY 569 30
```
The built-in Tier 2 names map to fixed task types:
```python
# binary -> breast_cancer
# multiclass -> iris, wine, digits
# regression -> diabetes, california_housing
```
Each `load` returns a `Dataset` dataclass with `X`, `y`, `task`, `target_name`, `feature_names`,
and a `metadata` dict (`source`, `n_rows`, `n_features`).
=== "Tier 1 — OpenML (benchmark suites, network)"
`OpenMLDatasetLoader` fetches by numeric id or by name using the `openml:` prefix. It needs the
`data` extra (`openml`) and network access; without the extra it raises `AdapterUnavailableError`.
```python
from fireflyframework_datascience.datasets.adapters import OpenMLDatasetLoader
loader = OpenMLDatasetLoader()
loader.can_load("openml:31") # True
loader.can_load("breast_cancer") # False (no openml: prefix)
ds = loader.load("openml:31") # by id, e.g. the 'credit-g' task
ds = loader.load("openml:credit-g") # by name
ds = loader.load("openml:31", target="class") # override the default target
print(ds.metadata["openml_id"], ds.task)
```
OpenML dataset ids are how Tier 1 suites (OpenML‑CC18, OpenML‑CTR23, AMLB) are addressed — each
suite is a curated set of these ids, so a credibility run is "load each id, fit, score, compare".
Install the extra:
```bash
pip install "fireflyframework-datascience[data]"
```
Dataset carries split and feature helpers used by the rest of the framework:
ds = SklearnDatasetLoader().load("iris")
train, test = ds.train_test_split(test_size=0.25, random_state=42) # (1)!
print(train.name, test.name) # iris[train] iris[test]
ds.has_target # True
ds.task.is_classification() # True- Classification targets are stratified automatically, so each split preserves the class balance — important on the small (~1000-row) datasets used in the Tier‑1 runs below.
When the target is unknown (OpenML without a declared task type), the loader infers it:
from fireflyframework_datascience.datasets import infer_task
infer_task([0, 1, 1, 0]) # TaskType.BINARY
infer_task([0.1, 2.3, 9.9, 4.2, ...]) # TaskType.REGRESSION (float, many uniques)When scikit-learn is on the path, the loaders are registered as beans automatically — no manual wiring. The OpenML bean only appears when openml is also importable.
# DatasetsAutoConfiguration registers:
# sklearn_dataset_loader (conditional_on_class "sklearn")
# openml_dataset_loader (conditional_on_class "openml")Both beans are typed as DatasetLoaderPort, so downstream code can depend on the port and let can_load route a source string to the right loader.
Tier 3 measures the agent, not a single estimator: given a task description and raw data, can the system produce a working, scoring solution end to end? The target suites are MLE‑bench and DSBench. These run in a sandbox on a periodic schedule rather than per-PR. As they land, they reuse the same DatasetLoaderPort contract — a new loader (e.g. a mlebench: adapter) plugs in exactly like SklearnDatasetLoader and OpenMLDatasetLoader without changing callers.
These are produced by running the harnesses — fixed random_state=0, default trainers, no manual tuning. Full table and reproduction steps: benchmarks/RESULTS.md.
To reproduce locally:
uv sync --extra tabular --extra data --extra validation
uv run python benchmarks/automl_benchmark.py # Tier-2 (offline, no network)
uv run python benchmarks/amlb_benchmark.py # Tier-1 (OpenML, needs network)!!! success "Expected — Tier-2 offline suite (automl_benchmark.py)"
`AutoML(cv=3, n_trials=1)` — the untuned default arm — over the default trainers (RandomForest,
Linear, HistGradientBoosting; + XGBoost / LightGBM / CatBoost when installed). The greedy OOF-measured
ensemble is default-on (`ensemble="auto"`, served only when it wins) and **TabPFN** joins the
leaderboard when it is installed and its weights are cached (≤ 1000 rows on CPU). Runs in seconds to
a minute, no network.
| Dataset | Task | Metric | CV | Holdout | Winner | Seconds |
|---|---|---|---:|---:|---|---:|
| breast_cancer | binary | roc_auc | 0.9972 | **0.9897** | greedy_weighted_ensemble | 56.8 |
| iris | multiclass | accuracy | 0.9557 | **1.0000** | tabpfn | 5.6 |
| wine | multiclass | accuracy | 1.0000 | **1.0000** | greedy_weighted_ensemble | 10.2 |
| diabetes | regression | rmse | −54.74 | **56.64** | greedy_weighted_ensemble | 2.8 |
| california_housing | regression | rmse | −0.454 | **0.442** | greedy_weighted_ensemble | 21.6 |
amlb_benchmark.py runs AutoML(cv=5, n_trials=1) — the untuned default arm — across real OpenML tasks with genuine categorical data (e.g. credit-g), exercising the dtype-aware preprocessing and string-target encoding. Holdout ROC-AUC:
| OpenML id | Dataset | CV | Holdout | Winner |
|---|---|---|---|---|
| 31 | credit-g | 0.7889 | 0.824 | greedy_weighted_ensemble |
| 37 | diabetes | 0.8389 | 0.868 | tabpfn |
| 1464 | blood-transfusion | 0.7697 | 0.736 | greedy_weighted_ensemble |
| 1480 | ilpd | 0.7592 | 0.778 | greedy_weighted_ensemble |
Out of the box, on real data with categorical features. Direct comparisons are claimed only where a rival actually ran: see the FLAML arm in the nested-CV evaluation below.
!!! note "On real finance & retail data (samples/industry_showcase.py)"
German credit risk (`credit-g`) reached **0.82** holdout ROC-AUC and bank-marketing campaign
conversion reached **0.92** holdout ROC-AUC (prior-cycle sample runs) — each a full
load → validate → AutoML → evaluate run on public OpenML data, no Kaggle account required.
Rerun `samples/industry_showcase.py` to reproduce current-engine numbers.
benchmarks/scientific_eval.py uses nested 5-fold CV (an inner CV selects the model on each outer fold's training data only; the untouched outer fold gives the unbiased estimate). Firefly AutoML is compared against a fixed LogReg, RandomForest and XGBoost, a tuned XGBoost at the same trial budget (n_trials=20), and — when flaml is installed — FLAML at a wall-clock budget matched to Firefly's measured fit time. The unit of analysis is the dataset: the five fold scores are aggregated to one value per dataset per system (Demšar 2006), a one-sided Wilcoxon signed-rank test runs across datasets, and Friedman + Holm handle the multi-system comparison. (The previous pooled-fold Wilcoxon over 25 correlated deltas was retired as statistically invalid.)
Nested 5-fold CV · ROC-AUC (mean ± std) · unbiased (selection on inner CV only) · n_trials=20 · FLAML arm on (matched wall-clock)
dataset LogReg RandomForest XGBoost XGBoost (tuned) Firefly AutoML FLAML Firefly picks
------------------------------------------------------------------------------------------------------------------------------------------------
credit-g 0.7913±0.024 0.7902±0.030 0.7761±0.032 0.8021±0.031 0.8036±0.025 0.8021±0.016 greedy_weighted_ensemble×5
phoneme 0.8115±0.013 0.9639±0.005 0.9573±0.003 0.9603±0.003 0.9651±0.004 0.9708±0.004 greedy_weighted_ensemble×5
diabetes 0.8337±0.028 0.8194±0.029 0.7836±0.024 0.8317±0.028 0.8370±0.028 0.8206±0.038 greedy_weighted_ensemble×5
ilpd 0.7494±0.032 0.7701±0.046 0.7292±0.024 0.7417±0.033 0.7588±0.030 0.7357±0.028 greedy_weighted_ensemble×5
blood-transfusion 0.7521±0.031 0.6802±0.068 0.6873±0.039 0.7278±0.049 0.7514±0.034 0.7251±0.041 greedy_weighted_ensemble×4, linear×1
Per-dataset statistics over n=5 datasets (one nested-CV mean per dataset per system)
Firefly AutoML vs LogReg mean Δ=+0.0356 | wins 4 / ties 1 / losses 0 | one-sided Wilcoxon (n=5) p=0.0625
Firefly AutoML vs RandomForest mean Δ=+0.0184 | wins 4 / ties 0 / losses 1 | one-sided Wilcoxon (n=5) p=0.09375
Firefly AutoML vs XGBoost mean Δ=+0.0365 | wins 5 / ties 0 / losses 0 | one-sided Wilcoxon (n=5) p=0.03125
Firefly AutoML vs XGBoost (tuned) mean Δ=+0.0105 | wins 5 / ties 0 / losses 0 | one-sided Wilcoxon (n=5) p=0.03125
Firefly AutoML vs FLAML mean Δ=+0.0123 | wins 4 / ties 0 / losses 1 | one-sided Wilcoxon (n=5) p=0.09375
Friedman p=0.03363 · mean ranks (1=best): LogReg=3.20, RandomForest=4.00, XGBoost=5.60, XGBoost (tuned)=3.40, Firefly AutoML=1.60, FLAML=3.20
Honest reading. Firefly AutoML earns the best mean rank (1.60 of 6 systems, Friedman p=0.034). It beats the tuned XGBoost on 5/5 datasets (one-sided Wilcoxon p=0.031), the untuned XGBoost 5/5 (p=0.031), and FLAML on 4/5 at matched wall-clock — adapting per dataset (an ensemble on non-linear phoneme, linear where linear genuinely wins on blood-transfusion). At n=5 datasets the power is limited: the per-comparison Wilcoxon tests are one-sided and the Holm-adjusted pairwise tests are not significant. We report the full per-dataset table rather than lean on one headline.
!!! note "Why nested CV"
An AutoML system that reports the cross-validated score of the model it *selected* is
optimistically biased — it is the maximum over many models scored on the same folds. Nested CV
removes that bias: model selection happens on the inner CV of each outer fold's training data, and
the outer fold — never seen during selection — gives the honest estimate.
benchmarks/genai_value.py isolates the GenAI contribution on a retail "high-value customer" task whose true driver (revenue = unit_price × units) is withheld from the model — a multiplicative interaction a linear learner cannot derive on its own. Four systems, 8 repeated train/test splits, real anthropic:claude-haiku-4-5:
| System | ROC-AUC (mean ± std) |
|---|---|
| linear (raw) | 0.9752 ± 0.006 |
| linear + GenAI | 0.9957 ± 0.002 |
| Firefly AutoML (raw) | 0.9969 ± 0.001 |
| Firefly AutoML + GenAI | 0.9969 ± 0.001 |
GenAI feature engineering lifts the linear model by +0.0205 ROC-AUC (0.975 → 0.996, Wilcoxon p = 0.0039 over 8 repeated splits of one dataset) — the LLM proposed and the serve-safe gate accepted multiplicative revenue features, rediscovering the withheld revenue = unit_price × units driver from the schema alone. On Firefly's tree-based AutoML the change is ≈0 (−0.0001): trees already approximate the interaction, so there is nothing left to add — and the gate accepts only measured train-CV wins. On real data (OpenML credit-g, semantic column names) the same serve-safe pipeline lifts Firefly from 0.8041 → 0.8105 ROC-AUC. Cost is reported from the agentic metering only — on this run the harness printed "metering unavailable", so no cost figure is claimed.
!!! tip "Pareto-safe accelerator"
GenAI feature engineering adds measurable, significant value where the data has structure a model
cannot reach on its own, surfaces interpretable domain features, and is gated to keep only measured
train-CV wins (holdout results are reported as measured). See [GenAI features](genai-features.md)
and the [agentic loop](agentic-loop.md) for the propose-measure-gate mechanism.