Retro

Retro is a tabular foundation model that makes predictions in context, without any fine-tuning. Rather than only stacking forward transformations, later stages revisit and recombine representations built by earlier layers, so that predictive refinement is allocated across depth instead of concentrated in the last layers. This repository ships the Fourier-hybrid Attn-Res architecture, the two released checkpoints, and the scikit-learn style API needed to run them.

Paper: arXiv:2610.10317 · PDF

Retro lies on the Pareto frontier of TabArena, TALENT and RelArena.

Installation

pip install -e .
# CSV command line interface
pip install -e ".[cli]"

Python >= 3.10 and PyTorch >= 2.4 are required.

Quick start

from retro import RetroClassifier, RetroRegressor

clf = RetroClassifier()          # downloads retro_cls.ckpt from Hugging Face Hub
clf.fit(X_train, y_train)
pred = clf.predict(X_test)
proba = clf.predict_proba(X_test)

reg = RetroRegressor()           # downloads retro_reg.ckpt from Hugging Face Hub
reg.fit(X_train, y_train)
pred = reg.predict(X_test)

Pass model_path="retro_cls.ckpt" to load a local checkpoint, device="cpu" to force CPU inference, or n_estimators=... / batch_size=... to override the defaults below.

Command line

python infer.py --task cls \
    --train train.csv --test test.csv --target label \
    --output predictions.csv

python infer.py --task reg \
    --train train.csv --test test.csv --target target \
    --model-path /path/to/retro_reg.ckpt

--test must contain the same features as --train but not the target column. For classification the output also holds one proba_<label> column per class.

Default configuration

A plain RetroClassifier() or RetroRegressor() runs 32 ensemble members with the settings used for the released checkpoints.

Setting	Classification	Regression
n_estimators	32	32
ensemble_mode	global	global
norm_methods	none, power, quantile, asinh (diverse)	none, power
batch_size	8	8
feat_shuffle_method	latin	latin
class_shuffle_method	shift	—
use_amp	True	True

ensemble_mode="legacy" restores the older cross-branch reuse policy.

Checkpoints

Task	File	URL
Classification	retro_cls.ckpt	https://huggingface.co/LAMDA-Tabular/Retro/blob/main/retro_cls.ckpt
Regression	retro_reg.ckpt	https://huggingface.co/LAMDA-Tabular/Retro/blob/main/retro_reg.ckpt

Checkpoints are downloaded automatically and cached by huggingface_hub when model_path is omitted. The architecture is fixed by this package, so the model_impl label stored inside a checkpoint is read for diagnostics only; weights are loaded with strict=True and any structural mismatch raises an error.

Architecture

The Retro architecture. A table encoder produces row representations for a retrospective ICL stack, whose layers revisit a retained depth history.

The table encoder applies three column-processing blocks and three row-processing blocks (hidden size 128, eight attention heads). The column stage uses 128 inducing vectors, the row stage aggregates with four CLS tokens, and their outputs are concatenated into a 512-dimensional representation per row. The encoder keeps the repeated feature grouping, target-aware embeddings and query-aware scalable softmax (QASSMax) of the base table encoder.

Row representations, augmented with observed support labels, enter a 12-layer ICL stack with eight attention heads and a feed-forward expansion ratio of two. Inside the stack, retrospective aggregation constructs the inputs to the attention and feed-forward sublayers, gated attention modulates contextual outputs before projection, and residual updates are grouped over pairs of layers. The final aggregation combines the initial encoding with six completed update groups, and a two-layer MLP with hidden dimension 1,024 produces query predictions. Classification and regression use separately trained models with task-specific target embeddings and output heads.

Results

Elo rankings across three benchmarks. Each panel shows the five highest-rated methods; candidate sets and scoring protocols differ, so ratings should be compared within panels.

Complete rankings, ablations and analyses are reported in the paper.

Layout

src/retro/
  _model/                          column embedding, row interaction, ICL stack
  _model_attn_res/                 Attn-Res residual routing
  _model_fourier_hybrid/           hybrid raw + Fourier cell embedding
  _model_gated/                    element-wise gated ICL attention
  _model_fourier_hybrid_attn_res/  the Retro architecture used by both checkpoints
  _sklearn/                        sklearn estimators, ensemble/multiview logic
infer.py                           CSV command line entry point

Citation

@misc{cai2026retro,
  title         = {Thinking in Depth: Retrospective Inference for Tabular Foundation Models},
  author        = {Cai, Hao-Run and Liu, Si-Yang and Cheng, Zi-Jian and Yu, Kun-Yang
                   and Sheng, Jin-Hao and Yu, Guo and Liu, Chonghan and Zhou, Zhi
                   and Jiang, Jun-Peng and Guo, Lan-Zhe and Ye, Han-Jia},
  year          = {2026},
  eprint        = {2610.10317},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2610.10317}
}

Acknowledgements

We thank the authors of TabICL, TabArena, TALENT and RelArena for their open-source models, benchmarks and evaluation tools.