[codelion]

Summary

Version 0.3.0. It bundles accuracy fixes (which change default predictions), new features for continual learning in production, packaging and CI work, and a much larger test suite. CHANGELOG.md has the full list; this is the shape of it.

Default behaviour changes (predictions change)

With fewer than 10 examples per class the default config scored 0.55-0.73 on perfectly separable synthetic data, where prototypes alone scored 0.96-1.0. Experiments traced it to three causes, all fixed:

• The neural head was barely trained (~10 optimiser steps, stopped at the first plateau). It could not learn a 2-D XOR from 120 examples (0.49-0.70). It now trains for at least head_steps (300) steps and solves it (1.00).
• Prototype scores were over-flattened (softmax(exp(-d)): nearest class ~0.45 vs ~0.21), so a confident head always outvoted them. Now softmax(-d / prototype_temperature), default 0.25.
• A fixed 0.3/0.7 split below 10 examples (head-heavy) created an accuracy cliff. The head's weight now ramps up from zero with the number of examples.

Classifiers saved by an earlier version keep their old scoring when loaded. prototype_temperature=None and fixed new_class_* weights restore the old behaviour.

Trade-offs: a class added with only a few examples is recognised less readily than before, because the old split biased the head toward new classes at the expense of existing ones. add_examples is slower on small memories (roughly 1-2 s vs ~0.1 s); head_steps is configurable.

Real-data check: 0.2.0 vs 0.3.0 defaults, adding classes one at a time

MiniLM encoder, ag_news (start with 2 classes) and emotion (start with 3), 16 examples per starting class, remaining classes added one at a time, 3 seeds, 600 test examples, identical few-shot examples in both runs. It is a trade-off, not a pure win:

	new classes have 4 examples	new classes have 16 examples
final accuracy, all classes	ag_news 0.604 -> 0.757; emotion 0.296 -> 0.351	ag_news 0.773 -> 0.762; emotion 0.379 -> 0.381 (no change within noise)
old-class accuracy lost when a class is added	ag_news 0.242 -> 0.063; emotion 0.203 -> 0.084	ag_news 0.107 -> 0.152; emotion 0.112 -> 0.135 (slightly worse)
recall on the class just added	ag_news 0.929 -> 0.766; emotion 0.720 -> 0.395 (worse)	ag_news 0.918 -> 0.928; emotion 0.611 -> 0.661

Pooled over the 12 configurations, final accuracy improves by 0.05 on average (better in 8 of 12). So: clearly better when new classes arrive with few examples (far less forgetting, much better overall accuracy) at the price of the new class being under-recognised; neutral overall when classes arrive well-populated, with somewhat more old-class loss. With 3 seeds and 2 datasets this is suggestive, not conclusive.

New features

• Calibration: calibrate() (temperature scaling, held-out ECE 0.20-0.34 -> 0.02-0.03 on synthetic overlapping classes), calibration_report(), and conformal predict_set().
• Active learning and drift: suggest_labels() (margin / entropy / least_confidence / ood, optional diversity) and drift_report(). Uncertainty sampling did not beat random labelling on heavily overlapping synthetic classes (0.67 vs 0.69); the docs say so.
• Forgetting and OOD: forget(), remove_examples(), ood_score() / is_ood(), and predict(..., abstain_below=, abstain_ood=).
• scikit-learn: SklearnAdaptiveClassifier (fit, partial_fit, predict_proba, works in Pipeline / cross_val_score / GridSearchCV).
• Serving: python -m adaptive_classifier.serving ./model (FastAPI; updates off by default and API-key protected), async methods, docker/Dockerfile, docs/serving.md.
• Deployment guide (docs/deployment.md) and examples/portable_inference.py, a NumPy + onnxruntime reference that reproduces predict() without PyTorch or FAISS. A saved directory now includes the tokenizer and the resolved pooling mode and loads without the Hub.

Bugs fixed (found by the new tests)

• Thread safety: predicting while another thread added/forgot classes crashed ("selected index k out of range"), and concurrent add_examples calls corrupted each other. The public operations of AdaptiveClassifier now hold a per-instance lock.
• Integer labels were accepted, then reloaded from disk as a duplicated mix of int and str classes. add_examples now rejects non-string labels (breaking for anyone passing ints; str(label) fixes it).
• predict(k=-1) silently dropped the last result; predict_batch(k=...) scored only k classes so its scores differed from predict; clear_memory(labels=...) left classes in the label map and head; loading a directory without config.json reported an invalid Hub repo id.
• Saves from before 0.2.0 that have classes with fewer than 10 examples would have loaded with a different blend than they had under 0.2.0 (found by loading one under both versions); the old fixed split is now pinned.

Packaging and CI

• PEP 621 pyproject.toml (setup.py and requirements.txt removed), Python >= 3.10, test / serve / scripts extras, tqdm no longer a core dependency.
• CI downloads the test models once (with retries) and runs the suite offline; an earlier version of this PR hit the Hugging Face 1000-requests/5-minutes limit because four matrix legs shared one quota. Matrix is Python 3.10 and 3.13. Weekly schedule added.

Testing

• Offline suite (~480 tests): tiny local model plus synthetic separable data; covers accuracy at several sizes, XOR, no catastrophic forgetting, class imbalance, label noise, property-based invariants (hypothesis), hostile and malformed input, damaged files, Hub failures, concurrency, the HTTP server, and parity of the portable reference with predict() for every scoring branch.
• Mutation-checked: I deliberately broke the library in ~70 places across these files; every break is caught. A few initially survived and exposed missing tests, which were added.
• Published classifiers: 10 of the classifiers in the adaptive-classifier Hugging Face org (saved before 0.2.0) give identical predictions under 0.2.0 and 0.3.0: 225 predictions, no changed labels, largest score difference 3e-8.
• CI is green (Python 3.10, 3.13, and the integration job), including the pre-existing real-model tests. Two of those encoded the old behaviour and were updated: the blend-weight tests and a 0.85 < confidence < 0.95 window in test_reported_confidence_drop.py, which now asserts a consistent blend that is not the saturated pure-neural value.

Known gaps

• docker/Dockerfile has not been built or run (no Docker daemon was available when it was written). The server it runs is covered by tests.
• The defaults were compared against the previous ones on real data only as described above: two datasets, one encoder, three seeds. Other encoders, domains and class counts are untested. ood_threshold (1.25) was chosen on synthetic data.
• MultiLabelAdaptiveClassifier's own methods (such as predict_multilabel) are not covered by the new lock. Strategic mode is not covered by the portable reference or by the new accuracy and calibration tests.

🤖 Generated with Claude Code