Skip to content

0.3.1: fix state-safety, EWC, merge, loading, multi-label, server and strategic-mode bugs; Node.js port; Docker check - #69

Merged
codelion merged 16 commits into
mainfrom
claude/zen-hawking-qj5dfq
Oct 9, 2026
Merged

codelion merged 16 commits into
mainfrom
claude/zen-hawking-qj5dfq

Conversation

@codelion

@codelion codelion commented Oct 9, 2026 •

Copy link
Copy Markdown
Owner

Summary

Version 0.3.1. Follow-up to #68, from a systematic hunt for bugs in features that had little or no test coverage: lifecycle operations, multi-label, strategic mode, the HTTP server, the scikit-learn wrapper, merge_classifiers, and the CI/release plumbing. Three independent reviewers (core library, feature modules, CI/packaging/docs) read the code cold and tried to break it; every finding below was reproduced with a script before being fixed. CHANGELOG.md has the full list.

Core library

  • add_examples is now all-or-nothing. An error or Ctrl-C while training used to leave the label map ahead of the neural head, so every later predict raised "selected index k out of range" and every retry failed; or left the search index stale, so predictions silently returned the wrong class. Texts are encoded before any state changes, and on failure everything is restored.
  • EWC did nothing. The penalty compared the old head with its own stored copy, so it was always exactly 0 and never reached the head being trained. It now penalises how far the live head's existing rows move. With the default strength (5.0) the effect is small: re-running the README's class-incremental experiment (MiniLM, ag_news and emotion, 3 seeds) moved final accuracy by -0.004 / +0.016, old-class loss by -0.002 / +0.002 and new-class recall by -0.012 / 0.000, all within noise. The README table therefore stands.
  • merge_classifiers was badly broken: the other classifier's classes were never searchable afterwards (the prototype index was not rebuilt; 0% accuracy on them), merging into an empty classifier raised, training counts were not combined, stale calibration was kept. Also fixed a deadlock when two threads merge each other (it never holds two classifiers' locks at once) and the other classifier's OOD radii are kept.
  • clear_memory() with no labels cleared only the prototypes; the label map, counts, calibration and neural head stayed, so predict kept answering from the head. It now empties the classifier.
  • A reloaded classifier re-averaged its prototype from the few kept examples on the first add_examples (moved by 0.32 versus 0.03 live for a 30-example class). Prototypes now fold in new examples with the weight of the examples they were saved from.
  • Re-saving an ONNX-loaded classifier wrote the ONNX folder as model_name; add_examples ignored batch_size; copy.deepcopy/pickle failed because of the instance lock; predict_batch disagreed with predict in strategic and multi-label modes; blank training texts/labels were silently learned (behaviour change: they now raise ValueError); remove_examples("str") removed nothing; PrototypeMemory.get_strategic_prototypes raised NameError.

Multi-label, strategic mode, server, scikit-learn

  • Multi-label: forget/remove_examples/load(use_onnx=...) crashed; an explicit threshold was ignored; a saved multi-label model loaded through AdaptiveClassifier.load (or the server) came back as single-label; the head was barely trained; calibrate/predict_set returned meaningless numbers (now NotImplementedError); bare-string labels were split into one class per character.
  • Strategic mode: the README's example config silently disabled it (costs must be a list with one entry per embedding dimension; the reason is now logged); the best response only explored the first few dimensions and was unseeded; scores depended on k; the robustness evaluation divided by zero and crashed unless the levels included 0.0 and 1.0.
  • Server: saves, updates and /info no longer block the event loop (a save with ONNX export froze /health for seconds); multi-label models are refused with a clear message instead of failing every request with 500.
  • scikit-learn: partial_fit(classes=...) now lists declared classes; labels that collapse to the same class name (1 and "1"), None and NaN are rejected.

New

  • examples/javascript/portable_inference.mjs: a Node.js port of the portable inference, tested against predict() for both pooling modes, every scoring branch and over-length inputs (tests/test_js_example.py, run in CI).
  • scripts/docker_smoke.py and .github/workflows/docker.yml: build the serving image and query it (path-filtered plus weekly, so ordinary PRs do not build it).
  • Release workflow now calls the full test workflow, only the publish job holds PyPI credentials, and the release tag must match the package version.
  • Docs: docs/API.md now covers the real signatures, MultiLabelAdaptiveClassifier, strategic mode and what a saved classifier keeps; README documents the new-class recall trade-off (new_class_example_threshold) with measurements.

Testing

  • 95 new test functions (more cases once parametrised): state safety, merge, multi-label, strategic mode, server, scikit-learn, Node port, memory. Most were checked to fail against the previous code.
  • The full offline suite passes locally (0 failures; two SetFit skips). Tests that download real models could not run in the authoring sandbox (Hugging Face rate-limits its IP), so CI's prefetched run is their first check.

Known gaps

  • Unverified until CI runs: the Docker build and smoke test (the sandbox's TLS proxy blocks the PyTorch download inside docker build), the onnxruntime-node path of the JavaScript example (only onnxruntime-web could be installed locally), and the reworked release workflow.
  • A reloaded classifier's neural head is still retrained from the few examples kept per class (documented in docs/API.md); only the prototypes were fixed.
  • Strategic mode is still not covered by the portable reference or the accuracy/calibration tests.

🤖 Generated with Claude Code

claude added 16 commits October 9, 2026 00:21
scripts/adaptive_router was an incomplete, pre-0.2.0 save (no examples.json,
tensors named tensors.safetensors, no tokenizer) that AdaptiveClassifier.load
could not read.
…inference

Work in progress: neither has been run yet. The Docker build cannot run in the
authoring sandbox (TLS proxy blocks the PyTorch wheel download), so the new CI
job is the first real check.
The port matches AdaptiveClassifier.predict for both pooling modes, legacy and
current scoring, fixed and ramped new-class weights, and over-length inputs.
Deliberately breaking the ramp or the truncation is caught.
Moved from the main test workflow to its own workflow with path filters (plus a
weekly schedule), so ordinary PRs do not build the image.
…locking

Found by probing MultiLabelAdaptiveClassifier with bad input and lifecycle calls:
- forget() and remove_examples() crashed (the head had no remove_classes)
- load(use_onnx=...) raised TypeError; thresholds and settings were not saved
- a bare-string label was split into one class per character
- max_labels=0 meant no limit, and min_predictions could exceed max_labels
- the head was trained for ~10 steps and later-added labels went through a
  softmax loss that does not apply to sigmoid outputs; it now trains with BCE for
  head_steps steps with a cosine schedule, and is retrained when labels are added
- public methods were not covered by the instance lock

28 new tests (27 fail on the old code).
…l and strategic mode

Strategic mode: validate cost_coefficients (the README example silently disabled it),
sample best-response candidates across all dimensions with a fixed seed, make
predict/predict_strategic/predict_robust scores independent of k, and fix the
robustness evaluation (division by zero, unknown labels, repeatability, lock).
Pin datasets/huggingface-hub in the benchmark and scripts extras.
…multi-label modes

The 0.3.0 instance lock made deepcopy and pickle fail. predict_batch skipped the
strategic blend and softmaxed multi-label sigmoid scores, so it disagreed with predict.
The prototype index was not rebuilt after a merge, so the other classifier's classes
were never found (0% accuracy on them). Also: empty target, training counts,
stale calibration, shared example objects.
…bugs found in review

Core: add_examples is all-or-nothing; EWC penalty now acts on the live head; clear_memory
really clears; reloaded prototypes fold in new examples by weight; re-saving an
ONNX-loaded classifier keeps the encoder name; merge_classifiers cannot deadlock and keeps
OOD radii; embeddings are encoded in batches.
Multi-label: explicit threshold honoured, base-class load returns the right type,
calibration refused, abstain options, thresholds follow merge/remove.
Server: updates/saves/info off the event loop, multi-label refused.
Strategic: input validation, gaming levels. sklearn: declared classes, label clashes.

18 + 40 new tests; the full suite run was still in progress when this was committed.
basic_usage.py no longer calls .eval() on an ONNX model; docs match real signatures and
drop the nonexistent --workers flag; the release workflow gates publishing on the full
test workflow, scopes PyPI credentials to the publish job and checks tag == version;
docker.yml triggers aligned.
@codelion
codelion merged commit c0b4a2d into main Oct 9, 2026
6 checks passed
@codelion
codelion deleted the claude/zen-hawking-qj5dfq branch October 9, 2026 03:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants