Upgrade to ScienceBeam Trainer DeLFT v1.x (PyTorch) - #695
Merged
Conversation
Every model the trainer now produces is a torch state dict, and the 0.0.x pin cannot load one, so serving a freshly trained model needs this first. The 0.0.x line is also closed to security and maintenance updates. TF-era `model_weights.hdf5` stays supported input: delft converts it to a torch state dict in memory on load and never touches the directory, so the model URLs in the shipped config are unchanged. Upstream measured the conversion against a reference capture for the eight CustomBidLSTM_CRF models the biorxiv_elife profile serves, which is why parity is not re-measured here. The graph loses tensorflow, tf-keras, tfa-nightly, keras, tensorboard, libclang, ml-dtypes and pydot, and gains pytorch-crf and sentencepiece; torch moves to 2.11.0 and torchvision to 0.26.0. torch stays declared in the cpu extra rather than left to arrive through delft. That declaration is what binds [tool.uv.sources] to the package name across the whole lock: uv applies a source to a direct dependency, so removing these lines as redundant - delft requires torch==2.11.0 on its own - would let torch resolve from PyPI instead, where the Linux wheel carries four nvidia-* packages and triton. The floors are raised to what actually resolves, since >=2.5.1 said nothing next to delft's exact pin. trainer-delft's own cpu extra is deliberately not requested. It re-pins torch==2.11.0, which delft requires anyway, and its index selection does not propagate to a consumer. The tf extra is removed. Nothing here installed it and it existed only to add TensorFlow for the delft engine, but it is published metadata, so it belongs in the release notes.
delft 1.0 picks CUDA whenever torch sees a GPU, where the TensorFlow-era path was CPU unless configured otherwise. An inference image should not change device because of the host it lands on. SCIENCEBEAM_DELFT_DEVICE is the whole mechanism, since Sequence is constructed here without a device argument. Set as an image default rather than baked in, so a GPU variant stays possible and an operator can override it per deployment.
The job installed sciencebeam-trainer-delft outside the lock and ran a hand-picked subset, because TensorFlow had no arm64 wheels and the delft extra would not install. Every distribution in that extra now has an aarch64 wheel, torch included, so arm64 installs like every other platform and stops being the one job testing an unlocked version. The selection widens to match macOS: the whole suite minus cv_models. Both native jobs now also deselect slow, which covers the cases that download model artifacts - the Docker pytest target runs those.
Upstream measured the conversion's scores, but nothing showed the load path works in this package's environment, and CI stopped exercising it when the default profile became wapiti. The biorxiv_elife header release is the model that exercises conversion and nothing else: TF-era model_weights.hdf5, and no word embeddings, so it needs nothing from the embedding registry. Loading the whole profile would be the wrong shape for CI, since its other models resolve glove-840B. The case goes through DelftModelImpl rather than the models API. The router is already covered by mocked tests and the default profile is wapiti, so a router-level case would need a config override to reach a delft model at all; the load-and-convert path is what was unproven. Input comes from HeaderDataGenerator rather than a hand-written feature matrix, which would risk a width mismatch against the model's config.json and fail for a reason unrelated to the upgrade. Assertions are shape only - a pair per token, labels drawn from the model's own set - because asserting particular labels would re-measure parity here and tie the test to output upstream has already checked. Marked slow: it downloads 6.6 MB, so the native jobs deselect it.
The delft extra pulls in torch, and on Linux the default PyPI wheel is the CUDA build. This project's index configuration is not part of its published metadata, so it cannot reach a downstream consumer; documentation is the only mechanism available. The non-obvious part is not the index block but that torch has to be declared as a direct dependency for a uv source to apply. A project that receives torch only through sciencebeam-parser has no binding at all, which is the same reason the cpu extra keeps its torch line. Also states that there is no TensorFlow extra and that model URLs need no change, since TF-era artifacts are converted on load.
1.0.1 is the first release that validates SCIENCEBEAM_DELFT_DEVICE and logs the device it resolved. Both runtime images set that variable, and until now a typo in it surfaced deep inside a tensor move rather than at construction, while a correct value left no record of which device was actually used. The floor states that dependency rather than leaving it to whatever resolves. It also drops the cpu and gpu extras, which were byte-identical once published and so promised a wheel choice they could not make. Nothing here requested them - torch is bound to the CPU index through this project's own direct declaration - so the resolution is unchanged: delft stays 1.0.1 and torch stays 2.11.0+cpu.
ScienceBeam Parser EvaluationOverall (60 docs across 6 corpora)grobid 0.9.0-crf (default): 60 docs | sciencebeam-parser:main-596b4687-20260818.1053 (grobid_crf): 60 docs | sciencebeam-parser:pr-695-33ee27ec-20260818.1216 (grobid_crf): 60 docs
biorxiv (10 docs)grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-596b4687-20260818.1053 (grobid_crf): 10 docs | sciencebeam-parser:pr-695-33ee27ec-20260818.1216 (grobid_crf): 10 docs
ore (10 docs)grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-596b4687-20260818.1053 (grobid_crf): 10 docs | sciencebeam-parser:pr-695-33ee27ec-20260818.1216 (grobid_crf): 10 docs
pkp (10 docs)grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-596b4687-20260818.1053 (grobid_crf): 10 docs | sciencebeam-parser:pr-695-33ee27ec-20260818.1216 (grobid_crf): 10 docs
scielo_br (10 docs)grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-596b4687-20260818.1053 (grobid_crf): 10 docs | sciencebeam-parser:pr-695-33ee27ec-20260818.1216 (grobid_crf): 10 docs
scielo_mx (10 docs)grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-596b4687-20260818.1053 (grobid_crf): 10 docs | sciencebeam-parser:pr-695-33ee27ec-20260818.1216 (grobid_crf): 10 docs
scielo_preprints-jats (10 docs)grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-596b4687-20260818.1053 (grobid_crf): 10 docs | sciencebeam-parser:pr-695-33ee27ec-20260818.1216 (grobid_crf): 10 docs
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The trainer and the parser have to agree on a weight format: every model the trainer
now produces is a torch state dict, and the 0.0.x pin cannot load one, so serving a
freshly trained model needs this first. The 0.0.x line is also closed to security and
maintenance updates.
Model URLs are unchanged. TF-era
model_weights.hdf5stays supported input — delftconverts it to a torch state dict in memory on load and never touches the model
directory.
What changes
sciencebeam-trainer-delft0.0.38→1.0.1delft0.4.3→1.0.1torch/torchvision2.5.1→2.11.0+cpu/0.20.1→0.26.0+cputensorflow,tf-keras,tfa-nightly,keras,tensorboard,libclang,ml-dtypes,pydotpytorch-crf,sentencepiece;transformers4.48.0→5.7.0nvidia-*, notritonBoth runtime images now set
SCIENCEBEAM_DELFT_DEVICE=cpu, overridable from theenvironment. delft 1.0 picks CUDA whenever torch sees a GPU, and an inference image
should not change device because of the host it lands on. 1.0.1 is the floor because it
is the first release that validates that variable and logs the device it resolved.
Linux arm64 installs
--extra delftlike every other platform, and its special-casedstep goes: it existed because TensorFlow had no arm64 wheels, and it installed
sciencebeam-trainer-delftoutside the lock, making arm64 the one job not testing thelocked version. Its selection widens to match macOS — the whole suite minus
cv_models— and both native jobs deselectslow.torchstays declared in thecpuextra rather than left to arrive through delft. Thatdeclaration is what binds
[tool.uv.sources]to the package name across the lock; uvapplies a source to a direct dependency, so removing those lines as redundant would let
torch resolve from PyPI, where the Linux wheel carries four
nvidia-*packages andtriton.A new case at
tests/models/delft_model_impl_test.pyloads a converted TF-era model andtags with it — nothing covered that path here, since CI stopped exercising delft when the
default profile became wapiti. It uses the
biorxiv_elifeheader release, which needs noword embeddings, so it exercises the conversion and nothing else. Marked
slowbecause itdownloads 6.6 MB; the Docker
pytesttarget runs it and the native jobs skip it.Parity with the TensorFlow build
biorxiv_elifeatsmoke, 60 documents across six corpora, run on this branch(
pr-695-5a9661fa) and on a branch offmainstill carrying TensorFlow(
pr-696-05c18a70). 146 of 147 metric rows areidentical, the overall section over all 60 documents included. One row differs:
ore,reference_title (edit_sim),partial_list, 0.340 on TensorFlow against 0.339 on torch.reference_title (levenshtein)over the same field and corpus is unchanged at 0.315, sowhatever differs is too small to move a match decision and washes out of the overall
figures — a float-level difference in one reference title in one of ten
oredocuments.Accepted as measured, not traced to a document.