Repository navigation
Whole-network finetuning in the browser: the last layer's loss settings, and augmentation - #142
Merged
Merged
Conversation
…d augmentation The whole network trained with masked BCE on the paint alone. It now takes the same loss fields as the head (head-trainer.ts batchLoss and distillationTerm, ported to TF.js in full/loss.ts): BCE, MSE or the margin loss, label smoothing, class balance split by the unsmoothed target, and λ × the mean squared distance of the logits from the starting network's (the base's weights, also on a restart) over the voxels the scope picks in each tile. Augmentation (augment.ts, the port of finetune/data/augment.py), off by default as on the server: each step's tiles get a brightness ×0.8 to ×1.2 and noise of 1% of the raw's range on the raw as stored, before it is normalized, then random flips and quarter turns of the raw and the paint alike. The targets are made from the turned paint, and the starting network is run on the turned tile, in lora_trainer.py's order, so an affinity channel still means its own direction; turning a target instead would need its channels permuted and shifted. A tile turns only where its plane's two voxel sizes are the same. tiles.ts keeps the raw before normalization (tileRaw, normalizeTile) so that it can be redrawn every step. The form shows the loss, distillation, smoothing and balance fields for the whole network, and the server's Augmentation box (full-also), with advice from the epochs; the job card says "augmented". The run's log names the settings it trained with and, at the end, how the tiles were flipped and turned. The head is unchanged and takes no augmentation: it trains on features computed once.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #137 (
browser-batch), where the browser's whole-network finetuning lives. Once #137 merges, this PR's base becomes main.The browser's whole-network finetune (the 2D models,
setup_53_2dandsetup_54_2d) used to train with masked BCE on the paint alone. It now takes the same loss settings as the last layer, and it gets an Augmentation box, off by default as on the server.Who sees a change
How augmentation works
It's the server's (
finetune/data/augment.py), in the same order aslora_trainer.py:Because the targets are made after the turn, an affinity channel still means its own direction. Turning a target instead would need its channels permuted and shifted. The browser makes binary and affinity targets only, and both come from the labels.
Code
web/src/finetune/augment.ts(new): the draws, the block transform (z, y, x, so a 3D tile can use it too), the intensity range, and a tally for the log.web/src/finetune/full/loss.ts(new): the head's supervised and distillation terms in TF.js, and the training step.full/run-full.ts: keeps each tile's raw and paint when augmenting and redraws them every step; runs a frozen copy of the starting network (the base's weights, also on a restart) when λ > 0.full/tiles.ts:tileRawandnormalizeTile, so the raw is kept before normalization;tileInputis unchanged in what it returns.routes/finetune.ts: onereadLossfor both modes; the whole network also readsaugment._finetune_tab.html,training-form.js,job-card.js), docs (browser.rst), CHANGELOG.Checks
npm run checkin web/: typecheck, 1,553 tests passed, build.test_importing_leaves_the_process_alone[blockwise-logging]failing, as on main in that environment.setup_53_2d:Not included
setup_53's 3D UNet at 172³, one PyTorch training step. LoRA of rank 8 on every layer kept more memory for the backward pass than training every weight (4.6 vs 4.2 GB) and was about 50% slower. The backward pass still needs every layer's activations. Training only the last decoder stage kept 0.5 GB, so that's the lever if memory gets tight.