Skip to content

Whole-network finetuning in the browser: the last layer's loss settings, and augmentation - #142

Merged
davidackerman merged 2 commits into
mainfrom
browser-full-augment
Oct 19, 2026
Merged

davidackerman merged 2 commits into
mainfrom
browser-full-augment

Conversation

@davidackerman

Copy link
Copy Markdown
Collaborator

Stacked on #137 (browser-batch), where the browser's whole-network finetuning lives. Once #137 merges, this PR's base becomes main.

The browser's whole-network finetune (the 2D models, setup_53_2d and setup_54_2d) used to train with masked BCE on the paint alone. It now takes the same loss settings as the last layer, and it gets an Augmentation box, off by default as on the server.

Who sees a change

  • The form, with Whole network picked: the Loss Function, Margin, Distillation Weight and Scope, Label Smoothing, Balance and Augmentation fields now show, and are used. Before, they were hidden and ignored.
  • The defaults change for a whole-network run: the form starts with class balance on and distillation at 0.01, so a run with untouched fields now trains with both, where before it had neither.
  • The log and the job card say what the run trained with. The card adds "augmented" when augmentation was on.
  • The last layer is unchanged. It trains on features computed once, so it takes no augmentation.

How augmentation works

It's the server's (finetune/data/augment.py), in the same order as lora_trainer.py:

  1. Brightness ×0.8–1.2 and noise of 1% of the raw's range, on the raw as stored, before normalization.
  2. Random flips and a quarter turn of the image and the paint together. A tile turns only where its plane's two voxel sizes are equal.
  3. Then the targets are made from the turned paint, and the starting network runs on the turned tile for distillation.

Because the targets are made after the turn, an affinity channel still means its own direction. Turning a target instead would need its channels permuted and shifted. The browser makes binary and affinity targets only, and both come from the labels.

Code

  • web/src/finetune/augment.ts (new): the draws, the block transform (z, y, x, so a 3D tile can use it too), the intensity range, and a tally for the log.
  • web/src/finetune/full/loss.ts (new): the head's supervised and distillation terms in TF.js, and the training step.
  • full/run-full.ts: keeps each tile's raw and paint when augmenting and redraws them every step; runs a frozen copy of the starting network (the base's weights, also on a restart) when λ > 0.
  • full/tiles.ts: tileRaw and normalizeTile, so the raw is kept before normalization; tileInput is unchanged in what it returns.
  • routes/finetune.ts: one readLoss for both modes; the whole network also reads augment.
  • Form (_finetune_tab.html, training-form.js, job-card.js), docs (browser.rst), CHANGELOG.

Checks

  • npm run check in web/: typecheck, 1,553 tests passed, build.
  • The full Python suite: 2,230 passed, with only test_importing_leaves_the_process_alone[blockwise-logging] failing, as on main in that environment.
  • New tests:
    • the image and the paint stay registered through flips and turns;
    • a boundary across x becomes one across y after a quarter turn;
    • brightness and noise stay in the raw's range and the padding stays 0;
    • the TF.js loss equals the head's for every loss type, balanced or not, smoothed or not;
    • the distillation term is exactly 0 before the first step;
    • the form's fields are read for the whole network.
  • Headless, on this workstation's GPU (WebGPU), jrc_hela-2 with setup_53_2d:
    • a 3-epoch run on 31 tiles of 300², with augmentation and distillation, at 244 ms a step;
    • the finetuned model served, and its chunks came back;
    • the log's tally: 93 tiles augmented, flips 51–57%, each turn 16–35%.
  • The draw itself, over 100,000 tiles: every turn 25%, every flip 50%.

Not included

  • LoRA in the browser: measured on setup_53's 3D UNet at 172³, one PyTorch training step. LoRA of rank 8 on every layer kept more memory for the backward pass than training every weight (4.6 vs 4.2 GB) and was about 50% slower. The backward pass still needs every layer's activations. Training only the last decoder stage kept 0.5 GB, so that's the lever if memory gets tight.
  • Good regions still aren't used by the whole network.

…d augmentation

The whole network trained with masked BCE on the paint alone. It now takes
the same loss fields as the head (head-trainer.ts batchLoss and
distillationTerm, ported to TF.js in full/loss.ts): BCE, MSE or the margin
loss, label smoothing, class balance split by the unsmoothed target, and
λ × the mean squared distance of the logits from the starting network's
(the base's weights, also on a restart) over the voxels the scope picks in
each tile.

Augmentation (augment.ts, the port of finetune/data/augment.py), off by
default as on the server: each step's tiles get a brightness ×0.8 to ×1.2
and noise of 1% of the raw's range on the raw as stored, before it is
normalized, then random flips and quarter turns of the raw and the paint
alike. The targets are made from the turned paint, and the starting network
is run on the turned tile, in lora_trainer.py's order, so an affinity
channel still means its own direction; turning a target instead would need
its channels permuted and shifted. A tile turns only where its plane's two
voxel sizes are the same. tiles.ts keeps the raw before normalization
(tileRaw, normalizeTile) so that it can be redrawn every step.

The form shows the loss, distillation, smoothing and balance fields for the
whole network, and the server's Augmentation box (full-also), with advice
from the epochs; the job card says "augmented". The run's log names the
settings it trained with and, at the end, how the tiles were flipped and
turned. The head is unchanged and takes no augmentation: it trains on
features computed once.
Base automatically changed from browser-batch to main October 19, 2026 14:39
@davidackerman
davidackerman marked this pull request as ready for review October 19, 2026 14:40
@davidackerman
davidackerman merged commit 88038ca into main Oct 19, 2026
7 checks passed
@davidackerman
davidackerman deleted the browser-full-augment branch October 19, 2026 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant