Repository navigation
feat(wasm): build both wasm FMU forms for every case, in CI - #486
Merged
Merged
Conversation
Every one of the 98 simulation cases is now also exported as a
WebAssembly-backed FMI 3.0 FMU, in both forms `--simCodeTarget=wasm-jit`
can emit, plus an ahead-of-time transpile of the portable one. Nothing
consumes these yet: they are built so that they are known to build, and
keep building, before any client work depends on them.
The native pipeline is untouched. These are additional `build-wasm-*`
DVC stages beside `build-case-*`; `make results`, the plots, Sphinx, the
ebooks, `exes.tar.gz` and `api/` all behave exactly as before.
Both forms, because they are not interchangeable and the choice does not
have to be made at build time:
dylink/ the model kernel alone, at binaries/wasm32-om-dylink/.
Median 24 KB, 8.3 MB for the whole book. Runs only in a
host that links it against a prebuilt adapter, which today
means an OpenModelica importer -- there is no browser host
for it.
component/ a portable fmi-ls-wasm component at binaries/wasm32-wasip2/.
Median 712 KB, and the form a browser can run today.
One `omc` run emits both (the only difference is --fmuDirectory), so
building both costs ~0.7 s per case.
The AOT step matters because a browser cannot instantiate a WebAssembly
Component -- OpenModelica's own fmi-simulator page therefore ships jco's
transpiler and converts in the page, which measured 1424 ms of a 1598 ms
first load. The book's model set is known at build time, so
tools/wasm/transpile.mjs does it once per case instead: load 1598 -> 169 ms,
first result 1854 -> 364 ms, shared download ~11.5 MB -> 2.2 MB (MIC-178).
Two things the export does not carry, and now does not pretend to:
- `setCommandLineOptions("+override=...")` was not a real omc option
(`Error: Unknown option +override.`), so 8 cases were silently built
with stock parameter values. buildModelFMU has no compile-time
equivalent; `-override` is a runtime flag on the native path too.
- buildModelFMU takes no simulation settings either, so an FMU carries
the model's own `experiment` annotation as its DefaultExperiment,
which differs from the case's stopTime for 74 of 97 cases.
Both belong beside the artifact rather than inside it, so specs.py now
emits results/cases.json and make-index.py folds it into
results/wasm/index.json: per case, the artifacts and the stopTime,
tolerance, interval count and overrides an importer has to apply.
CI runs this in ghcr.io/mtiller/omc-wasm, pinned to the OpenModelica
commit it was built from. No published image has this compiler:
--simCodeTarget=wasm-jit needs OM_OMC_ENABLE_RUST=ON, which defaults OFF,
and Docker Hub's openmodelica/openmodelica carries release tags only, all
pre-wasm. tools/wasm/Dockerfile adds the MSL and the build toolchain on
top of that base and rebuilds in seconds; producing the base itself is
the ~72 minute from-source build documented on MIC-178.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing consumes the wasm artifacts yet, so a 90-day retention would hold ~340 MB per run to no purpose. Rebuild cost is carried by the DVC cache (98 MB, content-addressed), not by the artifact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`make all` in text/results would otherwise require an omc with the wasm-jit backend and a node with the jco bundle, which the ordinary book toolchain has neither of. The DVC path and CI both name the wasm stages explicitly, so nothing else needs it there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… once jco names its core modules positionally (fmu.core.wasm, fmu.core2.wasm, ...) and the numbering shifts from model to model, which hid how much the 98 cases have in common. By content they barely differ: 186 MB of core modules is 12.8 MB of distinct bytes, and 1.85 MB of that -- the OpenModelica runtime -- is byte-identical in every single case. The glue is nearly shared too: fmu.js is 747 KB with only four distinct variants. Laid out per case none of that could be shared, because each aot/<case>/fmu.core9.wasm is its own URL: a reader opening a second figure would download the runtime again. pack.py rewrites the tree so every distinct blob has one content-addressed path: pack/blobs/<sha256[:16]>.wasm each distinct core module, once pack/blobs/<sha256[:16]>.js each distinct glue file, once pack/cases/<case>.json logical name -> blob path, + imports/exports 98 cases become 127 blobs, 16.3 MB -- 15.7x smaller -- in under a second. Blobs are hardlinked from the DVC outputs, so the packed tree costs no extra space, and it is built after `dvc repro` rather than as a stage, because a shared blob store cannot be the output of 98 separate stages. The logical names survive the rename because that is what the glue asks for: fmu-core.js builds `cores` as a Map keyed by jco's name and the generated module calls compile(name) against it. The client may fetch those bytes from any URL; it just needs the mapping. For a reader, assuming the blobs are served at those shared paths: about 0.75 MB gzipped for the first figure on a page, then ~16 KB for each figure after it, against 0.75 MB every time laid out per case. That also changes the Step 0 comparison the PR leaves open -- per figure the AOT form is ~33 KB against dylink's 24 KB, and unlike dylink it has a browser that can run it. CI now uploads pack/ and skips aot/, which is the same content with the duplicates removed rather than the same content twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
generate_results and generate_site were gated behind the same 'Full Build' label as deploy_site, so an ordinary pull request built nothing and a label meant to say "build this" also meant "publish this to production". Those are different questions and only one of them belongs on a PR. Now the three build jobs run everywhere and deploy_site is restricted to push events, which the workflow's own trigger already narrows to master. A pull request therefore cannot deploy, with or without a label. 'labeled' comes off the pull_request trigger with it: nothing keys off labels any more, so leaving it in would re-run a ~17 minute build every time any label was added or removed. The 'Full Build' label itself is now unused and can be deleted from the repo. Measured on the last two master runs: generate_results 16m, generate_site 1m, both green, and generate_wasm runs beside them in 7-14m. Minutes are free on a public repository. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`make results` runs `dvc repro dvc.yaml`, which reproduces *every* stage
in that file -- so adding the 196 build-wasm-* stages to text/dvc.yaml
quietly made the ordinary book build try to export wasm FMUs. The
book-builder image's omc is a release build with no wasm backend, so
generate_results failed:
Error: Invalid type of flag simCodeTarget, expected one of None, C,
Cpp, ... but got the string "wasm-jit".
Error: Unknown option +fmuDirectory.
Error: Unknown FMU version 3.0. Only version 1.0 & 2.0 are supported.
Diffing the generated stage definitions had shown them byte-identical to
refactor's, which was true and beside the point: what changed was not any
native stage but the set of stages `dvc repro dvc.yaml` reaches. Running
the job is what caught it.
specs.py now emits them to text/dvc-wasm/dvc.yaml instead, and
`make wasm` reproduces that file. Neither pipeline can reach the other.
Two things fall out of the split. text/dvc.lock goes back to exactly what
is on refactor, since the wasm stages now keep their own lock -- which
also drops the ~150 lines of YAML re-wrapping noise the shared lock had
picked up from a newer dvc. And `make wasm` got faster, 3m01 to 1m38 on a
warm cache, because dvc no longer walks 99 native stages to decide it has
nothing to do with them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Steps 0, 1 and 1.5 of the wasm plan: every case exporting as a WebAssembly FMU, in CI, before anything consumes it. Step 2 (the web-worker client that replaces the
api/round-trip) is deliberately not here.The astro/mystmd refactor is untouched and unblocked — that work is purely additive under
web-astro/andweb/, and this is purely additive insidetext/.The native build is a separate pipeline, and that matters
make resultsrunsdvc repro dvc.yaml, which reproduces every stage in that file. The first cut put the 196build-wasm-*stages there, which quietly made the ordinary book build try to export wasm FMUs with whateveromcit had — and the book-builder image's omc is a release build:specs.pynow emits them totext/dvc-wasm/dvc.yaml, its own pipeline with its own lock, andmake wasmreproduces that file. Neither pipeline can reach the other.text/dvc.lockis consequently byte-identical torefactor.Everything downstream of
make results— plots, Sphinx, ebooks,exes.tar.gz,api/— behaves exactly as before.Both export forms, because they are not interchangeable
--simCodeTarget=wasm-jitemits two quite different artifacts and--fmuDirectoryis the only difference. Oneomcrun emits both at ~0.7 s per case, so which to ship stays open rather than being decided by the build.wasm/dylink/<case>.fmubinaries/wasm32-om-dylink/. Needs a host that links it against a prebuilt adapter — today an OpenModelica importer. No browser host exists.wasm/component/<case>.fmubinaries/wasm32-wasip2/. What a browser can run today.wasm/aot/<case>/manifest.json.wasm/pack/Ahead-of-time transpile (Step 1.5)
A browser cannot instantiate a WebAssembly Component —
WebAssembly.instantiatetakes core modules. OpenModelica'sfmi-simulatorpage ships jco's transpiler (~9.3 MB) and converts in the page: 1424 ms of a 1598 ms first load (MIC-178). The book's model set is known at build time, sotools/wasm/transpile.mjsdoes it once per case.session.load()Content-addressed packing
jco names core modules positionally and the numbering shifts per model, hiding how much the cases share. By content they barely differ: 186 MB of core modules is 12.8 MB of distinct bytes, and 1.85 MB of that — the OpenModelica runtime — is byte-identical in every case.
fmu.jsis 747 KB with four distinct variants.Per case none of that can be shared, since each
aot/<case>/fmu.core9.wasmis its own URL.tools/wasm/pack.pyrewrites it:98 cases → 127 blobs, 16.3 MB — 15.7x, in 0.68 s, hardlinked so it costs no extra disk. The logical names survive because that is what the glue asks for:
fmu-core.jsbuildscoresas aMapkeyed by jco's name and callscompile(name).For a reader: ~0.75 MB gzipped for the first figure on a page, then ~16 KB per figure after — against 0.75 MB every time laid out per case. That also sharpens the open Step 0 question: per figure AOT is ~33 KB against dylink's 24 KB, and unlike dylink it has a browser that can run it.
Content-addressing the FMU forms was measured and rejected: it would save 8 KB across all of
dylink/(96 identical 85-byteartifact.jsonsidecars) and zero acrosscomponent/.Two defects fixed, and where the case settings went
gen_wasm.mosemittedsetCommandLineOptions("+override=..."), which is not an omc option — it answersError: Unknown option +override.and carries on — so 8 cases were silently exported with stock parameter values. There is no compile-time equivalent;-overrideis a runtime flag on the native path too.Separately,
buildModelFMUtakes no simulation settings, so an FMU carries the model's ownexperimentannotation as itsDefaultExperiment. For 74 of 97 cases that is not the case's stopTime.Both belong beside the artifact.
specs.pynow emitsresults/cases.json, andmake-index.pyfolds it intoresults/wasm/index.json— per case, the artifacts plus the stopTime, tolerance, intervals and overrides an importer must apply withfmi3Set*before initialization.CI
generate_wasmruns inghcr.io/mtiller/omc-wasm:3998b1f8, pinned to the OpenModelica commit it was built from. No published image has this compiler:--simCodeTarget=wasm-jitneedsOM_OMC_ENABLE_RUST=ON, which defaults OFF, and Docker Hub'sopenmodelica/openmodelicacarries release tags only, all pre-wasm.tools/wasm/Dockerfileadds the MSL and toolchain on top of that base and rebuilds in ~35 s; the base is the ~72 minute from-source build on MIC-178.Two things it needs that only running it revealed:
clang+lld.IIEVusesexternal "C"(interpolateVectorTable) with no wasm implementation, so omc compiles theIncludesources itself withclang --target=wasm32-wasip1. The wasi sysroot already ships in the base.HOME=/github/home, where omc finds no libraries at all, so the workflow links it to the image's MSL beforemake env.Gating also changed:
generate_resultsandgenerate_sitewere behind the sameFull Buildlabel asdeploy_site, so a label meaning "build this" also meant "publish to production". Now all three build jobs run on every PR anddeploy_siteis restricted topush, which the trigger already narrows tomaster— a PR cannot deploy, with or without a label.labeledcomes off the trigger too, since nothing keys off labels now; theFull Buildlabel is unused and can be deleted.Verification
Last run, all four jobs as intended:
generate_wasmgenerate_resultsgenerate_sitedeploy_site98/98 cases in all three forms, CI totals matching local byte for byte. Cold 14m30s, warm 3m. Artifact 75 MB at 7-day retention (
pack/is uploaded,aot/excluded — same content, duplicates removed).