diff --git a/Cargo.lock b/Cargo.lock index ec816d9e0..673ebdd63 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -867,6 +867,7 @@ dependencies = [ name = "rvt-wasm" version = "0.12.2" dependencies = [ + "js-sys", "rvt", "serde", "serde-wasm-bindgen", diff --git a/changelog.d/added/1195-viewer-progress-too-large.md b/changelog.d/added/1195-viewer-progress-too-large.md new file mode 100644 index 000000000..99f515ab2 --- /dev/null +++ b/changelog.d/added/1195-viewer-progress-too-large.md @@ -0,0 +1,21 @@ +- **The browser viewer shows the decoder's own progress, and says when a + model is too large for a tab.** The loading card names the stage the + decoder is in (the stages `rvt-gltf --progress` prints) and fills its bar + with that stage's count, where it used to sweep and guess "about N s" from + the file's size. `openRvtBytesWithDiagnosticsModeAndOpenLimits` takes an + optional fourth argument, a function called with the stage, the count done + and the count it is out of. `rvt-gltf --progress` now also prints a + `partition-scan` line for each `Partitions/*` stream as it is first + inflated, which is most of a project's open, where it printed the stage's + start and end only. A file whose size says it may need more than a tab's + 4 GiB (at least about ten times its size, the figure of the largest + reference model; the warning can fire on a file that fits) is warned about + before the decode and still tried. When the wasm instance runs out of + memory, the viewer says the model is too large for the browser, how far the + decode got and where, and that `rvt-ifc` and `rvt-gltf` with `--large-model` + (streams up to 2 GiB and, on a 64-bit machine, inflating up to 8 GiB, for a + file of up to 2 GiB), or Python with a larger `max_stream_bytes` and + `max_inflate_bytes`, can read a larger file, instead of + `RuntimeError: unreachable`. A stop a long way short of the ceiling says + "Ran out of memory" and that the file may be damaged, and a panic that is not + a memory failure keeps its own message (#1195). diff --git a/docs/status.md b/docs/status.md index 015b2439f..3006c4bc6 100644 --- a/docs/status.md +++ b/docs/status.md @@ -1052,10 +1052,11 @@ two fields are now pinned and two exports are byte-identical, as In the browser the same change takes the 33.7 MB Core Interior demo from about 28 s to **about 3 s** of decode, measured live on the deployed site (6.7 s including the 32 MB download). The viewer's loading card -derives its "about N s" hint from that measurement — +opened with an "about N s" hint from that measurement — `DECODE_BYTES_PER_SECOND` is 11 MB/s since #271, up from the 1.15 MB/s -the pre-#266 figure implied — and the progress bar stays indeterminate -because the decoder cannot report real progress. +the pre-#266 figure implied. It is a laptop's figure, and since #1195 the +card shows the decoder's own progress as soon as it reports (see +[`viewer-api.md`](viewer-api.md#decode-progress)). ## Capability Matrix diff --git a/docs/user-guide.md b/docs/user-guide.md index 53ff106f3..4f3a6f14f 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -350,8 +350,9 @@ at the default limits and, when a stream is not read, shows "Part of the file not read" in the File status panel with a "Retry with large-model limits" button, which reopens the same file with `--large-model`'s limits (streams up to 2 GiB). A browser tab can use at most 4 GiB of memory, so a very large -model can still fail to open there; `rvt-ifc --large-model` has no such -ceiling. +model can still fail to open there; `rvt-ifc --large-model` is not held to a +tab's 4 GiB (its caps are a 2 GiB stream and, on a 64-bit machine, 8 GiB +inflated, and a file over 2 GiB is rejected whatever the flags). Two exports of the same file are the same bytes except for the export time, which the IFC header and owner history record. Set `SOURCE_DATE_EPOCH` (Unix diff --git a/docs/viewer-api.md b/docs/viewer-api.md index 6bbf03f35..844d5b1b2 100644 --- a/docs/viewer-api.md +++ b/docs/viewer-api.md @@ -257,9 +257,180 @@ the emitted IFC is byte-identical. In the browser the 33.7 MB demo decodes in **about 3 s**, measured on the deployed site — 6.7 s including the 32 MB download. That measurement is what `DECODE_BYTES_PER_SECOND` in `viewer/src/main.ts` encodes (11 MB/s -since #271, up from 1.15 MB/s) to set the loading card's "about N s" -hint; the progress bar stays indeterminate because the decoder -cannot report real progress. +since #271, up from 1.15 MB/s) for the "about N s" hint the loading card +opens with. It is a laptop's figure: Deploy viewer's runner takes about +24 s to open the same file (the measurement in +[Models too large for the browser](#models-too-large-for-the-browser)), so +since #1195 the card shows the decoder's own progress as soon as it reports, +and the hint is only the first thing it says. + +## Decode progress + +`openRvtBytesWithDiagnosticsModeAndOpenLimits(bytes, mode, limits, progress)` +takes an optional `progress` function, called as `progress(stage, done, total)` +while the decoder works (#1195). They are the reports of the decoder's +`WalkerControl` (#1072), the ones `rvt-gltf --progress` prints, passed across +the wasm boundary: + +| `stage` | `done` of `total` | The card shows | +|---|---|---| +| `schema-parse` | 0, then 1 of 1 | Reading the schema | +| `candidate-scan` | bytes of `Global/Latest` scanned, of its size | Scanning for elements | +| `element-decode` | scan candidates decoded, of the candidates | Decoding elements | +| `partition-scan` | 0 with no total where the walk reaches the partitions; then each `Partitions/*` stream inflated for the first time, of all of them (the first pass over the partitions inflates every one, and the handle keeps them, #266); 1 of 1 where the walk ends | Reading partition streams | +| `saved-graphics` | 0, then 1 of 1 | Reading saved graphics | +| `saved-meshes` | elements meshed, of the elements | Drawing doors and windows from saved meshes | + +`total` is `null` where a stage has none yet. The decoder reports every +megabyte scanned and every 256th candidate; the binding calls `progress` at a +stage's start, at its end and each time a stage with a total passes another +whole percent, so a stage costs at most about a hundred calls and a decode a +few hundred. The function is held in a `thread_local` and the closure handed to +`WalkerControl::with_progress` captures nothing, because a `js_sys::Function` +is not `Send + Sync`. The worker posts each report from inside the wasm call +(`{ type: 'stage', stage, done, total, wasmMemoryBytes }`), and the browser +delivers it to the page while the call runs, because posting does not wait for +the worker's event loop. + +The bar fills while a stage has a total above one and sweeps otherwise. The +decoder reports nothing between the last partition stream's inflate and the +saved graphics (the model is built there) or after the saved meshes (the +diagnostics and the copy out of wasm), so the card names those stretches +"Building the model" and "Finishing the model" and sweeps; the steps after the +open call (scene graph, glTF, schedule) are the worker's own and sweep too. +How much of an open that is, measured, is under +[Models too large for the browser](#models-too-large-for-the-browser). The +viewer does not cancel a decode: the call is synchronous in the worker, so a +`CancelToken` has no one to set it. The "about N s" hint from the file's size +is what the card says until the first report. + +## Models too large for the browser + +A wasm32 instance addresses at most 4 GiB, and a single `Vec` cannot pass +2 GiB (`isize::MAX`). A `Vec` growing past that panics with `capacity overflow`, +which the panic hook writes to `console.error`; an allocation that +`memory.grow` cannot satisfy does not panic: Rust aborts, no hook runs, and +JavaScript sees `RuntimeError: unreachable` and nothing else. During an open the +file is also held twice in wasm memory: the binding copies `bytes` in, and +`RevitFile::open_bytes_with_limits(bytes.to_vec())` copies it again. + +What the viewer does (#1195): + +- **Before the decode**, from the file's size alone: it expects the open to need + at least about ten times the file (`PEAK_MEMORY_PER_FILE_BYTE` in + `viewer/src/main.ts`), and when that passes 4 GiB (a file over about 410 MiB) + the card says the file may be too large, with the estimate and the figure it + rests on, and the open goes ahead. It is a warning, never a refusal, and it + can fire on a file that fits (below). +- **When the instance stops**, the worker decides whether it ran out of memory: + a `RuntimeError` of the trap with no `rvt-rs wasm panic:` line on + `console.error` before it (`unreachable` in V8, `unreachable executed` in + Firefox, `Unreachable code should not be executed` in Safari, matched as + `/unreach/i`), or a panic or engine message that names an allocation + (`capacity overflow`, `memory allocation of N bytes failed`, `Array buffer + allocation failed`). Any other panic is a bug, and the page shows the panic's + own message and nothing about memory. The worker reports the wasm memory the + instance had reached, the decoder's last report (so a stop in the stretch it + reports nothing in says "building the model") or, once the open call has + returned, the worker step it was in, and the engine's own message, which the + status panel shows as "Engine message". The worker is ended afterwards. +- **What the page says of a memory stop** depends on how far the instance got. + When it reached half the ceiling, or the file's estimate passes the ceiling, + the page says "Too large for the browser": how far it got, that a tab holds at + most 4 GiB and an open has taken at least about ten times the file, and what + reads a larger file as the code takes it: `rvt-ifc` and `rvt-gltf` with + `--large-model` (streams up to 2 GiB and, on a 64-bit machine with the memory, + inflating up to 8 GiB; they have no flag for `max_file_bytes`, the 2 GiB cap on + the whole file, so a larger file is rejected whatever the flags), and the + Python `RevitFile` with a larger `max_stream_bytes` and `max_inflate_bytes`, + and `max_file_bytes` for a file over 2 GiB; it has no `--large-model` preset. + The threshold is half the ceiling because one wasm32 allocation cannot pass + 2 GiB (`isize::MAX`): an allocation that failed with under 2 GiB already held + was not stopped by the 4 GiB ceiling alone, while one that failed with 2 GiB + or more held can have been. A browser that gives a tab less than 4 GiB (a + phone, say) stops below half of the 4 GiB the page assumes, and gets the + hedged wording for a model that is truly too large for it. When the instance + stopped a long way short of the ceiling on a file whose estimate fits, such as + a damaged file tripping `capacity overflow` at 60 MiB, a claim that the file + needs more memory would be one nothing measured, so the page says "Ran out of + memory", that the file may be damaged or may have hit a limit other than the + browser's memory, and that running the CLI on it tells which. A tab the + browser kills outright sends nothing; the page shows an error for a worker + that stops. + +The pre-check's figure is measured, not assumed. Deploy viewer's +`tests/saved-meshes-measure.spec.ts` opens each reference model in a fresh +Chromium 156 context on GitHub's ubuntu-24.04 runner (4 CPUs, 16 GB) and +records the wasm linear memory after the open, which only grows and so is the +peak, beside the estimate (ten times the file) and their ratio (Deploy viewer run +[41700152800](https://github.com/DrunkOnJava/rvt-rs/actions/runs/41700152800), +saved meshes on; the same peaks with them off, and the same in the two runs +before it): + +| Model | File MiB | Wasm peak MiB | Peak / file (about) | Estimate MiB | Estimate / peak | +|---|---|---|---|---|---| +| 2024_Core_Interior | 32 | 324 | 10 | 322 | 0.99 | +| RE1-Electrical | 13 | 170 | 13 | 128 | 0.76 | +| RE1-Architecture | 6 | 100 | 16 | 63 | 0.63 | +| RE1-Plumbing | 3 | 60 | 18 | 33 | 0.56 | +| RE1-Mechanical | 3 | 71 | 22 | 32 | 0.45 | +| Revit_IFC5_Einhoven | 0.9 | 23 | 27 | 9 | 0.37 | + +The estimate is under the peak on every model (0.37 to 0.99), tightest on the +largest, because fixed costs weigh more on a small file; "at least about ten +times" is the lowest multiple measured, and it is the largest model's. The +multiple falls as files grow (27 times down to 10 across these six), no model +past 32 MiB is in CI, and the only larger evidence goes the other way: native +peaks of 3.7 GB for a 438 MB file (8.4 times, with the stream and inflate limits +raised) and 5.1 GB for a 1.13 GB file (4.5 times), both from #1075 and from +outside the browser. So the estimate under-counts every small model and may +over-count a large one: a 438 MB file is warned about (10 times is 4.1 GiB) +though 8.4 times is 3.4 GiB, and may fit once wasm's second copy of the file is +added (0.4 GiB more). The warning can therefore fire on a file that fits, which +is why it is a warning, and the stop is the verdict. The size of the file is the +basis because the sizes the reader knows before +inflating add nothing to it: a partition's stored size is nearly all of the +file (the largest stream is 96% of the 438 MB file and 98% of the 1.13 GB +one in #1075), and the inflated size is not stored (Revit's truncated gzip has +no `ISIZE` trailer). + +The options for opening a larger model, as of 2026-11-03. Only the first three +rows are measured here; the rest are read from the code and from the release +notes cited, and none is built: + +| Option | What it would do | Status | +|---|---|---| +| Say so, and point to the CLI | The viewer's answer today: warn from the file size, report the stop, name `rvt-ifc`, `rvt-gltf` and Python | Shipped (#1195) | +| Skip the saved-graphics pass | Free the memory of the pass that draws door and window meshes | No gain: peak memory is identical with it on and off on every reference model (#1150), so it stayed inside the export's peak | +| Read fewer streams | Leave `Partitions/*` streams unread | The streams are the model: `stream_not_read_resource_limit` (#1075) already leaves a model empty of what they hold | +| One copy of the file | A binding that takes the file as an owned `Vec` saves one copy of the file (a tenth of the peak at ten times) | Not done: retyping `bytes` is a breaking change to the stable wasm API (`tools/ci/wasm-api.py`), so it needs a new binding or a `!` PR | +| Stream partitions | Inflate, use and drop one partition at a time | A library change: `RevitFile::inflated_partition` keeps every partition for the handle's lifetime on purpose (#266 took Core Interior from 26 s to 1.7 s natively by inflating each once), and a dozen passes read them | +| wasm64 (memory64) | Raise the ceiling past 4 GiB | Chrome 133 and Firefox 134 ship memory64; Safari has it in Technology Preview, and an iOS Safari 27 report says a 64-bit `Memory` is accepted but a memory64 module does not compile ([wllama#297](https://github.com/ngxson/wllama/issues/297)). wasm-bindgen supports the `wasm64-unknown-unknown` target since 0.2.120 ([release notes](https://github.com/wasm-bindgen/wasm-bindgen/releases), 2026-04-28; the lock has 0.2.118). The target is tier 3 (nightly and `build-std`). It would be a second wasm build chosen by feature detection, with the 32-bit build kept for Safari | + +Where an open's time goes is in the second table of +`tests/saved-meshes-measure.spec.ts`, built from the reports the page receives, +and it limits what the progress bar can show. On Core Interior (Deploy viewer +[41700152800](https://github.com/DrunkOnJava/rvt-rs/actions/runs/41700152800), +the open takes 17 s alone in this run, and 22 s and 24 s in earlier runs on other +runners): the schema, the candidate scan and the element +decode take 0.1 s (the file has no `Global/Latest` candidates; its elements +come from the partition records), and the `partition-scan` stage covers 15.9 s, +of which the first 2.5 s are real counts. `tests/load-progress.spec.ts` opens +the same file while it samples the card, on a runner shared with another test, +so it is slower (31.6 s) but has the same shape: the card fills 12% for each of +the 8 partition streams inflated, from 0.65 s to 3.1 s, and after the last +inflate the decoder reports nothing until the saved graphics start, 26.4 s later +(84% of that open): the +partition-schema recovery (levels, materials, rooms, walls, doors, windows, +slabs and products, each a pass over the inflated partitions) and the export's +own passes. The saved graphics take 0.3 s and the saved meshes 0.3 s, and +the three worker steps 1.4 s. The RE1 models have the same shape: 4.6 s of +Architecture's 5.1 s and 2.6 s of Plumbing's 3.0 s are under `partition-scan`. +So the bar is determinate for a tenth of a Core Interior open, and the card says +"Building the model" with a sweep for the rest. Reporting progress through the +partition-schema recovery is the work that would change that; it needs a +control passed into those passes, which is a library change this one does not +make. ## Full frontend pipeline @@ -300,7 +471,7 @@ describing the error. | Binding | Arguments | Returns | What it does | |---|---|---|---| | `openRvtBytesWithDiagnosticsMode` | `bytes`, `mode: string` | `{ model, diagnostics }` | The same at an export quality mode: `scaffold`, `typed-no-geometry`, `geometry` or `strict`, the stronger ones failing when the file cannot meet them. Doors and windows are drawn from their saved meshes, as in the next row with the default options | -| `openRvtBytesWithDiagnosticsModeAndOpenLimits` | `bytes`, `mode: string`, `limits: { largeModel?, maxStreamBytes?, maxInflateBytes?, savedMeshes? }` or `null` | `{ model, diagnostics }` | `openRvtBytesWithDiagnosticsMode` under the reader's resource limits: `largeModel` is `rvt-ifc --large-model` (streams up to 2 GiB, inflating up to the 4 GiB wasm32 holds, the walker's scan cap lifted to match), `maxStreamBytes` and `maxInflateBytes` are whole numbers of bytes that win over it, and a field left out keeps its default. `savedMeshes` (default `true`) draws the doors and windows of Revit 2023 to 2027 files from the meshes Revit saved for them, as `rvt-ifc` does by default; `false` keeps their bounding boxes and skips the pass over the saved graphics. A stream a limit stops is listed in the diagnostics (`stream_not_read_resource_limit`, a `Not read whole:` warning each) and the viewer offers "Retry with large-model limits"; unknown fields and fractional or negative limits throw. When the saved graphics cannot be read the model keeps its boxes and the diagnostics carry a warning | +| `openRvtBytesWithDiagnosticsModeAndOpenLimits` | `bytes`, `mode: string`, `limits: { largeModel?, maxStreamBytes?, maxInflateBytes?, savedMeshes? }` or `null`, `progress?: (stage: string, done: number, total: number \| null) => void` | `{ model, diagnostics }` | `openRvtBytesWithDiagnosticsMode` under the reader's resource limits: `largeModel` is `rvt-ifc --large-model` (streams up to 2 GiB, inflating up to the 4 GiB wasm32 holds, the walker's scan cap lifted to match), `maxStreamBytes` and `maxInflateBytes` are whole numbers of bytes that win over it, and a field left out keeps its default. `savedMeshes` (default `true`) draws the doors and windows of Revit 2023 to 2027 files from the meshes Revit saved for them, as `rvt-ifc` does by default; `false` keeps their bounding boxes and skips the pass over the saved graphics. A stream a limit stops is listed in the diagnostics (`stream_not_read_resource_limit`, a `Not read whole:` warning each) and the viewer offers "Retry with large-model limits"; unknown fields and fractional or negative limits throw. When the saved graphics cannot be read the model keeps its boxes and the diagnostics carry a warning. `progress`, when given, is called as the decoder works with the stage name (`schema-parse`, `candidate-scan`, `element-decode`, `partition-scan`, `saved-graphics`, `saved-meshes`) and how many of how many it has done (`total` is `null` where the stage has no count yet), at a stage's start, its end and each whole percent between; see [Decode progress](#decode-progress). A function that throws is ignored | | `quickSummary` | `bytes` | `Summary` | Release, build, PartAtom and the stream inventory, read without decoding the model | | `fileMetadata` | `bytes` | `FileMetadata` | Document identity (release, worksharing, last save, GUID, save counter), from the two identity streams only | | `scheduleCsv` | `model`, `kind: string`, `metric: boolean`, `excel: boolean` | `string` | The element (`kind` `"elements"`) or room (`"rooms"`) schedule as CSV, the columns of the [user guide](user-guide.md#get-a-schedule-into-excel) | diff --git a/docs/viewer-build-pipeline.md b/docs/viewer-build-pipeline.md index 476ad8262..8cf3fd912 100644 --- a/docs/viewer-build-pipeline.md +++ b/docs/viewer-build-pipeline.md @@ -318,9 +318,10 @@ Natively, `rvt-ifc --mode geometry` on Core Interior went from Silicon, `/usr/bin/time -l`, best of three); in the browser the same 33.7 MB file decodes in **about 3 s**, measured on the deployed site. Output is byte-identical. The viewer's loading-card estimate -follows that measurement — `DECODE_BYTES_PER_SECOND` is 11 MB/s -since #271 — and the bar stays indeterminate, because a chunked -progress signal is still the unshipped work described above. +opened with that measurement — `DECODE_BYTES_PER_SECOND` is 11 MB/s +since #271 — and the bar swept, because the decoder reported no +progress. Since #1195 it does: the card shows the decoder's own stages +(see [`viewer-api.md`](viewer-api.md#decode-progress)). ## VW1-23 — Drag-and-drop user RVT support diff --git a/rvt-wasm/Cargo.toml b/rvt-wasm/Cargo.toml index 66c557270..fc415a646 100644 --- a/rvt-wasm/Cargo.toml +++ b/rvt-wasm/Cargo.toml @@ -22,6 +22,7 @@ workspace = true [dependencies] rvt = { path = ".." } wasm-bindgen = "0.2" +js-sys = "0.3" serde = { version = "1", features = ["derive"] } serde-wasm-bindgen = "0.6" diff --git a/rvt-wasm/src/lib.rs b/rvt-wasm/src/lib.rs index fd4c77c15..0cec0bcaf 100644 --- a/rvt-wasm/src/lib.rs +++ b/rvt-wasm/src/lib.rs @@ -19,10 +19,12 @@ //! compiled `.wasm` for those imports and fails the build if any //! appear. +use std::cell::RefCell; + use wasm_bindgen::prelude::*; use rvt::RevitFile; -use rvt::control::WalkerControl; +use rvt::control::{ProgressEvent, Stage, WalkerControl}; use rvt::ifc::{ ExportQualityMode, ExportResult, IfcModel, RvtDocExporter, gltf::model_to_glb, @@ -78,6 +80,7 @@ pub fn open_rvt_bytes_with_diagnostics_mode(bytes: &[u8], mode: &str) -> Result< OpenLimits::default(), rvt::walker::WalkerLimits::default(), true, + None, ) } @@ -106,15 +109,105 @@ pub fn open_rvt_bytes_with_diagnostics_mode(bytes: &[u8], mode: &str) -> Result< /// skipped item and a `Not read whole:` warning per stream, #1075), and the /// model is missing what it holds. Unknown fields and fractional, negative /// or non-finite byte limits throw. +/// +/// `progress`, when given, is a JS function `(stage, done, total)` called as +/// the decoder works: `stage` is a [`Stage`] name as `rvt-gltf --progress` +/// prints it (`schema-parse`, `candidate-scan`, `element-decode`, +/// `partition-scan`, `saved-graphics`, `saved-meshes`), `done` a count and +/// `total` the count it is out of, or `null` where the stage has no +/// denominator yet (#1195). It is called when a stage begins, when it ends, +/// and each time a stage with a total passes another whole percent, so a +/// stage costs at most about a hundred calls. An exception it throws is +/// ignored. #[wasm_bindgen(js_name = openRvtBytesWithDiagnosticsModeAndOpenLimits)] pub fn open_rvt_bytes_with_diagnostics_mode_and_open_limits( bytes: &[u8], mode: &str, limits: JsValue, + progress: Option, ) -> Result { install_panic_hook(); let (open_limits, walker_limits, saved_meshes) = limits_from_js(limits)?; - export_with_limits(bytes, mode, open_limits, walker_limits, saved_meshes) + export_with_limits( + bytes, + mode, + open_limits, + walker_limits, + saved_meshes, + progress, + ) +} + +/// The JS function a decode reports its progress to, and the last report it +/// forwarded. A `js_sys::Function` is not `Send + Sync`, which +/// [`WalkerControl::with_progress`] requires, so it sits here, where one +/// single-threaded wasm instance reads it, and the closure handed to the +/// control captures nothing. +struct ProgressSink { + callback: js_sys::Function, + last: Option<(Stage, Option)>, +} + +thread_local! { + static PROGRESS: RefCell> = const { RefCell::new(None) }; +} + +/// The control of one decode: reports to `progress` when a function is given +/// and does nothing otherwise. +fn progress_control(progress: Option) -> WalkerControl { + match progress { + Some(callback) => { + PROGRESS.with(|slot| { + *slot.borrow_mut() = Some(ProgressSink { + callback, + last: None, + }); + }); + WalkerControl::new().with_progress(forward_progress) + } + None => WalkerControl::default(), + } +} + +fn clear_progress() { + PROGRESS.with(|slot| { + slot.borrow_mut().take(); + }); +} + +/// Hands `event` to the JS function unless it is a report within the same +/// whole percent of the same stage as the last one handed over. The decoder +/// reports each megabyte scanned and every 256th candidate, which the page +/// has no use for; a stage with no total reports once, when it begins. +fn forward_progress(event: ProgressEvent) { + PROGRESS.with(|slot| { + let Ok(mut slot) = slot.try_borrow_mut() else { + return; + }; + let Some(sink) = slot.as_mut() else { + return; + }; + let percent = event + .total + .filter(|total| *total > 0) + .map(|total| event.done.saturating_mul(100) / total); + let finished = event.total.is_some_and(|total| event.done >= total); + let same_stage = sink.last.is_some_and(|(stage, _)| stage == event.stage); + let same_percent = sink.last.is_some_and(|(_, last)| last == percent); + if same_stage && same_percent && !finished { + return; + } + sink.last = Some((event.stage, percent)); + let total = event + .total + .map_or(JsValue::NULL, |total| JsValue::from_f64(total as f64)); + let _ = sink.callback.call3( + &JsValue::NULL, + &JsValue::from_str(event.stage.as_str()), + &JsValue::from_f64(event.done as f64), + &total, + ); + }); } fn export_with_limits( @@ -123,14 +216,41 @@ fn export_with_limits( open_limits: OpenLimits, walker_limits: rvt::walker::WalkerLimits, saved_meshes: bool, + progress: Option, +) -> Result { + let control = progress_control(progress); + let result = export_with_control( + bytes, + mode, + open_limits, + walker_limits, + saved_meshes, + &control, + ); + clear_progress(); + result +} + +fn export_with_control( + bytes: &[u8], + mode: &str, + open_limits: OpenLimits, + walker_limits: rvt::walker::WalkerLimits, + saved_meshes: bool, + control: &WalkerControl, ) -> Result { let quality_mode = ExportQualityMode::parse(mode).map_err(err_str)?; let mut rf = RevitFile::open_bytes_with_limits(bytes.to_vec(), open_limits).map_err(err_str)?; let mut result = RvtDocExporter - .export_with_diagnostics_mode_and_limits(&mut rf, quality_mode, walker_limits) + .export_with_diagnostics_mode_limits_and_control( + &mut rf, + quality_mode, + walker_limits, + control, + ) .map_err(err_str)?; if saved_meshes { - attach_saved_meshes(&mut rf, &mut result); + attach_saved_meshes(&mut rf, &mut result, control); } serde_wasm_bindgen::to_value(&result).map_err(err_str) } @@ -140,16 +260,12 @@ fn export_with_limits( /// 2027 only), and recount its diagnostics. Saved graphics only improve /// bodies: when they cannot be read the model keeps its bounding boxes and the /// diagnostics carry a warning saying so. -fn attach_saved_meshes(rf: &mut RevitFile, result: &mut ExportResult) { +fn attach_saved_meshes(rf: &mut RevitFile, result: &mut ExportResult, control: &WalkerControl) { let Some(scope) = rvt::ifc::saved_meshes::default_scope(rf) else { return; }; - match rvt::ifc::saved_meshes::attach_scoped_with_control( - rf, - &mut result.model, - scope, - &WalkerControl::default(), - ) { + match rvt::ifc::saved_meshes::attach_scoped_with_control(rf, &mut result.model, scope, control) + { Ok(_) => rvt::ifc::recount_exported(&mut result.diagnostics, &result.model), Err(e) => result.diagnostics.warnings.push(format!( "Saved graphics not read, door and window bodies unchanged: {e}" diff --git a/src/ifc/mod.rs b/src/ifc/mod.rs index 5a48f126f..7b5bf187e 100644 --- a/src/ifc/mod.rs +++ b/src/ifc/mod.rs @@ -1016,7 +1016,27 @@ impl RvtDocExporter { quality_mode: ExportQualityMode, limits: crate::walker::WalkerLimits, ) -> Result { - let model = self.export_with_mode_and_limits(rf, quality_mode, limits)?; + self.export_with_diagnostics_mode_limits_and_control( + rf, + quality_mode, + limits, + &crate::control::WalkerControl::default(), + ) + } + + /// Same as [`Self::export_with_diagnostics_mode_and_limits`], with the + /// walk's progress reported to `control` and a cancelled token stopping + /// the export with [`crate::Error::Cancelled`], as + /// [`Self::export_with_mode_limits_and_control`] does for the model alone. + /// The diagnostics are built after the walk and report nothing. + pub fn export_with_diagnostics_mode_limits_and_control( + &self, + rf: &mut crate::RevitFile, + quality_mode: ExportQualityMode, + limits: crate::walker::WalkerLimits, + control: &crate::control::WalkerControl, + ) -> Result { + let model = self.export_with_mode_limits_and_control(rf, quality_mode, limits, control)?; let diagnostics = build_export_diagnostics_with_limits( rf, &model, @@ -1113,6 +1133,19 @@ fn export_rvt_doc( quality_mode: ExportQualityMode, walker_limits: crate::walker::WalkerLimits, control: &crate::control::WalkerControl, +) -> Result { + rf.report_partition_inflates_to(control); + let result = export_rvt_doc_reporting(rf, mode, quality_mode, walker_limits, control); + rf.stop_reporting_partition_inflates(); + result +} + +fn export_rvt_doc_reporting( + rf: &mut crate::RevitFile, + mode: RvtDocExportMode, + quality_mode: ExportQualityMode, + walker_limits: crate::walker::WalkerLimits, + control: &crate::control::WalkerControl, ) -> Result { let policy = ExportContentPolicy::for_quality_mode(quality_mode); // Identity from PartAtom if present; fall back to the file name in diff --git a/src/reader.rs b/src/reader.rs index 3abfcf413..313fa620b 100644 --- a/src/reader.rs +++ b/src/reader.rs @@ -176,6 +176,14 @@ pub struct RevitFile { /// production walk, the partition MVP and the IFC export (twice) each ran /// the same scan over every partition. arc_wall_scan: Option, + /// Where an export in progress reports each `Partitions/*` stream it + /// inflates, and how many there are ([`Self::report_partition_inflates_to`]). + /// Held as `AssertUnwindSafe` so the handle keeps the unwind-safety it + /// has without one: a control wraps a caller's closure, which is neither. + partition_progress: Option<( + std::panic::AssertUnwindSafe, + u64, + )>, /// Memoised data objects (RE-153) of each partition stream: the /// original symbols, railing types and MEP systems each scanned every /// partition for them. @@ -433,6 +441,7 @@ impl RevitFile { phase_filter: None, walk: None, arc_wall_scan: None, + partition_progress: None, data_objects: std::collections::HashMap::new(), header_hits: std::collections::HashMap::new(), native_preamble: None, @@ -554,9 +563,37 @@ impl RevitFile { let inflated = Arc::new(inflated); self.inflated .insert(name.to_string(), Arc::clone(&inflated)); + if let Some((control, total)) = &self.partition_progress { + let done = self + .inflated + .keys() + .filter(|key| key.starts_with("Partitions/")) + .count() as u64; + control.report( + crate::control::Stage::PartitionScan, + done.min(*total), + Some(*total), + ); + } Ok(inflated) } + /// From now until [`Self::stop_reporting_partition_inflates`], reports to + /// `control` each time a `Partitions/*` stream is inflated for the first + /// time, as [`crate::control::Stage::PartitionScan`] with the number + /// inflated so far out of the number of partition streams. Whichever pass + /// reads the partitions first inflates them all, in turn, and on a project + /// that is most of an open's time, in passes that have no control of their + /// own. + pub(crate) fn report_partition_inflates_to(&mut self, control: &crate::control::WalkerControl) { + let total = self.partition_stream_names().len() as u64; + self.partition_progress = Some((std::panic::AssertUnwindSafe(control.clone()), total)); + } + + pub(crate) fn stop_reporting_partition_inflates(&mut self) { + self.partition_progress = None; + } + /// The data objects (RE-153) of partition stream `name`, each at its /// offset in the stream's inflated bytes ([`Self::inflated_partition`]), /// memoised. diff --git a/viewer/README.md b/viewer/README.md index df86c640f..1f8c6ef3f 100644 --- a/viewer/README.md +++ b/viewer/README.md @@ -73,6 +73,20 @@ none it runs that cargo build itself; when cargo is unavailable it warns `synthetic-mvp: missing (gen-fixture not found …)` and the gallery marks the demo unavailable. Nothing is downloaded either way. +## Progress while a model opens + +The loading card shows the decoder's own progress, the stages `rvt-gltf +--progress` prints: the bar fills with a stage's count while it has one +(scanning for elements, decoding elements, reading partition streams, drawing +doors and windows from saved meshes) and sweeps for the rest, and the card +names the stage and counts it. `src/worker.ts` passes a function to +`openRvtBytesWithDiagnosticsModeAndOpenLimits` and posts each report from +inside the wasm call; the page keeps the list on `#status` as +`data-load-progress`. The "about N s" estimate from the file's size is what +the card says until the first report. `tests/load-progress.spec.ts` watches +the card fill on Core Interior. The stages and what the card cannot show are +in [`docs/viewer-api.md`](../docs/viewer-api.md#decode-progress). + ## Large models and resource limits A file opens under the reader's default limits (256 MiB per stream and per @@ -80,9 +94,33 @@ inflated member). When a stream is over them, File status reports "Part of the file not read" and offers "Retry with large-model limits", which reads the same file again with the limits `rvt-ifc --large-model` uses (streams up to 2 GiB). A browser tab can use at most 4 GiB of memory, so a very large model can still -fail to open; `rvt-ifc --large-model` has no such ceiling. The button calls +fail to open; `rvt-ifc --large-model` is not held to a tab's 4 GiB (its caps are +a 2 GiB stream and, on a 64-bit machine with the memory, 8 GiB inflated, and a +file over 2 GiB is rejected whatever the flags). The button calls `openRvtBytesWithDiagnosticsModeAndOpenLimits` with `{ largeModel: true }`. +A file whose size says it may need more than a tab's 4 GiB (an open has taken +at least about ten times the file's size, and the warning can fire on a file +that fits) is warned about in the loading card before the decode, and still +tried. When the wasm instance does run out of memory, the viewer says the model +is too large for the browser, how much memory the tab reached, and what reads a +larger file (`rvt-ifc` and `rvt-gltf` with `--large-model`, up to a 2 GiB file; +the Python `RevitFile` with a larger `max_stream_bytes`, `max_inflate_bytes` and, +over 2 GiB, `max_file_bytes`), instead of +`RuntimeError: unreachable`; the engine's own message is in the status panel. +A stop a long way short of the ceiling on a small file says "Ran out of memory" +and that the file may be damaged, since nothing measured says it needs more. How +it tells, the measured estimate and the options for larger models are in +[`docs/viewer-api.md`](../docs/viewer-api.md#models-too-large-for-the-browser). +`?memoryCeilingBytes=` in the address lowers the ceiling the warning compares +with. `tests/memory-ceiling.spec.ts` uses it for the warning, and +`tests/memory-ceiling-capped.spec.ts` for the stop, in a Chromium whose V8 caps +wasm memory (`--js-flags=--wasm-max-mem-pages=256` is 16 MiB), where Einhoven +really stops, because no model of several hundred MiB is in CI. +`tests/worker-failures.spec.ts` covers the other engines' wording of the trap, a +panic that is not memory, and a stop after the open call, by failing one wasm +export in a replacement worker; nothing in it ships. + `?maxStreamBytes=` and `?maxInflateBytes=` in the address open the first file under those limits, so a small file reaches the same path. The browser test `tests/large-model-retry.spec.ts` opens the tier-1 fixture at diff --git a/viewer/index.html b/viewer/index.html index 291d48d97..2bacbb011 100644 --- a/viewer/index.html +++ b/viewer/index.html @@ -119,26 +119,36 @@

Demo gallery