feat(viewer): the loading card shows the decoder's own progress, and a model too large for the tab says so (#1195)

changelog.d/added/1195-viewer-progress-too-large.md

@@ -0,0 +1,21 @@
+- **The browser viewer shows the decoder's own progress, and says when a
+  model is too large for a tab.** The loading card names the stage the
+  decoder is in (the stages `rvt-gltf --progress` prints) and fills its bar
+  with that stage's count, where it used to sweep and guess "about N s" from
+  the file's size. `openRvtBytesWithDiagnosticsModeAndOpenLimits` takes an
+  optional fourth argument, a function called with the stage, the count done
+  and the count it is out of. `rvt-gltf --progress` now also prints a
+  `partition-scan` line for each `Partitions/*` stream as it is first
+  inflated, which is most of a project's open, where it printed the stage's
+  start and end only. A file whose size says it may need more than a tab's
+  4 GiB (at least about ten times its size, the figure of the largest
+  reference model; the warning can fire on a file that fits) is warned about
+  before the decode and still tried. When the wasm instance runs out of
+  memory, the viewer says the model is too large for the browser, how far the
+  decode got and where, and that `rvt-ifc` and `rvt-gltf` with `--large-model`
+  (streams up to 2 GiB and, on a 64-bit machine, inflating up to 8 GiB, for a
+  file of up to 2 GiB), or Python with a larger `max_stream_bytes` and
+  `max_inflate_bytes`, can read a larger file, instead of
+  `RuntimeError: unreachable`. A stop a long way short of the ceiling says
+  "Ran out of memory" and that the file may be damaged, and a panic that is not
+  a memory failure keeps its own message (#1195).

docs/status.md

@@ -1052,10 +1052,11 @@ two fields are now pinned and two exports are byte-identical, as
 In the browser the same change takes the 33.7 MB Core Interior demo from
 about 28 s to **about 3 s** of decode, measured live on the deployed
 site (6.7 s including the 32 MB download). The viewer's loading card
-derives its "about N s" hint from that measurement —
+opened with an "about N s" hint from that measurement —
 `DECODE_BYTES_PER_SECOND` is 11 MB/s since #271, up from the 1.15 MB/s
-the pre-#266 figure implied — and the progress bar stays indeterminate
-because the decoder cannot report real progress.
+the pre-#266 figure implied. It is a laptop's figure, and since #1195 the
+card shows the decoder's own progress as soon as it reports (see
+[`viewer-api.md`](viewer-api.md#decode-progress)).
 
 ## Capability Matrix
 

docs/user-guide.md

@@ -350,8 +350,9 @@ at the default limits and, when a stream is not read, shows "Part of the file
 not read" in the File status panel with a "Retry with large-model limits"
 button, which reopens the same file with `--large-model`'s limits (streams up
 to 2 GiB). A browser tab can use at most 4 GiB of memory, so a very large
-model can still fail to open there; `rvt-ifc --large-model` has no such
-ceiling.
+model can still fail to open there; `rvt-ifc --large-model` is not held to a
+tab's 4 GiB (its caps are a 2 GiB stream and, on a 64-bit machine, 8 GiB
+inflated, and a file over 2 GiB is rejected whatever the flags).
 
 Two exports of the same file are the same bytes except for the export time,
 which the IFC header and owner history record. Set `SOURCE_DATE_EPOCH` (Unix

docs/viewer-build-pipeline.md

@@ -318,9 +318,10 @@ Natively, `rvt-ifc --mode geometry` on Core Interior went from
 Silicon, `/usr/bin/time -l`, best of three); in the browser the same
 33.7 MB file decodes in **about 3 s**, measured on the deployed
 site. Output is byte-identical. The viewer's loading-card estimate
-follows that measurement — `DECODE_BYTES_PER_SECOND` is 11 MB/s
-since #271 — and the bar stays indeterminate, because a chunked
-progress signal is still the unshipped work described above.
+opened with that measurement — `DECODE_BYTES_PER_SECOND` is 11 MB/s
+since #271 — and the bar swept, because the decoder reported no
+progress. Since #1195 it does: the card shows the decoder's own stages
+(see [`viewer-api.md`](viewer-api.md#decode-progress)).
 
 ## VW1-23 — Drag-and-drop user RVT support
 

rvt-wasm/Cargo.toml

@@ -22,6 +22,7 @@ workspace = true
 [dependencies]
 rvt = { path = ".." }
 wasm-bindgen = "0.2"
+js-sys = "0.3"
 serde = { version = "1", features = ["derive"] }
 serde-wasm-bindgen = "0.6"
 

rvt-wasm/src/lib.rs

@@ -19,10 +19,12 @@
 //! compiled `.wasm` for those imports and fails the build if any
 //! appear.
 
+use std::cell::RefCell;
+
 use wasm_bindgen::prelude::*;
 
 use rvt::RevitFile;
-use rvt::control::WalkerControl;
+use rvt::control::{ProgressEvent, Stage, WalkerControl};
 use rvt::ifc::{
     ExportQualityMode, ExportResult, IfcModel, RvtDocExporter,
     gltf::model_to_glb,
@@ -78,6 +80,7 @@ pub fn open_rvt_bytes_with_diagnostics_mode(bytes: &[u8], mode: &str) -> Result<
         OpenLimits::default(),
         rvt::walker::WalkerLimits::default(),
         true,
+        None,
     )
 }
 
@@ -106,15 +109,105 @@ pub fn open_rvt_bytes_with_diagnostics_mode(bytes: &[u8], mode: &str) -> Result<
 /// skipped item and a `Not read whole:` warning per stream, #1075), and the
 /// model is missing what it holds. Unknown fields and fractional, negative
 /// or non-finite byte limits throw.
+///
+/// `progress`, when given, is a JS function `(stage, done, total)` called as
+/// the decoder works: `stage` is a [`Stage`] name as `rvt-gltf --progress`
+/// prints it (`schema-parse`, `candidate-scan`, `element-decode`,
+/// `partition-scan`, `saved-graphics`, `saved-meshes`), `done` a count and
+/// `total` the count it is out of, or `null` where the stage has no
+/// denominator yet (#1195). It is called when a stage begins, when it ends,
+/// and each time a stage with a total passes another whole percent, so a
+/// stage costs at most about a hundred calls. An exception it throws is
+/// ignored.
 #[wasm_bindgen(js_name = openRvtBytesWithDiagnosticsModeAndOpenLimits)]
 pub fn open_rvt_bytes_with_diagnostics_mode_and_open_limits(
     bytes: &[u8],
     mode: &str,
     limits: JsValue,
+    progress: Option<js_sys::Function>,
 ) -> Result<JsValue, JsValue> {
     install_panic_hook();
     let (open_limits, walker_limits, saved_meshes) = limits_from_js(limits)?;
-    export_with_limits(bytes, mode, open_limits, walker_limits, saved_meshes)
+    export_with_limits(
+        bytes,
+        mode,
+        open_limits,
+        walker_limits,
+        saved_meshes,
+        progress,
+    )
+}
+
+/// The JS function a decode reports its progress to, and the last report it
+/// forwarded. A `js_sys::Function` is not `Send + Sync`, which
+/// [`WalkerControl::with_progress`] requires, so it sits here, where one
+/// single-threaded wasm instance reads it, and the closure handed to the
+/// control captures nothing.
+struct ProgressSink {
+    callback: js_sys::Function,
+    last: Option<(Stage, Option<u64>)>,
+}
+
+thread_local! {
+    static PROGRESS: RefCell<Option<ProgressSink>> = const { RefCell::new(None) };
+}
+
+/// The control of one decode: reports to `progress` when a function is given
+/// and does nothing otherwise.
+fn progress_control(progress: Option<js_sys::Function>) -> WalkerControl {
+    match progress {
+        Some(callback) => {
+            PROGRESS.with(|slot| {
+                *slot.borrow_mut() = Some(ProgressSink {
+                    callback,
+                    last: None,
+                });
+            });
+            WalkerControl::new().with_progress(forward_progress)
+        }
+        None => WalkerControl::default(),
+    }
+}
+
+fn clear_progress() {
+    PROGRESS.with(|slot| {
+        slot.borrow_mut().take();
+    });
+}
+
+/// Hands `event` to the JS function unless it is a report within the same
+/// whole percent of the same stage as the last one handed over. The decoder
+/// reports each megabyte scanned and every 256th candidate, which the page
+/// has no use for; a stage with no total reports once, when it begins.
+fn forward_progress(event: ProgressEvent) {
+    PROGRESS.with(|slot| {
+        let Ok(mut slot) = slot.try_borrow_mut() else {
+            return;
+        };
+        let Some(sink) = slot.as_mut() else {
+            return;
+        };
+        let percent = event
+            .total
+            .filter(|total| *total > 0)
+            .map(|total| event.done.saturating_mul(100) / total);
+        let finished = event.total.is_some_and(|total| event.done >= total);
+        let same_stage = sink.last.is_some_and(|(stage, _)| stage == event.stage);
+        let same_percent = sink.last.is_some_and(|(_, last)| last == percent);
+        if same_stage && same_percent && !finished {
+            return;
+        }
+        sink.last = Some((event.stage, percent));
+        let total = event
+            .total
+            .map_or(JsValue::NULL, |total| JsValue::from_f64(total as f64));
+        let _ = sink.callback.call3(
+            &JsValue::NULL,
+            &JsValue::from_str(event.stage.as_str()),
+            &JsValue::from_f64(event.done as f64),
+            &total,
+        );
+    });
 }
 
 fn export_with_limits(
@@ -123,14 +216,41 @@ fn export_with_limits(
     open_limits: OpenLimits,
     walker_limits: rvt::walker::WalkerLimits,
     saved_meshes: bool,
+    progress: Option<js_sys::Function>,
+) -> Result<JsValue, JsValue> {
+    let control = progress_control(progress);
+    let result = export_with_control(
+        bytes,
+        mode,
+        open_limits,
+        walker_limits,
+        saved_meshes,
+        &control,
+    );
+    clear_progress();
+    result
+}
+
+fn export_with_control(
+    bytes: &[u8],
+    mode: &str,
+    open_limits: OpenLimits,
+    walker_limits: rvt::walker::WalkerLimits,
+    saved_meshes: bool,
+    control: &WalkerControl,
 ) -> Result<JsValue, JsValue> {
     let quality_mode = ExportQualityMode::parse(mode).map_err(err_str)?;
     let mut rf = RevitFile::open_bytes_with_limits(bytes.to_vec(), open_limits).map_err(err_str)?;
     let mut result = RvtDocExporter
-        .export_with_diagnostics_mode_and_limits(&mut rf, quality_mode, walker_limits)
+        .export_with_diagnostics_mode_limits_and_control(
+            &mut rf,
+            quality_mode,
+            walker_limits,
+            control,
+        )
         .map_err(err_str)?;
     if saved_meshes {
-        attach_saved_meshes(&mut rf, &mut result);
+        attach_saved_meshes(&mut rf, &mut result, control);
     }
     serde_wasm_bindgen::to_value(&result).map_err(err_str)
 }
@@ -140,16 +260,12 @@ fn export_with_limits(
 /// 2027 only), and recount its diagnostics. Saved graphics only improve
 /// bodies: when they cannot be read the model keeps its bounding boxes and the
 /// diagnostics carry a warning saying so.
-fn attach_saved_meshes(rf: &mut RevitFile, result: &mut ExportResult) {
+fn attach_saved_meshes(rf: &mut RevitFile, result: &mut ExportResult, control: &WalkerControl) {
     let Some(scope) = rvt::ifc::saved_meshes::default_scope(rf) else {
         return;
     };
-    match rvt::ifc::saved_meshes::attach_scoped_with_control(
-        rf,
-        &mut result.model,
-        scope,
-        &WalkerControl::default(),
-    ) {
+    match rvt::ifc::saved_meshes::attach_scoped_with_control(rf, &mut result.model, scope, control)
+    {
         Ok(_) => rvt::ifc::recount_exported(&mut result.diagnostics, &result.model),
         Err(e) => result.diagnostics.warnings.push(format!(
             "Saved graphics not read, door and window bodies unchanged: {e}"

src/ifc/mod.rs

@@ -1016,7 +1016,27 @@ impl RvtDocExporter {
         quality_mode: ExportQualityMode,
         limits: crate::walker::WalkerLimits,
     ) -> Result<ExportResult> {
-        let model = self.export_with_mode_and_limits(rf, quality_mode, limits)?;
+        self.export_with_diagnostics_mode_limits_and_control(
+            rf,
+            quality_mode,
+            limits,
+            &crate::control::WalkerControl::default(),
+        )
+    }
+
+    /// Same as [`Self::export_with_diagnostics_mode_and_limits`], with the
+    /// walk's progress reported to `control` and a cancelled token stopping
+    /// the export with [`crate::Error::Cancelled`], as
+    /// [`Self::export_with_mode_limits_and_control`] does for the model alone.
+    /// The diagnostics are built after the walk and report nothing.
+    pub fn export_with_diagnostics_mode_limits_and_control(
+        &self,
+        rf: &mut crate::RevitFile,
+        quality_mode: ExportQualityMode,
+        limits: crate::walker::WalkerLimits,
+        control: &crate::control::WalkerControl,
+    ) -> Result<ExportResult> {
+        let model = self.export_with_mode_limits_and_control(rf, quality_mode, limits, control)?;
         let diagnostics = build_export_diagnostics_with_limits(
             rf,
             &model,
@@ -1113,6 +1133,19 @@ fn export_rvt_doc(
     quality_mode: ExportQualityMode,
     walker_limits: crate::walker::WalkerLimits,
     control: &crate::control::WalkerControl,
+) -> Result<IfcModel> {
+    rf.report_partition_inflates_to(control);
+    let result = export_rvt_doc_reporting(rf, mode, quality_mode, walker_limits, control);
+    rf.stop_reporting_partition_inflates();
+    result
+}
+
+fn export_rvt_doc_reporting(
+    rf: &mut crate::RevitFile,
+    mode: RvtDocExportMode,
+    quality_mode: ExportQualityMode,
+    walker_limits: crate::walker::WalkerLimits,
+    control: &crate::control::WalkerControl,
 ) -> Result<IfcModel> {
     let policy = ExportContentPolicy::for_quality_mode(quality_mode);
     // Identity from PartAtom if present; fall back to the file name in

src/reader.rs

@@ -176,6 +176,14 @@ pub struct RevitFile {
     /// production walk, the partition MVP and the IFC export (twice) each ran
     /// the same scan over every partition.
     arc_wall_scan: Option<CachedArcWallScan>,
+    /// Where an export in progress reports each `Partitions/*` stream it
+    /// inflates, and how many there are ([`Self::report_partition_inflates_to`]).
+    /// Held as `AssertUnwindSafe` so the handle keeps the unwind-safety it
+    /// has without one: a control wraps a caller's closure, which is neither.
+    partition_progress: Option<(
+        std::panic::AssertUnwindSafe<crate::control::WalkerControl>,
+        u64,
+    )>,
     /// Memoised data objects (RE-153) of each partition stream: the
     /// original symbols, railing types and MEP systems each scanned every
     /// partition for them.
@@ -433,6 +441,7 @@ impl RevitFile {
             phase_filter: None,
             walk: None,
             arc_wall_scan: None,
+            partition_progress: None,
             data_objects: std::collections::HashMap::new(),
             header_hits: std::collections::HashMap::new(),
             native_preamble: None,
@@ -554,9 +563,37 @@ impl RevitFile {
         let inflated = Arc::new(inflated);
         self.inflated
             .insert(name.to_string(), Arc::clone(&inflated));
+        if let Some((control, total)) = &self.partition_progress {
+            let done = self
+                .inflated
+                .keys()
+                .filter(|key| key.starts_with("Partitions/"))
+                .count() as u64;
+            control.report(
+                crate::control::Stage::PartitionScan,
+                done.min(*total),
+                Some(*total),
+            );
+        }
         Ok(inflated)
     }
 
+    /// From now until [`Self::stop_reporting_partition_inflates`], reports to
+    /// `control` each time a `Partitions/*` stream is inflated for the first
+    /// time, as [`crate::control::Stage::PartitionScan`] with the number
+    /// inflated so far out of the number of partition streams. Whichever pass
+    /// reads the partitions first inflates them all, in turn, and on a project
+    /// that is most of an open's time, in passes that have no control of their
+    /// own.
+    pub(crate) fn report_partition_inflates_to(&mut self, control: &crate::control::WalkerControl) {
+        let total = self.partition_stream_names().len() as u64;
+        self.partition_progress = Some((std::panic::AssertUnwindSafe(control.clone()), total));
+    }
+
+    pub(crate) fn stop_reporting_partition_inflates(&mut self) {
+        self.partition_progress = None;
+    }
+
     /// The data objects (RE-153) of partition stream `name`, each at its
     /// offset in the stream's inflated bytes ([`Self::inflated_partition`]),
     /// memoised.