diff --git a/docs/streaming/README.md b/docs/streaming/README.md index 7ef84f5..5685ae4 100644 --- a/docs/streaming/README.md +++ b/docs/streaming/README.md @@ -890,14 +890,45 @@ the uninterrupted run's output; and the kernel refuses 80 sampled bit flips of a native checkpoint, its first and last byte and a wrong identity (83 refusals), as the native build does. +The R2007 object stream crosses both ways too: `run_r2007_objects` walks for a +number of steps and leaves a `DWGQ` checkpoint, and `resume_r2007_objects` restores +one, given the saved payload CRC-32 and length, and finishes the walk, printing the +lines `stream_r2007_objects` prints. `tools/streaming/verify_wasm_r2007_objects.py` +is its acceptance run (above, in the R2007 object records section). + ## Checkpoint contract -A checkpoint contains the source identity and length, phase, byte position, -current target, locator count/index/mask, failure marker and all retained -partial header bytes. The wire form is canonical, versioned and little -endian. Its current maximum is 157 bytes (97 consumed header bytes plus 60 -bytes of metadata/checksum). The holder and every scratch buffer fit within -one declared 256-byte chunk. +Every walker can be stopped between two steps and resumed in a fresh process, +or in the wasm32 kernel, from its checkpoint and the source identity alone. +There are ten kinds. Each has its own canonical, versioned, little-endian wire +form with a four-byte marker and a CRC-32 of everything before it as its last +four bytes: + +| Marker | Type | Holds | Size | +| --- | --- | --- | --- | +| `DWGH` | `Checkpoint` | the §3.2 header parser: identity, length, phase, position, locator state, retained header bytes | at most 157 bytes | +| `DWGW` | `WalkCheckpoint` | the R13 to R2000 container walk: the header checkpoint (replayed) and the walk cursors | at most 256 bytes | +| `DWG4` | `R2004Checkpoint` | the §4.1 header of an R2004-family file | at most 316 bytes (bounded at 336) | +| `DWGZ` | `Lz77Checkpoint` | the §4.7 decoder between calls: cursors, pending run, totals and the last `min(produced, 0xC000)` output bytes | `54 + kept + 4`, at most 49,210 bytes | +| `DWGM` | `R2004MapCheckpoint` | the header checkpoint, the decoder checkpoint while a page is decoded, and the page map and section map cursors | at most about 50 KB | +| `DWGS` | `R2004SectionCheckpoint` | one section's fields, the page in hand, the cursors and, mid LZ77 page, the decoder checkpoint | at most about 50 KB | +| `DWGF` | `R2004WalkCheckpoint` | the whole-file walk: the section table (at most 64 slots), the map walker's checkpoint and, mid page, the section walker's | at most about 100 KB | +| `DWGO` | `R2004ObjectCheckpoint` | the `DWGF` checkpoint, the object section's id, the record framer's cursors and the section bytes not yet framed (at most 256) | at most about 100 KB | +| `DWGR` | `R2007Checkpoint` | the R2007 walk: cursors, the verified 0x110-byte header and each reader's progress; page geometry is derived on restore, never stored | at most `2 * R2007_WINDOW + 8192` bytes | +| `DWGQ` | `R2007ObjectCheckpoint` | the `DWGR` checkpoint, the object section's index, the framer's cursors and the section bytes not yet framed (at most 256) | at most about 140 KB | + +The sizes are the compile-time bounds in `src/streaming/checkpoint.rs` and +`r2007_checkpoint.rs` as each walker's section above states them; they do not +grow with the file. Restore checks the complete length and the marker and +version before copying, checks the CRC-32, rejects a source identity other than +the one the caller passes (every kind but `DWGZ`, which holds no source: where +its compressed bytes come from is the caller's), and then requires a state the +walk can reach. Embedded checkpoints are restored by their own restore. No +source byte is read to rebuild state. Each walker's section says what its +reachability check covers; the two object-record sections also say what a +forged checkpoint can get past. + +The `DWGH` layout, the smallest, as the others are built: | Byte offset | Encoding | | --- | --- | @@ -912,24 +943,29 @@ one declared 256-byte chunk. | 56 | Consumed prefix, including any partial field | | Last 4 | CRC32 of all preceding checkpoint bytes | -Restore checks the complete length before copying, rejects incompatible -versions and source identities, checks CRC, then validates phase and counters -by replaying only the retained bytes **inside the checkpoint**. No source -bytes are read to reconstruct state. Failure snapshots restore the same -failure. The next request starts at the first unconsumed source byte. +A `DWGH` checkpoint's maximum is 157 bytes (97 consumed header bytes plus 60 +bytes of metadata and checksum); the holder and every scratch buffer fit within +one declared 256-byte chunk. Restore replays the retained bytes inside the +checkpoint to validate the phase and counters. Failure snapshots restore the +same failure; the next request starts at the first unconsumed source byte. The caller supplies identity from a trusted content manifest or immutable -object version; the parser does not reread a multi-gigabyte source to hash -it. CRC detects accidental checkpoint corruption and provides no authenticity +object version; the parser does not reread a multi-gigabyte source to hash it. +CRC detects accidental checkpoint corruption and provides no authenticity guarantee. The caller owns atomic custody of saved state and any already -committed output. `header()` is an inspection API, not an exactly-once event -queue. Full object/decompression checkpoints remain open in #207. +committed output: what a walk emitted (its output cursors, and the digest and +length of an open record's payload) is the caller's state, and the example entry +points save it beside the checkpoint (`.caller`). `header()` is an +inspection API, not an exactly-once event queue. -The R2007 object stream crosses both ways too: `run_r2007_objects` walks for a -number of steps and leaves a `DWGQ` checkpoint, and `resume_r2007_objects` restores -one, given the saved payload CRC-32 and length, and finishes the walk, printing the -lines `stream_r2007_objects` prints. `tools/streaming/verify_wasm_r2007_objects.py` -is its acceptance run (above, in the R2007 object records section). +Fresh-process resume, rejection of corrupt, truncated, resealed and wrong-identity +checkpoints, and native-to-wasm32 restore are acceptance steps of the required +Streaming E2E job, one per walker (see Current evidence). + +What no checkpoint holds is a partly decoded object: the streaming grammar frames +records and decodes no object body (#205), so there is no object-level state to +save. **NOT MEASURED:** a restore inside a browser (the kernel runs under node, +V8), and a source larger than the manifests' largest file (1,312,281 bytes). ## Repeatable acceptance @@ -967,36 +1003,60 @@ python3 tools/streaming/verify_wasm_r2007_objects.py target/debug/examples/strea python3 tools/streaming/probe_r2007_checks.py ``` -Run these on the project's build host. The workflow uploads -`streaming-evidence.json` and the generated schedule JSONL files. It checks +Run these on the project's build host. The workflow uploads the +`streaming-evidence` artifact: one `*-evidence.json` per step and the generated +schedule JSONL files. The header step checks 12 synthetic version/count combinations, the canonical R2000 fixture, bytewise and short delivery, truncation at every header byte, malformed input, two hash-pinned real R14/R2000 headers, 128 generated schedules per real file, 98 fresh-process resume positions, actual source-read continuity, pending delivery, corrupt and inconsistent checkpoints, source mismatch and failure-state restore. -The real-file manifest records header fields, not object counts. No real R13 -file, multi-gigabyte model or independent object-count oracle has passed -this reader. The kernel WASM compile does not establish browser runtime; -cross-runtime checkpoint acceptance is established only for the R2004 walks, -the R2004 object stream, the R2007 walk and the R2007 object stream (above). +The header manifest records header fields only; the walk manifest records object +counts and the LZ77 manifest page counts and decompressed bytes, and each +walker's output is compared at run time with an independent Python parse and, where +the legacy reader reads the same thing, with the legacy reader (the sections +above). **NOT MEASURED:** a real R13 file (the corpus +holds none that AutoCAD wrote, #345), a multi-gigabyte model, and any browser +runtime: the kernel WASM compile does not establish one. Cross-runtime +checkpoint acceptance is established for the R2004 walks, the R2004 object +stream, the R2007 walk and the R2007 object stream (above). ## Current evidence -[Run 40234765141](https://github.com/DrunkOnJava/dwg-rs/actions/runs/40234765141) -passed on `a6dc6baa2d449e3f33db3bba0e3998803cc87834`: native executable -acceptance, proptest schedules, fresh-process resume and the native/WASM -no-allocator kernel builds. The evidence artifact identifies the built -binary by SHA-256. The same commit also passed the existing three WASM -wrapper build targets. Repository-wide checks are reported separately by the -PR; these claims do not establish a complete model parse or release readiness. +[Run 40532346151](https://github.com/DrunkOnJava/dwg-rs/actions/runs/40532346151) +passed the required Streaming E2E job on `main` at `e7c53e1` (October 27, +2026): the grammar-isolation and bounded-state gate, the native and wasm32 +`no_std` no-allocator kernel builds, and every acceptance step below. Its +`streaming-evidence` artifact holds one `*-evidence.json` per step, each naming +the SHA-256 of the binary it ran, and the generated schedule JSONL files. + +| Step | Evidence file | Measured | +| --- | --- | --- | +| §3.2 header | `streaming-evidence.json` | 16 cases passed | +| R13 to R2000 container walk | `walk-evidence.json` | 5,835 records over 17 files, each equal to an independent framing parse and the legacy reader | +| §4.7 LZ77 decoder | `lz77-evidence.json` | 25 files, 266 pages, 6,097,392 decompressed bytes | +| R2004-family header | `r2004-evidence.json` | 25 files | +| Page map and section map | `map-evidence.json` | 25 files, 371 pages, 306 sections, 321 section pages | +| Data pages and sections | `sections-evidence.json` | 26 files, 313 sections, 332 pages, 1,902,760 bytes, 4 holes, 2 zero pages | +| Whole R2004-family file | `file-evidence.json` | 28 files (25 pinned), 327 sections, 354 pages, 1,916,832 bytes | +| R2004-family object records | `objects-evidence.json` | 32 pinned files, 8,027 records, 6,858,006 object bytes | +| Whole R2007 file | `r2007-evidence.json` | 11 files (8 pinned, 3 built), 115 sections, 153 pages, 4,966 data codewords, 3,067,696 bytes | +| R2007 object records | `r2007-objects-evidence.json` | 16 files (8 pinned, 8 built), 2,496 records, 2,465,701 object bytes | +| `AsyncSource` over HTTP range | `source-evidence.json` | 5 cases passed (R13 to R2000 walk only) | +| Native to wasm32 and back | `wasm-evidence.json`, `wasm-objects-evidence.json`, `wasm-r2007-evidence.json`, `wasm-r2007-objects-evidence.json` | all passed, under node | + +These claims do not establish a complete model parse, a browser runtime or +release readiness. **NOT MEASURED:** a multi-gigabyte model, a real R13 file, and +the allocation behaviour of the streaming buffers beyond their compile-time +bounds. The canonical R2000 fixture's writer used to omit the file-header CRC and sentinel, so the strict reader rejected it ([#214](https://github.com/DrunkOnJava/dwg-rs/issues/214)). The generator now writes both and the acceptance run checks that file -under every chunk size from 1 to 32 bytes. Its other sections remain -placeholders. The audit and complete ordered work -are in [audit.md](audit.md) and [#204](https://github.com/DrunkOnJava/dwg-rs/issues/204). +under every chunk size from 1 to 32 bytes. The audit and complete ordered work +are in [audit.md](audit.md), [task-ledger.md](task-ledger.md) and +[#204](https://github.com/DrunkOnJava/dwg-rs/issues/204). ## Throughput diff --git a/docs/streaming/task-ledger.md b/docs/streaming/task-ledger.md index 7b2ced2..39ba7d8 100644 --- a/docs/streaming/task-ledger.md +++ b/docs/streaming/task-ledger.md @@ -9,29 +9,31 @@ CLI probe, streaming CI gate, and streaming documentation. Existing parser and peer worktrees remain under their existing custody. Build artifacts go to CI; verification receipts go to `docs/streaming/` or uploaded CI artifacts. +Last reviewed October 27, 2026 against `main` at `e7c53e1` and the Streaming +E2E job of [run 40532346151](https://github.com/DrunkOnJava/dwg-rs/actions/runs/40532346151) +(success; artifact `streaming-evidence`, whose `*-evidence.json` files the +counts below come from). A claim with no run or file behind it is marked +**NOT MEASURED**. + | Work | State | Evidence / next action | | --- | --- | --- | | Review startup drift, health queue, Git and active jobs | complete | Counts reviewed; no Rust build; existing clients preserved; isolated clean worktree | | Audit current reader against requested invariants | complete | Whole-file reads in reader; whole-section buffers; WASM copies uploaded file | | File ordered gaps in GitHub, link existing issues | complete | Epic #204 and ordered gaps #205–#212 | -| Hand-written byte-demand state machine | in progress | Header core passed remote E2E at 9a93736 (run 40234332022). R13–R2000 container walk (sections, object map, record framing, type, handle) matches an independent parse and the legacy reader on 3,546 records in 15 files; object bodies, R2004+ containers and decompression remain #205. §4.1 R2004-family header matches an independent parse and the legacy reader on 25 files with every documented check enforced; the §4.3–§4.5 page map and section map walk (checksum-verified, records committed) matches both on 25 files (371 pages, 306 sections, 321 section pages); §4.7 LZ77 decoder matches the legacy decoder and an independent parse on 266 pages of 25 R2004–R2013 files; the R2004+ walk that drives it remains #205 | -| Transport abstraction | in progress | File/slice adapters and real-file E2E passed at 422dc67 (run 40234514166); `AsyncSource` (host-driven asynchronous adapter) drives the walk over an HTTP/1.1 range worker with output equal to the file run on 3 files, 9,206 fetches each after a Pending, fresh-process resume with no refetch (including held bytes carried with the checkpoint), and 416/Content-Range/short-body/ignored-Range/oversized-206 faults typed; TLS, remote hosts and a browser worker (#209) remain | -| Portable checkpointing | in progress | Header state passed fresh-process restore at 98 byte positions, failed-state restore and malformed checkpoint rejection at a6dc6ba (run 40234765141). Walk state resumes at every feed of the canonical file and 66 per real file with no re-read or duplicate output; LZ77 decoder state resumes at every call of the canonical page and 45–46 sampled calls of three real pages; object-body state and a combined R2004+ walk checkpoint remain #207 | -| Spec compiler upgrade | pending | Only after a working hand-written reader; never patch generated output | -| WASM streaming API and native/WASM gate | pending | Existing full-buffer WASM is insufficient | -| Boundary fuzzing through real entry point | pending | proptest-driven executable, one byte and random splits | -| Golden real corpus, large-file benchmarks and comparisons | pending | No fabricated counts, fixtures or performance claims | -| Spec change watcher and automatic invariant walls | pending | Fail closed; no vacuous gate | -| Known prior prunable worktree | preserved peer artifact | Preserve; not owned by this work unit | - -Discovered: canonical `synthetic_2000.dwg` had zero header CRC and zero sentinel, so the strict §3.2 core rejected it. Fixed under #214: the generator writes both, the fixture was regenerated, and the acceptance run reads it through the strict header entry point. - -Real inputs acquired read-only outside Git: nextgis `line_R14.dwg` (122353 bytes, SHA-256 `4ae12afb4052f1be913003b84021054c0e07b08e6aef5e430bf523c6f85c0903`) and `line_2000.dwg` (116899 bytes, `f1936ccc5e26dd1aae3f6a26bf5803bdf53d8e0b24f8c18bf3fb96fc540918e0`). Independent prefix CRC/sentinel checks agree with §3.2.6. Native parser measurement pending. +| Hand-written byte-demand state machine (#205) | in progress | On `main`: the §3.2 header (#213), the R13–R2000 container walk (#240), the §4.7 LZ77 decoder (#246), the R2004-family header (#260), page map and section map (#276), data pages (#299), whole file (#312) and object records (#333), and the whole R2007 file (#340) and its object records (#363). Run 40532346151: R13–R2000 walk, 5,835 records over 17 files, each equal to an independent framing parse and the legacy reader (`walk-evidence.json`); LZ77, 25 files, 266 pages, 6,097,392 decompressed bytes (`lz77-evidence.json`); maps, 25 files, 371 pages, 306 sections (`map-evidence.json`); whole R2004-family file, 28 files, 327 sections, 354 pages (`file-evidence.json`); R2004-family object records, 32 pinned files, 8,027 records (`objects-evidence.json`); R2007 walk, 11 files (8 pinned, 3 built), 153 pages, 4,966 data codewords (`r2007-evidence.json`); R2007 object records, 16 files, 2,496 records (`r2007-objects-evidence.json`). Speed: the throughput harness (#343) and five changes (#360, #362, #364, #365, #366) made the walks 2.32 to 4.25 times faster, profile in #367 (README, Throughput). **Open:** the streaming grammar frames records and decodes no object body, so entities, tables and objects still come from the legacy whole-section reader, and #205's item that every path migrates off it is not met | +| Transport abstraction (#206) | in progress | File and slice adapters, and `AsyncSource` (#255), a host-driven adapter that owns no thread or connection. `source-evidence.json` (5 cases, passed): the R13–R2000 walk through `AsyncSource` and an HTTP/1.1 range worker equals the file-backed run on the canonical R2000 fixture and two real files, every fetched byte is delivered once, a fresh-process resume refetches 0 bytes, and the range faults are typed errors. **NOT MEASURED:** the R2004-family and R2007 walks through `AsyncSource` (no driver does it), TLS, a remote host, latency or throughput, and a browser worker (#209) | +| Portable checkpointing (#207) | in progress | Ten checkpoint kinds, each with its own marker and a CRC-32 trailer (README, Checkpoint contract): `DWGH`, `DWGW`, `DWG4`, `DWGZ`, `DWGM`, `DWGS`, `DWGF`, `DWGO`, `DWGR`, `DWGQ`. Fresh-process resume, corrupt, truncated and resealed-forgery rejection, wrong identity and zero source re-reads are checked by the walkers' acceptance steps in the Streaming E2E job (each walker's section of the README says which). Cross-runtime restore is no longer open (#301): checkpoints made natively are restored and finished by a `no_std` wasm32 kernel and the reverse, for the R2004 maps, sections, whole file and objects and for the R2007 walk and objects (`wasm-evidence.json`, `wasm-objects-evidence.json`, `wasm-r2007-evidence.json`, `wasm-r2007-objects-evidence.json`, all passed). **Open:** the kernel runs under node (V8), the data-page section walk crosses runtimes in one direction only, and no checkpoint holds a partly decoded object because none is decoded | +| Spec compiler upgrade (#208) | pending | Only after a working hand-written reader; never patch generated output. No machine-readable grammar or compiler is on `main` | +| WASM streaming API and native/WASM gate (#209) | in progress | The native/wasm32 gate is four required steps of Streaming E2E (the evidence files above). **Open:** `wasm/src/lib.rs` `DwgFile::open` still copies the whole uploaded buffer and `docs/viewer/index.html` still reads `file.arrayBuffer()`, so there is no browser or worker API that requests chunks. **NOT MEASURED:** browser memory and responsiveness | +| Boundary fuzzing through real entry point (#210) | in progress | `examples/streaming_schedules.rs` (proptest, 128 generated schedules per file, each attempted schedule kept as a `*-schedules-*.jsonl` artifact) fronts the header, R13–R2000 walk, LZ77, R2004 header, maps, sections, whole-file, object-record, R2007 and R2007 object-record verifiers. **NOT MEASURED:** allocation instrumentation of the streaming buffers (the only allocator tooling on `main` is `examples/dhat_profile.rs`, outside the gate), and no legacy-migration inventory is written | +| Golden real corpus, large-file benchmarks and comparisons (#211) | in progress | `tools/corpus/pins.tsv` (#225) pins 52 public drawings by sha256, R14 to R2018, fetched in CI, and the streaming manifests (`tools/streaming/*-corpus.json`) name all 52. Throughput against the legacy reader is in the README (best of 40, in memory, one Apple M2 Max under load); the files timed are 1.06 to 1.31 MB. **NOT MEASURED:** a multi-gigabyte model, peak memory, time to first object, a comparison with independently runnable open readers, and an AutoCAD-written R13 drawing (#345) | +| Spec change watcher and automatic invariant walls (#212) | partial | The invariant gate (`tools/streaming/check_invariants.py`, step "Enforce grammar isolation and bounded state") is required and scoped to `src/streaming/`. **Pending:** no spec watcher workflow exists on `main` | +| Known prior prunable worktree | complete | `git worktree list --porcelain` shows no prunable entry on October 27, 2026 | -Discovered live-vs-file ruleset drift: committed main-protection has an admin bypass and fewer required checks; live policy has no bypass, adds macOS/Windows/MSRV and a PR requirement. Track under existing #117; preserve all live restrictions when adding the requested streaming gate. +Discovered: canonical `synthetic_2000.dwg` had zero header CRC and zero sentinel, so the strict §3.2 core rejected it. Fixed under #214 (#215): the generator writes both, the fixture was regenerated, and the acceptance run reads it through the strict header entry point. -Ready for foundation delivery: remote acceptance at a6dc6ba passed (native/WASM no-allocator kernel, 256 proptest schedules across two real files, 98 fresh-process byte positions, malformed and failed-state restore). Saved artifacts retain binary SHA and exact schedules. This is header-only proof; compiler, object/decompression migration, browser runtime, real R13 and multi-gigabyte corpus remain open, in requested order. +Real inputs acquired read-only outside Git: nextgis `line_R14.dwg` (122353 bytes, SHA-256 `4ae12afb4052f1be913003b84021054c0e07b08e6aef5e430bf523c6f85c0903`) and `line_2000.dwg` (116899 bytes, `f1936ccc5e26dd1aae3f6a26bf5803bdf53d8e0b24f8c18bf3fb96fc540918e0`). Both are pinned in `tools/corpus/pins.tsv` now, and their headers and walks pass in Streaming E2E (`streaming-evidence.json`, `walk-evidence.json`). -Owned integration path added: `.github/rulesets/main-protection.json`. Export current live restrictions unchanged, add `Streaming E2E`, and apply/verify the live gate only against a checked candidate. Existing #117 tracks ruleset drift; #214 tracks invalid canonical header. +Live-vs-file ruleset drift, found October 25: resolved. `.github/rulesets/main-protection.json` now lists no bypass actor and the twelve required checks, `Streaming E2E` among them, as the live ruleset does (#220, #292). -Foundation documentation and required-check integration are in progress for PR #213. The full architecture program is not complete; its ordered acceptance issues stay open. Later stages are deliberately not skipped while full container/object migration remains unresolved. +Foundation delivered in #213, accepted remotely at a6dc6ba (run 40234765141): native and WASM no-allocator kernels, 256 proptest schedules across two real files, 98 fresh-process byte positions, malformed and failed-state restore. Everything after it is listed in the table. The architecture program is not complete: #205 to #211 stay open, each with the open items above, and the later stages were not skipped.