feat(rust)!: decompress and validate the prebuilt fullmap in Rust, matching a force build

CHANGELOG.md

@@ -26,6 +26,7 @@ All notable changes to this project are documented in this file.
   **Migration:** add an `effect_type` to any config declaring an effect size (`method: value` when every row shares one statistic, `method: column` when the table provides it); its value is coerced to the permissible `EffectTypes` set as before. Drop a lone `effect_type` that had no effect size — the build was already discarding it.
 
 ### Changed
+- **`build-fullmap`'s prebuilt extraction now runs in the Rust extension, and extracted archives are validated against the force-build contract before install.** The prebuilt download (and its `sha256sum.txt` check) is unchanged, but the extraction is no longer Python-side (`tarfile` zstd with an installed-`zstd`-binary fallback): the extension streams the archive through zstd → tar — the multi-GB decompressed tar is never materialized on disk — with the GIL released, extracts into a temp directory on the output's filesystem, and validates the bundle BEFORE renaming anything into place: the primary's `meta` schema tag must be exactly `tablassert.fullmap.v5` (older schemas are rejected), a `build_id` must be present, the shard files must be exactly the set the primary advertises — no gaps, no extras — and every shard's `build_id` must equal the primary's. Only a passing bundle is atomically renamed beside `--output` (primary → `--output`, shard `i` → `<output stem>.s<i>.redb`); unsupported entry types (symlinks, devices) and PAX-sparse archives are rejected loudly, and path traversal is blocked. Any failure raises, so the command falls back to the from-scratch BABEL build exactly as a failed download does (the archive is kept for that retry). For end users the visible effect is faster extraction and the guarantee that an invalid prebuilt can never land a broken database at `--output`; the progress callback keeps firing its step-detail strings (`opening archive`, `extracting <entry>`, `validating`). The private Python helpers this replaces — `_extract_zst_tar` / `_stream_tar` — are gone; they were never public API, so the removal breaks only code that reached into those internals.
 - **The `build-kg --qc` study no longer flags `original_*` fields for leading/trailing whitespace.** Those slots are verbatim copies of the source-table cell (written by `Tcode.encoding` under an `original_` prefix before any regex/normalization runs), so retaining the cell's whitespace is faithful to the source, not a defect. The whitespace assertion now skips any key prefixed `original_`, while every other field is still checked exactly as before.
 
 ## 10.1.0 - 2026-08-13

docs/cli.md

@@ -114,9 +114,13 @@ tablassert build-fullmap --aria2c --output /data/fullmap/fullmap.redb
 By default `build-fullmap` looks for a prebuilt `fullmap.tar.zst` at
 `https://stars.renci.org/var/babel_outputs/<babel-version>/fullmap/<tablassert-version>/` (the version
 directory is the **installed Tablassert package version**, never hardcoded), verifies it against the
-published `sha256sum.txt`, and stream-extracts it beside `--output`. If no prebuilt exists for this
-version (or the download/extract fails), it falls back to a from-scratch BABEL build and logs a
-warning. A database already present at `--output` is reused as-is; pass `--force` to rebuild.
+published `sha256sum.txt`, and extracts it beside `--output` in the Rust extension — streaming zstd →
+tar with the GIL released (the decompressed tar never touches disk), then validating the extracted
+primary + shards against the force-build contract (exact `v5` schema, a recorded `build_id`, the exact
+shard set, and per-shard `build_id` equality) before atomically renaming them into place. If no
+prebuilt exists for this version (or the download or extraction fails), it falls back to a
+from-scratch BABEL build and logs a warning. A database already present at `--output` is reused as-is;
+pass `--force` to rebuild.
 
 See [Fullmap](fullmap.md) for the data pipeline, output schema, and graph-config usage.
 

docs/fullmap.md

@@ -32,10 +32,17 @@ and the `--force` / `-f` rebuild flag), their defaults, and more examples.
 By default, `build-fullmap` first downloads a **prebuilt** database published for this Tablassert
 version — a `fullmap.tar.zst` under `.../fullmap/<tablassert-version>/` (the version directory is the
 installed package version, never hardcoded), verified against a co-published `sha256sum.txt` and
-stream-extracted beside `--output`. If none is available for this version it falls back to the
-from-scratch build below; `--force` / `-f` skips the prebuilt attempt and always builds. The optional
-`--aria2c` / `-a` accelerates **either** download — the multi-GB prebuilt archive is the ideal aria2
-use case.
+extracted beside `--output` entirely in the Rust extension: it streams the archive through zstd → tar
+(the multi-GB decompressed tar is never materialized on disk) with the GIL released, extracts into a
+temp directory on the output's filesystem, and — before renaming anything into place — validates the
+bundle against the same contract a `--force` build must satisfy: the `meta` schema tag is exactly
+`tablassert.fullmap.v5`, a `build_id` is recorded, the shard files are exactly the set the primary
+advertises (no gaps, no extras), and every shard's `build_id` equals the primary's. Only a bundle that
+passes is atomically renamed into place (primary → `--output`, shards beside it); any failure raises
+and the command falls back to the from-scratch build below. If no prebuilt is published for this
+version it falls back the same way; `--force` / `-f` skips the prebuilt attempt and always builds. The
+optional `--aria2c` / `-a` accelerates **either** download — the multi-GB prebuilt archive is the ideal
+aria2 use case.
 
 Two facts matter most when planning a build:
 

rust/Cargo.toml

@@ -37,16 +37,23 @@ rlimit = "0.10"
 rustc-hash = "1"
 serde = { version = "1", features = ["derive"] }
 serde_json = { version = "1", features = ["preserve_order"] }
+tar = "0.4"
 uuid = { version = "1", features = ["v3"] }
 xxhash-rust = { version = "0.8", features = ["xxh64", "xxh3"] }
+# Streaming zstd decompression for the prebuilt `fullmap.tar.zst` archive
+# (`extract_prebuilt_fullmap`); the C-backed decoder is the fastest option and
+# `cc` is available in the build environment.
+zstd = "0.13"
 
 [dev-dependencies]
 bincode = "1"
 flate2 = { version = "1", features = ["zlib-rs"], default-features = false }
 redb = { git = "https://github.com/cberner/redb.git", rev = "a35e7cc86f191d08a444d5973469b34673d586fc", features = ["experimental_cursor"] }
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"
+tar = "0.4"
 tempfile = "3"
+zstd = "0.13"
 
 [lints.rust]
 unused_imports = "deny"