From 036eff4c9a057460d8e4367534664d6bfeb623eb Mon Sep 17 00:00:00 2001 From: Martin Hutchinson Date: Tue, 27 Oct 2026 14:27:39 +0000 Subject: [PATCH 1/3] docs(vindex): add subcomponent design documentation - Add subcomponent design READMEs across client/, mapfn/, and internal/{ingest,kvstore,tree,server,coordinator,auditor,hammer}/. - Link subcomponent design docs from vindex/README.md and vindex/docs/ARCHITECTURE.md. --- vindex/README.md | 12 ++ vindex/client/README.md | 190 ++++++++++++++++++++++++++ vindex/docs/ARCHITECTURE.md | 20 +-- vindex/docs/tlog-vindex.md | 6 +- vindex/internal/auditor/README.md | 177 ++++++++++++++++++++++++ vindex/internal/coordinator/README.md | 189 +++++++++++++++++++++++++ vindex/internal/hammer/README.md | 121 ++++++++++++++++ vindex/internal/ingest/README.md | 112 +++++++++++++++ vindex/internal/kvstore/README.md | 179 ++++++++++++++++++++++++ vindex/internal/server/README.md | 98 +++++++++++++ vindex/internal/tree/README.md | 167 ++++++++++++++++++++++ vindex/mapfn/README.md | 175 ++++++++++++++++++++++++ 12 files changed, 1433 insertions(+), 13 deletions(-) create mode 100644 vindex/client/README.md create mode 100644 vindex/internal/auditor/README.md create mode 100644 vindex/internal/coordinator/README.md create mode 100644 vindex/internal/hammer/README.md create mode 100644 vindex/internal/ingest/README.md create mode 100644 vindex/internal/kvstore/README.md create mode 100644 vindex/internal/server/README.md create mode 100644 vindex/internal/tree/README.md create mode 100644 vindex/mapfn/README.md diff --git a/vindex/README.md b/vindex/README.md index 6bda417..b36db5d 100644 --- a/vindex/README.md +++ b/vindex/README.md @@ -75,6 +75,18 @@ The VIndex topology bounds search indexing between two append-only logs—a stru ## 6. Documentation +### Core Specification & Architecture - [**C2SP `tlog-vindex` Specification**](./docs/tlog-vindex.md): Normative wire formats, MPT and value-log Merkle definitions, HTTP lookup endpoints, and client/auditor verification rules. - [**Verifiable Index Applications**](./docs/APPLICATIONS.md): How VIndex applies across Certificate Transparency, Merkle Tree Certificates, Go SumDB, Sigstore, and Sigsum—who monitors each log, illustrative canonical key shapes, prefix queries, and epoch rotation (`from > 0`). - [**Verifiable Index Architecture**](./docs/ARCHITECTURE.md): High-level pipeline stages, checkpoint promotion invariants, concurrency model, Zero-WAL crash recovery, auditing boundaries, and architectural trade-offs. + +### Subcomponent Designs +- [**Client SDK & Query Proof Verification**](./client/README.md): Public Go lookup client API, backward value-log streaming, and offline cryptographic proof verification. +- [**WASM Guest SDK & Plugin Interface**](./mapfn/README.md): WebAssembly mapping function SDK, sandbox host interface, and verification tooling. +- [**Ingestion Pipeline & Bundle Cache**](./internal/ingest/README.md): Input Log checkpoint verification, local entry-bundle cache, worker pool orchestration, and chronological resequencing. +- [**Inverted Chunk Storage Engine**](./internal/kvstore/README.md): Embedded LSM value-log chunk layout, compact range persistence, and snapshot-bounded reads. +- [**Authenticated State Commitment**](./internal/tree/README.md): Variable-length MPT wrapper, non-blocking root prediction, Output Log publication, and serving-snapshot promotion. +- [**Read Serving & Query Endpoints**](./internal/server/README.md): `/keys` and `/values` HTTP handlers, prefix policy guardrails, and reader snapshot isolation. +- [**Coordinator & Lifecycle Engine**](./internal/coordinator/README.md): Sync-cycle orchestration, checkpoint ratcheting, Genesis Backfill, and Zero-WAL startup recovery. +- [**Independent Log Auditor**](./internal/auditor/README.md): Continuous Output Log tailing, independent Input Log replay, and divergence alerting. +- [**Hammer Load Testing & Verification**](./internal/hammer/README.md): Synthetic workload generation and continuous end-to-end lookup verification. diff --git a/vindex/client/README.md b/vindex/client/README.md new file mode 100644 index 0000000..872540f --- /dev/null +++ b/vindex/client/README.md @@ -0,0 +1,190 @@ +# Client SDK & Query Proof Verification + +This document defines the lightweight public Go client SDK (`vindex/client`) for querying VIndex servers over HTTP and cryptographically verifying point, non-inclusion, prefix, and paginated value responses through snapshot-scoped lookup handles. + +--- + +## 1. Context & Objectives + +### 1.1 Problem Statement: Zero-Trust Lookup Clients +In verifiable transparency systems, a lookup client querying an index cannot trust the index operator's response. A compromised, coerced, or malfunctioning server could return falsified Input Log entry indices, omit matching records, or present stale state commitments. + +To achieve cryptographic security without requiring lookup clients to download entire databases or execute WebAssembly indexing pipelines, VIndex provides a **lightweight, goroutine-safe HTTP verification client** (`vindex/client`). External Go callers import only `vindex/client`, issue exact or prefix lookups that return an immutable `*LookupResult` snapshot, and lazily stream verified Input Log entry indices per matched key via `MatchedKey.Values(ctx)` against configured Output Log checkpoint verification policies (such as [C2SP `tlog-policy`](https://c2sp.org/tlog-policy)). + +### 1.2 Goals & Non-Goals +- **Goals**: + - Provide a clean, minimal public Go package (`vindex/client`) as the sole import required by external Go lookup clients, CLIs, and monitors. + - Expose a stateless, concurrency-safe HTTP `Client` targeting `` (`New(indexURL, cfg, opts ...Option)` with `WithHTTPClient(*http.Client)` for custom proxies, TLS, auth, timeouts, and test transports while enforcing protocol `3xx` redirect rejection and HTTP status semantics). + - Configure Output Log verification via `Config` (`ExpectedOutputLogOrigin`, `OutputLogPolicy`, and `ExpectedInputLogOrigin`), supporting both [C2SP `tlog-policy`](https://c2sp.org/tlog-policy) witness quorum policies and programmatic log/witness verifier policies without coupling `/keys` lookups to Input Log signature schemes. + - Authenticate state commitments on `LookupKey` and `LookupPrefix` by verifying the `c2sp.org/tlog-proof@v1` bundle against `ExpectedOutputLogOrigin` and `OutputLogPolicy` (enforcing canonical decimal `index == output_size - 1`) and parsing the strict 6-line `c2sp.org/tlog-vindex-state@v1` record (`input-origin`, `from`, `input-size`, `input-root`, `mpt-root`) into an immutable `*LookupResult`. + - Verify binary MPT inclusion, non-inclusion, and two-sided prefix proofs (`internal/mpt`, enforcing `b <= 8192`, terminal `path_key` canonicality, and 32-byte `path_val` bounds on top of `filippo.io/torchwood/mpt`) over canonical keys (`1..1024` printable ASCII bytes in `0x21..0x7E`). + - Scope value enumeration to immutable `MatchedKey` values inside `LookupResult.Keys` via a single lazy iterator method, `MatchedKey.Values(ctx context.Context) iter.Seq2[uint64, error]`, which paginates `/values` backward from `before = InputSize`, verifies each page's RFC 6962 Value Log compact range against the expected `value_root`, and yields verified Input Log entry indices **newest-first** so callers can `break` early at a watermark without unbounded slice allocation. + - Maintain a minimal dependency graph with zero dependencies on storage engines (`pebble`), WebAssembly runtimes (`wazero`), or internal ingestion packages (`internal/ingest`, `internal/kvstore`). +- **Non-Goals**: + - **No Unbounded `AllValues` Slice Helper**: Hot keys can match millions of Input Log entries; `vindex/client` intentionally omits an `AllValues` slice-drain helper so callers stream newest-first via `MatchedKey.Values(ctx)` and apply explicit bounds when collecting into memory. + - **No Cross-Lookup Mutable State**: `Client` caches zero state across `LookupKey` / `LookupPrefix` calls; monitors that track progress across polling intervals persist their own `InputSize` watermark externally. + - **No Full-Log Auditing**: The lookup client does not replay raw Input Log entries or reconstruct global MPT state; continuous full-log auditing is performed by [`internal/auditor`](../internal/auditor/README.md). + - **No Output Log Tile Polling**: Lookup clients query only `/keys` and `/values`; they do not configure `` or fetch `/checkpoint` and `/tile/...` endpoints directly. + +### 1.3 Security Boundary +- **Witness Policy Trust Floor**: Lookup client security on `/keys` and `/values` is bounded by the configured `OutputLogPolicy` (and, when a caller fetches matched raw entries from the Input Log, by Merkle authentication against `(InputSize, InputRoot)` and optional Merkle consistency verification against a signed Input Log checkpoint). If Output Log witnesses equivocate or cosign a fraudulent Output Log checkpoint, lookup clients will accept proofs anchored to that checkpoint until an independent auditor ([`internal/auditor`](../internal/auditor/README.md)) detects the invalid state transition. + +--- + +## 2. Detailed Design + +### 2.1 Public Architecture & Package Decoupling + +External Go callers import `vindex/client` exclusively. Shared wire and MPT proof primitives live in internal packages decoupled from the indexing and storage subsystems: + +| Responsibility | Package | Purpose | +| :--- | :--- | :--- | +| **HTTP Transport & Verification Client API** | `vindex/client` | Public snapshot-scoped lookup client (`Client`, `LookupResult`, `MatchedKey`) with zero dependencies on `pebble`, `wazero`, or `tessera`. | +| **C2SP Plaintext Wire & Value Log Codecs** | `vindex/internal/wire` | C2SP `tlog-vindex` plaintext parsers/formatters, checkpoint policy verification, and RFC 6962 Value Log compact range folding. | +| **MPT Point & Prefix Proof Verification** | `vindex/internal/mpt` | Binary MPT point inclusion/non-inclusion and two-sided prefix proof verification (`b <= 8192` split-bit bounds and canonical terminal leaf validation). | + +### 2.2 Snapshot-Scoped Client API (`Client`, `LookupResult`, `MatchedKey`) + +`Client` is immutable after construction and safe for concurrent use by multiple goroutines. Each `LookupKey` or `LookupPrefix` call returns an isolated `*LookupResult` capturing the verified state commitment and any `MatchedKey` handles bound to that snapshot: + +```go +type Config struct { + ExpectedOutputLogOrigin string + OutputLogPolicy CheckpointPolicy + ExpectedInputLogOrigin string +} + +func New(indexURL string, cfg Config, opts ...Option) (*Client, error) +func WithHTTPClient(hc *http.Client) Option + +func (c *Client) LookupKey(ctx context.Context, key string) (*LookupResult, error) +func (c *Client) LookupPrefix(ctx context.Context, prefix string) (*LookupResult, error) + +type LookupResult struct { + OutputSize uint64 + OutputRoot [32]byte + InputOrigin string + From uint64 + InputSize uint64 + InputRoot [32]byte + MPTRoot [32]byte + Keys []MatchedKey // 0..1 for LookupKey; 0..N (lexicographically sorted) for LookupPrefix +} + +type MatchedKey struct { + Key string + ValueRoot [32]byte + // unexported snapshot context: client, from, inputSize +} + +func (k MatchedKey) Values(ctx context.Context) iter.Seq2[uint64, error] + +var ( + ErrVerificationFailed = errors.New("vindex: verification failed") + ErrBadRequest = errors.New("vindex: bad request") + ErrTooManyKeys = errors.New("vindex: prefix matched too many keys") + ErrUnavailable = errors.New("vindex: server unavailable") +) +``` + +`OutputLogPolicy` may be parsed from a standard [C2SP `tlog-policy`](https://c2sp.org/tlog-policy) text file (supporting arbitrary `quorum` threshold expressions across witnesses) or constructed programmatically from a primary log verifier and witness threshold, pinned to `ExpectedOutputLogOrigin`. `WithHTTPClient` clones the caller-supplied `*http.Client` (preserving custom `Transport`, proxies, TLS settings, and timeouts) while overriding `CheckRedirect` to reject all `3xx` redirects per [`docs/tlog-vindex.md`](../docs/tlog-vindex.md). + +Because snapshot context (`ValueRoot`, `From`, `InputSize`, and `*Client`) is encapsulated on each immutable `MatchedKey`, callers can selectively enumerate a subset of keys from a prefix lookup, stop early at a watermark, and reuse the same `*Client` across unrelated lookups without mutating shared client state: + +```go +res, err := cli.LookupPrefix(ctx, "com.google.") +if err != nil { + return err +} +if res.From > requiredHorizon { + return fmt.Errorf("index starts at %d, need <= %d", res.From, requiredHorizon) +} +for _, mk := range res.Keys { + if mk.Key != "com.google.mail." && mk.Key != "com.google.maps." { + continue // skip uninterested keys with zero /values requests + } + for entryIdx, err := range mk.Values(ctx) { + if err != nil { + return err + } + if entryIdx < lastProcessedSize { + break // newest-first stream: stops without fetching older /values pages + } + // Authenticate raw entry at entryIdx against (res.InputSize, res.InputRoot) + } +} +``` + +### 2.3 End-to-End HTTP & Cryptographic Verification Sequence + +Verification executes across 4 protocol steps matching [`docs/tlog-vindex.md`](../docs/tlog-vindex.md)—Steps 1–3 eagerly inside `LookupKey` / `LookupPrefix`, and Step 4 lazily per page inside `MatchedKey.Values`: + +| Step | Phase | Operations | Cryptographic & Protocol Assertion | +| :--- | :--- | :--- | :--- | +| **1** | **Output Log Proof (`c2sp.org/tlog-proof@v1`)** | Percent-encodes `` or `` per RFC 3986 (`unreserved` only, escaping `+` as `%2B`), fetches `GET /keys?...` (rejecting `3xx`, mapping `400`/`413`/`503`, bounding response body size), splits on `\n\n` into `c2sp.org/tlog-vindex-keys@v1`, `c2sp.org/tlog-proof@v1` (base64 `extra` `state_entry` and canonical decimal `index`), and ``, and computes `SHA-256(0x00 \|\| state_entry)`. | Verifies `` against `cfg.ExpectedOutputLogOrigin` and `cfg.OutputLogPolicy` to extract `output_size` and `output_root`, asserts `output_size >= 1` and `index == output_size - 1`, and verifies the Merkle inclusion proof for `SHA-256(0x00 \|\| state_entry)` at `index`. | +| **2** | **State Commitment (`c2sp.org/tlog-vindex-state@v1`)** | Parses `state_entry` as a strict 6-line `c2sp.org/tlog-vindex-state@v1` record (`input-origin`, `from`, `input-size`, `input-root`, `mpt-root`) with no extension lines or trailing data. | Asserts `input_origin == cfg.ExpectedInputLogOrigin` and `0 <= from <= input_size`. Because a lookup response only attests to Input Log entries in `[from, input_size)`, callers requiring coverage back to a specific entry index check `LookupResult.From` against that bound. | +| **3** | **MPT Key / Prefix Proof (`LookupKey` / `LookupPrefix`)** | Validates canonical key syntax (`1..1024` bytes in `0x21..0x7E`), and verifies the binary MPT proof (`internal/mpt`) against `mpt_root` (enforcing `b <= 8192` split-bit bounds and canonical `path_key` / 32-byte `path_val` checks on terminal leaves). | **Inclusion**: Single matched `(key, value_root)` with `key == target` hashes to `mpt_root`.
**Non-Inclusion**: `mpt_root == empty_mpt_hash` (`0`-byte proof) or terminal MPT leaf `mpt_leaf_hash(path_key, path_val)` with `path_key != target` hashes to `mpt_root` (`len(res.Keys) == 0`).
**Prefix**: On `0` matches, `mpt_root == empty_mpt_hash` (`0`-byte proof) or terminal MPT leaf with `!HasPrefix(path_key, target)` hashes to `mpt_root`; on `n >= 1` matches, `HasPrefix` + strictly ascending key order + `b_1 < 8*len(target)` + subtree root + path to `mpt_root`. | +| **4** | **Lazy Backward Value Pagination (`MatchedKey.Values`)** | Initializes local iterator state `(before = inputSize, expectedValueRoot = k.ValueRoot)`. Fetches `GET /values?key=&before=` and parses `c2sp.org/tlog-vindex-values@v1` (`preceding`, `popcount(preceding)` compact range hashes, and strictly increasing `Indices`). | Treats `HTTP 404` on an MPT-included key or promised continuation page (`preceding > 0`) as `ErrVerificationFailed`. Asserts `len(Indices) >= 1`, strictly increasing order in `[from, before)`, and `preceding <= Indices[0] - from`; appends `value_log_leaf_hash(entry_idx)` into the running compact range (merging interior nodes via `value_log_node_hash(left, right)` on trailing `1` bits of the running leaf count) and folds the resulting compact range right-to-left to assert equality with `expectedValueRoot` **before** yielding `Indices` in reverse (`len-1` down to `0`, newest-first). If the caller continues and `preceding > 0`, folds the page's initial `popcount(preceding)` compact range hashes right-to-left into `expectedValueRoot`, sets `before = Indices[0]`, and fetches the next backward page. | + +### 2.4 In-Situ Invariants & Performance Optimizations + +- **[Correctness Invariant] Pre-Yield Cryptographic & Protocol Assertion**: + - *Rule*: `LookupKey` and `LookupPrefix` never return a `*LookupResult` until Output Log policy verification, `index == output_size - 1`, `c2sp.org/tlog-vindex-state@v1` bounds, and binary MPT proof verification all succeed; `MatchedKey.Values` verifies the entire `/values` page's compact range root before yielding a single `entry_idx` from that page, and rejects `3xx` redirects or unexpected `404` responses on `/values`. + - *Rationale*: HTTP responses and status codes from `` are untrusted input; yielding unverified indices or silently treating an unexpected `404` as empty history would let a faulty or malicious server truncate results. + - *Consequence ("Or Else")*: A compromised index operator could forge entry indices or omit matching records without detection. + +- **[Correctness Invariant] Snapshot-Scoped Lookup Isolation**: + - *Rule*: `*Client` holds zero mutable per-lookup or cross-lookup state. Each `*LookupResult` and its `MatchedKey` values capture an immutable `(ValueRoot, from, inputSize)` snapshot, and pagination cursors `(before, expectedValueRoot)` exist strictly as local stack variables inside each `MatchedKey.Values(ctx)` invocation. + - *Rationale*: Allows a single `*Client` (and even a single `MatchedKey`) to be used concurrently across goroutines, across unrelated key/prefix lookups, and against load-balanced or short-TTL-cached server replicas where consecutive `/keys` responses may arrive slightly out of order without invalidating any proof. + - *Consequence ("Or Else")*: Mutating shared state on `*Client` across lookups causes data races or false regression errors under concurrent goroutines and load-balanced replicas, and tangles cursor state when interleaving multiple prefix enumerations. + +- **[Performance Optimization] Lazy Newest-First Streaming & Zero-Allocation Proof Evaluation**: + - *Mechanism*: `MatchedKey.Values(ctx)` streams verified entry indices newest-first via `iter.Seq2[uint64, error]` one server-bounded page at a time (omitting any `AllValues` full-slice helper), while `internal/mpt` and `internal/wire` evaluate binary MPT path steps and RFC 6962 compact range folds linearly over byte slices without allocating intermediate tree node graphs. + - *Impact*: Callers inspecting recent occurrences or polling down to a watermark execute a single `/values` round-trip with constant per-page memory even on hot keys with millions of historical occurrences, completing proof verification in roughly 1 ms per query on commodity CPUs. + +### 2.5 Failure Taxonomy & Client Handling + +| Failure Category | Root Cause | Client Action | +| :--- | :--- | :--- | +| **Untrusted Checkpoint / Quorum Failure** | Output Log proof or checkpoint fails verification against `ExpectedOutputLogOrigin` and `OutputLogPolicy`, or `input-origin` in `state_entry` does not match `ExpectedInputLogOrigin`. | Return `ErrVerificationFailed`; abort lookup without `*LookupResult`. | +| **Latest-Entry Proof Violation** | `c2sp.org/tlog-proof@v1` proves an entry at `index != output_size - 1` or `output_size == 0`. | Return `ErrVerificationFailed`; abort lookup (server attempted to serve a non-tip Output Log entry). | +| **MPT Proof Mismatch** | Binary MPT path steps or prefix subtree reconstruction do not evaluate to `mpt_root`, or returned keys violate canonical key / prefix bounds. | Return `ErrVerificationFailed`; abort lookup. | +| **Value Log Root or Range Mismatch** | Reconstructed RFC 6962 root from `preceding` compact range hashes and `Indices` does not equal `expectedValueRoot`, or `Indices` fall outside `[from, before)` or are not strictly increasing. | Yield `(0, ErrVerificationFailed)` and terminate iterator without yielding unverified page indices. | +| **Unexpected `/values` `HTTP 404` on Present Key** | `/values` returns `404 Not Found` for a key proven present in the MPT at `before = InputSize` (or when `preceding > 0` promised older entries). | Yield `(0, ErrVerificationFailed)` and terminate iteration (server state violates the witnessed MPT or Value Log commitment). | +| **Prefix Cardinality or Policy Rejection (`HTTP 413` / `HTTP 400`)** | `/keys?prefix=` matched more keys than the server's configured prefix limit (`413`), or violated the server's prefix policy (`400`). | Surface typed HTTP status error (`ErrTooManyKeys` / `ErrBadRequest`) so the caller can narrow the queried prefix. | +| **Upstream Server Unavailable (`HTTP 503`)** | Server has not yet published its first Output Log checkpoint or lags `before`. | Surface `ErrUnavailable` (preserving `Retry-After` metadata when present) so the caller can retry with backoff or fail over. | + +### 2.6 Client Contract & API Requirements + +- **Inputs**: + - Index base URL (``, serving `/keys` and `/values`). + - `Config` containing non-empty `ExpectedOutputLogOrigin` (`string`), non-nil `OutputLogPolicy` (`CheckpointPolicy`), and non-empty `ExpectedInputLogOrigin` (`string`), plus optional `WithHTTPClient(*http.Client)`. + - Canonical key (`key`) or canonical key prefix (`prefix`) of `1..1024` printable ASCII bytes (`0x21..0x7E`), validated locally before sending and percent-encoded per RFC 3986 (`unreserved`) on the wire. +- **Outputs**: + - `*LookupResult` exposing verified state commitment metadata (`OutputSize`, `OutputRoot`, `InputOrigin`, `From`, `InputSize`, `InputRoot`, `MPTRoot`) and lexicographically ordered `Keys []MatchedKey` (`0..1` for `LookupKey`, `0..N` for `LookupPrefix`). + - Lazy `MatchedKey.Values(ctx) iter.Seq2[uint64, error]` yielding verified `uint64` Input Log entry indices in `[From, InputSize)` in **strictly decreasing (newest-first)** order across backward pages (callers requiring historical coverage back to a specific entry index check `LookupResult.From`, and callers fetching raw entries from the Input Log authenticate them against `(LookupResult.InputSize, LookupResult.InputRoot)`). +- **Guarantees**: + - All HTTP redirects (`3xx`) on `/keys` and `/values` are rejected per `tlog-vindex`. + - `*Client`, `*LookupResult`, and `MatchedKey` contain no mutable cross-lookup state and are safe for concurrent use across goroutines. + - Any Output Log policy verification failure, `input-origin` or `from` violation, non-tip Output Log proof, MPT path/prefix defect, unexpected `404` on `/values`, entry index bound violation, or compact range mismatch returns an immediate error and withholds unverified data. + +--- + +## 3. Alternatives Considered (or Tried) + +### 3.1 Stateless Snapshot Handles vs. Mutable Cross-Lookup Client State +- **Proposed**: Caching observed `(output_size, input_size)` watermarks inside `Client` across lookups or exposing unbounded slice-draining helpers for value pagination. +- **Rejection**: + - **Cross-Lookup Watermark Races**: Caching observed `(output_size, input_size)` watermarks inside `Client` verifies no Merkle consistency proof (witnesses and auditors provide that guarantee), causes false regression errors when concurrent goroutines or load-balanced/short-TTL-cached server replicas return checkpoints out of arrival order, and tangles state across unrelated key lookups. + - **Unbounded Slice Allocations on Hot Keys**: Keys appearing hundreds of thousands or millions of times would trigger unbounded sequential HTTP pagination and large slice allocations, whereas callers needing a slice can collect from `iter.Seq2` explicitly with their own bounds. +- **Chosen Design**: `Client` is stateless across lookups and returns snapshot-scoped `*LookupResult` handles whose `MatchedKey.Values(ctx)` `iter.Seq2[uint64, error]` iterator streams verified entry indices newest-first with early `break` support. + +--- + +## 4. Verification Gate + +All client proof verification, HTTP transport/redirect/status handling, checkpoint policy quorum enforcement, backward pagination streaming, and package dependency boundary tests are verified via: + +```bash +go test -race ./vindex/client/... +``` diff --git a/vindex/docs/ARCHITECTURE.md b/vindex/docs/ARCHITECTURE.md index 27b4a93..bc1a7a7 100644 --- a/vindex/docs/ARCHITECTURE.md +++ b/vindex/docs/ARCHITECTURE.md @@ -20,16 +20,16 @@ VIndex operates as a **Map Sandwich** bounded between two append-only transparen Data flows sequentially through five stages governed by a central coordinator, verified externally by lookup clients and independent auditors: -| Stage / Role | Responsibility | -| :--------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **1. Ingest & Bundle Cache** | Polls Input Log checkpoints, verifies checkpoint policies and Merkle consistency, downloads and authenticates entry bundles into a local disk cache, and prunes cached bundles once downstream storage and MPT state are durable. | -| **2. Sandboxed Mapping** | Executes the index's deterministic mapping function inside a zero-I/O WebAssembly sandbox pool, validating emitted [canonical keys](./tlog-vindex.md#canonical-keys) and resequencing mapped batches into strict Input Log entry order. | -| **3. Value-Log Storage** | Partitions each key's matching Input Log entry indices into append-only chunks in an embedded LSM key-value store, precomputing RFC 6962 compact range hashes and syncing updates to disk at target checkpoint barriers. | -| **4. State Commitment** | Predicts the candidate `mpt_root` over un-hashed canonical keys under a shared read lock, appends the state commitment entry to the Output Log, collects witness cosignatures, and atomically swaps the active serving snapshot. | -| **5. Read Serving** | Serves `/keys` (exact key and bounded prefix proofs from the in-memory MPT) and `/values` (lock-free backward value-log pagination from the KV store), strictly isolated to witnessed snapshots. | -| **Coordinator** | Freezes target Input Log checkpoints, drives stages 1–4 in monotonic order, governs safe cache-pruning watermarks, and runs Zero-WAL startup recovery. | -| **Lookup Client** | Queries `/keys` and `/values` and verifies Output Log inclusion proofs, MPT inclusion/non-inclusion/prefix proofs, and value-log compact range roots offline against an Output Log checkpoint verification policy. | -| **Independent Auditor** | Tails the Output Log from entry `0`, verifies Input Log and Output Log checkpoints, independently replays Input Log entries from `from` through the mapping function, and halts and alerts if any published `mpt-root` diverges. | +| Stage / Role | Responsibility | +| :----------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [**1. Ingest & Bundle Cache**](../internal/ingest/README.md) | Polls Input Log checkpoints, verifies checkpoint policies and Merkle consistency, downloads and authenticates entry bundles into a local disk cache, and prunes cached bundles once downstream storage and MPT state are durable. | +| [**2. Sandboxed Mapping**](../mapfn/README.md) | Executes the index's deterministic mapping function inside a zero-I/O WebAssembly sandbox pool, validating emitted [canonical keys](./tlog-vindex.md#canonical-keys) and resequencing mapped batches into strict Input Log entry order. | +| [**3. Value-Log Storage**](../internal/kvstore/README.md) | Partitions each key's matching Input Log entry indices into append-only chunks in an embedded LSM key-value store, precomputing RFC 6962 compact range hashes and syncing updates to disk at target checkpoint barriers. | +| [**4. State Commitment**](../internal/tree/README.md) | Predicts the candidate `mpt_root` over un-hashed canonical keys under a shared read lock, appends the state commitment entry to the Output Log, collects witness cosignatures, and atomically swaps the active serving snapshot. | +| [**5. Read Serving**](../internal/server/README.md) | Serves `/keys` (exact key and bounded prefix proofs from the in-memory MPT) and `/values` (lock-free backward value-log pagination from the KV store), strictly isolated to witnessed snapshots. | +| [**Coordinator**](../internal/coordinator/README.md) | Freezes target Input Log checkpoints, drives stages 1–4 in monotonic order, governs safe cache-pruning watermarks, and runs Zero-WAL startup recovery. | +| [**Lookup Client**](../client/README.md) | Queries `/keys` and `/values` and verifies Output Log inclusion proofs, MPT inclusion/non-inclusion/prefix proofs, and value-log compact range roots offline against an Output Log checkpoint verification policy. | +| [**Independent Auditor**](../internal/auditor/README.md) | Tails the Output Log from entry `0`, verifies Input Log and Output Log checkpoints, independently replays Input Log entries from `from` through the mapping function, and halts and alerts if any published `mpt-root` diverges. | --- diff --git a/vindex/docs/tlog-vindex.md b/vindex/docs/tlog-vindex.md index 07425d3..45d53a2 100644 --- a/vindex/docs/tlog-vindex.md +++ b/vindex/docs/tlog-vindex.md @@ -243,8 +243,8 @@ example.com/outputlog 13 jR10gZzRRHNJbpwE1qNYfgk7jAv2Ta+5Ud8SpKcWZ6w= -- example.com/outputlog opLqBQsREYCgu6xQkYQwJr9fo45a62DN9EdmHXnZdXNqlcVGlCum2Wks+49/V6267UEjw6QUXTS5Rovnzv++qbSzm9Q= -- witness.example/w1 CMp+6LWBU0anHGH5aNDTJkH/gj79sG+T6+iP2ThYN5krrDJbR1HDnucjL39QsZTSvVjyQLrdk3DXDqI5G2HgLatVs0pWh6Up69HVOw== +— example.com/outputlog opLqBQsREYCgu6xQkYQwJr9fo45a62DN9EdmHXnZdXNqlcVGlCum2Wks+49/V6267UEjw6QUXTS5Rovnzv++qbSzm9Q= +— witness.example/w1 CMp+6LWBU0anHGH5aNDTJkH/gj79sG+T6+iP2ThYN5krrDJbR1HDnucjL39QsZTSvVjyQLrdk3DXDqI5G2HgLatVs0pWh6Up69HVOw== ``` #### Binary MPT proofs @@ -335,7 +335,7 @@ To authenticate the state commitment in a `keys` response: 3. Verify the `c2sp.org/tlog-proof@v1` block against that leaf hash, ``, and the client's output log checkpoint verification policy according to [tlog-proof][]. Let `output_size` be the verified output log checkpoint tree size. Check that `output_size >= 1` and that the proven entry `index` equals `output_size - 1`. -4. Parse `state_entry` as a `c2sp.org/tlog-vindex-state@v1` record (see [Output log entries](#output-log-entries)) to obtain `input_origin`, `from`, `input_size`, `input_root`, and `mpt_root`. Check that `input_origin` equals the client's configured `` and that `0 <= from <= input_size`. If `from = input_size`, check that `mpt_root` equals `empty_mpt_hash`. Because a lookup response only attests to input log entries in `[from, input_size)`, callers that require coverage back to a specific entry index MUST check that `from` does not exceed that bound. +4. Parse `state_entry` as a `c2sp.org/tlog-vindex-state@v1` record (see [Output log entries](#output-log-entries)) to obtain `input_origin`, `from`, `input_size`, `input_root`, and `mpt_root`. Check that `input_origin` equals the client's configured `` and that `0 <= from <= input_size`. Because a lookup response only attests to input log entries in `[from, input_size)`, callers that require coverage back to a specific entry index MUST check that `from` does not exceed that bound. If the client subsequently fetches matched entries from the input log, it MUST authenticate those entries against `(input_size, input_root)` according to the input log's specification, and unless relying solely on the witnessed output log and auditors for input log checkpoint validity, verify that `(input_size, input_root)` is Merkle-consistent with a signed input log [checkpoint][] verified under its input log policy. diff --git a/vindex/internal/auditor/README.md b/vindex/internal/auditor/README.md new file mode 100644 index 0000000..6eaccb3 --- /dev/null +++ b/vindex/internal/auditor/README.md @@ -0,0 +1,177 @@ +# Independent Log Auditor + +This document defines the architecture, continuous audit loop, root mismatch alerting, state-preserving halt protocols, and adversarial threat model for the **Auditor Subsystem** (`vindex/internal/auditor`). + +--- + +## 1. Context & Objectives + +### 1.1 Problem Statement & The Operator Trust Dilemma +The index operator publishes an Output Log asserting that its verifiable index accurately, completely, and deterministically represents the underlying Input Log over `[from, input_size)`. + +However, in a verifiable transparency architecture, lookup clients and ecosystem monitors cannot simply trust the index operator's claims. A compromised, coerced, or buggy index operator could: +1. **Targeted Omission Attacks**: Commit an invalid `mpt_root` that silently conceals specific certificates, packages, or identities, while returning valid-looking non-inclusion proofs against that forged `mpt_root`. +2. **Equivocation & Split-Views**: Present conflicting Output Log checkpoints to different monitors or lookup clients. +3. **Historical Rollbacks & Parameter Drift**: Regress tree sizes or mutate `input-origin` or `from` across Output Log entries. + +To enforce cryptographic integrity, independent third parties must be able to audit the index operator's Output Log state commitments from `from` to the Output Log head without trusting the `/keys` or `/values` HTTP endpoints. This reference auditor implementation reuses the Pebble inverted chunk store ([`internal/kvstore`](../kvstore/README.md)) and disk-backed MPT (`internal/mpt`) from the indexer to share the high-throughput ingestion/recovery pipeline and preserve inspectable on-disk forensic state immediately prior to any divergence. However, reusing the indexer's storage and trie packages is not required by the protocol—and relying solely on a single codebase would only verify that the operator ran the same code, not that the code itself is free of shared bugs. Independent clean-room auditors (written in other languages or using different storage and mapping implementations conforming to [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md#auditing-the-index)) should also audit the Output Log to provide true implementation diversity. + +### 1.2 Goals, Non-Goals & Future Work +- **Goals**: + - Continuously audit published Output Log index state commitments (`c2sp.org/tlog-vindex-state@v1`) against authenticated Input Log entries starting from the configured `from` index (default `0`). + - Verify Output Log and Input Log checkpoints against configured checkpoint verification policies (such as [C2SP `tlog-policy`](https://c2sp.org/tlog-policy)). + - Assert cryptographic equality between the independently reconstructed local `mpt_root` and the index operator's committed `mpt_root` at every Output Log entry. + - Signal divergence immediately via Prometheus metrics, structured logs, and readiness probe degradation (`/readyz` on the internal operational listener). + - Freeze physical database state on root mismatch to preserve forensic post-mortem evidence. +- **Non-Goals**: + - **No Automatic Retry / Self-Healing on Root Mismatch**: If a root mismatch occurs, the auditor does not guess or retry; it halts immediately to preserve evidence of operator fraud or software bugs. + - **No Operator Checkpoint Generation**: The auditor does not sign or publish Output Log checkpoints; it is strictly an independent consumer and auditor. + - **No HTTP Lookup Serving**: The auditor does not serve `/keys` or `/values` lookup queries; verified read-mirror serving is out of scope. +- **Future Work (Verified Mirror Serving)**: + - Because an auditor already maintains a locally verified MPT and Pebble chunk store, it could in principle also serve C2SP lookup queries (`/keys` and `/values`) as a read mirror. Mirror serving is intentionally omitted for now: `c2sp.org/tlog-vindex` does not specify mirror semantics, and serving lookups from an auditor introduces protocol design questions around which Output Log checkpoint is served and how to satisfy the `index == output_size - 1` latest-entry proof requirement when an auditor lags or halts behind the index operator's Output Log head. This capability can be revisited in future work once specified. + +### 1.3 Requirements, Dependencies & Constraints +- **Dependencies**: Integrates checkpoint policy and Merkle verification (`internal/wire`), the tile fetcher and local tile cache ([`internal/ingest`](../ingest/README.md)), the KV chunk store ([`internal/kvstore`](../kvstore/README.md)), and the MPT (`internal/mpt`). +- **Hardware Sizing Parity**: Running a full-log auditor requires the same RAM and NVMe disk capacity as the index operator to maintain the local MPT and Pebble chunk store. +- **Historical Entry Availability**: Auditing an Output Log from `from` requires historical Input Log entry bundles across `[from, input_size)` to remain retrievable. + +--- + +## 2. Detailed Design + +### 2.1 Ecosystem Verification Architecture & Policy Configuration + +VIndex verification operates across two complementary boundaries: + +| Role & Component | Execution Scope | Target Consumer | Threat Coverage | +| :--- | :--- | :--- | :--- | +| **Lookup Client Verification** ([`client/`](../../client/README.md)) | Verifies individual `/keys` and `/values` responses against witnessed Output Log checkpoints. | Lookup clients, end-user apps, CLIs. | Verifies inclusion/non-inclusion/prefix proofs for queried keys; verifies Output Log checkpoint policies and witness quorums; verifies backward Value Log pagination chains. | +| **Full-Log Independent Auditor** (`vindex/internal/auditor`) | Independently replays all Input Log entries in `[from, input_size)` and reconstructs the full MPT across every Output Log entry. | Standalone monitoring daemon (`vindex-auditor`). | Detects operator equivocation, forged `mpt_root` commitments, `input-origin`/`from` drift, targeted omission of unqueried keys, and tree rollbacks; enforces checkpoint witness quorums and Merkle consistency. | + +#### Auditor Configuration & Checkpoint Policy Enforcement +The auditor daemon is configured with ``, the expected `` and ``, the Input Log checkpoint and entry retrieval endpoint, the compiled WASM `MapFn` module, the configured start index `from` (`0` by default, or `> 0` for suffix indices), and checkpoint verification policies (such as C2SP `tlog-policy` policies) for both the Output Log and the Input Log. Every Output Log checkpoint and polled Input Log checkpoint is verified against its expected origin and checkpoint policy before any entries are processed. + +### 2.2 The Continuous Audit Loop +Initialized with an empty local MPT and `prev_size = from` (or, on process restart, after running local Phase 2 MPT catch-up across `[max(MPT_Durable_Size, from), audited_input_size)` if the MPT's persisted `(version, exact)` lags `audited_input_size`), the auditor polls `/checkpoint` and verifies `(input_size, input_root)` from each `c2sp.org/tlog-vindex-state@v1` entry against a signed Input Log checkpoint (`latest_input_size`, `latest_input_root`, re-fetched whenever `input_size > latest_input_size`), executing the 5-step audit loop defined in [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md): + +| Step | Phase | Action & Operations | Verification Check | +| :--- | :--- | :--- | :--- | +| **1** | **Poll Output & Input Checkpoints** | Fetches `/checkpoint` and a signed Input Log `/checkpoint` (re-fetching the Input Log checkpoint as needed when auditing newer Output Log entries). | Verifies Output Log and Input Log checkpoint origins, primary signatures, and witness cosignatures against their respective checkpoint verification policies, and verifies Output Log Merkle consistency against the previously audited Output Log checkpoint. | +| **2** | **Stream Output Entries** | Fetches new Output Log entry bundles from `/tile/entries/...` from the last audited `output_size` up to the new `output_size`. | Authenticates entry bundles against the verified Output Log checkpoint root hash per `tlog-tiles`, parses each 6-line `c2sp.org/tlog-vindex-state@v1` entry (`input-origin`, `from`, `input-size`, `input-root`, `mpt-root`), asserts `input_origin` and `from` match configured parameters, checks `prev_size <= input_size <= latest_input_size`, and verifies that `(input_size, input_root)` is Merkle-consistent with `(latest_input_size, latest_input_root)` (checking `input_root == latest_input_root` when `input_size == latest_input_size`). | +| **3** | **Fetch, Map & Stage Input Entries** | Downloads Input Log entries in `[prev_size, input_size)` (slicing non-256-aligned initial `from` or partial tail bundles), filters already-persisted entries, stages chunk mutations into an uncommitted indexed Pebble batch, and collects modified `(canonicalKey, value_root)` mappings in memory. | Authenticates Input Log entry bundles against `input_root` (or `latest_input_root`), executes sandboxed `MapFn`, checks that every emitted key is a valid canonical key (`1..1024` bytes in `0x21..0x7E`), deduplicates keys per `entry_idx`, appends `value_log_leaf_hash(entry_idx)` to each key's Value Log inside the uncommitted batch, and enforces replay deduplication. | +| **4** | **Predict Candidate MPT Root** | Evaluates the accumulated in-memory `(canonicalKey, value_root)` pairs against the local MPT without mutating verified tree state. | Computes the candidate local `mpt_root` in memory under a shared read lock. | +| **5** | **Assert Root Equality & Commit** | Asserts that the candidate local `mpt_root` equals the committed `mpt_root` in the Output Log entry. | **Match**: Applies the staged Pebble batch (`'c'` chunks + `audited_output_size` + `audited_input_size`) with durable `fsync`, commits and syncs MPT mutations, and sets `prev_size = input_size`.
**Mismatch**: Triggers state-preserving halt, discards the uncommitted Pebble batch and clears the in-memory active chunk cache, freezes database files, and emits alerts. | + +#### Step Details: +1. **Discover & Authenticate Checkpoints**: Polls `/checkpoint`, verifies `` and witness cosignatures against the Output Log checkpoint policy, and verifies a Merkle consistency proof from the previously audited Output Log checkpoint. Fetches and verifies a signed Input Log `/checkpoint` against `` and the Input Log checkpoint policy to obtain `(latest_input_size, latest_input_root)` (re-fetching as needed when auditing newer Output Log entries). If policy or consistency verification fails, the checkpoint is rejected before reading any entries. +2. **Stream State Commitments & Verify Input Log Consistency**: Fetches new Output Log entry bundles from `/tile/entries/...` and authenticates them against the Output Log checkpoint root hash according to `tlog-tiles`. For each 6-line `c2sp.org/tlog-vindex-state@v1` Output Log entry in sequential order, parses `(input_origin, from, input_size, input_root, mpt_root)`, checks that `input_origin` and `from` match the auditor's configured parameters and `prev_size <= input_size <= latest_input_size`, and verifies that `(input_size, input_root)` is Merkle-consistent with `(latest_input_size, latest_input_root)`. +3. **Fetch, Map & Stage Input Entries**: For the Input Log entry range `[prev_size, input_size)`: + - Fetches Input Log entry bundles and authenticates them against `input_root` (or `latest_input_root`) according to `tlog-tiles` or `static-ct-api`, slicing boundary 256-entry bundles to `[prev_size, input_size)` so entries `< from` on non-256-aligned initial bundles and entries `>= input_size` on partial tail bundles are excluded. + - Executes the WASM `MapFn` on each Input Log entry at `entry_idx` in sequential order, validates that every emitted key is a canonical key (`1..1024` printable ASCII bytes in `0x21..0x7E`), and deduplicates keys emitted for the same `entry_idx`. + - Stages chunk mutations into an uncommitted **indexed Pebble batch** (`stagedBatch`), which reads through uncommitted batch state and the active chunk cache to return the updated `(canonicalKey, value_root)` mappings directly in memory without writing to the underlying database. + - **Replay Deduplication Invariant**: On restart or crash replay, any Input Log entry indices `< from` or already persisted in Pebble are filtered out prior to appending `value_log_leaf_hash(entry_idx)` to active chunks while still reconstructing their point-in-time `(canonicalKey, value_root)` pairs at `input_size` from the KV store for Step 4, ensuring idempotent crash recovery without false root mismatches. +4. **Predict Candidate MPT Root**: Evaluates the accumulated in-memory `(canonicalKey, value_root)` mappings at `input_size` against the local MPT using non-mutating prediction to compute the candidate local `mpt_root`. +5. **Root Equality Assertion & Commit**: Checks that the candidate local `mpt_root` equals the Output Log entry's `mpt_root`: + - If equal: atomically applies the staged Pebble batch (`'c'` chunks together with `audited_output_size` and `audited_input_size`, advancing `prev_size = input_size`) with a durable disk sync before committing and syncing MPT mutations (`KV_CP >= MPT_Durable_Size`). If the process crashes between the Pebble commit and MPT `fsync`, startup MPT catch-up replays `[max(MPT_Durable_Size, from), audited_input_size)` from local storage before auditing the next Output Log entry. + - If divergent: closes and discards the uncommitted Pebble batch, clears the in-memory active chunk cache, and enters the state-preserving halt protocol without advancing watermarks or mutating the verified MPT and Pebble state on disk. + +### 2.3 Root Mismatch Alerting & State-Preserving Halt +If the candidate local `mpt_root` differs from the committed `mpt_root`, or an upstream consistency/policy fault is detected, the auditor immediately executes a coordinated containment response: + +1. **Sync Engine Halt**: + - The background audit loop terminates immediately. + - Verified watermarks are **not** advanced on disk. +2. **Forensic State Freeze**: + - Discards the uncommitted Pebble staging batch and clears the in-memory chunk cache while leaving verified Pebble DB and MPT files intact on disk, freezing the exact pre-divergence state for forensic triage and public misbehavior reporting. +3. **Metrics & Structured Logging**: + - Increments the root mismatch counter metric and sets the root mismatch gauge to `1`. + - Emits structured error logs recording the divergent output entry index, target `input_size`, candidate local root, and committed root. +4. **Health and Readiness Probes (Internal Operational Listener)**: + - `/healthz`: Remains **HTTP 200 OK** as long as the auditor process is alive, preventing container orchestrators from restarting the halted process and disturbing frozen forensic state or diagnostic endpoints. + - `/readyz`: Returns **HTTP 503 Service Unavailable** on the internal operational listener with structured JSON diagnostics: + ```json + { + "status": "degraded", + "error": "auditor root hash mismatch", + "output_index": 12480, + "input_size": 3194880, + "local_root": "a4f1...", + "committed_root": "b7c2..." + } + ``` + +### 2.4 Adversarial Threat Vector Matrix + +| Threat Vector | Adversary Action | Detection Mechanism | Auditor Action | +| :--- | :--- | :--- | :--- | +| **Operator Equivocation** | Index operator serves conflicting Output Log checkpoints to different auditors or witnesses. | Output Log checkpoint policy verification; Merkle consistency proofs between Output Log checkpoints. | Audit loop halts immediately; alert logged; database frozen. | +| **Tree Size Rollback** | Index operator serves an Output Log checkpoint (or `input-size` commitment) smaller than previously audited watermarks. | Monotonic `output_size` and `prev_size <= input_size <= latest_input_size` assertions during audit poll. | Audit loop halts; watermarks frozen; `/healthz` remains HTTP 200; `/readyz` reports `503` degraded status. | +| **Forged MPT Root (Omission)** | Index operator commits an invalid `mpt_root` in an Output Log entry to hide or fabricate records. | Local MPT reconstruction from authenticated Input Log entries; root equality check against the Output Log entry. | Sets root mismatch alert metrics; discards uncommitted batch and freezes disk state; `/readyz` reports `503` degraded status. | +| **Corrupted Input Entries** | Network attacker or corrupted CDN alters raw Input Log entry bundles. | Entry bundle Merkle authentication fails against `input_root` / `latest_input_root` verified under the signed Input Log checkpoint. | Pipeline rejects entry bundle; retries transient fetch errors without advancing state. | +| **Unauthenticated Output Entries** | Index operator serves Output Log entry bundles that do not match the Output Log checkpoint root. | `tlog-tiles` entry bundle Merkle authentication fails against the Output Log checkpoint root hash. | Entry bundle rejected; audit loop terminates with an authentication error. | +| **Parameter Drift (`input-origin` / `from`)** | Index operator publishes an Output Log state commitment altering `input-origin` or `from`. | State commitment parser checks `input-origin` and `from` against the auditor's configured parameters on every Output Log entry. | Audit loop halts immediately; state frozen; alerts emitted. | + +### 2.5 In-Situ Invariants & Performance Optimizations + +- **[Correctness Invariant] Strict Root Equality Assertion**: + - *Rule*: For every processed Output Log entry, the candidate `mpt_root` computed from authenticated Input Log entries in `[from, input_size)` must equal the `mpt_root` committed in the `c2sp.org/tlog-vindex-state@v1` payload. + - *Rationale*: Guarantees that the index operator's published index state is 100% faithful to the Input Log. + - *Consequence ("Or Else")*: Accepting mismatched roots would allow a malicious or buggy index operator to conceal or fabricate key occurrences without detection. + +- **[Correctness Invariant] Monotonic Progression & Log Consistency**: + - *Rule*: Each successive Output Log entry must satisfy `from <= prev_size <= input_size <= latest_input_size`, Merkle consistency across consecutive Output Log checkpoints, and Merkle consistency of `(input_size, input_root)` against the verified signed Input Log checkpoint `(latest_input_size, latest_input_root)`. + - *Rationale*: Transparency logs are append-only; historical tree prefixes and sizes cannot shrink or rewrite prior entries. + - *Consequence ("Or Else")*: An index operator or upstream log could roll back or rewrite historical entries undetected. + +- **[Performance Optimization] High-Throughput Catch-Up Architecture**: + - *Mechanism*: Reuses the core batching pipeline (256-entry bundle fetching, Wazero worker pool, and KV batch staging). + - *Impact*: Sustains 90,000 to 240,000+ entries/sec, allowing a newly provisioned auditor to catch up over tens of millions of historical Input Log entries in hours. + +### 2.6 Operational Runbook: Responding to Mismatches +When a root mismatch alert fires (mismatch metric `== 1` or `/readyz` returning `HTTP 503` with `"auditor root hash mismatch"`): +1. **Triage & Containment**: The auditor has automatically halted forward sync and frozen database state on disk. Do not restart the auditor with cleared storage, as the disk preserves forensic state immediately prior to the divergent Output Log entry. +2. **Inspect Structured Logs**: Examine auditor logs and `/readyz` diagnostics to record the divergent output entry index, target `input_size`, `local_root`, and `committed_root`. +3. **Forensic State Extraction**: Inspect local Pebble DB inverted chunks (`'c'`) and compare against raw Input Log entries in `[prev_size, input_size)` to determine whether the index operator omitted entries, altered canonical keys, or suffered a non-deterministic `MapFn` bug. +4. **Reproduce via Standalone One-Shot Audit**: Run the auditor in one-shot mode with an isolated scratch directory against the target Output Log checkpoint to confirm deterministic reproducibility. +5. **Publish Misbehavior Bundle**: If `MapFn` and the Input Log entries are valid, publish the cryptographic proof bundle (witnessed Output Log checkpoint, Output Log inclusion proof, raw `c2sp.org/tlog-vindex-state@v1` entry, and the signed Input Log checkpoint + consistency proof) to transparency witnesses and public monitoring channels. + +### 2.7 Auditor Contract + +The auditor subsystem provides continuous full-log state verification: + +- **Inputs**: + - Output Log URL (``), expected ``, and Output Log checkpoint verification policy. + - Expected ``, Input Log checkpoint & entry retrieval endpoint, and Input Log checkpoint verification policy (used to verify signed Input Log checkpoints, `(input_size, input_root)` consistency proofs, and entry bundles). + - Compiled WASM `MapFn` module (pinned out-of-band by the auditor operator) and configured start index `from` (default `0`). + - Local Pebble DB and MPT storage directories. + - Internal operational listener serving `/healthz`, `/readyz`, `/metrics`, and `/debug/pprof/*`. +- **Outputs**: + - Continuous audit telemetry and `/readyz` JSON diagnostics on the internal operational listener. +- **Safety Guarantees**: + - Never signs or publishes Output Log checkpoints. + - Never commits unverified state to disk: candidate MPT roots are predicted from in-memory staged `(canonicalKey, value_root)` mappings and checked against the Output Log entry commitment before applying the staged Pebble batch, mutating the verified MPT, or advancing watermarks. + - Immediately halts forward sync and preserves forensic state on any checkpoint policy failure, Merkle consistency failure, tree size regression, `input-origin` or `from` mismatch, entry bundle authentication failure, invalid canonical key emission, or `mpt_root` mismatch. + +--- + +## 3. Alternatives Considered (or Tried) + +### 3.1 Lookup Client Verification Alone vs. Full-Log Independent Auditing +- **Proposed**: Relying exclusively on lookup clients (`vindex/client`) to verify individual query responses, omitting a full-log auditor. +- **Theoretical Rejection**: Lookup client verification proves inclusion or non-inclusion against `mpt_root` only when a key is queried, and cannot verify that `mpt_root` itself was faithfully constructed from the Input Log without omissions across unqueried keys. +- **Chosen Design**: Complementary verification: lightweight lookup client SDK ([`client/`](../../client/README.md)) for per-query proof verification against witnessed checkpoints, paired with `vindex/internal/auditor` for continuous full-log replay auditing. + +### 3.2 Ephemeral In-Memory Auditing vs. State-Preserving Forensic Freezing +- **Proposed**: Storing auditor state ephemerally in RAM or wiping local database files upon detecting an upstream error. +- **Operational Rejection**: Wiping or discarding state upon detecting a forged `mpt_root` destroys the exact pre-divergence state needed to pinpoint omitted Input Log entries and produce a reproducible misbehavior report against the index operator. +- **Chosen Design**: State-preserving halt: the auditor freezes verified Pebble DB and MPT files in-place, exposes structured JSON diagnostics on `/readyz`, keeps `/healthz` at `HTTP 200 OK` so orchestrators do not restart the halted process, and stops forward sync. + +--- + +## 4. Verification Gate + +All auditor replay, checkpoint policy verification, and root mismatch forensic freeze tests are verified via: + +```bash +go test -race ./vindex/internal/auditor/... +``` diff --git a/vindex/internal/coordinator/README.md b/vindex/internal/coordinator/README.md new file mode 100644 index 0000000..b6a9aed --- /dev/null +++ b/vindex/internal/coordinator/README.md @@ -0,0 +1,189 @@ +# Coordinator & Lifecycle Engine + +This document defines the batch orchestration engine, watermark governance, Zero-WAL startup recovery protocols, load-bearing invariants, and verified performance optimizations for the **Coordinator Subsystem** (`vindex/internal/coordinator`). + +--- + +## 1. Context & Objectives + +### 1.1 Problem Statement & Orchestration Dilemma +The VIndex architecture decomposes indexing into five specialized, asynchronous subsystems: +1. **`internal/ingest`**: Network fetching and local 256-entry bundle tile caching. +2. **`mapfn` & `internal/wasmhost`**: Isolated WebAssembly guest SDK (`mapfn`) and Wazero sandbox host execution (`internal/wasmhost`). +3. **`internal/kvstore`**: Pebble LSM inverted chunk storage. +4. **`internal/mpt` & `internal/tree`**: Authenticated MPT state trie (`internal/mpt`) and Output Log state commitment publication (`internal/tree`). +5. **`internal/server`**: C2SP HTTP read serving. + +Without a central orchestrator managing these components behind strict black-box interfaces, three critical failure modes emerge: +1. **Watermark Drift & Ambiguity**: If individual subsystems track state independently, cross-subsystem boundaries drift out of alignment. A crash between storage sync and Output Log publication leaves the node unable to determine where recovery begins. +2. **Moving-Goalpost Starvation**: In high-velocity logs appending thousands of entries per second, continuously polling the log head prevents the pipeline from ever finalizing a batch, causing synchronization starvation. +3. **Network-Dependent Crash Recovery**: If startup recovery relies on downloading historical tiles over the network, node restart times become unpredictable and fail completely during upstream network outages. + +### 1.2 Centralized Watermark Authority & Separation of Concerns +The Coordinator serves as the **single source of truth** for system lifecycle and watermark governance: +- Worker subsystems expose clean, black-box functional interfaces and do not inspect each other's internal metadata or storage engines. +- The Coordinator alone tracks the global progression chain, evaluates persistence boundaries, and calculates the authoritative `SafeWatermark` (`max(min(KV_CP.Size, MPT_Durable_Size), from)`) communicated to the tile cache reaper. + +### 1.3 Goals & Non-Goals +- **Goals**: + - Drive the steady-state batch processing loop across all pipeline stages in strictly monotonic sequence. + - Prevent moving-goalpost starvation by freezing target sync checkpoints prior to batch ingestion. + - Enforce the synchronous handoff: persist the KV checkpoint to disk, ratchet forward, predict the MPT change, and promote the state commitment entry to the Output Log. + - Guarantee deterministic, network-free Zero-WAL crash recovery (< 5ms clean start, < 500ms dirty crash recovery after `torchwood/mpt` journal replay completes on open). +- **Non-Goals**: + - **No Distributed Consensus**: Operates strictly within a single process; does not run Raft, Paxos, or distributed leader election. + - **No Direct In-Memory Mutation**: The Coordinator does not touch trie nodes or LSM chunk bytes directly; it coordinates subsystem boundaries. + - **No Arbitrary Runtime Mode Switching**: Operates Normal Serving Mode for all serving logs; Genesis Backfill is used strictly during unserved initial bootstrap when the Output Log is empty (`Output_CP.Size == 0`). + +### 1.4 Requirements & Dependencies +- **Dependencies**: Orchestrates the ingestion fetcher and local tile cache, WASM mapping pipeline (`internal/wasmhost`), KV chunk store (`internal/kvstore`), MPT state trie (`internal/mpt`), Output Log publisher (`internal/tree`), and checkpoint policy verification (`internal/wire`). + +--- + +## 2. Detailed Design + +### 2.1 The Master Batch Loop +The coordinator executes batch synchronization through a structured, 6-step sequential lifecycle: + +| Step | Phase | Action & Subsystem Transition | Durability Transition | +| :- | :--- | :--- | :--- | +| **1** | **Poll & Freeze** | Polls upstream Input Log `/checkpoint`, verifies the Input Log checkpoint against `` and the configured Input Log checkpoint policy (e.g., C2SP `tlog-policy`), checks `Target_CP.Size >= max(prev_Target_CP.Size, from)` and Input Log Merkle consistency against `prev_Target_CP`, and writes `Target_CP` to Pebble metadata. | Freezes target sync boundary. | +| **2** | **Tile Ingestion** | Downloads missing 256-entry bundles across `[max(KV_CP.Size, from), Target_CP.Size)` into the local tile cache. | Durably stages verified entry bundles on disk. | +| **3** | **Parallel Mapping** | Streams 256-entry bundles to WASM workers (`internal/wasmhost`), validates un-hashed canonical keys (`1..1024` bytes in `0x21..0x7E`), and resequences batches into multi-tile commit slabs. | In-memory key-index aggregation. | +| **4** | **Storage Commit** | Streams commit slabs into the KV store (committing intermediate slabs without disk sync while accumulating modified `(key, value_root)` mappings across the target window, and executing a blocking disk `fsync` barrier on the final slab reaching `Target_CP.Size`). | **Ratchets persisted KV storage watermark durably to disk and persists `KV_CP` at `Target_CP.Size`.** | +| **5** | **Commitment & Promotion** | Once `Target_CP.Size` is reached and synced in Pebble, predicts candidate `mpt_root` over accumulated modified `(key, value_root)` mappings, appends one 6-line state commitment entry (`c2sp.org/tlog-vindex-state@v1`: `input-origin`, `from`, `input-size`, `input-root`, and `mpt-root`) for `Target_CP` to the Output Log (while retaining the signed Input Log checkpoint durably out-of-band), collects witness cosignatures per Output Log `tlog-policy`, and ratchets `Serving_CP` under the exclusive tree lock (< 5ms). | **Promotes state to Output Log (`Output_CP`) & Serving (`Serving_CP`).** | +| **6** | **Pruning Notification** | Computes `SafeWatermark` (`max(min(KV_CP.Size, MPT_Durable_Size), from)`) and notifies the tile cache reaper. | Bounded disk cache garbage collection. | + +### 2.2 Checkpoint Governance & Progression Chain + +#### The 4 Authoritative Checkpoints + +| Checkpoint | Log Type | Committed Size | Persistence | Advancement Mechanism | Role in Pipeline | +| :--- | :--- | :--- | :--- | :--- | :--- | +| **`Target_CP`** | Input Log | `Target_CP.Size` (`input_size`) | Durable (synced in Pebble metadata) | Upstream poll, Input Log checkpoint policy & Merkle consistency check (`>= max(prev_Target_CP.Size, from)`) | Upper goalpost for current sync cycle; prevents moving-goalpost starvation. | +| **`KV_CP`** | Input Log | `KV_CP.Size` | Durable (synced in Pebble LSM) | Atomic flush of `'c'` chunk records | Durably binds indexed `'c'` chunk records to the verified Input Log state they cover. | +| **`Output_CP`** | Output Log | `Output_CP.Size` (`output_size`), committing `Output_CP.InputSize` (`input_size` in state commitment entry) | Durable (Tessera Output Log storage) | Append state commitment entry & witness cosigning | Cryptographically commits to index state (`input-origin`, `from`, `input-size`, `input-root`, and `mpt-root`), with the signed Input Log checkpoint retained out-of-band. | +| **`Serving_CP`** | Output Log Entry | `Serving_CP.InputSize` (`input_size`) | Volatile (In-memory atomic snapshot pointer) | Pointer swap via atomic state promotion under tree lock (< 5ms) | Active state exposed to client HTTP readers. | + +**Monotonic Checkpoint Progression Invariant**: +```text +Target_CP.Size >= KV_CP.Size >= Output_CP.InputSize >= Serving_CP.InputSize +``` +Every relation is `>=`: +- `Target_CP.Size >= KV_CP.Size`: Batch aggregation advances toward the target checkpoint in discrete commit slices. +- `KV_CP.Size >= Output_CP.InputSize`: KV storage persistence strictly precedes Output Log publication. +- `Output_CP.InputSize >= Serving_CP.InputSize`: Output Log publication and witness collection execute outside reader locking; `Output_CP` advances before the in-memory `Serving_CP` pointer is swapped via atomic state promotion under the tree lock. + +#### Intermediate Buffers & Trailing Durability +- `Target_CP -> KV_CP`: Ingest stages 256-entry bundles on disk and through worker mapping queues; the persisted KV storage watermark accumulates across write batches until synced at `KV_CP.Size`. +- `KV_CP -> Output_CP`: Non-blocking MPT prediction calculates candidate `mpt_root`. +- `Output_CP -> Serving_CP`: In-memory atomic swap ratchets `Serving_CP` (< 5ms critical section). +- `Trailing Serving_CP`: `MPT_Durable_Size` `fsync`s to disk in the background, satisfying `Serving_CP.InputSize >= MPT_Durable_Size`. + +### 2.3 Authoritative Safe Pruning Watermark for Tile Cache Reclamation +To guarantee that the ingestion cache never deletes 256-entry bundles needed for crash recovery, the Coordinator calculates the authoritative safe pruning boundary `SafeWatermark = max(min(KV_CP.Size, MPT_Durable_Size), from)`. Because storage persistence strictly precedes Output Log publication and MPT persistence (`KV_CP.Size >= Output_CP.InputSize >= MPT_Durable_Size`), clamping the lower bound to `from` ensures that suffix indices (`from > 0`) never retain or replay pre-`from` tiles before their first durable MPT sync, while keeping all 256-entry bundles in `[max(MPT_Durable_Size, from), KV_CP.Size)` cached on disk until the MPT confirms a durable `fsync`. + +### 2.4 Moving-Goalpost Prevention +On high-traffic transparency logs, the log head advances continuously. If the coordinator polled unverified checkpoints on every iteration, the sync target would constantly move, starving downstream commitment. + +The Coordinator prevents this by writing the verified target checkpoint into Pebble DB metadata (`Target_CP`) prior to batch processing. The entire pipeline processes that fixed entry range to completion before the coordinator advances to a new target. + +### 2.5 Zero-WAL Startup Recovery Sequence & Uncommitted KV Writes +On daemon launch, the HTTP read server gates public lookup endpoints with `503 Service Unavailable` while opening the MPT replays any uncompacted `torchwood/mpt` journal frames on disk. Once journal replay finishes, the Coordinator computes `recoveryTarget = Output_CP.InputSize` when `Output_CP.Size > 0` (or `KV_CP.Size` when `Output_CP.Size == 0`) and executes a deterministic 3-phase recovery sequence: + +1. **Phase 1: Instant Warm Start (< 5ms after MPT open)**: + - Entered if and only if `Output_CP.Size > 0`, the MPT's persisted version equals `(Output_CP.InputSize, true)` (`exact == true` and `MPT_Durable_Size == Output_CP.InputSize`), and the trie root matches `mpt-root` in the latest Output Log state commitment entry. + - Clean shutdown is verified: activates `Serving_CP`, initializes watermarks, and **opens the HTTP Read Server immediately (< 5ms)**. +2. **Phase 2: Fast-Forward Tile Replay (< 500ms after MPT open)**: + - Entered whenever the persisted MPT lags `recoveryTarget` or is inexact (`version < recoveryTarget || !exact`): + - Streams missing historical 256-entry bundles across `[max(MPT_Durable_Size, from), recoveryTarget)` directly from the local disk tile cache. + - Maps replayed entries via `map_bundle` to collect distinct modified un-hashed canonical keys. + - Reconstructs value log roots (`value_root`) for modified keys at `recoveryTarget` from the KV store with **zero writes to the database**. + - Updates in-memory MPT nodes with sorted `(key, value_root)` pairs and flushes MPT persistence to disk. When `Output_CP.Size > 0`, asserts that the resulting `mpt_root` strictly matches the latest Output Log state commitment entry, activates `Serving_CP`, and **opens the Read Server (< 500ms)**; when `Output_CP.Size == 0`, public endpoints remain `503 Service Unavailable`. +3. **Phase 3: Background Catch-Up & Uncommitted KV Writes**: + - Resumes forward ingestion from `max(Serving_CP.InputSize, from)` when `Output_CP.Size > 0` (or from `max(KV_CP.Size, from)` in Genesis Backfill when `Output_CP.Size == 0`—including clean restarts mid-backfill where `version == KV_CP.Size && exact == true`) toward `Target_CP.Size`. + - **Handling Uncommitted KV Writes**: If a crash occurred in Normal Serving Mode after Pebble durably synced a batch to disk but before Output Log commitment (`KV_CP.Size > max(Serving_CP.InputSize, from)`), the next sync cycle streams starting from `max(Serving_CP.InputSize, from)`. When re-processing Input Log entries that were already written to disk, the KV store filters out already-persisted entry indices and reconstructs their `value_root` digests directly from Pebble without re-writing storage. This allows uncommitted batches to be safely published to the Output Log upon initial sync without duplicate entries or storage churn. + +### 2.6 In-Situ Invariants & Performance Optimizations + +- **[Correctness Invariant] Synchronous Stage Promotion**: + - *Rule*: Inverted chunk storage writes must complete a durable disk sync and commit `KV_CP` before Output Log publication is permitted to execute (`KV_CP.Size >= Output_CP.InputSize`). + - *Rationale*: Guarantees `KV_CP.Size >= Output_CP.InputSize` across all crash scenarios. + - *Consequence ("Or Else")*: A crash between Output Log publishing and storage sync would publish a state commitment entry referencing missing or rolled-back KV chunks, permanently breaking inclusion proofs for witnessed checkpoints. + +- **[Correctness Invariant] Zero-WAL Startup Durability Guarantee**: + - *Rule*: Because `KV_CP.Size >= Output_CP.InputSize` (and cached tiles cover `[max(MPT_Durable_Size, from), KV_CP.Size)`), crash recovery must resolve all lagging MPT state using the local tile cache and local Pebble DB records without issuing external network requests. + - *Rationale*: Protects node availability against upstream log outages and guarantees predictable cold-boot timing. + - *Consequence ("Or Else")*: A node recovering from a crash during an upstream network partition would fail to boot, causing extended service downtime. + +- **[Performance Optimization] Multi-Tile Slab Aggregation & Target-Level Witnessing**: + - *Mechanism*: Buffers mapped Input Log entries into multi-tile commit slabs without per-slab disk syncs on intermediate slabs, executing a single durable KV sync and Output Log witness cycle at `Target_CP.Size`. + - *Impact*: Amortizes disk write latency and external witness network RPCs across the sync window, sustaining high indexing throughput (>90,000 entries/sec). + +### 2.7 Coordination Contract & State Guarantees + +The coordinator acts as the single source of truth for pipeline progression and lifecycle management: + +- **Inputs**: + - Input Log fetcher and configured Input Log checkpoint policy. + - Configured `` and start index `from` (default `0`). + - Local tile cache. + - Mapping pipeline (`internal/wasmhost`). + - KV store (`internal/kvstore`). + - MPT state trie (`internal/mpt`). + - Output Log publisher (`internal/tree`) and configured Output Log checkpoint policy. +- **Recovery Contract**: + - Enforces 3-phase startup recovery: + 1. Verifies whether persisted trie state matches the committed Output Log watermark (`Output_CP.InputSize`). + 2. Replays uncommitted 256-entry bundles across `[max(MPT_Durable_Size, from), recoveryTarget)` from the local disk cache to align the MPT with zero database writes. + 3. Resumes ingestion without duplicating persisted writes. +- **Durability Invariant**: + - Storage durability strictly precedes public Output Log commitment (`KV_CP.Size >= Output_CP.InputSize`). +- **Concurrency**: + - The background polling loop runs synchronously per iteration; catch-up and steady-state ingestion are strictly single-threaded. + +### 2.8 Genesis Backfill Lifecycle (Unserved Bootstrap) + +When bootstrapping an unserved index from its configured start index `from` (`Output_CP.Size == 0`) against a large Input Log, accumulating every modified `(key, value_root)` pair across `[from, Target_CP.Size)` in memory before a single final commit would exhaust host RAM, while publishing and witnessing an Output Log state commitment entry on every intermediate batch would incur thousands of unnecessary external witness roundtrips before the index is ready to serve. + +When `Output_CP.Size == 0`, the Coordinator executes a synchronous **Genesis Backfill** flow: + +1. **Gating & Serving Plane Isolation (`Output_CP.Size == 0`)**: + - Activated automatically if and only if the Output Log has zero state commitment entries (`Output_CP.Size == 0`). + - Throughout Genesis Backfill, public `/keys` and `/values` endpoints and readiness probes remain `503 Service Unavailable` per [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md). + +2. **Bounded Coalescing & Synchronous Coarse Checkpointing (Suppressing Intermediate Output Log Appends)**: + - Multi-tile batches commit inverted `'c'` chunk updates to Pebble without per-batch `fsync` while coalescing modified `(key, value_root)` mappings in a bounded in-memory map and **suppressing intermediate Output Log appends and witness roundtrips**. + - Whenever the modified-key count or entry threshold is reached, the Coordinator executes a synchronous Pebble disk sync (advancing `KV_CP.Size`) followed immediately by a synchronous MPT commit and disk sync (advancing `MPT_Durable_Size` strictly after Pebble `fsync` so `KV_CP.Size >= MPT_Durable_Size` always holds even across power loss), clears the modified-key map, and updates `SafeWatermark` (`max(min(KV_CP.Size, MPT_Durable_Size), from)`) so the tile cache reaper prunes superseded tiles. + +3. **Genesis Finalization & Output Log `Entry 0` Publication**: + - Upon reaching `Target_CP.Size`, the Coordinator executes a final synchronous Pebble sync and MPT commit/sync, publishes Output Log `Entry 0` (`c2sp.org/tlog-vindex-state@v1` binding `input-origin`, `from`, `input-size`, `input-root`, and `mpt-root`), collects witness cosignatures, promotes `Serving_CP`, and permanently transitions to Normal Serving Mode (`Output_CP.Size > 0`). + +4. **Standard Phase 2 Crash Recovery**: + - If the process crashes during Genesis Backfill (`Output_CP.Size == 0` and `version < KV_CP.Size || !exact`), startup reuses standard Phase 2 recovery across `[max(MPT_Durable_Size, from), KV_CP.Size)`—replaying cached 256-entry bundles and reconstructing `value_root` digests from Pebble—with zero special-case backfill recovery machinery before resuming synchronous backfill from `KV_CP.Size` toward `Target_CP.Size`. + +--- + +## 3. Alternatives Considered (or Tried) + +### 3.1 Decentralized Subsystem Watermarking vs. Centralized Coordinator Governance +- **Proposed**: Allowing each subsystem to independently determine when to advance watermarks and prune historical state. +- **Theoretical Rejection**: Decentralized state tracking creates race conditions during dirty crashes. Subsystems cannot establish whether sibling components successfully persisted data, leading to either unrecoverable data loss or redundant disk storage. +- **Chosen Design**: Centralized Coordinator governance with a single authoritative progression chain. + +### 3.2 Distributed Consensus Orchestration (Raft/Paxos) vs. Single-Host Coordinator +- **Proposed**: Coordinating indexing across a distributed cluster using Raft consensus. +- **Theoretical & Architectural Rejection**: + - The MPT already requires single-host RAM/mmap locality; clustering the coordinator adds immense operational complexity without solving tree state synchronization. + - High availability is achieved externally: independent operators run standalone VIndex instances against the shared Input Log. +- **Chosen Design**: Single-process, single-host embedded coordinator. + +--- + +## 4. Verification Gate + +All coordinator recovery, genesis backfill, and batch loop invariants are verified via: + +```bash +go test -race ./vindex/internal/coordinator/... +``` diff --git a/vindex/internal/hammer/README.md b/vindex/internal/hammer/README.md new file mode 100644 index 0000000..0c0a002 --- /dev/null +++ b/vindex/internal/hammer/README.md @@ -0,0 +1,121 @@ +# Hammer Load Testing & Verification Framework + +This document defines the architecture, workload generation models, drip-feed scheduling protocols, cryptographic invariant verification rules, and operational telemetry for the internal **Hammer Integration Load-Testing & Verification Package** (`vindex/internal/hammer`). + +--- + +## 1. Context & Objectives + +### 1.1 Problem Statement & Integration Testing Dilemma +Unit tests verify isolated package logic, but critical failure modes in a verifiable index only emerge under **sustained, concurrent read-write load**: +1. **LSM Compaction Stalls**: Heavy write ingestion triggers background Pebble LSM compactions that can spike read latency or stall batch commits. +2. **Lock Contention & Race Conditions**: High-frequency updates to hot keys stress in-memory MPT path compression and exclusive writer lock transitions, risking deadlocks or tail-latency regressions against concurrent HTTP lookups. +3. **Chunk Boundary Rollovers**: Inverted Value Log chunks roll over every 65,536 Input Log entries; verifying that multi-chunk backward pagination reconstructs complete, unbroken Value Logs requires driving individual hot keys across multiple 65,536-entry boundaries. + +Validating that `vindexd` preserves cryptographic integrity, zero data loss, and sub-millisecond read serving requires an internal integration load-testing and verification package (`vindex/internal/hammer`) that simultaneously acts as the upstream Input Log producer and a pool of concurrent lookup clients. + +### 1.2 Goals & Non-Goals +- **Goals**: + - Provide an internal integration load-testing and verification harness (`vindex/internal/hammer`) for stressing `vindexd` under multi-million-entry workloads with heavy Zipfian key skew. + - Act simultaneously as the upstream Input Log producer (serving signed Input Log checkpoints and `tlog-tiles` entry bundles under steady, burst, and pause/catch-up drip schedules) and a pool of concurrent lookup clients. + - Assert [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md) cryptographic invariants (Output Log checkpoint policy and inclusion proof verification, MPT point/prefix proofs, Value Log compact range roots, and monotonic history) on 100% of lookup queries. + - Render real-time terminal telemetry tracking ingestion throughput (entries/sec), read QPS, latency percentiles, and invariant checks. +- **Non-Goals**: + - **Not a Public SDK Package**: `vindex/internal/hammer` is strictly an internal test and benchmark harness, not an external library API. + - **Not a Production Log Sequencer**: Designed exclusively for integration testing, benchmarking, and chaos verification. + - **Not an In-Memory Mock**: Executes real HTTP requests over loopback sockets against real POSIX files. + +### 1.3 Requirements & Dependencies +- **Dependencies**: Integrates `tessera` (POSIX storage, appender, publication awaiter) and [`client/`](../../client/README.md) (`vindex/client`). + +--- + +## 2. Detailed Design + +### 2.1 Component Architecture & Pipeline Map + +| Subsystem Component | Core Responsibility | +| :--- | :--- | +| **Synthetic Entry Generator** | Emits structured synthetic Input Log entries across Zipfian/Pareto and uniform key distributions in configurable leaf formats (`raw`, `sumdb`, or `ct`), and provides disjoint non-inclusion probe keys for reader workers. | +| **Tessera Input Log Sequencer** | Writes generated Input Log entries into a local POSIX append-only log, computes Merkle roots, and signs Input Log checkpoints with a configurable or generated Ed25519 signer (printing the verifier key on startup for `vindexd`). | +| **Drip-Feed HTTP Server** | Serves standard C2SP `tlog-tiles` endpoints (`/checkpoint` and `/tile/...`) for the synthetic Input Log and schedules checkpoint visibility across steady, burst, and pause/catch-up modes. | +| **Verifying Concurrent Readers** | Queries `vindexd` (`GET /keys` and `GET /values`) via [`client/`](../../client/README.md) across hot, uniform, non-inclusion, and full-pagination query mixes, cryptographically verifying 100% of lookup responses. | +| **Metrics Analyzer & Dashboard** | Aggregates latency percentiles, read QPS, ingestion lag, and invariant assertions into periodic terminal summaries. | + +--- + +### 2.2 Synthetic Entry Generation & Leaf Formats +The entry generator synthesizes Input Log entries in raw key/value (`raw`), Go SumDB (`sumdb`, ` h1:`), or Certificate Transparency (`ct`, X.509/precertificate) formats across two write distributions: + +1. **Zipfian / Pareto Skew (`alpha > 1`)**: + - Models realistic registries where the hottest 1% of canonical keys account for over 80% of all Input Log entries. + - **Targeted Stress**: Drives individual hot keys across multiple 65,536-entry chunk boundaries, verifying that chunk rollovers (`^0 -> ^1`), Pebble SSTable compactions, and multi-page backward `/values` scans preserve unbroken Value Log continuity. +2. **Uniform Distribution**: + - Spreads Input Log entries evenly across a wide canonical keyspace (`[0, K)`) to stress MPT branch fanout and single-occurrence chunk creation. + +--- + +### 2.3 Tessera Input Log Sequencer & Drip-Feed HTTP Server +To evaluate `vindexd` under varied upstream conditions, the hammer controls Input Log checkpoint visibility: + +- **POSIX Storage & Signer**: Appends generated Input Log entries to a local directory using Tessera POSIX storage, signing Input Log checkpoints with a configurable or freshly generated Ed25519 key (and printing the corresponding verifier key at startup so `vindexd` can configure its Input Log checkpoint policy). +- **Drip-Feed Schedules**: + - **Steady Drip**: Releases queued Input Log checkpoints at a fixed rate, simulating steady upstream log integration. + - **Burst Mode**: Buffers generated checkpoints and releases them in sudden bursts, simulating upstream batch publications. + - **Pause & Catch-Up Mode**: Holds checkpoint publication for a configurable window so `vindexd` idles, then exposes a deep backlog of 256-entry bundles to benchmark catch-up throughput. + +--- + +### 2.4 Verifying Concurrent Readers +A pool of concurrent reader workers issues continuous point and prefix lookups against `vindexd` (`GET /keys?key=`, `GET /keys?prefix=`, and `GET /values?key=&before=`), validating every response through [`client/`](../../client/README.md) configured with the expected ``, ``, and Output Log checkpoint policy. + +#### Reader Query Mix: +Each reader query selects a target key according to a weighted workload mix: +- **Hot Keys (~60%)**: Samples heavily skewed Zipfian keys and verifies the newest `/values` page (single-page break). +- **Uniform Keys (~25%)**: Samples uniformly across the active keyspace and verifies the newest `/values` page. +- **Non-Inclusion Probes (~10%)**: Samples synthetic keys (`nonexistent/`) never appended to the Input Log, exercising cryptographic MPT non-inclusion and empty-prefix proofs (`0` matched key lines in `c2sp.org/tlog-vindex-keys@v1`). +- **Full Backward Pagination (~5%)**: Samples a hot key and walks `MatchedKey.Values(ctx)` all the way back to `preceding == 0` across multiple 65,536-entry chunks. + +#### Query Verification Sequence: +1. **Output Log Proof & State Commitment**: `Client.LookupKey` / `Client.LookupPrefix` verifies `` and `c2sp.org/tlog-proof@v1` against `ExpectedOutputLogOrigin` and `OutputLogPolicy` (asserting `output_size >= 1` and canonical decimal `index == output_size - 1`) and parses the 6-line `c2sp.org/tlog-vindex-state@v1` record (`input-origin == ExpectedInputLogOrigin` and `0 <= from <= input_size`). +2. **MPT Proof Verification**: Verifies the `c2sp.org/tlog-vindex-keys@v1` binary MPT inclusion, non-inclusion, or two-sided prefix proof against `mpt_root`, returning an immutable `*client.LookupResult` snapshot. +3. **Value Log Compact Range Verification**: Iterates `MatchedKey.Values(ctx)` (either stopping after the newest page on normal hot/uniform probes or walking to `preceding == 0` on full-pagination probes), reconstructing and verifying the RFC 6962 Value Log root from `preceding` compact range hashes and returned Input Log entry indices on every fetched page. +4. **Monotonic History & Parameter Assertions**: Asserts that `LookupResult.From` never regresses across queries. Whenever a query observes the **complete** Value Log history for canonical key `K` (either a single-page result with `preceding == 0` or a full backward pagination walk to `preceding == 0`), the reader records the chronologically ordered entry index slice `I_new` at `input_size_new` and asserts that for any prior complete observation `I_old` at `input_size_old <= input_size_new`, `I_new` is a prefix-preserving superset of `I_old` (`len(I_new) >= len(I_old)` and `I_new[:len(I_old)] == I_old`). + +--- + +### 2.5 In-Situ Invariants & Performance Assertions + +- **[Correctness Invariant] Verification & Consistency Fault Discrimination**: + - *Rule*: Any verification error from `vindex/client` (`ErrVerificationFailed`) or reader-side consistency failure (`From` regression or monotonic history superset violation) immediately records a fatal invariant violation and fails the hammer run, whereas transient operational status codes (`503 Service Unavailable` during startup/backfill or `413 Content Too Large` on broad prefix limits) are tracked in telemetry without failing the run. + - *Rationale*: A verifiable index must never serve forged, torn, or unverifiable responses under any read/write concurrency stress, while legitimately returning `503` before its first Output Log checkpoint is published. + - *Consequence ("Or Else")*: Tolerating cryptographic proof errors would mask race conditions between MPT snapshots, Output Log witnessing, and Pebble chunk flushes, while treating initial `503` responses as cryptographic faults would cause false failures during cold startup. + +- **[Correctness Invariant] Monotonic History Superset on Complete Scans**: + - *Rule*: For repeated complete Value Log observations (`preceding == 0` reached) on canonical key `K` where `input_size_new >= input_size_old`, `I_new` must be a superset of `I_old` with an identical prefix (`I_new[:len(I_old)] == I_old`). + - *Rationale*: Input Logs and per-key Value Logs are strictly append-only; committed Input Log entry indices can never vanish, shrink, or reorder. + - *Consequence ("Or Else")*: Any violation proves chunk rollover corruption, out-of-order batch commits, or broken snapshot isolation. + +--- + +## 3. Alternatives Considered (or Tried) + +### 3.1 In-Process Go Unit Test Mocks vs. Out-of-Process End-to-End Harness +- **Proposed**: Stress-testing `vindexd` purely in-memory using mocked storage and channel harnesses. +- **Practical Rejection**: In-memory mocks miss real concurrency and I/O dynamics—loopback HTTP connection pooling, OS scheduler preemption, `mmap` page flushing, and Pebble LSM compaction stalls. +- **Chosen Design**: Real HTTP client/server execution backed by POSIX Tessera tiles, Pebble SSTables, and `mmap` MPT files. + +### 3.2 Static Pre-Recorded Log Dumps vs. Dynamic Drip-Feed Sequencer +- **Proposed**: Replaying a fixed snapshot of an existing public transparency log. +- **Practical Rejection**: Static dumps cannot modulate write rates on demand, dial exact Zipfian skew to force 65,536-entry chunk rollovers, inject targeted non-inclusion probes, or orchestrate pause/burst checkpoint schedules. +- **Chosen Design**: Deterministic synthetic entry generator paired with a programmable drip-feed `tlog-tiles` sequencer. + +--- + +## 4. Verification Gate + +All internal hammer generator, sequencer, and concurrent client verification tests are verified via: + +```bash +go test -race ./vindex/internal/hammer/... +``` diff --git a/vindex/internal/ingest/README.md b/vindex/internal/ingest/README.md new file mode 100644 index 0000000..6029db2 --- /dev/null +++ b/vindex/internal/ingest/README.md @@ -0,0 +1,112 @@ +# Ingestion Pipeline & Local Tile Cache + +This document defines the architecture, data structures, concurrency model, load-bearing invariants, and verified performance optimizations for the **Ingestion Subsystem** (`vindex/internal/ingest`). + +--- + +## 1. Context & Objectives + +### 1.1 Problem Statement & Real-World Friction +The ingestion engine sits at the front of the VIndex pipeline. It must continuously poll remote append-only transparency logs over HTTP, verify checkpoint policies and Merkle consistency, download 256-entry bundles, execute untrusted mapping logic inside WebAssembly sandboxes (`internal/wasmhost`), and feed sorted batches to downstream storage. + +If ingestion is naive, four severe operational failure modes emerge: +1. **Host-Sandbox Boundary Overhead**: Invoking WebAssembly guest functions per entry incurs severe boundary-crossing penalties. Every transition across the Foreign Function Interface (FFI)—switching execution contexts and copying memory between the Go host process and the WebAssembly sandbox—costs CPU cycles. When executed entry-by-entry, FFI boundary overhead alone consumes ~23% of total host CPU time. +2. **Out-of-Order Corruption**: Executing mapping across parallel worker threads causes batches to finish at different times. If out-of-order batches reach the storage engine, the database commits non-contiguous entry index sequences, permanently corrupting the RFC 6962 compact range hashes of affected keys' value logs. +3. **Upstream Rate Limiting vs. Catch-Up Bandwidth**: Ingesting high-velocity logs from genesis (tens of millions of entries) requires saturating network bandwidth, but upstream CDNs and log endpoints enforce strict rate limits (HTTP 429 Too Many Requests). Ingestion must maximize parallel download concurrency while gracefully backing off to avoid upstream bans. +4. **Network-Free Restart Guarantee (Cache Durability)**: Network egress is expensive and external logs can be transiently unreachable. Ingestion stages consumed tiles locally in a bounded staging buffer until downstream storage and trie persistence confirm they are no longer needed, ensuring daemon restarts and crash recoveries complete with zero network roundtrips. + +### 1.2 Goals & Non-Goals +- **Goals**: + - Ingest upstream Input Logs in native 256-entry bundles matching the underlying tile structure. + - Maximize upstream download throughput with adaptive concurrency while respecting HTTP 429 rate limits. + - Parallelize CPU-heavy parsing across worker pools while emitting strictly contiguous Input Log entry sequences. + - Maintain a bounded local staging buffer and crash-recovery cache, enabling restarts and Zero-WAL recovery with zero network egress (not an immutable log of record). + - Bound local disk growth by having the background tile reaper actively prune cached tiles older than `SafeWatermark`. +- **Non-Goals**: + - **No Upstream Checkpoint Creation**: The ingestion engine does not publish checkpoints or modify upstream log state; it is strictly a read-only consumer. + - **No In-Guest Storage Writes**: The WASM sandbox is forbidden from performing disk or network I/O; guest modules solely transform entry bytes into canonical keys. + - **No Cross-Subsystem Watermark Calculation**: The ingestion cache does not inspect the internal state of Pebble DB or the MPT; watermark authority resides strictly with the Coordinator. + +### 1.3 Requirements, Dependencies & Constraints +- **Upstream Log Contract**: Supports standard tile-based logs ([C2SP tlog-tiles](https://c2sp.org/tlog-tiles) `/tile/entries/...` and [C2SP static-ct-api](https://c2sp.org/static-ct-api) `/tile/data/...`) as well as custom leaf-by-leaf log adapters that group entries into 256-entry bundles, with checkpoints verified against configured checkpoint verification policies (such as C2SP [tlog-policy](https://c2sp.org/tlog-policy)). +- **WASM Host Runtime**: Pure Go WebAssembly sandbox execution via `internal/wasmhost` backed by `github.com/tetratelabs/wazero` (zero CGO dependencies). +- **Resequencer Straggler Buffering**: If a single WASM worker stalls on an anomalous entry bundle, subsequent workers buffer finished batches in the in-memory min-heap until the straggler bundle completes. + +--- + +## 2. Detailed Design + +### 2.1 The Assembly Line Sorter Architecture +Checkpoint polling and `Target_CP` persistence are driven by the Coordinator using the upstream log fetcher; once a verified `Target_CP` is established, the Coordinator streams `[max(persisted_watermark, from), Target_CP.Size)` through the three-stage ingestion pipeline: + +1. **Tile Fetcher**: Queries the upstream Input Log's `/checkpoint` endpoint and verifies checkpoints against `` and the configured Input Log checkpoint policy. Downloads 256-entry bundles (`/tile/entries/[.p/]` for `tlog-tiles`, `/tile/data/[.p/]` for `static-ct-api` using partial tile width `.p/` on unsealed tail tiles, or synthesized 256-entry bundles via a leaf-by-leaf adapter) and authenticates each bundle against its Level-0 tree tile and upper-level tree tiles up to `Target_CP.Root` across `[max(persisted_watermark, from), Target_CP.Size)`. +2. **Local Tile Cache**: Writes downloaded, Merkle-authenticated 256-entry bundles to the local filesystem via atomic temp-file rename. When `Target_CP.Size` advances within an unsealed 256-entry tail tile across consecutive sync cycles, any cached partial bundle narrower than required (`bundleEnd < Target_CP.Size` and `len < 256`) is treated as a cache miss, re-fetched at the wider width, and atomically overwritten on disk. +3. **WASM Worker Pool**: A configurable pool of parallel Wazero sandboxes (`internal/wasmhost`, scaling across available host CPU capacity) running the compiled `.wasm` plugin. Each worker slices boundary bundles to `[max(bundleStart, start), min(bundleEnd, Target_CP.Size))`—excluding entries `< from`, entries already persisted in a partially indexed tail bundle, and entries `>= Target_CP.Size`—and maps the slice in a single `map_bundle` invocation via the in-guest SDK (`mapfn/sdk`). +4. **Canonical Key Validation & Per-Entry Deduplication**: Upon unpacking the WASM output slab, the host validates every emitted key against the canonical key bounds in [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md) (`1..1024` printable ASCII bytes in `0x21..0x7E`, containing no `NUL` or whitespace bytes) and sorts and deduplicates canonical keys per entry (`entryIdx`). If the WASM module traps, overflows its linear memory slabs, or emits any key violating canonical key bounds (`1..1024` bytes in `0x21..0x7E`), the host returns a fatal validation error that immediately **`HALT`s** ingestion—never silently dropping invalid keys—preserving bit-for-bit `mpt_root` equivalence with [`internal/auditor`](../auditor/README.md) and [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md). Valid deduplicated un-hashed canonical keys are passed downstream to storage and the MPT. +5. **Resequencer (Min-Heap)**: Receives completed batches out of order from workers and buffers them in a priority queue. Emits strictly ascending entry batches across `[max(persisted_watermark, from), input_size)` to the downstream storage channel. +6. **Tile Reaper**: Periodically receives the authorized `SafeWatermark` from the Coordinator (`max(min(KV_CP.Size, MPT_Durable_Size), from)`) and actively prunes cached tiles fully covered by that watermark (`(tileIndex + 1) * 256 <= SafeWatermark`) to bound local disk usage. + +### 2.2 In-Situ Invariants & Performance Optimizations + +- **[Correctness Invariant] Target Checkpoint Policy & Consistency Verification**: + - *Rule*: Upstream Input Log checkpoints must be cryptographically validated against `` and the configured Input Log checkpoint policy (enforcing primary log signature and witness cosignature quorum), must satisfy `Target_CP.Size >= max(prev_Target_CP.Size, from)`, and (after the first verified checkpoint) must verify Merkle consistency against `prev_Target_CP` before the size advancement is accepted. + - *Rationale*: Guarantees that the indexer only ingests authentic, append-only entries committed by the legitimate log authority and witnesses at or above `from`. + - *Consequence ("Or Else")*: If an unauthenticated, regressed, or split-view checkpoint were accepted, an adversary could feed forged or rolled-back entries, causing the indexer to commit bogus state and permanently diverge from independent auditors. + +- **[Correctness Invariant] Fatal `HALT` on WASM Trap or Invalid Canonical Key**: + - *Rule*: Every key emitted by the WASM guest must satisfy canonical key bounds (`1..1024` bytes in `0x21..0x7E`). If the WASM module traps or emits an invalid canonical key, the host must return a fatal validation error that immediately **`HALT`s** ingestion rather than dropping or sanitizing the offending key. + - *Rationale*: Preserves bit-for-bit root equivalence with [`internal/auditor`](../auditor/README.md) and the auditing rules in [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md). + - *Consequence ("Or Else")*: Silently dropping invalid keys would allow a defective mapping module to publish Output Log state commitments that spec-compliant auditors reject. + +- **[Correctness Invariant] Strictly Monotonic Resequencing**: + - *Rule*: Batches pushed to downstream storage must strictly follow ascending Input Log entry index (`entryIdx`) order without gaps, where each emitted batch's starting entry index equals the preceding batch's ending entry index. + - *Rationale*: Inverted chunks maintain cumulative RFC 6962 compact range hashes over value log leaves, which require strictly increasing Input Log entry indices. + - *Consequence ("Or Else")*: Committing out-of-order batches breaks the Merkle compact range tree structure, computing invalid `value_root` digests that fail client inclusion proofs. + +- **[Correctness Invariant] Tile Reaper Authority & Pruning Bounds**: + - *Rule*: The background tile reaper must strictly prune only those tile files whose highest entry index is covered by `SafeWatermark` (`(tileIndex + 1) * 256 <= SafeWatermark`), where `SafeWatermark` is authoritative input dictated by the Coordinator. + - *Rationale*: Preserves un-synced historical 256-entry bundles on disk so that Zero-WAL crash recovery can fast-forward lagging MPT state without network requests. + - *Consequence ("Or Else")*: Pruning ahead of the durable MPT watermark causes dirty crash recovery to fail, forcing the daemon to re-download historical tiles over the network and violating the sub-500ms time-to-first-serve SLO. + +- **[Performance Optimization] Native 256-Entry Tile Bundling**: + - *Mechanism*: Fetches and caches data in native 256-entry bundles rather than issuing individual HTTP requests per entry. + - *Impact*: Reduces upstream network requests by 256x compared to per-entry REST APIs. + +- **[Performance Optimization] Bundled WebAssembly Invocation (`map_bundle`)**: + - *Mechanism*: Transfers an entire 256-entry bundle across the WASM boundary in a single FFI call. + - *Impact*: Reduces FFI boundary crossings from 768 to 2–3 per tile, dropping FFI CPU overhead from ~23% of host CPU time to < 1%. + +### 2.3 Ingestion Pipeline Contract + +The ingestion subsystem manages upstream data retrieval, sandboxed mapping, and local tile caching: + +- **Inputs**: + - Upstream Input Log HTTP endpoint, expected ``, and Input Log checkpoint policy. + - Local tile cache directory. + - Sandboxed WASM mapping runner (`internal/wasmhost`). + - Worker concurrency. +- **Outputs**: + - Stream of mapped `(canonicalKey, entryIdx)` batches ordered monotonically by Input Log entry index. +- **Pruning Contract**: + - Local tile cache retains 256-entry bundles strictly within the active lag window; tiles covered by `SafeWatermark` are deleted by the background reaper. + +--- + +## 3. Alternatives Considered (or Tried) + +### 3.1 Ephemeral In-Memory Fetching vs. Persistent Local Tile Cache +- **Proposed**: Streaming 256-entry bundles directly from HTTP responses into the WASM worker pool in memory without persisting verified bundles on local disk. +- **Theoretical & Operational Rejection**: + - Tightly couples upstream HTTP jitter to downstream mapping and storage throughput. + - Any daemon restart or dirty crash where `MPT_Durable_Size` lags behind `KV_CP.Size` or `Output_CP.InputSize` would force the node to re-download historical 256-entry bundles over the network—risking upstream HTTP 429 rate limits, stalling startup completely during upstream outages, and violating the sub-500ms Zero-WAL recovery SLO. +- **Chosen Design**: Persist Merkle-authenticated 256-entry bundles on local disk up to `SafeWatermark` (`max(min(KV_CP.Size, MPT_Durable_Size), from)`), enabling offline Zero-WAL crash replay while bounding disk usage via the background tile reaper. + +--- + +## 4. Verification Gate + +All tile fetching, cryptographic tile verification, cache durability, reaper, and WASM pipeline tests are verified via: + +```bash +go test -race ./vindex/internal/ingest/... +``` diff --git a/vindex/internal/kvstore/README.md b/vindex/internal/kvstore/README.md new file mode 100644 index 0000000..7198262 --- /dev/null +++ b/vindex/internal/kvstore/README.md @@ -0,0 +1,179 @@ +# Inverted Chunk Storage Engine + +This document defines the storage architecture, LSM-tree key-value layout, binary serialization schemas, load-bearing invariants, verified performance optimizations, and empirical benchmarks for the **Storage Subsystem** (`vindex/internal/kvstore`). + +--- + +## 1. Context & Objectives + +### 1.1 Problem Statement & Skewed Cardinality Dilemma +The storage engine persists the per-key value logs mapping canonical keys (`1..1024` bytes in `0x21..0x7E`) to chronologically ordered sequences of Input Log entry indices (`entryIdx`), indexing physical Pebble chunks internally by 32-byte `KeyHash = SHA-256(key)` digests and returning modified `(canonicalKey, value_root)` pairs for MPT state commitment. + +Transparency logs exhibit extreme key cardinality skew: +- **Long-tail keys**: The vast majority of domains or packages appear only once or twice in the entire log. +- **Hot keys**: Popular keys (e.g. major cloud provider domains, popular open-source packages) accumulate hundreds of thousands of entries over years. + +In an LSM-tree database (such as Pebble), storing this mapping naively leads to two severe failure modes: +1. **O(N^2) Write Amplification**: Appending to flat lists per key requires rewriting the entire historical list on every new entry. +2. **Scan Latency on Deep Keys**: If chunks are stored in chronological forward order (`chunk_0, chunk_1, ...`), locating the active chunk to append a new entry requires seeking past years of historical SSTables, introducing a 7.5x latency penalty. + +### 1.2 Goals & Non-Goals +- **Goals**: + - Provide single-seek active chunk location for batch appends regardless of key depth or historical occurrence count. + - Eliminate disk I/O on missing keys via full-table 33-byte prefix Bloom filters. + - Maximize storage density by encoding occurrence entry indices as 16-bit relative offsets within 65,536-entry chunks. + - Guarantee zero-overhead point-in-time `value_root` calculation to support non-blocking MPT prediction and Zero-WAL crash recovery. + - Enforce synchronous durability before state commitment handoff to guarantee `KV_CP.Size >= Output_CP.InputSize`. +- **Non-Goals**: + - **No Distributed Storage**: Operates strictly as a single-host embedded database; distributed clustering and remote RPC storage backends are explicitly out of scope. + - **No Cross-Key Range Scans**: Supports point queries by exact 32-byte `KeyHash`; lexical prefix scanning across distinct canonical keys inside Pebble is out of scope (all prefix discovery is served by un-hashed canonical keys in the MPT). + - **No Deletions or Tombstones**: Data is strictly append-only; records are never deleted or overwritten once sealed. + +### 1.3 Requirements & Dependencies +- **Storage Engine**: Embedded [Pebble](https://github.com/cockroachdb/pebble) LSM-tree with a custom 33-byte prefix splitter and 10-bit Bloom filter policy. +- **Merkle Tree Utilities**: RFC 6962 value log hashing and compact range hashes via `internal/wire` (`vindex/internal/wire`) and `github.com/transparency-dev/merkle`. + +--- + +## 2. Detailed Design + +### 2.1 Logical Model & Inverted Chunk Key Layout +To eliminate write amplification and support direct window-based seeks, the Input Log entry index space is partitioned into fixed global windows of 65,536 entries. Each occurrence at Input Log entry index `entryIdx` maps deterministically to: +- **Chunk Number**: `chunkNum = entryIdx / 65536` +- **Relative Offset**: `relOffset = uint16(entryIdx % 65536)` + +While the MPT stores un-hashed variable-length canonical keys (`1..1024` bytes in `0x21..0x7E`), Pebble allocates chunks only for windows in which a canonical key actually occurs, keyed by fixed-width 41-byte records: `'c'` (1 byte) + `KeyHash` (32 bytes, `SHA-256(key)`) + `^BigEndian(chunkNum)` (8 bytes, bitwise inverted so newer chunks sort first): + +```text +Key (41 bytes) = [Prefix 'c' (1B)] + [KeyHash (32B)] + [^BigEndian(chunkNum) (8B)] +``` +where `^BigEndian(chunkNum) = 0xFFFFFFFFFFFFFFFF - chunkNum`. + +#### The Inverted Ordering Mechanism: +Because chunk numbers are bitwise-inverted, the **highest (newest) chunk has the lexicographically smallest key** among all chunks for that key prefix. +- Pebble's custom key comparer splits keys at byte 33 (`'c' + KeyHash`). +- Full-table 10-bit Bloom filters are indexed on this 33-byte prefix across all SSTable levels. +- Performing a prefix-bounded seek on `'c' + KeyHash` achieves two simultaneous benefits: + 1. If the key does not exist, the Bloom filter prunes all SSTables, returning empty with **zero disk I/O**. + 2. If the key exists, the iterator lands **directly on the latest active chunk**, completely skipping all older historical chunks. + +### 2.2 Delimitless Binary Chunk Serialization & Compact Ranges + +Values store a cumulative RFC 6962 compact range covering all occurrences of that canonical key across prior chunks (`chunk < chunkNum`), plus a dense array of 16-bit relative offsets for occurrences within the current 65,536-entry window (`chunkNum`): + +```text +Preceding uint64 (8 bytes, BigEndian) || CompactRange (32 bytes * popcount(Preceding)) || RelativeIndices (uint16, 2 bytes BigEndian * N) +``` + +| Byte Offset | Field Name | Data Type | Description | +| :--- | :--- | :--- | :--- | +| `0` | `Preceding` | `uint64` (8 bytes, BigEndian) | Cumulative number of occurrences (value log leaves) for this key across all prior chunks. | +| `8` | `CompactRange` | `[K][32]byte` | Compact range hashes committing to the `Preceding` older value log leaves of this key, where `K = popcount(Preceding)` (32 bytes * K). | +| `8 + 32*K` | `RelativeIndices` | `[]uint16` (2 bytes BigEndian * N) | Dense 2-byte offsets: `uint16(entryIdx % 65536)` for each occurrence of this key within global 65,536-entry window `chunkNum` (`entryIdx = chunkNum*65536 + uint64(relOffset)`). | + +#### Per-Entry Key Deduplication, Crash-Replay Idempotency & Value Log Leaf Hashing: +Each leaf in a key's RFC 6962 value log commits to a distinct Input Log entry index (`entryIdx`) where the canonical key occurred: +- **Per-Entry Deduplication**: While the WASM host already sorts and deduplicates canonical keys per entry on output slab unpacking, batch indexing also defensively sorts and deduplicates `(canonicalKey, entryIdx)` pairs (hashing `canonicalKey` internally to `KeyHash = SHA-256(key)` for Pebble storage). Each `entryIdx` is appended at most once to a key's value log, ensuring strictly increasing entry indices within every chunk (`relOffset_0 < relOffset_1 < ...`). +- **Crash-Replay Idempotency (`entryIdx < KV_CP.Size`)**: Every batch write atomically records the batch's ending entry watermark alongside the modified `'c'` chunk records in Pebble (with the full `KV_CP` checkpoint note and durable `fsync` committed on the final slab reaching `Target_CP.Size`). During startup crash recovery or batch replay across `[Output_CP.InputSize, KV_CP.Size)`, any `entryIdx` below the persisted KV watermark—or whose `(chunkNum, relOffset)` is `<=` the active chunk's highest persisted offset—is skipped. When all occurrences of a key in a replayed batch are already persisted in Pebble, the storage engine skips chunk mutation and reconstructs the key's point-in-time `value_root` as of the target size so the MPT receives the exact modified `(canonicalKey, value_root)` mapping without duplicating chunk offsets. +- **Value Log Leaf Hashing**: Each absolute Input Log entry index (`entryIdx = chunkNum*65536 + uint64(relOffset)`) is encoded as an 8-byte big-endian integer and hashed with RFC 6962 leaf domain separator `0x00` (`value_log_leaf_hash(entryIdx) = SHA-256(0x00 || uint64_be(entryIdx))`). +- **Interior Nodes & Chunk Rollover**: Value log interior nodes are hashed with RFC 6962 node domain separator `0x01` (`value_log_node_hash(left, right) = SHA-256(0x01 || left || right)`). When a new occurrence for a key falls into a higher global 65,536-entry window, the previous chunk is sealed and a new chunk is initialized with `Preceding` and `CompactRange` snapshotted from the key's running compact range hashes across all prior chunks. + +#### Delimitless Deserialization: +Because the number of compact range hashes is mathematically determined by the population count (number of set bits) of `Preceding` (`K = popcount(Preceding)`), the byte offset where `RelativeIndices` begins is always `8 + 32*K`. This layout eliminates field delimiters, length prefixes, and serialization padding, achieving optimal binary packing density. + +### 2.3 Two-Generational Active Chunk Cache & Running Compact Range +To avoid repeatedly reading active chunk descriptors from Pebble or re-hashing historical offsets within the active 65,536-entry window during sequential batching, the store maintains a bounded in-memory two-generational chunk cache: +- **Cached Descriptor & Running Compact Range**: Each cache entry holds both the active chunk record (`Preceding`, `CompactRange`, `RelativeIndices`) and its cumulative in-memory running compact range across all occurrences up to the latest batch. Incremental batch appends update the running compact range directly in O(1) amortized leaf hashes per new occurrence instead of re-hashing prior `RelativeIndices` from index 0; on cache eviction or node restart, the running compact range is lazily reconstructed once when the active chunk is loaded from Pebble. +- **Current & Previous Generations**: The current generation stores chunk descriptors accessed or modified during the active batch, while the previous generation retains descriptors from the immediately preceding batch. +- **Eviction**: When the current generation reaches capacity (scaling to hundreds of thousands of hot keys), the previous generation is replaced by the current generation, and a fresh current generation map is allocated. This absorbs repeated writes to hot keys without LRU lock contention, eliminating >90% of Pebble block cache read I/O during high-throughput ingestion. + +### 2.4 In-Situ Invariants & Performance Optimizations + +- **[Correctness Invariant] Synchronous Persistence Barrier**: + - *Rule*: Within a target sync cycle, intermediate commit slabs may write to Pebble without an immediate `fsync`, but the final slab reaching `Target_CP.Size` (or an explicit flush barrier) must execute a durable disk sync to flush all modified `'c'` records and metadata (`KV_CP`) to disk before returning to the Coordinator for Output Log publication. + - *Rationale*: Enforces `KV_CP.Size >= Output_CP.InputSize` across all crash and power-loss scenarios while amortizing disk `fsync` overhead across multi-slab sync cycles. + - *Consequence ("Or Else")*: If the node crashed after appending a state commitment entry to the Output Log but before Pebble synced to disk, the published root would reference missing KV chunks. The node would be permanently unable to produce inclusion proofs for its own witnessed checkpoints. + +- **[Correctness Invariant] Per-Entry Key Deduplication**: + - *Rule*: The KV store must deduplicate identical `(canonicalKey, entryIdx)` emissions before appending relative offsets or updating the key's running compact range. + - *Rationale*: Guarantees that a key's value log contains each matching Input Log entry index at most once in strictly ascending order. + - *Consequence ("Or Else")*: Duplicate relative offsets for the same Input Log entry would be appended to chunk records, corrupting the value log compact range hashes and violating pagination cursor monotonicity (`before=`). + +- **[Correctness Invariant] Point-in-Time Value Log Root Isolation**: + - *Rule*: When computing or reconstructing `value_root` digests for target Input Log size `input_size`, the storage engine strictly evaluates records with `entryIdx < input_size`, ignoring any chunks or entry indices written ahead of `input_size`. + - *Rationale*: Allows startup recovery and reader lookups to safely inspect the database even if the KV store has synced batches ahead of the active serving state. + - *Consequence ("Or Else")*: Query responses would leak unwitnessed or in-flight entry indices, violating reader snapshot isolation. + +- **[Performance Optimization] Bitwise Key Inversion (`^chunkNum`) & 33-Byte Bloom Filters**: + - *Mechanism*: Bitwise inversion places the active chunk first under the prefix; the 33-byte prefix Bloom filter prunes non-existent keys. + - *Impact*: Delivers single-seek active chunk lookups, eliminating the 7.5x append latency penalty observed on deep keys in forward-ordered layouts. + +- **[Performance Optimization] 16-Bit Relative Offset Encoding**: + - *Mechanism*: Stores entry index offsets as `uint16(entryIdx % 65536)` within each global 65,536-entry window instead of 8-byte integers. + - *Impact*: Saves 75% disk storage on entry index occurrence lists within each 65,536-entry chunk. + +- **[Performance Optimization] Two-Generational Active Chunk Cache**: + - *Mechanism*: Caches active chunk descriptors across sequential batches. + - *Impact*: Cuts Pebble block read I/O by >90% during sustained high-concurrency ingestion. + +- **[Performance Optimization] Chunk-Aligned Backward Lookup & Zero-Hashing Compact Range Reuse**: + - *Mechanism*: Seeks directly to the highest valid chunk (`(before - 1) / 65536`), aligns page boundaries to whole 65,536-entry chunks until `totalMatched >= targetPageSize` (default `100`) or `chunksVisited >= maxChunksPerRequest` (default `16`), and copies the `c2sp.org/tlog-vindex-values@v1` `preceding` count and compact range hashes directly from the oldest visited chunk's stored `Preceding` and `CompactRange` with zero dynamic SHA-256 hashing on the read path. + - *Impact*: Bounds SSTable chunk reads (`maxChunksPerRequest`) per request while eliminating intra-chunk splitting and dynamic Merkle re-hashing across deep keys. + +- **[Performance Optimization] Key-Partitioned In-Memory Value Log Hashing**: + - *Mechanism*: Partitions distinct keys within each incoming mapped batch across a worker pool. Workers independently compute value log compact range updates, `value_root` digests (`internal/wire`), and serialized chunk records in parallel in memory, while the committer sequentially applies the precomputed writes into an atomic Pebble batch. + - *Impact*: Eliminates single-thread CPU bottlenecks on RFC 6962 value log Merkle hashing during high-throughput ingestion, preventing upstream pipeline backpressure on mapping workers while keeping database writes strictly sequential. + +### 2.5 Storage Engine Contract + +The storage subsystem manages physical chunk persistence, value log compact range hashes, and point lookups: + +- **Physical Format & Key Layout**: + - Chunks are partitioned by global 65,536-entry Input Log windows (`chunkNum = entryIdx / 65536`) and keyed by 41 bytes: `'c'` (1 byte) + `KeyHash` (32 bytes, `SHA-256(key)`) + `^BigEndian(chunkNum)` (8 bytes bitwise inverted). + - Bitwise inversion places the newest active chunk first under the key prefix, enabling single-seek active chunk lookups with 33-byte prefix Bloom filtering. + - Chunk values use a delimitless binary serialization format: an 8-byte big-endian `Preceding` count (total occurrences for that key across prior chunks), followed by RFC 6962 `CompactRange` hashes committing to those `Preceding` prior value log leaves, followed by dense 2-byte relative entry index offsets (`uint16(entryIdx % 65536)`). +- **Write Contract & Rollover**: + - Monotonic batch writes accept `(canonicalKey, entryIdx)` pairs, compute `KeyHash = SHA-256(key)` internally, deduplicate per Input Log entry, accumulate occurrences across multi-tile commit slabs, return the modified `(canonicalKey, value_root)` mappings for MPT commitment, and execute a durable disk sync at the target checkpoint boundary before Output Log commitment. + - When an occurrence for a key crosses into a new global 65,536-entry window, the prior chunk is sealed as immutable historical state and a new active chunk is allocated with the cumulative `Preceding` and `CompactRange` of all prior occurrences. +- **Point-in-Time `value_root` Reconstruction**: + - To reconstruct a modified key's 32-byte `value_root` as of any target `input_size` (used during crash recovery and uncommitted batch staging for keys with `>= 1` occurrences in `[from, input_size)`), the store seeks to `'c' + KeyHash + ^BigEndian((input_size - 1) / 65536)`, skips any chunk where `chunkNum * 65536 >= input_size`, clones that chunk's `Preceding` and `CompactRange`, appends `SHA-256(0x00 || uint64_be(entryIdx))` into the running compact range for each relative offset where `chunkNum * 65536 + relOffset < input_size`, and folds the compact range right-to-left into the 32-byte `value_root` (keys with zero total occurrences in `[from, input_size)`—where `Preceding` plus matching offsets in the chunk equals `0`—are never inserted into the MPT per [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md)). +- **Lookup Contract & Chunk-Aligned Backward Traversal**: + - The HTTP serving layer rejects `before > Serving_CP.InputSize` with `503 Service Unavailable` prior to storage lookup; once `before <= Serving_CP.InputSize` is established, point lookups accept `canonicalKey` (hashed internally to `KeyHash = SHA-256(key)`) and retrieve Input Log entry indices in `[from, before)` (ignoring any uncommitted chunks written ahead of `Serving_CP.InputSize`) using server-governed chunk-aligned page boundaries per [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md) (`targetPageSize` default `100`, `maxChunksPerRequest` default `16`). + - **Direct Seek**: Computes the target window (`(before - 1) / 65536`) and seeks directly to that inverted chunk key under `'c' + KeyHash`, landing on the newest candidate chunk without scanning future or non-matching SSTables. + - **Whole-Chunk Reverse Traversal**: Iterates forward through inverted keys (`^chunkNum`), moving backward chronologically. Any chunk where `chunkNum * 65536 >= before` or where zero offsets fall in `[from, before)` is skipped without incrementing `chunksVisited` or copying `Preceding`/`CompactRange`. Each visited chunk with `>= 1` occurrences in `[from, before)` emits **all** matching offsets in that 65,536-entry window without intra-chunk splitting, halting once `totalMatched >= targetPageSize` or `chunksVisited >= maxChunksPerRequest` (default `16`). Consequently, a single dense chunk may return up to `65,536` entry indices while sparse chunks return fewer than `targetPageSize`, returning empty (`404 Not Found` at the HTTP layer) if zero matching occurrences exist in `[from, before)`. When `preceding > 0` on the returned page, clients paginate backward using the first (smallest) returned entry index (`entry_idx_0`) as the next `before` cursor. + - **Zero-Hashing Compact Range Reuse**: Because page boundaries always align to whole chunks, when `Preceding > 0` on the oldest visited matching chunk, `preceding` and the compact range hashes are copied directly from that chunk's precomputed `Preceding` and `CompactRange` with zero dynamic SHA-256 hashing on the read path. + +--- + +## 3. Alternatives Considered (or Tried) + +### 3.1 Evaluated Storage Layouts & Chunk Sizing +1. **Flat Lists per Key**: Storing an unbounded flat list of 8-byte integers under `KeyHash` caused `O(N^2)` write amplification on hot keys as full lists were rewritten on every batch. +2. **One Record per Occurrence**: Appending a discrete key-value pair per occurrence (`KeyHash + BigEndian(entryIdx)`) inflated SSTable metadata and slowed range scans. +3. **Chunk Size (`65,536` vs. `256` / `1,024`)**: Small windows (`256` or `1,024`) caused frequent chunk rollovers and inflated Pebble's block index; `65,536` caps uncompressed chunk values at ~130 KB and enables 16-bit relative offset encoding. +4. **Inverted vs. Forward Chunk Ordering**: Forward ordering (`chunk_0, chunk_1, ...`) required scanning past historical chunks to locate the active tail, whereas bitwise inversion (`^chunkNum`) places the newest chunk first under `'c' + KeyHash` for single-seek access. + +### 3.2 Distributed NoSQL (Cloud Bigtable, Spanner, Cassandra) vs. Embedded Pebble LSM +- **Proposed**: Backing the inverted chunk store with a distributed NoSQL engine. +- **Theoretical & Architectural Rejection**: + - Remote RPC roundtrips introduce 5–20ms network latency per batch seek, destroying ingestion throughput. + - The authenticated MPT already requires single-host RAM/mmap locality; placing the storage engine on a distributed cluster introduces operational complexity without solving tree state replication. +- **Chosen Design**: Single-host embedded Pebble LSM engine. + +### 3.3 Concurrent Database Iterators vs. Key-Partitioned In-Memory Hashing +- **Proposed (Concurrent Database Iterators)**: Partition keys across worker goroutines where each worker manages an independent database iterator to read, mutate, and write chunks concurrently to disk. +- **Empirical Rejection**: Concurrent Pebble iterators introduced lock contention, allocator overhead, and cache thrashing, yielding only an isolated ~14% speedup on 4 workers while degrading performance beyond 8 workers. Furthermore, concurrent disk I/O competed directly with background LSM compactions. +- **Adopted Design (Key-Partitioned In-Memory Hashing)**: Decouples CPU-bound Merkle calculations from I/O-bound database mutations: + 1. *Parallel In-Memory Hashing*: Distinct keys in each batch are distributed across worker goroutines. Each worker computes compact range appends and serializes chunk entries independently in pure memory with zero shared state. + 2. *Sequential Database Commit*: The precomputed key-value writes are merged into a single atomic Pebble batch and committed sequentially. +- **Impact**: Delivers linear multi-core scaling for CPU-intensive value log hashing on high-throughput workloads, eliminating pipeline stalls while preserving zero-contention, deterministic single-threaded database durability. + +--- + +## 4. Verification Gate + +All inverted chunk storage, chunk-aligned paging, point-in-time root reconstruction, and batch staging tests are verified via: + +```bash +go test -race ./vindex/internal/kvstore/... +``` diff --git a/vindex/internal/server/README.md b/vindex/internal/server/README.md new file mode 100644 index 0000000..d8be974 --- /dev/null +++ b/vindex/internal/server/README.md @@ -0,0 +1,98 @@ +# Read Serving & C2SP Query Protocol + +This document defines the HTTP serving architecture, endpoint admission rules, reader snapshot isolation, load-bearing invariants, and verified performance optimizations for the **Serving Subsystem** (`vindex/internal/server`). For the full `tlog-vindex` wire format specifications (`c2sp.org/tlog-vindex-keys@v1`, `c2sp.org/tlog-vindex-values@v1`, `c2sp.org/tlog-vindex-state@v1`) and client-side cryptographic verification rules, see [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md) and [`client/README.md`](../../client/README.md). + +--- + +## 1. Context & Objectives + +### 1.1 Problem Statement & Verifiable Pagination Dilemma +The read serving engine exposes HTTP endpoints for point and prefix lookups over variable-length canonical keys (`1..1024` printable ASCII bytes in `0x21..0x7E`). In a verifiable index, serving search results is fundamentally different from a traditional database: +1. **Omission Resistance**: Lookup clients cannot trust the index operator. Every query response must deliver cryptographic inclusion, non-inclusion, or two-sided prefix proofs, proving that no matching Input Log entries were omitted. +2. **Cardinality Skew & Pagination**: When a canonical key accumulates tens of thousands of occurrences (e.g. popular domains in Certificate Transparency), returning all Input Log entry indices in a single HTTP response causes memory exhaustion, serialization stalls, and denial-of-service amplification. +3. **Backward Pagination over Compact Ranges**: If the server paginated forward in time (`start=X`), proving that an intermediate page did not omit records would require computing Merkle consistency proofs across arbitrary Value Log prefixes or scanning from entry `0`. By serving Value Log pages backward from `input_size` (`before=`) aligned to whole 65,536-entry Pebble chunks, each page embeds the oldest visited chunk's precomputed `preceding` count and compact range hashes (`c2sp.org/tlog-vindex-values@v1`) with zero dynamic SHA-256 hashing on the read path. + +### 1.2 Goals & Non-Goals +- **Goals**: + - Implement the two-endpoint C2SP `tlog-vindex` HTTP read protocol separating key/prefix proof generation (`GET /keys?key=` or `GET /keys?prefix=`) from lock-free Pebble Value Log pagination (`GET /values?key=&before=`). + - Return verifiable point inclusion proofs, non-inclusion proofs, and two-sided prefix proofs (`c2sp.org/tlog-vindex-keys@v1`) paired with the latest Output Log inclusion proof (`c2sp.org/tlog-proof@v1`). + - Remain completely application-agnostic while enforcing an optional caller-supplied prefix policy callback (`400 Bad Request`, e.g. CT/MTC eTLD+1 policies configured by the binary) alongside a configurable maximum prefix match limit (`413 Content Too Large`, default `10,000`). + - Enforce strict reader snapshot isolation: lookup clients observe state strictly bounded by witnessed Output Log checkpoints (`Serving_CP.InputSize`). + - Stream responses using direct plaintext formatting (`internal/wire`) without JSON reflection or intermediate heap allocations. +- **Non-Goals**: + - **No Unbounded Full-Tree Scans**: Prefix scans are guarded by the optional prefix policy callback and maximum prefix match limit. + - **No Uncommitted / Read-Uncommitted Mode**: Lookups never observe unwitnessed or in-flight Input Log entries ahead of `Serving_CP.InputSize`. + - **No In-Server Client Verification**: The server generates proofs; cryptographic verification is executed exclusively by the lookup client ([`client/`](../../client/README.md)). + +### 1.3 Requirements, Dependencies & Constraints +- **Protocol Standards**: Implements [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md) and [C2SP `tlog-tiles`](https://c2sp.org/tlog-tiles@v1.0.0). +- **Dependencies**: Reads from the Pebble inverted chunk store ([`internal/kvstore`](../kvstore/README.md)), the in-memory MPT ([`internal/tree`](../tree/README.md)), and the active serving snapshot (`Serving_CP`). +- **Sequential Roundtrips for Deep Keys**: Because each `/values` continuation cursor (`before = Indices[0]`) is derived from the oldest entry index returned on the preceding page, callers fetching the full history of a key spanning multiple 65,536-entry chunks must issue `/values` requests sequentially rather than in parallel. + +--- + +## 2. Detailed Design + +### 2.1 HTTP Endpoints +The public read server exposes the spec-defined C2SP `tlog-vindex` lookup endpoints under `` (optionally mounted under a configured URL path prefix without `3xx` redirects) alongside `tlog-tiles` Output Log endpoints when the Output Log directory is co-hosted on the same mux, while operational probes and telemetry are isolated onto an internal administrative/metrics listener: + +#### A. Public C2SP `tlog-vindex` & Co-Hosted `tlog-tiles` Endpoints + +| Endpoint | Method | Purpose, Status Codes & Caching | +| :--- | :--- | :--- | +| `/keys` | `GET` | Exact key or key prefix lookup over canonical keys (`1..1024` printable ASCII bytes in `0x21..0x7E`).
Query params: either `key=` (exact lookup) or `prefix=` (prefix lookup), mutually exclusive with duplicate or unknown params rejected, parsed via strict RFC 3986 percent-decoding (never converting literal `+` to space `0x20`).
On `prefix=`, validates `` against the optional prefix policy callback and scans up to `maxPrefixKeys + 1` matching `(key, value_root)` pairs **before** writing `200 OK` headers (returning `413 Content Too Large` if a `(maxPrefixKeys + 1)`-th match exists), then writes `200 OK` (`text/plain; charset=utf-8`, `Cache-Control: no-store`) with `c2sp.org/tlog-vindex-keys@v1` followed by `c2sp.org/tlog-proof@v1` (see [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md)).
Returns `400 Bad Request` if neither/both/duplicate/unknown params are present, if percent-decoding fails or the unescaped key/prefix is not a valid canonical key, or if the configured prefix policy rejects ``; `413 Content Too Large` if matched keys exceed the configured prefix limit; or `503 Service Unavailable` (with `Retry-After: 30`) if the server has not yet published its first Output Log checkpoint. | +| `/values` | `GET` | Lock-free Value Log pagination for `key`.
Query params: `key` (**required**, parsed via strict RFC 3986 percent-decoding without `+`-to-space conversion) and `before` (**required** canonical positive ASCII decimal integer `1 <= before < 2^64` with no leading zeros or `+`). Page boundaries are server-governed via whole 65,536-entry chunk alignment (`targetPageSize` default `100`, `maxChunksPerRequest` default `16`), returning at least `1` entry index whenever occurrences in `[from, before)` exist.
Acquires **zero MPT locks**; returns `200 OK` (`text/plain; charset=utf-8`, `Cache-Control: public, max-age=31536000, immutable`) with `c2sp.org/tlog-vindex-values@v1` (see [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md)).
Returns `400 Bad Request` on missing/unknown/duplicate/invalid params, non-canonical or non-positive `before`, or malformed `%`-escapes; `503 Service Unavailable` (with `Retry-After: 30`) if uninitialized or `before > Serving_CP.InputSize`; or `404 Not Found` once `before <= Serving_CP.InputSize` and zero occurrences exist in `[from, before)`. | +| `/checkpoint` | `GET` | Co-hosted `tlog-tiles` endpoint returning the latest signed, witnessed Output Log checkpoint note from `Serving_CP` (`503 Service Unavailable` with `Retry-After: 30` if uninitialized). | +| `/tile/...` | `GET` | Co-hosted `tlog-tiles` internal hash tiles and `/tile/entries/...` Output Log entry bundles served from the local Tessera Output Log directory for auditors. | + +The server implements only the spec-defined protocol paths above; other paths under `` (such as `/` and `/index.html` for an optional embedded single-page HTML explorer UI) are left open and are not part of the protocol. All C2SP `tlog-vindex` endpoints never serve `3xx` redirect responses. + +#### B. Non-Public Operational Endpoints (Admin / Metrics Listener Only) + +Operational liveness, readiness, Prometheus telemetry, and Go runtime profiling handlers are registered on a separate administrative mux and are never exposed on the public listener: + +| Endpoint | Method | Purpose | +| :--- | :--- | :--- | +| `/healthz` | `GET` | Liveness probe. Returns **HTTP 200 OK** (`ok\n`) as long as the process and optional health callback report healthy. | +| `/readyz` | `GET` | Readiness probe. Returns **HTTP 200 OK** (`ok\n`) when `Serving_CP` is initialized and ready to serve, or **HTTP 503** (with `Retry-After: 30` and JSON status diagnostics) during MPT initialization, startup recovery, or Genesis Backfill. | +| `/metrics` | `GET` | Exposes Prometheus telemetry metrics. | +| `/debug/pprof/*` | `GET` | Go runtime profiling endpoints. | + +### 2.2 Reader Snapshot Isolation & Concurrency +To ensure lookup clients never observe uncommitted or unwitnessed records, the serving engine enforces strict watermark filtering (`Serving_CP.InputSize <= Output_CP.InputSize`): + +1. **Brief Shared Read Lock on `/keys`**: Every `/keys` query acquires a shared read lock on the in-memory MPT, reads the atomic `Serving_CP` snapshot under that lock, generates the binary MPT point or prefix proof (`0` bytes when `mpt_root == empty_mpt_hash`, single-leaf inclusion, or full-tree prefix match), and releases the read lock before formatting and writing the response. +2. **Zero-Lock Pebble Reads on `/values`**: Every `/values` query loads the atomic `Serving_CP` pointer without acquiring any MPT lock, rejects `before > Serving_CP.InputSize` with `503 Service Unavailable` (`Retry-After: 30`), hashes `` to `SHA-256(key)`, and performs a chunk-aligned backward lookup in Pebble restricted to `[from, before)`. +3. **Decoupled from Background Ingestion**: Because Output Log appends, witness network roundtrips, and MPT disk `fsync` calls execute outside the in-memory MPT lock (see [`internal/tree`](../tree/README.md)), concurrent `/keys` and `/values` requests never block on ingestion I/O. + +### 2.3 In-Situ Invariants & Performance Optimizations + +- **[Correctness Invariant] Serving Snapshot Isolation**: + - *Rule*: Query responses evaluate state strictly bounded by the latest witnessed Output Log commitment (`Serving_CP.InputSize <= Output_CP.InputSize`). + - *Rationale*: Guarantees that lookup clients only observe state committed in the Output Log and cosigned by witnesses. + - *Consequence ("Or Else")*: Lookup clients would observe uncommitted or unwitnessed entries, causing client-side `/keys` or `/values` proof verification to fail against witnessed checkpoints. + +- **[Correctness Invariant] Pre-Header Prefix Admission**: + - *Rule*: On `GET /keys?prefix=`, the server validates `` against the configured prefix policy and collects at most `maxPrefixKeys + 1` matching keys from the MPT **before** writing HTTP response headers, returning `400 Bad Request` or `413 Content Too Large` cleanly if rejected. + - *Rationale*: Prevents the server from committing a `200 OK` status header and then truncating a response stream mid-body if a broad prefix exceeds `maxPrefixKeys`. + - *Consequence ("Or Else")*: Mid-stream truncation after `200 OK` would deliver an incomplete key set that fails two-sided prefix proof verification on the client. + +- **[Performance Optimization] Zero-Reflection Plaintext Formatting**: + - *Mechanism*: Formats C2SP plaintext responses (`c2sp.org/tlog-vindex-keys@v1`, `c2sp.org/tlog-proof@v1`, and `c2sp.org/tlog-vindex-values@v1`) directly into contiguous byte buffers without JSON reflection. + - *Impact*: Delivers sub-millisecond P50 read latency (< 1 ms) and sustains high lookup throughput under concurrent ingestion. + +### 2.4 Serving Contract + +- **Non-Blocking Read Path**: `/keys` acquires only a brief shared read lock on the in-memory MPT (`Cache-Control: no-store`), while `/values` acquires zero MPT locks and reads immutable/append-only Pebble inverted chunks directly (`Cache-Control: public, max-age=31536000, immutable`). +- **Application-Agnostic Prefix Guardrails**: Contains no hardcoded domain or ecosystem rules; callers may supply an optional prefix policy callback (`400 Bad Request` on policy rejection) alongside a configurable maximum prefix key limit (`413 Content Too Large` when exceeded, default `10,000`). +- **Strict RFC 3986 Query Validation & Chunk-Aligned Paging**: Rejects malformed `%`-escapes, unknown parameters, or duplicate query parameters with `400 Bad Request`, never decodes literal `+` into space `0x20`, validates canonical key syntax (`1..1024` bytes in `0x21..0x7E`), and validates `before` as a canonical positive ASCII decimal integer (`1 <= before < 2^64`) without leading zeros or `+`. On `/values`, page boundaries align to whole 65,536-entry Pebble chunks (`targetPageSize` default `100`, `maxChunksPerRequest` default `16`). + +--- + +## 3. Verification Gate + +All server endpoint, prefix policy, chunk-aligned paging, and C2SP wire response tests are verified via: + +```bash +go test -race ./vindex/internal/server/... +``` diff --git a/vindex/internal/tree/README.md b/vindex/internal/tree/README.md new file mode 100644 index 0000000..21164cc --- /dev/null +++ b/vindex/internal/tree/README.md @@ -0,0 +1,167 @@ +# Authenticated State Commitment (MPT & Output Log) + +This document defines the cryptographic commitment engine, tree synchronization protocols, concurrency model, load-bearing invariants, and verified performance optimizations for the **MPT Subsystem** (`vindex/internal/mpt`) and **Output Log Commitment Subsystem** (`vindex/internal/tree`). + +--- + +## 1. Context & Objectives + +### 1.1 Problem Statement, Package Decoupling & Concurrency Dilemma +The commitment plane is split into two focused packages: +- **`internal/mpt`** (`vindex/internal/mpt`): Owns the `filippo.io/torchwood/mpt` state trie, canonical key validation, and binary MPT proof codecs—point inclusion/non-inclusion and two-sided prefix subtree proofs—with zero dependencies on `pebble` or `tessera` so it can be imported directly by [`client`](../../client/README.md) and server subsystems. On top of `torchwood/mpt`, `internal/mpt` enforces the strict `tlog-vindex` proof invariants: split-bit bounds `0 <= b <= 8192` on all path steps, canonical key (`1..1024` bytes in `0x21..0x7E`) and 32-byte value validation on terminal leaves in non-inclusion/empty-prefix proofs, and two-sided prefix verification (prefix match + strictly ascending key order + subtree root check). +- **`internal/tree`** (`vindex/internal/tree`): Owns Output Log publication, Tessera POSIX Output Log storage, and atomic `Serving_CP` promotion via the 3-step commitment lifecycle. + +The tree subsystem binds the search index into an authenticated 32-byte cryptographic root (`mpt_root`), anchors it in an append-only Output Log, and serves cryptographic inclusion, non-inclusion, and prefix proofs to HTTP readers. + +Because readers must verify proofs against a coherent snapshot, all mutations to the in-memory MPT require taking an exclusive tree lock. This creates a severe operational dilemma: +1. **The Reader Starvation Trap**: A full commit requires computing tree updates, persisting MPT files to disk (5–20ms), appending a state commitment entry to the Output Log, and awaiting cosignatures from external witnesses over the network (50–200ms). If the writer lock is held across this entire sequence, concurrent HTTP readers are starved. Under heavy traffic, query availability collapses and lookups time out. +2. **The Equivocation Trap**: If the system writes a state commitment entry to the append-only Output Log and *subsequently* encounters an in-memory trie mutation error or calculation mismatch, the node has permanently committed an unprovable root to public witnesses, destroying operator trust. + +### 1.2 Goals & Non-Goals +- **Goals**: + - Deliver cryptographic inclusion, non-inclusion, and two-sided prefix proofs with compact path sizes (`uvarint(b) || sib_hash` steps over split bits `0 <= b <= 8192`, bounded by at most `min(N - 1, 8193)` steps for `N` active keys). + - Minimize reader lock contention during continuous ingestion, keeping write lock duration under 5 milliseconds. + - Eliminate split-view equivocation by binding `mpt_root` commitments to `(input-origin, from, input-size, input-root)` inside witnessed Output Log entries (while retaining signed Input Log checkpoints out-of-band). + - Maintain an immutable on-disk MPT log for Zero-WAL crash recovery. +- **Non-Goals**: + - **No Dynamic Rollbacks**: The MPT is strictly append-only; state transitions progress monotonically forward. + - **No In-Memory Tree Duplication**: Does not duplicate full in-memory trie trees across buffers; memory constraints require mutating a single in-memory trie under short-lived locks. + - **No Multi-Master Consensus**: Tree sequencing is driven by a single local Index Operator process. + +### 1.3 Requirements, Dependencies & Constraints +- **Dependencies**: MPT via `filippo.io/torchwood/mpt`, checkpoint policy verification (`internal/wire`), Tessera POSIX Output Log writer, and external witness clients. +- **RAM Sizing for Branch Locality**: Variable-length canonical keys scatter updates across the trie keyspace. `torchwood/mpt` holds inner trie nodes (`52 bytes/node`) in anonymous virtual memory backed by alternating checksummed patch-frame journal files (`mpt.tree1`, `mpt.tree2`) and an append-only leaf value file (`mpt.disk`), scaling from ~1–1.5 GB for 10M keys to 100–150 GB for 1B keys. +- **Cold-Start Journal Replay**: On process restart, opening the MPT replays uncompacted `torchwood/mpt` journal frames on disk before coordinator recovery begins (while the HTTP server returns `503 Service Unavailable` until replay and startup recovery complete). + +--- + +## 2. Detailed Design + +### 2.1 Variable-Length MPT (`internal/mpt`) +The commitment plane maps un-hashed variable-length canonical keys (`1..1024` printable ASCII bytes in `0x21..0x7E`) to 32-byte value log roots (`value_root`): + +1. **Variable-Length Radix Structure & Proof Wrapper**: `filippo.io/torchwood/mpt` stores variable-length canonical keys (up to 1024 bytes) in bitwise big-endian paths. Keys sharing a common byte prefix share a single subtrie, enabling both point proofs and two-sided prefix subtree proofs, with `internal/mpt` enforcing `0 <= b <= 8192` split-bit bounds, canonical `path_key` / 32-byte `path_val` checks on terminal no-match leaves, and two-sided prefix completeness/soundness. +2. **Journaled Anonymous Memory & Leaf Storage**: `torchwood/mpt` stores inner trie nodes (`52 bytes/node`) in anonymous memory backed by two alternating checksummed patch-frame journal files (`mpt.tree1`, `mpt.tree2`) and stores variable-length MPT leaf nodes (`mpt_leaf_hash`) in an append-only leaf file (`mpt.disk`), wrapped with deferred `fsync` file descriptors so slow kernel `fsync(2)` calls never block in-memory trie reads. +3. **Value Log Roots (`value_root`)**: The 32-byte value committed at each MPT leaf node is the RFC 6962 value log root computed by the KV storage layer (`internal/wire`), authenticating all Input Log entry indices at which that canonical key occurred. + +### 2.2 The 3-Step Atomic Commitment Sequence +The Output Log publisher coordinates the progression from KV storage to public commitment through a structured 3-step sequence designed to satisfy two strict system constraints: + +1. **System Constraint 1 (Zero Unwitnessed State)**: Readers must never observe or obtain proofs against an in-memory root that has not yet been committed to the append-only Output Log and signed by a threshold of external witnesses. Mutating the live trie before publishing would leak unwitnessed state to readers. +2. **System Constraint 2 (Sub-Millisecond Read Latency)**: Readers must never block on external network I/O. Because appending to the Output Log and collecting witness cosignatures takes 50ms–200ms+, holding an exclusive tree write lock across network calls would stall concurrent client queries and violate read latency SLOs (< 15ms P99). +3. **Solution (Separate Prediction from Mutation)**: Pre-compute the deterministic candidate root under a shared read lock without mutating the live trie, release the read lock, and then append the state commitment entry to the Output Log and gather witness cosignatures while readers query unblocked. Once proofs and signatures are secured, acquire the exclusive tree lock briefly (< 5ms) to commit in-memory mutations, assert that the committed root equals the predicted root, and atomically ratchet the `Serving_CP` snapshot. +4. **Why Halt on Divergence**: Candidate root prediction and in-place mutation are deterministic functions over identical inputs. Any divergence indicates an internal invariant violation or memory corruption in the trie. Halting immediately prevents the node from serving proofs that contradict the witnessed Output Log. + +#### Batch Publication Locking & Concurrency: +Batch commits are serialized by the single-writer coordinator. Within each commit cycle, lock scope is minimized so readers never block on network or disk I/O: + +| Step | Phase | Operations | Lock State | Reader Impact | +| :--- | :--- | :--- | :--- | :--- | +| **1** | **Non-Blocking Prediction & Witnessing** | 1. Compute candidate `mpt_root` over modified `(key, value_root)` mappings.
2. Append state commitment entry to Output Log.
3. Obtain witness cosignatures satisfying the Output Log checkpoint policy (e.g., C2SP `tlog-policy`). | Shared read lock held only during in-memory prediction; **no lock held** during Output Log append and witness network calls. | **Zero blocking**; shared read locks allow concurrent HTTP proof generation to run in parallel. | +| **2** | **Atomic Mutation & Ratchet** | 1. Acquire exclusive persistence and tree locks.
2. Commit batch mutations and flush the in-memory journal patch buffer into the OS page cache (via deferred-`fsync` file wrappers).
3. Assert committed root equals predicted `mpt_root`.
4. Ratchet active `Serving_CP` pointer.
5. Release exclusive tree lock. | Exclusive persistence & tree locks held. | **< 5ms** critical section duration. | +| **3** | **Background Durability** | 1. Execute kernel `fsync(2)` on dirty journal/leaf files (`mpt.tree1`, `mpt.tree2`, `mpt.disk`).
2. Advance durable trie watermark (`MPT_Durable_Size`). | Exclusive persistence lock held; exclusive tree lock released. | **Zero blocking**; disk `fsync(2)` runs outside the exclusive tree lock without racing in-memory trie state. | + +#### Step Details: +1. **Non-Blocking Prediction & Witnessing**: + - The publisher acquires a shared read lock on the trie to compute candidate hashes over cloned branch nodes without mutating the live tree, then immediately releases the read lock. + - The canonical 6-line `c2sp.org/tlog-vindex-state@v1` state commitment entry is appended to the Output Log with zero tree or persistence locks held (while retaining the corresponding signed Input Log checkpoint durably out-of-band): + ```text + c2sp.org/tlog-vindex-state@v1 + input-origin + from + input-size + input-root + mpt-root + ``` + - The resulting Output Log checkpoint is submitted to external witnesses to gather cosignatures satisfying the configured Output Log checkpoint policy quorum (e.g., C2SP `tlog-policy`). + - **Throughout this entire phase, concurrent HTTP readers acquire shared read locks and generate proofs without blocking.** +2. **Atomic Mutation & Ratchet**: + - The publisher acquires the exclusive persistence lock and exclusive tree lock. + - Mutates in-memory MPT nodes and flushes `torchwood/mpt`'s in-memory header/journal patch buffer into the OS page cache (against deferred-`fsync` file wrappers) while holding the exclusive tree lock so header and patch-frame compaction state never race concurrent proof or scan readers. + - Asserts that the committed root equals the predicted root. If mismatched, the process terminates immediately. + - Ratchets the active `Serving_CP` pointer to the newly witnessed Output Log checkpoint and inclusion proof. + - Releases the exclusive tree lock (keeping the persistence lock for Step 3). + - **Total critical section duration: < 5 milliseconds.** +3. **Background Durability**: + - Still under the independent persistence lock—with zero tree lock held—`internal/mpt` executes blocking kernel `fsync(2)` calls on the dirty underlying file descriptors (`mpt.tree1`, `mpt.tree2`, `mpt.disk`). + - Once durable on disk, `MPT_Durable_Size` advances atomically, allowing the tile cache reaper to prune superseded 256-entry bundles. + +### 2.3 Concurrency Architecture & Split-Locking +The subsystem enforces strict isolation between in-memory proof serving and background disk persistence using two distinct locks: + +| Conceptual Lock | Synchronization Model | Scope & Protected State | Typical Duration | +| :--- | :--- | :--- | :--- | +| **In-Memory Tree Lock** | Read-write lock | Protects in-memory MPT nodes and the active `Serving_CP` snapshot. Acquired in shared read mode by HTTP proof readers and candidate root prediction; acquired in exclusive write mode only during the in-memory batch commit and `Serving_CP` ratchet. | **< 5ms** (exclusive write)
**< 50µs** (shared read) | +| **Disk Persistence Lock** | Exclusive mutex | Serializes in-memory trie mutations against background snapshot serialization and disk `fsync` operations. Never held during Output Log appends or witness network calls. | **5ms – 20ms** | + +Because disk persistence and network witnessing are decoupled from the exclusive tree lock, slow disk syncs and witness roundtrips never block HTTP readers. + +### 2.4 In-Situ Invariants & Performance Optimizations + +- **[Correctness Invariant] Fatal Halt on Root Prediction Divergence**: + - *Rule*: After the Output Log append succeeds, when the exclusive tree lock is acquired to apply the batch mutations in memory, the resulting trie root must equal the predicted root committed to the Output Log; any mismatch must immediately terminate the process. + - *Rationale*: The Output Log is immutable. If the committed in-memory root diverges from the predicted root that was already appended and witnessed, execution must halt immediately. + - *Consequence ("Or Else")*: Continuing execution would publish an equivocal state commitment entry to the Output Log that cannot be cryptographically proven by the node's internal state. + +- **[Correctness Invariant] Single-Timeline Equivocation Resistance**: + - *Rule*: Every published state commitment entry binds `input-origin`, `from`, `input-size`, `input-root`, and the exact `mpt_root` (`mpt-root`), while the operator retains the signed Input Log checkpoint out-of-band. + - *Rationale*: Independent witnesses cosign the Output Log checkpoint. The index operator cannot present conflicting views of index state to different clients without producing split-view signatures detectable by public monitors, and retaining the signed Input Log checkpoint proves upstream Input Log equivocation if a root conflict ever arises. + - *Consequence ("Or Else")*: An operator attempting to selectively hide certificates or packages would be mathematically detected by any monitor cross-checking witness cosignatures. + +- **[Correctness Invariant] Serving Isolation & Atomic Ratcheting**: + - *Rule*: HTTP readers evaluate state strictly bounded by the active `Serving_CP` snapshot (`Serving_CP.InputSize <= Output_CP.InputSize`). + - *Rationale*: HTTP requests capture an atomic `Serving_CP` snapshot and filter all storage reads to Input Log entry indices strictly below the serving checkpoint's `input_size` (`InputSize`). + - *Consequence ("Or Else")*: Readers would observe uncommitted, unwitnessed Input Log entries ahead of the latest published Output Log entry, breaking client-side Merkle proof verification. + +- **[Performance Optimization] Non-Blocking MPT Prediction**: + - *Mechanism*: Computes candidate trie roots under a shared read lock and performs Output Log writes and witness network roundtrips with no tree locks held. + - *Impact*: Reduces exclusive writer lock duration from ~250ms to < 5ms per commit cycle. + +- **[Performance Optimization] Split-Locking Concurrency Engine**: + - *Mechanism*: Isolates background disk `fsync` operations from in-memory trie read locks. + - *Impact*: Sustains 678,000 read queries/sec during active disk commits, compared to 53,000 reads/sec under coarse global locking (12.8x throughput improvement). + +### 2.5 State Trie & Commitment Contract + +The state commitment subsystem provides authenticated indexing over canonical keys and coordinates public Output Log commitments: + +- **Inputs**: + - Batch of modified `(key, value_root)` mappings over un-hashed canonical keys (`1..1024` bytes in `0x21..0x7E`). + - Configured `` and start index `from` (`0 <= from <= input_size`). + - Verified Input Log checkpoint (`input_size` and `input_root`, validated against the Input Log checkpoint policy and retained out-of-band). +- **Outputs**: + - Witnessed Output Log state commitment entry (`c2sp.org/tlog-vindex-state@v1`). + - Verified Output Log Merkle inclusion proof (`c2sp.org/tlog-proof@v1` at `index == output_size - 1`). + - Atomically promoted `Serving_CP` snapshot (exposing `InputSize` and `OutputSize`). +- **Concurrency Contract**: + - Readers are wait-free and never block on disk `fsync` or network calls. + - State commits are strictly serialized (single-writer); state ratcheting promotes `Serving_CP` atomically. + +--- + +## 3. Alternatives Considered (or Tried) + +### 3.1 Coarse Global Locking vs. Split-Locking with Prediction +- **Proposed**: Holding a single global lock across the entire commitment cycle (prediction, Output Log append, witness network calls, in-memory trie mutation, and disk `fsync`). +- **Empirical Rejection**: Benchmark profiling under heavy concurrent read traffic showed that coarse locking dropped HTTP read throughput to **53,000 queries/sec**, with P99 read latency ballooning past 250ms due to writer lock starvation. +- **Chosen Design**: Split-locking with lock-free prediction holds the exclusive tree lock for < 5ms, sustaining **678,000 queries/sec** (a 12.8x speedup) with sub-millisecond P50 read latency. + +### 3.2 Sparse Merkle Trees (SMT) vs. Merkle Patricia Trie (MPT) +- **Proposed**: Using a standard Sparse Merkle Tree (SMT) with fixed 256-level depth. +- **Theoretical & Empirical Rejection**: Standard SMTs require generating and traversing 256 levels of nodes per operation. Even with default empty node optimizations, SMTs incur massive memory footprint and excessive disk I/O during random-insert batch workloads. +- **Chosen Design**: Path-compressed binary MPT (`filippo.io/torchwood/mpt`), which collapses single-child paths over variable-length canonical keys, bounding tree depth to the active key set and enabling fast `mmap` persistence. + +### 3.3 Post-Commit Output Log Publishing +- **Proposed**: Mutating and syncing the in-memory MPT to disk *first*, and then appending the state commitment entry to the Output Log. +- **Theoretical Rejection**: If the node crashes or power fails after the MPT disk sync but before the Output Log append completes, the local MPT has advanced to a state that was never committed or witnessed in the Output Log. On restart, the node would serve state ahead of the Output Log, violating the universal invariant `MPT_Durable_Size <= Output_CP.InputSize`. +- **Chosen Design**: Predict root lock-free -> append state commitment entry to Output Log -> commit to MPT under short lock. + +--- + +## 4. Verification Gate + +All MPT variable-length canonical key, prefix proof, and Output Log publisher tests are verified via: + +```bash +go test -race ./vindex/internal/mpt/... ./vindex/internal/tree/... +``` diff --git a/vindex/mapfn/README.md b/vindex/mapfn/README.md new file mode 100644 index 0000000..1503ce1 --- /dev/null +++ b/vindex/mapfn/README.md @@ -0,0 +1,175 @@ +# WASM Guest SDK & Plugin Interface + +This document defines the WebAssembly guest ABI, developer contract, memory management protocols, load-bearing invariants, and verified performance optimizations across VIndex's mapping packages: +- **`vindex/mapfn/sdk`** *(public Go package)*: External zero-allocation Go guest SDK imported by WASM plugin authors. +- **`vindex/internal/wasmhost`** *(internal Go package)*: Internal WASI-free Wazero sandbox host used by `vindexd`, `vindex-auditor`, and `vindex-wasm`. + +--- + +## 1. Context & Objectives + +### 1.1 Architectural Separation: User Callback vs. Guest SDK vs. Host Runtime +Mapping in VIndex operates across three decoupled layers: +1. **User Logic (`MapFn`)**: The pure mapping logic (`func(entry []byte, emit func(key []byte))` in Go, or any language targeting the slab ABI) written by ecosystem developers. In Go, user logic has no knowledge of WebAssembly, linear memory pointers, or host runtimes. +2. **Public Guest SDK (`vindex/mapfn/sdk`, Go) / Language-Agnostic Slab ABI**: `vindex/mapfn/sdk` is the public Go guest harness compiled into `.wasm` alongside Go `MapFn` callbacks. Because the underlying `input_buffer` / `map_bundle` slab ABI (§2.2) uses flat little-endian linear memory, guest plugins can equally be written in Rust (`wasm32-unknown-unknown` / `wasm32-wasip1`), C/C++, Zig, or TinyGo without `mapfn/sdk`. +3. **Internal Host Sandbox Runtime (`vindex/internal/wasmhost` & `vindex/internal/ingest`)**: `internal/wasmhost` provides the internal, WASI-free Wazero sandbox pool that loads the compiled `.wasm` plugin, executes `map_bundle` over 256-entry bundles, strictly validates every emitted canonical key (`1..1024` bytes in `0x21..0x7E`, returning a fatal error that immediately `HALT`s the daemon on violation), sorts and deduplicates canonical keys per entry (`entry_idx`) immediately upon unpacking the output slab, and passes the deduplicated un-hashed canonical keys downstream. + +### 1.2 Problem Statement & Real-World Friction +Every transparency ecosystem indexes different payloads (e.g. DER X.509 certificates in Certificate Transparency, newline-delimited strings in Go SumDB, or JSON provenance records in Sigstore). VIndex must allow ecosystem authors to define custom extraction logic without compromising node security, crashing the host process, or causing state divergence across independent auditors. + +Relying on in-process native Go plugins or shared libraries introduces three severe failure modes: +1. **Toolchain Version Drift & Non-Determinism**: If an Index Operator and an Auditor compile their code with different Go versions (e.g. Go 1.23 vs. Go 1.25) or execute on different operating systems, standard library behavior can diverge. A subtle parsing change in ASN.1 DER certificate decoding or Unicode normalization across compiler versions on even **one entry out of 100 million** causes the computed `mpt_root` to diverge permanently, triggering false split-view alarms and breaking consensus. +2. **Host Process Fragility**: Untrusted or malformed entries in public logs could cause out-of-bounds memory accesses, nil pointer panics, or unbounded recursion. Running native code directly in the host engine risks corrupting host memory. +3. **Foreign Function Interface (FFI) Overhead**: Naively executing foreign functions across language boundaries on a per-entry basis introduces excessive context-switching and memory-copying overhead, consuming ~23% of total host CPU time purely in FFI crossings. + +### 1.3 Goals & Non-Goals +- **Goals**: + - Provide a high-level, idiomatic public Go developer contract (`func(entry []byte, emit func(key []byte))`) allowing ecosystem authors to write pure functions without managing raw WebAssembly linear memory pointers. + - Provide a zero-allocation public guest runtime SDK (`vindex/mapfn/sdk`) that binds the developer's function to the high-performance `map_bundle` ABI. + - Freeze compiler and standard library bytecode inside a hermetic `.wasm` binary, guaranteeing bit-for-bit identical execution across all host architectures, operating systems, and toolchain versions. + - Amortize boundary-crossing overhead by processing full 256-entry bundles in a single FFI call using contiguous shared-memory slabs. + - Provide an offline CLI (`vindex-wasm`) to inspect ABI exports and `.wasm` artifact SHA-256 digests, test canonical key extraction, and benchmark throughput prior to deployment. +- **Non-Goals**: + - **No Ingestion Concurrency in `mapfn/sdk`**: `vindex/mapfn/sdk` contains strictly in-guest ABI harnesses, while `vindex/internal/wasmhost` provides the internal zero-I/O Wazero module runner; multi-worker pool orchestration lives in the Ingest subsystem (`internal/ingest`). + - **No In-Place Dynamic Upgrades**: Once a compiled plugin binary is deployed for an Output Log, mapping logic is immutable for the lifetime of that Output Log. Fixing a parsing bug requires deploying a new index instance from its configured `from` start index (`0` or a suffix start index). + - **No Host I/O or State**: Guest modules are strictly pure functions; they have zero access to disk, network, environment variables, or host clocks. + +### 1.4 Requirements & Constraints +- **Host Runtime**: Pure Go WebAssembly runtime via `github.com/tetratelabs/wazero` (`GOOS=wasip1 GOARCH=wasm` guest binaries instantiated in `internal/wasmhost` with stubbed, zero-I/O WASI imports and zero CGO dependencies). +- **WASM Memory Constraints**: Guest modules execute within a bounded linear memory space (tens of megabytes by default, scalable via configuration), with pre-allocated multi-megabyte static input and output slabs. +- **Standard Go WASM Binary Size**: Standard Go (`GOOS=wasip1 GOARCH=wasm`) bundles the complete Go runtime and garbage collector into the binary (~1.9 MB to ~2.5 MB), whereas TinyGo produces ~200 KB binaries but lacks support for certain standard library packages (such as full `crypto/x509`). + +--- + +## 2. Detailed Design + +### 2.1 The High-Level Developer Experience +Ecosystem developers write standard, idiomatic Go against `vindex/mapfn/sdk`. They do not write assembly, manage WebAssembly linear pointers, or handle memory layout serialization. + +The developer implements a pure mapping function using a streaming `emit` callback and registers it in `init()` (Go guest plugins compile as WASI reactors with `GOOS=wasip1 GOARCH=wasm -buildmode=c-shared`, isolating `//go:wasmexport` bindings in `mapfn/sdk` behind `//go:build wasm` so native `go test` runs cleanly on host OSes; the host instantiates modules without running `_start` / `main()` and invokes `_initialize` if exported so both WASI reactors and `wasm32-unknown-unknown` guests initialize deterministically): + +```go +type EmitMapFunc func(entry []byte, emit func(key []byte)) + +func RegisterEmit(fn EmitMapFunc) +``` + +Every emitted `key` MUST be a valid canonical key (`1..1024` printable ASCII bytes in `0x21..0x7E`, containing no `NUL` or whitespace bytes; see [`../docs/tlog-vindex.md`](../docs/tlog-vindex.md#canonical-keys) and [`../docs/APPLICATIONS.md`](../docs/APPLICATIONS.md) for ecosystem-specific canonicalization rules). Guest mappers MUST skip malformed or unindexable fields before calling `emit`: emitting an invalid canonical key (`len == 0`, `len > 1024`, or any byte outside `0x21..0x7E`) causes the host (`internal/wasmhost`) to return a fatal validation error that immediately **`HALT`s** the Index Operator (`vindexd`) or Auditor (`vindex-auditor`). + +The `sdk.RegisterEmit` harness automatically binds the user's pure function to the low-level `map_bundle` FFI export. Each call to `emit(key)` writes the length-prefixed canonical key directly into the pre-allocated output memory slab in-place. This eliminates intermediate slice allocations (`[]string` or `[][]byte`), dynamic array resizing, and garbage collector overhead inside the WebAssembly guest. + +#### WASM Guest ABI Contract: +To exchange data across the FFI boundary while eliminating per-entry allocation overhead: +- **Memory Lifecycle**: The guest exports `input_buffer() uint32` returning the base pointer of a pre-allocated multi-megabyte input slab in linear memory. The host serializes the 256-entry bundle directly into this buffer, invokes `map_bundle`, and reads and validates the resulting canonical keys from the returned output slab pointer. +- **Statelessness Invariant**: Guest code must maintain zero mutable state across invocations. All mapping logic must be deterministic and pure; execution depends strictly on the bytes of each individual Input Log entry. + +--- + +### 2.2 Language-Agnostic Low-Level WASM ABI & The Pack-and-Wipe Memory Slab +To eliminate the 23% CPU overhead of per-entry FFI boundary crossings—and allow guest plugins to be authored in any WASM-capable systems language (Go via `mapfn/sdk`, Rust, C/C++, Zig, or TinyGo)—the host (`internal/wasmhost`) and guest communicate through contiguous pre-allocated linear memory slabs using the `input_buffer` and `map_bundle` exports: + +```text +//go:wasmexport input_buffer +func input_buffer() uint32 + +//go:wasmexport map_bundle +func map_bundle(inputPtr uint32, inputLen uint32) uint64 +``` + +#### Memory Slab Lifecycle: +1. **Pack (Host)**: The Go host (`internal/wasmhost`) queries `input_buffer()` once on instance initialization (falling back to an exported `allocate(size)` function if `input_buffer` is not exported) and serializes all entries of a 256-entry bundle into the contiguous input slab in the guest's linear memory. If the framed input bundle exceeds the static input slab capacity, the host immediately returns a fatal error (`HALT`). +2. **Execute (FFI Call)**: The host invokes `map_bundle(inputPtr, inputLen)`. +3. **Stream (Guest SDK)**: Inside WASM, the SDK harness reads the `[Count+1]uint32` offset table (`entry := payload[offsets[i]:offsets[i+1]]`), iterates over all entries in a tight in-guest loop, and invokes `userMapFn(entry, emit)`. Each call to `emit` copies the length-prefixed canonical key directly into the pre-allocated output slab at the current write cursor, avoiding intermediate collection allocations. If the input buffer cannot be decoded or `emit` overflows the static output slab capacity, `map_bundle` aborts and returns `0` (`outputPtr = 0, outputLen = 0`). +4. **Return, Validate, Deduplicate & Unpack (Host)**: The guest returns a packed 64-bit integer `uint64(outputPtr)<<32 | uint64(outputLen)`. If the return value is `0` (indicating input decoding failure or output slab overflow) or falls outside guest linear memory bounds, the host returns a fatal error (`HALT`). Otherwise, the host reads all extracted canonical keys directly from guest linear memory, validates every key (`1 <= len(key) <= 1024`, all bytes in `0x21..0x7E`; any violation returns a fatal error that immediately **`HALT`s** the indexer or auditor), and **sorts and deduplicates the canonical keys emitted for each individual entry** before passing the batch downstream. +5. **Wipe (Guest/Host)**: Output slab write cursors are reset (`offset = 0`) at the start of the next `map_bundle` call, completely eliminating guest heap allocations and GC pauses. + +#### Binary Slab Layouts (All `uint32` fields encoded in Little-Endian): + +| Memory Buffer | Offset | Field Name | Data Type | Description | +| :--- | :--- | :--- | :--- | :--- | +| **Input Slab** | `0` | `EntryCount` | `uint32` (4B, LE) | Number of entries in the 256-entry bundle (`Count`, 1 to 256). | +| | `4` | `OffsetTable` | `[Count+1]uint32` (LE) | `Count + 1` byte offsets relative to `PayloadBlock`: `OffsetTable[0] = 0`, followed by the end offset of each entry (`entry[i] = PayloadBlock[OffsetTable[i]:OffsetTable[i+1]]`). | +| | `8 + 4*Count` | `PayloadBlock` | `[]byte` | Raw, contiguous Input Log entry bytes concatenated end-to-end. | +| **Output Slab** | `0` | `EntryCount` | `uint32` (4B, LE) | Number of entries processed (`Count`). | +| | `4` | `KeyCountTable` | `[Count]uint32` (LE) | Number of canonical keys emitted by each respective entry. | +| | `4 + 4*Count` | `KeyBlock` | `[]byte` | Concatenated length-prefixed canonical keys (`uint32 length` (LE) + raw canonical key bytes). | + +--- + +### 2.3 In-Situ Invariants & Performance Optimizations + +- **[Correctness Invariant] Pinned Artifact Determinism**: + - *Rule*: Within `vindex`, the `MapFn` `.wasm` module is compiled once (e.g. reproducibly via `go build -trimpath -buildvcs=false -buildmode=c-shared` or an equivalent Rust/Zig/C toolchain) and executed identically by `vindexd` and WASM-based `vindex-auditor` instances, pinned out-of-band by its `SHA-256(wasm_bytes)` artifact digest (`vindex-wasm inspect`). At the `tlog-vindex` protocol layer, `c2sp.org/tlog-vindex-state@v1` intentionally omits an in-band `map-fn` commitment pre-1.0 (see [`docs/tlog-vindex.md`](../docs/tlog-vindex.md#design-log)), and independent auditors may either verify and execute the pinned `.wasm` artifact or audit via a native clean-room implementation matching the mapping specification. + - *Rationale*: Running pinned `.wasm` bytecode across `vindex` nodes eliminates subtle compiler, standard library, or operating system drift (e.g. updates to ASN.1 parsers or Unicode tables), while keeping `tlog-vindex` open to multi-language guests and native non-WASM auditors. + - *Consequence ("Or Else")*: If Index Operators and WASM-based Auditors ran unpinned native builds compiled with drifting toolchains, edge-case parsing discrepancies on a single entry would produce conflicting `mpt_root` commitments and false split-view alarms. + +- **[Correctness Invariant] Hermetic Isolation, Canonical Key Validation & Fatal HALT Policy**: + - *Rule*: Guest WASM execution runs inside a hermetically sealed WASI-free sandbox (`internal/wasmhost`) with zero host filesystem access, zero network sockets, and deterministic frozen clocks. If the guest encounters an unhandled panic, memory fault, execution trap, or emits an invalid canonical key (`len == 0`, `len > 1024`, or any byte outside `0x21..0x7E`), `internal/wasmhost` returns a fatal validation error that immediately **`HALT`s** the indexer or auditor. + - *Rationale*: Ensures that execution is 100% deterministic, spec-compliant, and reproducible across all Index Operators and Auditors. + - *Consequence ("Or Else")*: Silently swallowing guest traps or dropping malformed canonical keys produces divergent key mappings, resulting in false non-inclusion proofs or `mpt-root` audit mismatches. + +- **[Performance Optimization] Bundled FFI Invocation (`map_bundle`)**: + - *Mechanism*: Packs an entire 256-entry bundle into the input slab and executes a single FFI call per bundle, rather than crossing the boundary once per entry. + - *Impact*: Reduces FFI transitions from 768 to 2–3 per 256-entry bundle, slashing FFI CPU overhead from ~23% of host CPU time to < 1%. + +- **[Performance Optimization] Pack-and-Wipe Slab Allocation**: + - *Mechanism*: Slabs are allocated once in guest linear memory and reused across sequential bundle invocations with simple pointer resets. + - *Impact*: Eliminates in-guest heap allocations and garbage collector pauses, keeping memory usage bounded and CPU dedicated to parsing. + +--- + +### 2.4 Offline Developer Verification Tooling (`vindex-wasm`) +The mapping subsystem provides a standalone CLI tool (`vindex/cmd/vindex-wasm`) so plugin authors can inspect, test, and benchmark their `.wasm` binaries before production deployment: + +- **`vindex-wasm inspect`**: Instantiates the module via `internal/wasmhost`, validates the required `input_buffer` and `map_bundle` ABI exports and linear memory bounds, and prints the 32-byte `.wasm` artifact SHA-256 digest in both canonical base64 and hex for out-of-band deployment and auditor pinning. +- **`vindex-wasm test`**: Executes the module against literal entry strings or binary entry files, validates every emitted canonical key against `1..1024` printable ASCII bounds (`0x21..0x7E`), and prints extracted canonical keys in plain text or JSON. +- **`vindex-wasm bench`**: Measures single-worker and multi-bundle mapping throughput (`entries/sec`) and per-bundle latency over repeated 256-entry bundles. + +```bash +# Validate ABI exports, linear memory bounds, and print .wasm SHA-256 digest (base64 & hex) +vindex-wasm inspect --wasm=map.wasm + +# Test canonical key extraction against a literal entry string or binary entry file +vindex-wasm test --wasm=map.wasm --input="golang.org/x/crypto v0.1.0 h1:..." +vindex-wasm test --wasm=map.wasm --input_file=entry.bin --format=json + +# Run performance and throughput benchmarks (entries/sec) +vindex-wasm bench --wasm=map.wasm --input="golang.org/x/crypto v0.1.0 h1:..." --iterations=1000 +``` + +--- + +## 3. Alternatives Considered (or Tried) + +### 3.1 Per-Entry Mapping (`map_leaf`) vs. Bundled Mapping (`map_bundle`) +- **Proposed**: Invoking an exported `map_leaf(ptr, len)` guest function individually for every Input Log entry. +- **Empirical Rejection**: Generated 768 FFI calls per 256-entry bundle (memory allocation, entry mapping, and arena reset transitions x 256). Profiling revealed that ~23% of total host CPU time was spent purely on boundary-crossing overhead. +- **Chosen Design**: Bundled mapping (`map_bundle`) processes all 256 entries in a single invocation via shared memory slabs, reducing FFI overhead to < 1%. + +### 3.2 In-Process Go Plugins / CGO vs. WebAssembly Sandboxing +- **Proposed**: Dynamically loading Go plugins (`plugin.Open`) or shared C libraries (`cgo`). +- **Theoretical Rejection**: + - In-process native plugins do not guarantee determinism across toolchain versions (Go compiler upgrades can alter standard library parsing behavior). + - Memory corruptions or panics inside the plugin crash the main daemon. + - Go plugins have poor cross-platform support and require identical dependency version trees between the host and plugin. +- **Chosen Design**: Standardized on Wazero WebAssembly sandboxing (`internal/wasmhost`). + +--- + +## Appendix: Guest Parsing Optimization Notes (Certificate Transparency) + +While WebAssembly provides sandboxed memory safety and cross-platform determinism, compiling reflection-heavy standard library decoders (`crypto/x509`, `encoding/asn1`, `encoding/json`, `regexp`) into `wasip1/wasm` incurs high heap allocation churn and unnecessary cryptographic validation (such as public-key and signature decoding) irrelevant to canonical key extraction. + +- **[Performance Optimization] Zero-Allocation In-Guest Parsing**: + - *Mechanism*: Ecosystem plugins should prefer targeted, in-place byte scanners over reflection-heavy standard library decoders (for example, scanning raw ASN.1 TLV streams for Subject Common Name OID `2.5.4.3` and SAN `dNSName` OID `2.5.29.17` before applying canonical DNS label normalization per [`../docs/APPLICATIONS.md`](../docs/APPLICATIONS.md)). + - *Impact*: Eliminates in-guest AST heap allocations and extraneous cryptographic validation (~7.5x throughput improvement over `crypto/x509` in `wasip1/wasm`), preventing guest WASM execution from starving the downstream storage and trie pipeline. + +--- + +## 4. Verification Gate + +All public guest SDK (`mapfn/sdk`), internal sandbox host (`internal/wasmhost`), example mapper (`ct`, `mtc`, `sumdb`), and `vindex-wasm` CLI tests are verified via: + +```bash +go test -race ./vindex/mapfn/... ./vindex/internal/wasmhost/... ./vindex/cmd/vindex-wasm/... +``` From 621f03b56e3fc718be33bf11958281237b545612 Mon Sep 17 00:00:00 2001 From: Martin Hutchinson Date: Mon, 2 Nov 2026 09:56:44 +0000 Subject: [PATCH 2/3] docs(vindex): increase signal-to-noise ratio in subcomponent specs --- vindex/README.md | 4 +- vindex/client/README.md | 108 ++++--------- vindex/docs/ARCHITECTURE.md | 2 +- vindex/internal/auditor/README.md | 180 ++++++---------------- vindex/internal/coordinator/README.md | 211 +++++++------------------- vindex/internal/hammer/README.md | 136 +++++------------ vindex/internal/ingest/README.md | 112 +++----------- vindex/internal/kvstore/README.md | 187 ++++++----------------- vindex/internal/server/README.md | 93 +++--------- vindex/internal/tree/README.md | 173 +++++---------------- vindex/mapfn/README.md | 160 ++++++------------- 11 files changed, 334 insertions(+), 1032 deletions(-) diff --git a/vindex/README.md b/vindex/README.md index b36db5d..5e6cc56 100644 --- a/vindex/README.md +++ b/vindex/README.md @@ -35,7 +35,7 @@ Earlier systems attempted to act as verifiable state machines that copied, aggre ## 3. The Map Sandwich Architecture -The VIndex topology bounds search indexing between two append-only logs—a structure known as the **"Map Sandwich"**: +The VIndex topology bounds search indexing between two append-only logs, a structure known as the **"Map Sandwich"**: | Component | Physical Role | Cryptographic Contract | | :--- | :--- | :--- | @@ -77,7 +77,7 @@ The VIndex topology bounds search indexing between two append-only logs—a stru ### Core Specification & Architecture - [**C2SP `tlog-vindex` Specification**](./docs/tlog-vindex.md): Normative wire formats, MPT and value-log Merkle definitions, HTTP lookup endpoints, and client/auditor verification rules. -- [**Verifiable Index Applications**](./docs/APPLICATIONS.md): How VIndex applies across Certificate Transparency, Merkle Tree Certificates, Go SumDB, Sigstore, and Sigsum—who monitors each log, illustrative canonical key shapes, prefix queries, and epoch rotation (`from > 0`). +- [**Verifiable Index Applications**](./docs/APPLICATIONS.md): How VIndex applies across Certificate Transparency, Merkle Tree Certificates, Go SumDB, Sigstore, and Sigsum: who monitors each log, illustrative canonical key shapes, prefix queries, and epoch rotation (`from > 0`). - [**Verifiable Index Architecture**](./docs/ARCHITECTURE.md): High-level pipeline stages, checkpoint promotion invariants, concurrency model, Zero-WAL crash recovery, auditing boundaries, and architectural trade-offs. ### Subcomponent Designs diff --git a/vindex/client/README.md b/vindex/client/README.md index 872540f..d79b12f 100644 --- a/vindex/client/README.md +++ b/vindex/client/README.md @@ -1,51 +1,30 @@ # Client SDK & Query Proof Verification -This document defines the lightweight public Go client SDK (`vindex/client`) for querying VIndex servers over HTTP and cryptographically verifying point, non-inclusion, prefix, and paginated value responses through snapshot-scoped lookup handles. +`vindex/client` is a lightweight, concurrency-safe Go SDK for querying VIndex servers via their public API. It is safe by default: every lookup and paginated value response is cryptographically verified before data is returned to the caller. This allows Go applications to query a Verifiable Index without trusting the server operator, and serves as a reference model for client implementations in other languages. ---- - -## 1. Context & Objectives - -### 1.1 Problem Statement: Zero-Trust Lookup Clients -In verifiable transparency systems, a lookup client querying an index cannot trust the index operator's response. A compromised, coerced, or malfunctioning server could return falsified Input Log entry indices, omit matching records, or present stale state commitments. - -To achieve cryptographic security without requiring lookup clients to download entire databases or execute WebAssembly indexing pipelines, VIndex provides a **lightweight, goroutine-safe HTTP verification client** (`vindex/client`). External Go callers import only `vindex/client`, issue exact or prefix lookups that return an immutable `*LookupResult` snapshot, and lazily stream verified Input Log entry indices per matched key via `MatchedKey.Values(ctx)` against configured Output Log checkpoint verification policies (such as [C2SP `tlog-policy`](https://c2sp.org/tlog-policy)). +Clients must configure an appropriate [C2SP `tlog-policy`](https://c2sp.org/tlog-policy) (or programmatic verifier policy) for trusting the Output Log: lookup verification guarantees that results match a witnessed Output Log commitment, while end-to-end security relies on third-party auditors (such as [`internal/auditor`](../internal/auditor/README.md)) verifying that those commitments were honestly constructed from the Input Log (and, when a caller fetches matched raw entries from the Input Log, authenticating them against the committed `(InputSize, InputRoot)`). -### 1.2 Goals & Non-Goals -- **Goals**: - - Provide a clean, minimal public Go package (`vindex/client`) as the sole import required by external Go lookup clients, CLIs, and monitors. - - Expose a stateless, concurrency-safe HTTP `Client` targeting `` (`New(indexURL, cfg, opts ...Option)` with `WithHTTPClient(*http.Client)` for custom proxies, TLS, auth, timeouts, and test transports while enforcing protocol `3xx` redirect rejection and HTTP status semantics). - - Configure Output Log verification via `Config` (`ExpectedOutputLogOrigin`, `OutputLogPolicy`, and `ExpectedInputLogOrigin`), supporting both [C2SP `tlog-policy`](https://c2sp.org/tlog-policy) witness quorum policies and programmatic log/witness verifier policies without coupling `/keys` lookups to Input Log signature schemes. - - Authenticate state commitments on `LookupKey` and `LookupPrefix` by verifying the `c2sp.org/tlog-proof@v1` bundle against `ExpectedOutputLogOrigin` and `OutputLogPolicy` (enforcing canonical decimal `index == output_size - 1`) and parsing the strict 6-line `c2sp.org/tlog-vindex-state@v1` record (`input-origin`, `from`, `input-size`, `input-root`, `mpt-root`) into an immutable `*LookupResult`. - - Verify binary MPT inclusion, non-inclusion, and two-sided prefix proofs (`internal/mpt`, enforcing `b <= 8192`, terminal `path_key` canonicality, and 32-byte `path_val` bounds on top of `filippo.io/torchwood/mpt`) over canonical keys (`1..1024` printable ASCII bytes in `0x21..0x7E`). - - Scope value enumeration to immutable `MatchedKey` values inside `LookupResult.Keys` via a single lazy iterator method, `MatchedKey.Values(ctx context.Context) iter.Seq2[uint64, error]`, which paginates `/values` backward from `before = InputSize`, verifies each page's RFC 6962 Value Log compact range against the expected `value_root`, and yields verified Input Log entry indices **newest-first** so callers can `break` early at a watermark without unbounded slice allocation. - - Maintain a minimal dependency graph with zero dependencies on storage engines (`pebble`), WebAssembly runtimes (`wazero`), or internal ingestion packages (`internal/ingest`, `internal/kvstore`). -- **Non-Goals**: - - **No Unbounded `AllValues` Slice Helper**: Hot keys can match millions of Input Log entries; `vindex/client` intentionally omits an `AllValues` slice-drain helper so callers stream newest-first via `MatchedKey.Values(ctx)` and apply explicit bounds when collecting into memory. - - **No Cross-Lookup Mutable State**: `Client` caches zero state across `LookupKey` / `LookupPrefix` calls; monitors that track progress across polling intervals persist their own `InputSize` watermark externally. - - **No Full-Log Auditing**: The lookup client does not replay raw Input Log entries or reconstruct global MPT state; continuous full-log auditing is performed by [`internal/auditor`](../internal/auditor/README.md). - - **No Output Log Tile Polling**: Lookup clients query only `/keys` and `/values`; they do not configure `` or fetch `/checkpoint` and `/tile/...` endpoints directly. - -### 1.3 Security Boundary -- **Witness Policy Trust Floor**: Lookup client security on `/keys` and `/values` is bounded by the configured `OutputLogPolicy` (and, when a caller fetches matched raw entries from the Input Log, by Merkle authentication against `(InputSize, InputRoot)` and optional Merkle consistency verification against a signed Input Log checkpoint). If Output Log witnesses equivocate or cosign a fraudulent Output Log checkpoint, lookup clients will accept proofs anchored to that checkpoint until an independent auditor ([`internal/auditor`](../internal/auditor/README.md)) detects the invalid state transition. +### Non-Goals +- **No Unbounded In-Memory Value Collection**: Hot keys can match millions of Input Log entries; the SDK streams verified entry indices lazily page-by-page rather than providing helpers that drain all historical matches into memory. +- **No Cross-Lookup State**: The client caches zero state across lookups; monitors tracking progress across polling intervals persist their own watermarks externally. +- **No Full-Log Auditing**: The client verifies individual query proofs against witnessed commitments; replaying the Input Log to verify index construction is performed by an auditor ([`internal/auditor`](../internal/auditor/README.md)). +- **No Direct Output Log Polling**: Lookup clients query only the VIndex server's lookup endpoints; they do not fetch Output Log `/checkpoint` or `/tile/...` endpoints directly. --- -## 2. Detailed Design +## 1. Design -### 2.1 Public Architecture & Package Decoupling +### 1.1 Public Architecture & Package Decoupling -External Go callers import `vindex/client` exclusively. Shared wire and MPT proof primitives live in internal packages decoupled from the indexing and storage subsystems: +External Go callers import `vindex/client` exclusively. Shared wire and MPT proof primitives live in internal packages that evaluate binary MPT path steps and RFC 6962 compact range folds linearly over byte slices without allocating intermediate tree node graphs (~1 ms per query on commodity CPUs): | Responsibility | Package | Purpose | | :--- | :--- | :--- | -| **HTTP Transport & Verification Client API** | `vindex/client` | Public snapshot-scoped lookup client (`Client`, `LookupResult`, `MatchedKey`) with zero dependencies on `pebble`, `wazero`, or `tessera`. | +| **HTTP Transport & Verification Client API** | `vindex/client` | Public snapshot-scoped lookup client (`Client`, `LookupResult`, `MatchedKey`) with zero dependencies on `pebble`, `wazero`, `tessera`, `internal/ingest`, or `internal/kvstore`. | | **C2SP Plaintext Wire & Value Log Codecs** | `vindex/internal/wire` | C2SP `tlog-vindex` plaintext parsers/formatters, checkpoint policy verification, and RFC 6962 Value Log compact range folding. | -| **MPT Point & Prefix Proof Verification** | `vindex/internal/mpt` | Binary MPT point inclusion/non-inclusion and two-sided prefix proof verification (`b <= 8192` split-bit bounds and canonical terminal leaf validation). | - -### 2.2 Snapshot-Scoped Client API (`Client`, `LookupResult`, `MatchedKey`) +| **MPT Point & Prefix Proof Verification** | `vindex/internal/mpt` | Binary MPT point inclusion/non-inclusion and two-sided prefix proof verification (`b <= 8192` split-bit bounds and canonical terminal leaf validation on top of `filippo.io/torchwood/mpt`). | -`Client` is immutable after construction and safe for concurrent use by multiple goroutines. Each `LookupKey` or `LookupPrefix` call returns an isolated `*LookupResult` capturing the verified state commitment and any `MatchedKey` handles bound to that snapshot: +### 1.2 Snapshot-Scoped Client API (`Client`, `LookupResult`, `MatchedKey`) ```go type Config struct { @@ -87,9 +66,8 @@ var ( ) ``` -`OutputLogPolicy` may be parsed from a standard [C2SP `tlog-policy`](https://c2sp.org/tlog-policy) text file (supporting arbitrary `quorum` threshold expressions across witnesses) or constructed programmatically from a primary log verifier and witness threshold, pinned to `ExpectedOutputLogOrigin`. `WithHTTPClient` clones the caller-supplied `*http.Client` (preserving custom `Transport`, proxies, TLS settings, and timeouts) while overriding `CheckRedirect` to reject all `3xx` redirects per [`docs/tlog-vindex.md`](../docs/tlog-vindex.md). - -Because snapshot context (`ValueRoot`, `From`, `InputSize`, and `*Client`) is encapsulated on each immutable `MatchedKey`, callers can selectively enumerate a subset of keys from a prefix lookup, stop early at a watermark, and reuse the same `*Client` across unrelated lookups without mutating shared client state: +- **Configuration & Transport**: `New` requires non-empty `ExpectedOutputLogOrigin`, non-nil `OutputLogPolicy` (parsed from a [C2SP `tlog-policy`](https://c2sp.org/tlog-policy) witness quorum text file or constructed programmatically from a primary log verifier and witness threshold), and non-empty `ExpectedInputLogOrigin`. `WithHTTPClient` clones the caller-supplied `*http.Client` (preserving custom `Transport`, proxies, TLS settings, and timeouts) while overriding `CheckRedirect` to reject all `3xx` redirects per [`docs/tlog-vindex.md`](../docs/tlog-vindex.md). +- **Snapshot-Scoped Isolation**: `*Client`, `*LookupResult`, and `MatchedKey` hold zero mutable cross-lookup state. Pagination cursors `(before, expectedValueRoot)` exist strictly as stack-local variables inside each `MatchedKey.Values(ctx)` invocation, allowing a single `*Client` or `MatchedKey` to be used concurrently across goroutines and against load-balanced or short-TTL-cached server replicas where consecutive `/keys` responses may arrive slightly out of order. ```go res, err := cli.LookupPrefix(ctx, "com.google.") @@ -115,34 +93,18 @@ for _, mk := range res.Keys { } ``` -### 2.3 End-to-End HTTP & Cryptographic Verification Sequence +### 1.3 End-to-End HTTP & Cryptographic Verification Sequence -Verification executes across 4 protocol steps matching [`docs/tlog-vindex.md`](../docs/tlog-vindex.md)—Steps 1–3 eagerly inside `LookupKey` / `LookupPrefix`, and Step 4 lazily per page inside `MatchedKey.Values`: +`LookupKey` and `LookupPrefix` never return a `*LookupResult` until Steps 1–3 succeed, and `MatchedKey.Values` verifies each `/values` page's compact range root before yielding any `entry_idx` from that page: | Step | Phase | Operations | Cryptographic & Protocol Assertion | | :--- | :--- | :--- | :--- | -| **1** | **Output Log Proof (`c2sp.org/tlog-proof@v1`)** | Percent-encodes `` or `` per RFC 3986 (`unreserved` only, escaping `+` as `%2B`), fetches `GET /keys?...` (rejecting `3xx`, mapping `400`/`413`/`503`, bounding response body size), splits on `\n\n` into `c2sp.org/tlog-vindex-keys@v1`, `c2sp.org/tlog-proof@v1` (base64 `extra` `state_entry` and canonical decimal `index`), and ``, and computes `SHA-256(0x00 \|\| state_entry)`. | Verifies `` against `cfg.ExpectedOutputLogOrigin` and `cfg.OutputLogPolicy` to extract `output_size` and `output_root`, asserts `output_size >= 1` and `index == output_size - 1`, and verifies the Merkle inclusion proof for `SHA-256(0x00 \|\| state_entry)` at `index`. | +| **1** | **Output Log Proof (`c2sp.org/tlog-proof@v1`)** | Validates local canonical key/prefix syntax (`1..1024` bytes in `0x21..0x7E`), percent-encodes per RFC 3986 (`unreserved` only, escaping `+` as `%2B`), fetches `GET /keys?...` (rejecting `3xx`, mapping `400`/`413`/`503`, bounding response body size), splits on `\n\n` into `c2sp.org/tlog-vindex-keys@v1`, `c2sp.org/tlog-proof@v1` (base64 `extra` `state_entry` and canonical decimal `index`), and ``, and computes `SHA-256(0x00 \|\| state_entry)`. | Verifies `` against `cfg.ExpectedOutputLogOrigin` and `cfg.OutputLogPolicy` to extract `output_size` and `output_root`, asserts `output_size >= 1` and `index == output_size - 1`, and verifies the Merkle inclusion proof for `SHA-256(0x00 \|\| state_entry)` at `index`. | | **2** | **State Commitment (`c2sp.org/tlog-vindex-state@v1`)** | Parses `state_entry` as a strict 6-line `c2sp.org/tlog-vindex-state@v1` record (`input-origin`, `from`, `input-size`, `input-root`, `mpt-root`) with no extension lines or trailing data. | Asserts `input_origin == cfg.ExpectedInputLogOrigin` and `0 <= from <= input_size`. Because a lookup response only attests to Input Log entries in `[from, input_size)`, callers requiring coverage back to a specific entry index check `LookupResult.From` against that bound. | -| **3** | **MPT Key / Prefix Proof (`LookupKey` / `LookupPrefix`)** | Validates canonical key syntax (`1..1024` bytes in `0x21..0x7E`), and verifies the binary MPT proof (`internal/mpt`) against `mpt_root` (enforcing `b <= 8192` split-bit bounds and canonical `path_key` / 32-byte `path_val` checks on terminal leaves). | **Inclusion**: Single matched `(key, value_root)` with `key == target` hashes to `mpt_root`.
**Non-Inclusion**: `mpt_root == empty_mpt_hash` (`0`-byte proof) or terminal MPT leaf `mpt_leaf_hash(path_key, path_val)` with `path_key != target` hashes to `mpt_root` (`len(res.Keys) == 0`).
**Prefix**: On `0` matches, `mpt_root == empty_mpt_hash` (`0`-byte proof) or terminal MPT leaf with `!HasPrefix(path_key, target)` hashes to `mpt_root`; on `n >= 1` matches, `HasPrefix` + strictly ascending key order + `b_1 < 8*len(target)` + subtree root + path to `mpt_root`. | +| **3** | **MPT Key / Prefix Proof (`LookupKey` / `LookupPrefix`)** | Validates canonical key syntax (`1..1024` bytes in `0x21..0x7E`) on returned keys, and verifies the binary MPT proof (`internal/mpt`) against `mpt_root` (enforcing `b <= 8192` split-bit bounds and canonical `path_key` / 32-byte `path_val` checks on terminal leaves). | **Inclusion**: Single matched `(key, value_root)` with `key == target` hashes to `mpt_root`.
**Non-Inclusion**: `mpt_root == empty_mpt_hash` (`0`-byte proof) or terminal MPT leaf `mpt_leaf_hash(path_key, path_val)` with `path_key != target` hashes to `mpt_root` (`len(res.Keys) == 0`).
**Prefix ([two-sided](../docs/tlog-vindex.md#two-sided-prefix-verification))**: On `0` matches, `mpt_root == empty_mpt_hash` (`0`-byte proof) or terminal MPT leaf with `!HasPrefix(path_key, target)` hashes to `mpt_root`; on `n >= 1` matches, `HasPrefix(key, target)` on every returned key proves no keys outside `target` were included from a wider ancestor subtree, while strictly ascending key order + first ancestor split bit `b_1 < 8*len(target)` + reconstructed subtree root + ancestor path to `mpt_root` proves no keys inside `target` were omitted. | | **4** | **Lazy Backward Value Pagination (`MatchedKey.Values`)** | Initializes local iterator state `(before = inputSize, expectedValueRoot = k.ValueRoot)`. Fetches `GET /values?key=&before=` and parses `c2sp.org/tlog-vindex-values@v1` (`preceding`, `popcount(preceding)` compact range hashes, and strictly increasing `Indices`). | Treats `HTTP 404` on an MPT-included key or promised continuation page (`preceding > 0`) as `ErrVerificationFailed`. Asserts `len(Indices) >= 1`, strictly increasing order in `[from, before)`, and `preceding <= Indices[0] - from`; appends `value_log_leaf_hash(entry_idx)` into the running compact range (merging interior nodes via `value_log_node_hash(left, right)` on trailing `1` bits of the running leaf count) and folds the resulting compact range right-to-left to assert equality with `expectedValueRoot` **before** yielding `Indices` in reverse (`len-1` down to `0`, newest-first). If the caller continues and `preceding > 0`, folds the page's initial `popcount(preceding)` compact range hashes right-to-left into `expectedValueRoot`, sets `before = Indices[0]`, and fetches the next backward page. | -### 2.4 In-Situ Invariants & Performance Optimizations - -- **[Correctness Invariant] Pre-Yield Cryptographic & Protocol Assertion**: - - *Rule*: `LookupKey` and `LookupPrefix` never return a `*LookupResult` until Output Log policy verification, `index == output_size - 1`, `c2sp.org/tlog-vindex-state@v1` bounds, and binary MPT proof verification all succeed; `MatchedKey.Values` verifies the entire `/values` page's compact range root before yielding a single `entry_idx` from that page, and rejects `3xx` redirects or unexpected `404` responses on `/values`. - - *Rationale*: HTTP responses and status codes from `` are untrusted input; yielding unverified indices or silently treating an unexpected `404` as empty history would let a faulty or malicious server truncate results. - - *Consequence ("Or Else")*: A compromised index operator could forge entry indices or omit matching records without detection. - -- **[Correctness Invariant] Snapshot-Scoped Lookup Isolation**: - - *Rule*: `*Client` holds zero mutable per-lookup or cross-lookup state. Each `*LookupResult` and its `MatchedKey` values capture an immutable `(ValueRoot, from, inputSize)` snapshot, and pagination cursors `(before, expectedValueRoot)` exist strictly as local stack variables inside each `MatchedKey.Values(ctx)` invocation. - - *Rationale*: Allows a single `*Client` (and even a single `MatchedKey`) to be used concurrently across goroutines, across unrelated key/prefix lookups, and against load-balanced or short-TTL-cached server replicas where consecutive `/keys` responses may arrive slightly out of order without invalidating any proof. - - *Consequence ("Or Else")*: Mutating shared state on `*Client` across lookups causes data races or false regression errors under concurrent goroutines and load-balanced replicas, and tangles cursor state when interleaving multiple prefix enumerations. - -- **[Performance Optimization] Lazy Newest-First Streaming & Zero-Allocation Proof Evaluation**: - - *Mechanism*: `MatchedKey.Values(ctx)` streams verified entry indices newest-first via `iter.Seq2[uint64, error]` one server-bounded page at a time (omitting any `AllValues` full-slice helper), while `internal/mpt` and `internal/wire` evaluate binary MPT path steps and RFC 6962 compact range folds linearly over byte slices without allocating intermediate tree node graphs. - - *Impact*: Callers inspecting recent occurrences or polling down to a watermark execute a single `/values` round-trip with constant per-page memory even on hot keys with millions of historical occurrences, completing proof verification in roughly 1 ms per query on commodity CPUs. - -### 2.5 Failure Taxonomy & Client Handling +### 1.4 Failure Taxonomy & Client Handling | Failure Category | Root Cause | Client Action | | :--- | :--- | :--- | @@ -154,36 +116,20 @@ Verification executes across 4 protocol steps matching [`docs/tlog-vindex.md`](. | **Prefix Cardinality or Policy Rejection (`HTTP 413` / `HTTP 400`)** | `/keys?prefix=` matched more keys than the server's configured prefix limit (`413`), or violated the server's prefix policy (`400`). | Surface typed HTTP status error (`ErrTooManyKeys` / `ErrBadRequest`) so the caller can narrow the queried prefix. | | **Upstream Server Unavailable (`HTTP 503`)** | Server has not yet published its first Output Log checkpoint or lags `before`. | Surface `ErrUnavailable` (preserving `Retry-After` metadata when present) so the caller can retry with backoff or fail over. | -### 2.6 Client Contract & API Requirements - -- **Inputs**: - - Index base URL (``, serving `/keys` and `/values`). - - `Config` containing non-empty `ExpectedOutputLogOrigin` (`string`), non-nil `OutputLogPolicy` (`CheckpointPolicy`), and non-empty `ExpectedInputLogOrigin` (`string`), plus optional `WithHTTPClient(*http.Client)`. - - Canonical key (`key`) or canonical key prefix (`prefix`) of `1..1024` printable ASCII bytes (`0x21..0x7E`), validated locally before sending and percent-encoded per RFC 3986 (`unreserved`) on the wire. -- **Outputs**: - - `*LookupResult` exposing verified state commitment metadata (`OutputSize`, `OutputRoot`, `InputOrigin`, `From`, `InputSize`, `InputRoot`, `MPTRoot`) and lexicographically ordered `Keys []MatchedKey` (`0..1` for `LookupKey`, `0..N` for `LookupPrefix`). - - Lazy `MatchedKey.Values(ctx) iter.Seq2[uint64, error]` yielding verified `uint64` Input Log entry indices in `[From, InputSize)` in **strictly decreasing (newest-first)** order across backward pages (callers requiring historical coverage back to a specific entry index check `LookupResult.From`, and callers fetching raw entries from the Input Log authenticate them against `(LookupResult.InputSize, LookupResult.InputRoot)`). -- **Guarantees**: - - All HTTP redirects (`3xx`) on `/keys` and `/values` are rejected per `tlog-vindex`. - - `*Client`, `*LookupResult`, and `MatchedKey` contain no mutable cross-lookup state and are safe for concurrent use across goroutines. - - Any Output Log policy verification failure, `input-origin` or `from` violation, non-tip Output Log proof, MPT path/prefix defect, unexpected `404` on `/values`, entry index bound violation, or compact range mismatch returns an immediate error and withholds unverified data. - --- -## 3. Alternatives Considered (or Tried) +## 2. Alternatives Considered -### 3.1 Stateless Snapshot Handles vs. Mutable Cross-Lookup Client State -- **Proposed**: Caching observed `(output_size, input_size)` watermarks inside `Client` across lookups or exposing unbounded slice-draining helpers for value pagination. +### 2.1 Stateless Snapshot Handles vs. Mutable Cross-Lookup Client State +- **Proposed**: Caching observed `(output_size, input_size)` watermarks inside `Client` across lookups or exposing an `AllValues` full-slice helper for value pagination. - **Rejection**: - - **Cross-Lookup Watermark Races**: Caching observed `(output_size, input_size)` watermarks inside `Client` verifies no Merkle consistency proof (witnesses and auditors provide that guarantee), causes false regression errors when concurrent goroutines or load-balanced/short-TTL-cached server replicas return checkpoints out of arrival order, and tangles state across unrelated key lookups. - - **Unbounded Slice Allocations on Hot Keys**: Keys appearing hundreds of thousands or millions of times would trigger unbounded sequential HTTP pagination and large slice allocations, whereas callers needing a slice can collect from `iter.Seq2` explicitly with their own bounds. -- **Chosen Design**: `Client` is stateless across lookups and returns snapshot-scoped `*LookupResult` handles whose `MatchedKey.Values(ctx)` `iter.Seq2[uint64, error]` iterator streams verified entry indices newest-first with early `break` support. + - **Cross-Lookup Watermark Races**: Caching watermarks inside `Client` verifies no Merkle consistency proof (witnesses and auditors provide that guarantee), causes false regression errors when concurrent goroutines or load-balanced/short-TTL-cached server replicas return checkpoints out of arrival order, and tangles state across unrelated key lookups. + - **Unbounded Slice Allocations on Hot Keys**: Hot keys can match millions of entries, triggering unbounded sequential HTTP pagination and large slice allocations, whereas callers needing a slice can collect from `iter.Seq2` explicitly with their own bounds. +- **Chosen Design**: `Client` is stateless across lookups and returns snapshot-scoped `*LookupResult` handles whose `MatchedKey.Values(ctx)` iterator streams verified entry indices newest-first with early `break` support. --- -## 4. Verification Gate - -All client proof verification, HTTP transport/redirect/status handling, checkpoint policy quorum enforcement, backward pagination streaming, and package dependency boundary tests are verified via: +## 3. Verification Gate ```bash go test -race ./vindex/client/... diff --git a/vindex/docs/ARCHITECTURE.md b/vindex/docs/ARCHITECTURE.md index bc1a7a7..460cef3 100644 --- a/vindex/docs/ARCHITECTURE.md +++ b/vindex/docs/ARCHITECTURE.md @@ -89,7 +89,7 @@ Ecosystem authors express mapping logic as a deterministic function from raw Inp VIndex separates trie structure from per-key occurrence lists using a **hybrid keying model**: -- The **MPT** indexes **un-hashed variable-length canonical keys** directly so keys sharing a string prefix form a contiguous subtrie for [two-sided prefix proofs](./tlog-vindex.md#verifying-a-keys-response). +- The **MPT** indexes **un-hashed variable-length canonical keys** directly so keys sharing a string prefix form a contiguous subtrie for [prefix subtree proofs with two-sided verification](./tlog-vindex.md#two-sided-prefix-verification). - The **KV store** indexes per-key value-log chunks by fixed-size **`SHA-256(key)` digests** computed on the host, keeping LSM keys uniform and decoupled from variable canonical key lengths. Within the KV store, each key's matching `entry_idx` sequence is partitioned into fixed-window chunks ordered newest-first under `SHA-256(key)`. Each chunk stores the RFC 6962 compact range hashes of all older occurrences preceding that chunk alongside compact relative offsets within the window: diff --git a/vindex/internal/auditor/README.md b/vindex/internal/auditor/README.md index 6eaccb3..e8c3ed2 100644 --- a/vindex/internal/auditor/README.md +++ b/vindex/internal/auditor/README.md @@ -1,94 +1,39 @@ # Independent Log Auditor -This document defines the architecture, continuous audit loop, root mismatch alerting, state-preserving halt protocols, and adversarial threat model for the **Auditor Subsystem** (`vindex/internal/auditor`). +Lookup clients ([`client/`](../../client/README.md)) verify that individual query responses match a witnessed Output Log commitment, but they cannot verify that the committed `mpt_root` itself was honestly constructed from the Input Log without omitting or fabricating unqueried keys. `vindex/internal/auditor` (`vindex-auditor`) closes that security loop: it independently replays the Input Log from `from` through the mapping function, recomputes every key's Value Log and the global MPT, and asserts that every state commitment published in the Output Log matches the reconstructed root. ---- +This reference auditor reuses the indexer's Pebble chunk store ([`internal/kvstore`](../kvstore/README.md)) and disk-backed MPT (`internal/mpt`) to share the high-throughput ingestion pipeline (90,000 to 240,000+ entries/sec, requiring equivalent RAM and NVMe sizing) and preserve inspectable on-disk forensic state immediately prior to any divergence. However, reusing the indexer's storage and trie code is not required by the protocol, and relying solely on a single codebase would only verify that the operator ran the same code rather than catching shared implementation bugs. Independent clean-room auditors (written in other languages or using different storage and mapping implementations conforming to [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md#auditing-the-index)) should also audit the Output Log to provide true implementation diversity. -## 1. Context & Objectives - -### 1.1 Problem Statement & The Operator Trust Dilemma -The index operator publishes an Output Log asserting that its verifiable index accurately, completely, and deterministically represents the underlying Input Log over `[from, input_size)`. - -However, in a verifiable transparency architecture, lookup clients and ecosystem monitors cannot simply trust the index operator's claims. A compromised, coerced, or buggy index operator could: -1. **Targeted Omission Attacks**: Commit an invalid `mpt_root` that silently conceals specific certificates, packages, or identities, while returning valid-looking non-inclusion proofs against that forged `mpt_root`. -2. **Equivocation & Split-Views**: Present conflicting Output Log checkpoints to different monitors or lookup clients. -3. **Historical Rollbacks & Parameter Drift**: Regress tree sizes or mutate `input-origin` or `from` across Output Log entries. - -To enforce cryptographic integrity, independent third parties must be able to audit the index operator's Output Log state commitments from `from` to the Output Log head without trusting the `/keys` or `/values` HTTP endpoints. This reference auditor implementation reuses the Pebble inverted chunk store ([`internal/kvstore`](../kvstore/README.md)) and disk-backed MPT (`internal/mpt`) from the indexer to share the high-throughput ingestion/recovery pipeline and preserve inspectable on-disk forensic state immediately prior to any divergence. However, reusing the indexer's storage and trie packages is not required by the protocol—and relying solely on a single codebase would only verify that the operator ran the same code, not that the code itself is free of shared bugs. Independent clean-room auditors (written in other languages or using different storage and mapping implementations conforming to [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md#auditing-the-index)) should also audit the Output Log to provide true implementation diversity. - -### 1.2 Goals, Non-Goals & Future Work -- **Goals**: - - Continuously audit published Output Log index state commitments (`c2sp.org/tlog-vindex-state@v1`) against authenticated Input Log entries starting from the configured `from` index (default `0`). - - Verify Output Log and Input Log checkpoints against configured checkpoint verification policies (such as [C2SP `tlog-policy`](https://c2sp.org/tlog-policy)). - - Assert cryptographic equality between the independently reconstructed local `mpt_root` and the index operator's committed `mpt_root` at every Output Log entry. - - Signal divergence immediately via Prometheus metrics, structured logs, and readiness probe degradation (`/readyz` on the internal operational listener). - - Freeze physical database state on root mismatch to preserve forensic post-mortem evidence. -- **Non-Goals**: - - **No Automatic Retry / Self-Healing on Root Mismatch**: If a root mismatch occurs, the auditor does not guess or retry; it halts immediately to preserve evidence of operator fraud or software bugs. - - **No Operator Checkpoint Generation**: The auditor does not sign or publish Output Log checkpoints; it is strictly an independent consumer and auditor. - - **No HTTP Lookup Serving**: The auditor does not serve `/keys` or `/values` lookup queries; verified read-mirror serving is out of scope. -- **Future Work (Verified Mirror Serving)**: - - Because an auditor already maintains a locally verified MPT and Pebble chunk store, it could in principle also serve C2SP lookup queries (`/keys` and `/values`) as a read mirror. Mirror serving is intentionally omitted for now: `c2sp.org/tlog-vindex` does not specify mirror semantics, and serving lookups from an auditor introduces protocol design questions around which Output Log checkpoint is served and how to satisfy the `index == output_size - 1` latest-entry proof requirement when an auditor lags or halts behind the index operator's Output Log head. This capability can be revisited in future work once specified. - -### 1.3 Requirements, Dependencies & Constraints -- **Dependencies**: Integrates checkpoint policy and Merkle verification (`internal/wire`), the tile fetcher and local tile cache ([`internal/ingest`](../ingest/README.md)), the KV chunk store ([`internal/kvstore`](../kvstore/README.md)), and the MPT (`internal/mpt`). -- **Hardware Sizing Parity**: Running a full-log auditor requires the same RAM and NVMe disk capacity as the index operator to maintain the local MPT and Pebble chunk store. -- **Historical Entry Availability**: Auditing an Output Log from `from` requires historical Input Log entry bundles across `[from, input_size)` to remain retrievable. +### Non-Goals & Future Work +- **No Automatic Retry / Self-Healing on Root Mismatch**: On any root mismatch or consistency violation, the auditor halts immediately and freezes local disk state to preserve forensic evidence. +- **No Checkpoint Signing or Publication**: Strictly an independent consumer and verifier of the Output Log and Input Log. +- **No HTTP Lookup Serving (Future Work)**: Although an auditor maintains a verified MPT and Pebble chunk store locally, `c2sp.org/tlog-vindex` does not yet specify read-mirror semantics (such as which Output Log checkpoint a lagging or halted auditor serves to satisfy the `index == output_size - 1` latest-entry proof check). Verified mirror serving is deferred to future work. --- -## 2. Detailed Design +## 1. Design + +### 1.1 The Continuous Audit Loop -### 2.1 Ecosystem Verification Architecture & Policy Configuration +The auditor is configured with ``, the Input Log retrieval endpoint (requiring historical entry bundles across `[from, input_size)` to remain retrievable), expected `` and ``, checkpoint verification policies (such as [C2SP `tlog-policy`](https://c2sp.org/tlog-policy)) for both logs, the compiled WASM `MapFn` module, and the start index `from` (default `0`). -VIndex verification operates across two complementary boundaries: +Initialized with an empty local MPT and `prev_size = from` (or, on restart, after replaying local Phase 2 MPT catch-up across `[max(MPT_Durable_Size, from), audited_input_size)` if the MPT's persisted `(version, exact)` lags `audited_input_size`), the auditor executes the 4-step loop defined in [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md): -| Role & Component | Execution Scope | Target Consumer | Threat Coverage | +| Step | Phase | Operations | Verification & Persistence Rules | | :--- | :--- | :--- | :--- | -| **Lookup Client Verification** ([`client/`](../../client/README.md)) | Verifies individual `/keys` and `/values` responses against witnessed Output Log checkpoints. | Lookup clients, end-user apps, CLIs. | Verifies inclusion/non-inclusion/prefix proofs for queried keys; verifies Output Log checkpoint policies and witness quorums; verifies backward Value Log pagination chains. | -| **Full-Log Independent Auditor** (`vindex/internal/auditor`) | Independently replays all Input Log entries in `[from, input_size)` and reconstructs the full MPT across every Output Log entry. | Standalone monitoring daemon (`vindex-auditor`). | Detects operator equivocation, forged `mpt_root` commitments, `input-origin`/`from` drift, targeted omission of unqueried keys, and tree rollbacks; enforces checkpoint witness quorums and Merkle consistency. | +| **1** | **Poll Output & Input Checkpoints** | Fetches `/checkpoint` and a signed Input Log `/checkpoint` (`latest_input_size`, `latest_input_root`, re-fetched whenever a newer Output Log entry commits `input_size > latest_input_size`). | Verifies both checkpoints against their expected origins and checkpoint policies, and verifies Output Log Merkle consistency against the previously audited Output Log checkpoint before reading any entries. | +| **2** | **Stream Output Entries** | Fetches new Output Log entry bundles from `/tile/entries/...` from the last audited `output_size` up to the new `output_size`. | Authenticates bundles against the verified Output Log checkpoint root per `tlog-tiles`, parses each 6-line `c2sp.org/tlog-vindex-state@v1` entry (`input-origin`, `from`, `input-size`, `input-root`, `mpt-root`) in order, asserts `input_origin` and `from` match configured parameters and `prev_size <= input_size <= latest_input_size`, and verifies that `(input_size, input_root)` is Merkle-consistent with `(latest_input_size, latest_input_root)` (checking `input_root == latest_input_root` when `input_size == latest_input_size`). | +| **3** | **Fetch, Map & Calculate Local MPT Root** | Downloads Input Log entry bundles covering `[prev_size, input_size)` (slicing boundary 256-entry bundles to exclude entries `< from` on non-256-aligned initial bundles and entries `>= input_size` on partial tail bundles), maps entries via `MapFn`, updates modified Value Logs and `(canonicalKey, value_root)` mappings in the local MPT, and calculates the resulting local `mpt_root`. | Authenticates Input Log bundles against `input_root` (or `latest_input_root`), validates canonical key bounds (`1..1024` bytes in `0x21..0x7E`), deduplicates keys per `entry_idx`, appends `value_log_leaf_hash(entry_idx)` to each key's Value Log, and updates the local MPT to compute the root at `input_size`. On crash replay, any `entry_idx` already persisted in Pebble is skipped while reconstructing its point-in-time `(canonicalKey, value_root)` at `input_size`. | +| **4** | **Assert Root Equality** | Compares the calculated local `mpt_root` against `mpt_root` in the Output Log entry. | **Match**: Persists the updated chunk state, MPT snapshot, and `audited_output_size` / `audited_input_size` watermarks (`KV_CP >= MPT_Durable_Size`), and sets `prev_size = input_size`.
**Mismatch**: Executes the state-preserving halt protocol without advancing verified watermarks, freezing disk state and emitting alerts. | -#### Auditor Configuration & Checkpoint Policy Enforcement -The auditor daemon is configured with ``, the expected `` and ``, the Input Log checkpoint and entry retrieval endpoint, the compiled WASM `MapFn` module, the configured start index `from` (`0` by default, or `> 0` for suffix indices), and checkpoint verification policies (such as C2SP `tlog-policy` policies) for both the Output Log and the Input Log. Every Output Log checkpoint and polled Input Log checkpoint is verified against its expected origin and checkpoint policy before any entries are processed. +### 1.2 Root Mismatch Alerting, State-Preserving Halt & Runbook -### 2.2 The Continuous Audit Loop -Initialized with an empty local MPT and `prev_size = from` (or, on process restart, after running local Phase 2 MPT catch-up across `[max(MPT_Durable_Size, from), audited_input_size)` if the MPT's persisted `(version, exact)` lags `audited_input_size`), the auditor polls `/checkpoint` and verifies `(input_size, input_root)` from each `c2sp.org/tlog-vindex-state@v1` entry against a signed Input Log checkpoint (`latest_input_size`, `latest_input_root`, re-fetched whenever `input_size > latest_input_size`), executing the 5-step audit loop defined in [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md): +On any root mismatch or upstream consistency/policy violation, the auditor halts and freezes local database state on disk for forensic triage: -| Step | Phase | Action & Operations | Verification Check | -| :--- | :--- | :--- | :--- | -| **1** | **Poll Output & Input Checkpoints** | Fetches `/checkpoint` and a signed Input Log `/checkpoint` (re-fetching the Input Log checkpoint as needed when auditing newer Output Log entries). | Verifies Output Log and Input Log checkpoint origins, primary signatures, and witness cosignatures against their respective checkpoint verification policies, and verifies Output Log Merkle consistency against the previously audited Output Log checkpoint. | -| **2** | **Stream Output Entries** | Fetches new Output Log entry bundles from `/tile/entries/...` from the last audited `output_size` up to the new `output_size`. | Authenticates entry bundles against the verified Output Log checkpoint root hash per `tlog-tiles`, parses each 6-line `c2sp.org/tlog-vindex-state@v1` entry (`input-origin`, `from`, `input-size`, `input-root`, `mpt-root`), asserts `input_origin` and `from` match configured parameters, checks `prev_size <= input_size <= latest_input_size`, and verifies that `(input_size, input_root)` is Merkle-consistent with `(latest_input_size, latest_input_root)` (checking `input_root == latest_input_root` when `input_size == latest_input_size`). | -| **3** | **Fetch, Map & Stage Input Entries** | Downloads Input Log entries in `[prev_size, input_size)` (slicing non-256-aligned initial `from` or partial tail bundles), filters already-persisted entries, stages chunk mutations into an uncommitted indexed Pebble batch, and collects modified `(canonicalKey, value_root)` mappings in memory. | Authenticates Input Log entry bundles against `input_root` (or `latest_input_root`), executes sandboxed `MapFn`, checks that every emitted key is a valid canonical key (`1..1024` bytes in `0x21..0x7E`), deduplicates keys per `entry_idx`, appends `value_log_leaf_hash(entry_idx)` to each key's Value Log inside the uncommitted batch, and enforces replay deduplication. | -| **4** | **Predict Candidate MPT Root** | Evaluates the accumulated in-memory `(canonicalKey, value_root)` pairs against the local MPT without mutating verified tree state. | Computes the candidate local `mpt_root` in memory under a shared read lock. | -| **5** | **Assert Root Equality & Commit** | Asserts that the candidate local `mpt_root` equals the committed `mpt_root` in the Output Log entry. | **Match**: Applies the staged Pebble batch (`'c'` chunks + `audited_output_size` + `audited_input_size`) with durable `fsync`, commits and syncs MPT mutations, and sets `prev_size = input_size`.
**Mismatch**: Triggers state-preserving halt, discards the uncommitted Pebble batch and clears the in-memory active chunk cache, freezes database files, and emits alerts. | - -#### Step Details: -1. **Discover & Authenticate Checkpoints**: Polls `/checkpoint`, verifies `` and witness cosignatures against the Output Log checkpoint policy, and verifies a Merkle consistency proof from the previously audited Output Log checkpoint. Fetches and verifies a signed Input Log `/checkpoint` against `` and the Input Log checkpoint policy to obtain `(latest_input_size, latest_input_root)` (re-fetching as needed when auditing newer Output Log entries). If policy or consistency verification fails, the checkpoint is rejected before reading any entries. -2. **Stream State Commitments & Verify Input Log Consistency**: Fetches new Output Log entry bundles from `/tile/entries/...` and authenticates them against the Output Log checkpoint root hash according to `tlog-tiles`. For each 6-line `c2sp.org/tlog-vindex-state@v1` Output Log entry in sequential order, parses `(input_origin, from, input_size, input_root, mpt_root)`, checks that `input_origin` and `from` match the auditor's configured parameters and `prev_size <= input_size <= latest_input_size`, and verifies that `(input_size, input_root)` is Merkle-consistent with `(latest_input_size, latest_input_root)`. -3. **Fetch, Map & Stage Input Entries**: For the Input Log entry range `[prev_size, input_size)`: - - Fetches Input Log entry bundles and authenticates them against `input_root` (or `latest_input_root`) according to `tlog-tiles` or `static-ct-api`, slicing boundary 256-entry bundles to `[prev_size, input_size)` so entries `< from` on non-256-aligned initial bundles and entries `>= input_size` on partial tail bundles are excluded. - - Executes the WASM `MapFn` on each Input Log entry at `entry_idx` in sequential order, validates that every emitted key is a canonical key (`1..1024` printable ASCII bytes in `0x21..0x7E`), and deduplicates keys emitted for the same `entry_idx`. - - Stages chunk mutations into an uncommitted **indexed Pebble batch** (`stagedBatch`), which reads through uncommitted batch state and the active chunk cache to return the updated `(canonicalKey, value_root)` mappings directly in memory without writing to the underlying database. - - **Replay Deduplication Invariant**: On restart or crash replay, any Input Log entry indices `< from` or already persisted in Pebble are filtered out prior to appending `value_log_leaf_hash(entry_idx)` to active chunks while still reconstructing their point-in-time `(canonicalKey, value_root)` pairs at `input_size` from the KV store for Step 4, ensuring idempotent crash recovery without false root mismatches. -4. **Predict Candidate MPT Root**: Evaluates the accumulated in-memory `(canonicalKey, value_root)` mappings at `input_size` against the local MPT using non-mutating prediction to compute the candidate local `mpt_root`. -5. **Root Equality Assertion & Commit**: Checks that the candidate local `mpt_root` equals the Output Log entry's `mpt_root`: - - If equal: atomically applies the staged Pebble batch (`'c'` chunks together with `audited_output_size` and `audited_input_size`, advancing `prev_size = input_size`) with a durable disk sync before committing and syncing MPT mutations (`KV_CP >= MPT_Durable_Size`). If the process crashes between the Pebble commit and MPT `fsync`, startup MPT catch-up replays `[max(MPT_Durable_Size, from), audited_input_size)` from local storage before auditing the next Output Log entry. - - If divergent: closes and discards the uncommitted Pebble batch, clears the in-memory active chunk cache, and enters the state-preserving halt protocol without advancing watermarks or mutating the verified MPT and Pebble state on disk. - -### 2.3 Root Mismatch Alerting & State-Preserving Halt -If the candidate local `mpt_root` differs from the committed `mpt_root`, or an upstream consistency/policy fault is detected, the auditor immediately executes a coordinated containment response: - -1. **Sync Engine Halt**: - - The background audit loop terminates immediately. - - Verified watermarks are **not** advanced on disk. -2. **Forensic State Freeze**: - - Discards the uncommitted Pebble staging batch and clears the in-memory chunk cache while leaving verified Pebble DB and MPT files intact on disk, freezing the exact pre-divergence state for forensic triage and public misbehavior reporting. -3. **Metrics & Structured Logging**: - - Increments the root mismatch counter metric and sets the root mismatch gauge to `1`. - - Emits structured error logs recording the divergent output entry index, target `input_size`, candidate local root, and committed root. -4. **Health and Readiness Probes (Internal Operational Listener)**: - - `/healthz`: Remains **HTTP 200 OK** as long as the auditor process is alive, preventing container orchestrators from restarting the halted process and disturbing frozen forensic state or diagnostic endpoints. - - `/readyz`: Returns **HTTP 503 Service Unavailable** on the internal operational listener with structured JSON diagnostics: +1. **State Freeze & Telemetry**: Terminates the background audit loop without advancing verified watermarks, leaves local Pebble and MPT files intact on disk for inspection, sets the root mismatch gauge metric to `1`, and logs the divergent `output_index`, `input_size`, `local_root`, and `committed_root`. +2. **Operational Probes (Internal Admin Listener)**: + - `/healthz`: Remains **`200 OK`** while the process is alive so container orchestrators do not restart the halted auditor and disturb frozen disk state. + - `/readyz`: Returns **`503 Service Unavailable`** with structured JSON diagnostics: ```json { "status": "degraded", @@ -99,78 +44,39 @@ If the candidate local `mpt_root` differs from the committed `mpt_root`, or an u "committed_root": "b7c2..." } ``` +3. **Operator Runbook**: + - Do not wipe storage; inspect `/readyz` and structured logs for the divergent `(output_index, input_size, local_root, committed_root)`. + - Compare local Pebble `'c'` chunks against raw Input Log entries in `[prev_size, input_size)` and re-run the auditor in one-shot mode with an isolated scratch directory to confirm deterministic reproduction. + - If `MapFn` and Input Log entries are valid, publish the misbehavior proof bundle (witnessed Output Log checkpoint, inclusion proof, raw `c2sp.org/tlog-vindex-state@v1` entry, and signed Input Log checkpoint + consistency proof) to witnesses and monitoring channels. -### 2.4 Adversarial Threat Vector Matrix +### 1.3 Adversarial Threat Matrix | Threat Vector | Adversary Action | Detection Mechanism | Auditor Action | | :--- | :--- | :--- | :--- | -| **Operator Equivocation** | Index operator serves conflicting Output Log checkpoints to different auditors or witnesses. | Output Log checkpoint policy verification; Merkle consistency proofs between Output Log checkpoints. | Audit loop halts immediately; alert logged; database frozen. | -| **Tree Size Rollback** | Index operator serves an Output Log checkpoint (or `input-size` commitment) smaller than previously audited watermarks. | Monotonic `output_size` and `prev_size <= input_size <= latest_input_size` assertions during audit poll. | Audit loop halts; watermarks frozen; `/healthz` remains HTTP 200; `/readyz` reports `503` degraded status. | -| **Forged MPT Root (Omission)** | Index operator commits an invalid `mpt_root` in an Output Log entry to hide or fabricate records. | Local MPT reconstruction from authenticated Input Log entries; root equality check against the Output Log entry. | Sets root mismatch alert metrics; discards uncommitted batch and freezes disk state; `/readyz` reports `503` degraded status. | -| **Corrupted Input Entries** | Network attacker or corrupted CDN alters raw Input Log entry bundles. | Entry bundle Merkle authentication fails against `input_root` / `latest_input_root` verified under the signed Input Log checkpoint. | Pipeline rejects entry bundle; retries transient fetch errors without advancing state. | -| **Unauthenticated Output Entries** | Index operator serves Output Log entry bundles that do not match the Output Log checkpoint root. | `tlog-tiles` entry bundle Merkle authentication fails against the Output Log checkpoint root hash. | Entry bundle rejected; audit loop terminates with an authentication error. | -| **Parameter Drift (`input-origin` / `from`)** | Index operator publishes an Output Log state commitment altering `input-origin` or `from`. | State commitment parser checks `input-origin` and `from` against the auditor's configured parameters on every Output Log entry. | Audit loop halts immediately; state frozen; alerts emitted. | - -### 2.5 In-Situ Invariants & Performance Optimizations - -- **[Correctness Invariant] Strict Root Equality Assertion**: - - *Rule*: For every processed Output Log entry, the candidate `mpt_root` computed from authenticated Input Log entries in `[from, input_size)` must equal the `mpt_root` committed in the `c2sp.org/tlog-vindex-state@v1` payload. - - *Rationale*: Guarantees that the index operator's published index state is 100% faithful to the Input Log. - - *Consequence ("Or Else")*: Accepting mismatched roots would allow a malicious or buggy index operator to conceal or fabricate key occurrences without detection. - -- **[Correctness Invariant] Monotonic Progression & Log Consistency**: - - *Rule*: Each successive Output Log entry must satisfy `from <= prev_size <= input_size <= latest_input_size`, Merkle consistency across consecutive Output Log checkpoints, and Merkle consistency of `(input_size, input_root)` against the verified signed Input Log checkpoint `(latest_input_size, latest_input_root)`. - - *Rationale*: Transparency logs are append-only; historical tree prefixes and sizes cannot shrink or rewrite prior entries. - - *Consequence ("Or Else")*: An index operator or upstream log could roll back or rewrite historical entries undetected. - -- **[Performance Optimization] High-Throughput Catch-Up Architecture**: - - *Mechanism*: Reuses the core batching pipeline (256-entry bundle fetching, Wazero worker pool, and KV batch staging). - - *Impact*: Sustains 90,000 to 240,000+ entries/sec, allowing a newly provisioned auditor to catch up over tens of millions of historical Input Log entries in hours. - -### 2.6 Operational Runbook: Responding to Mismatches -When a root mismatch alert fires (mismatch metric `== 1` or `/readyz` returning `HTTP 503` with `"auditor root hash mismatch"`): -1. **Triage & Containment**: The auditor has automatically halted forward sync and frozen database state on disk. Do not restart the auditor with cleared storage, as the disk preserves forensic state immediately prior to the divergent Output Log entry. -2. **Inspect Structured Logs**: Examine auditor logs and `/readyz` diagnostics to record the divergent output entry index, target `input_size`, `local_root`, and `committed_root`. -3. **Forensic State Extraction**: Inspect local Pebble DB inverted chunks (`'c'`) and compare against raw Input Log entries in `[prev_size, input_size)` to determine whether the index operator omitted entries, altered canonical keys, or suffered a non-deterministic `MapFn` bug. -4. **Reproduce via Standalone One-Shot Audit**: Run the auditor in one-shot mode with an isolated scratch directory against the target Output Log checkpoint to confirm deterministic reproducibility. -5. **Publish Misbehavior Bundle**: If `MapFn` and the Input Log entries are valid, publish the cryptographic proof bundle (witnessed Output Log checkpoint, Output Log inclusion proof, raw `c2sp.org/tlog-vindex-state@v1` entry, and the signed Input Log checkpoint + consistency proof) to transparency witnesses and public monitoring channels. - -### 2.7 Auditor Contract - -The auditor subsystem provides continuous full-log state verification: - -- **Inputs**: - - Output Log URL (``), expected ``, and Output Log checkpoint verification policy. - - Expected ``, Input Log checkpoint & entry retrieval endpoint, and Input Log checkpoint verification policy (used to verify signed Input Log checkpoints, `(input_size, input_root)` consistency proofs, and entry bundles). - - Compiled WASM `MapFn` module (pinned out-of-band by the auditor operator) and configured start index `from` (default `0`). - - Local Pebble DB and MPT storage directories. - - Internal operational listener serving `/healthz`, `/readyz`, `/metrics`, and `/debug/pprof/*`. -- **Outputs**: - - Continuous audit telemetry and `/readyz` JSON diagnostics on the internal operational listener. -- **Safety Guarantees**: - - Never signs or publishes Output Log checkpoints. - - Never commits unverified state to disk: candidate MPT roots are predicted from in-memory staged `(canonicalKey, value_root)` mappings and checked against the Output Log entry commitment before applying the staged Pebble batch, mutating the verified MPT, or advancing watermarks. - - Immediately halts forward sync and preserves forensic state on any checkpoint policy failure, Merkle consistency failure, tree size regression, `input-origin` or `from` mismatch, entry bundle authentication failure, invalid canonical key emission, or `mpt_root` mismatch. +| **Operator Equivocation** | Serves conflicting Output Log checkpoints to different auditors or witnesses. | Output Log checkpoint policy verification and Merkle consistency proofs between Output Log checkpoints. | Halts audit loop; logs alert; freezes database. | +| **Tree Size Rollback** | Serves an Output Log checkpoint (or `input-size` commitment) smaller than previously audited watermarks. | Monotonic `output_size` and `prev_size <= input_size <= latest_input_size` assertions. | Halts audit loop; freezes watermarks; `/readyz` reports `503`. | +| **Forged MPT Root (Omission)** | Commits an invalid `mpt_root` in an Output Log entry to hide or fabricate records. | Local MPT root calculation from authenticated Input Log entries compared against `mpt_root`. | Freezes disk state; sets mismatch metric; `/readyz` reports `503`. | +| **Corrupted Input Entries** | Alters raw Input Log entry bundles in transit or on CDN. | Entry bundle Merkle authentication fails against verified `input_root` / `latest_input_root`. | Rejects bundle; retries transient fetch errors without advancing state. | +| **Unauthenticated Output Entries** | Serves Output Log entry bundles that do not match the Output Log checkpoint root. | `tlog-tiles` entry bundle Merkle authentication fails against Output Log checkpoint root. | Rejects bundle; terminates audit loop with authentication error. | +| **Parameter Drift (`input-origin` / `from`)** | Publishes an Output Log state commitment altering `input-origin` or `from`. | Checks `input-origin` and `from` against configured parameters on every Output Log entry. | Halts audit loop immediately; freezes state; emits alerts. | --- -## 3. Alternatives Considered (or Tried) +## 2. Alternatives Considered -### 3.1 Lookup Client Verification Alone vs. Full-Log Independent Auditing -- **Proposed**: Relying exclusively on lookup clients (`vindex/client`) to verify individual query responses, omitting a full-log auditor. -- **Theoretical Rejection**: Lookup client verification proves inclusion or non-inclusion against `mpt_root` only when a key is queried, and cannot verify that `mpt_root` itself was faithfully constructed from the Input Log without omissions across unqueried keys. -- **Chosen Design**: Complementary verification: lightweight lookup client SDK ([`client/`](../../client/README.md)) for per-query proof verification against witnessed checkpoints, paired with `vindex/internal/auditor` for continuous full-log replay auditing. +### 2.1 Lookup Client Verification Alone vs. Full-Log Independent Auditing +- **Proposed**: Relying solely on lookup clients ([`client/`](../../client/README.md)) to verify individual query responses without running a full-log auditor. +- **Rejection**: Lookup clients verify proofs against `mpt_root` only for queried keys and cannot detect whether `mpt_root` omitted or fabricated entries across unqueried keys. +- **Chosen Design**: Complementary verification: per-query proof verification in [`client/`](../../client/README.md) paired with continuous full-log replay auditing in `vindex/internal/auditor` (and independent clean-room auditors). -### 3.2 Ephemeral In-Memory Auditing vs. State-Preserving Forensic Freezing -- **Proposed**: Storing auditor state ephemerally in RAM or wiping local database files upon detecting an upstream error. -- **Operational Rejection**: Wiping or discarding state upon detecting a forged `mpt_root` destroys the exact pre-divergence state needed to pinpoint omitted Input Log entries and produce a reproducible misbehavior report against the index operator. -- **Chosen Design**: State-preserving halt: the auditor freezes verified Pebble DB and MPT files in-place, exposes structured JSON diagnostics on `/readyz`, keeps `/healthz` at `HTTP 200 OK` so orchestrators do not restart the halted process, and stops forward sync. +### 2.2 Ephemeral In-Memory Auditing vs. State-Preserving Forensic Freeze +- **Proposed**: Storing auditor state purely in RAM or wiping local database files upon detecting a divergence. +- **Rejection**: Destroys the local state needed to isolate which Input Log entries were omitted or corrupted in `[prev_size, input_size)`. +- **Chosen Design**: State-preserving halt that freezes local Pebble and MPT files in-place without advancing verified watermarks. --- -## 4. Verification Gate - -All auditor replay, checkpoint policy verification, and root mismatch forensic freeze tests are verified via: +## 3. Verification Gate ```bash go test -race ./vindex/internal/auditor/... diff --git a/vindex/internal/coordinator/README.md b/vindex/internal/coordinator/README.md index b6a9aed..f701b8f 100644 --- a/vindex/internal/coordinator/README.md +++ b/vindex/internal/coordinator/README.md @@ -1,188 +1,81 @@ # Coordinator & Lifecycle Engine -This document defines the batch orchestration engine, watermark governance, Zero-WAL startup recovery protocols, load-bearing invariants, and verified performance optimizations for the **Coordinator Subsystem** (`vindex/internal/coordinator`). +`vindex/internal/coordinator` is the single-threaded lifecycle orchestrator and watermark authority for `vindexd`. Worker subsystems (tile ingestion, WASM mapping, KV chunk storage, MPT state commitment, and HTTP serving) expose narrow interfaces and never inspect each other's internal state. The Coordinator drives the end-to-end indexing loop in monotonic order, freezes target checkpoints so high-velocity logs cannot starve batch completion, computes the safe pruning watermark for the local tile cache, and orchestrates network-free Zero-WAL startup recovery and unserved Genesis Backfill. ---- - -## 1. Context & Objectives - -### 1.1 Problem Statement & Orchestration Dilemma -The VIndex architecture decomposes indexing into five specialized, asynchronous subsystems: -1. **`internal/ingest`**: Network fetching and local 256-entry bundle tile caching. -2. **`mapfn` & `internal/wasmhost`**: Isolated WebAssembly guest SDK (`mapfn`) and Wazero sandbox host execution (`internal/wasmhost`). -3. **`internal/kvstore`**: Pebble LSM inverted chunk storage. -4. **`internal/mpt` & `internal/tree`**: Authenticated MPT state trie (`internal/mpt`) and Output Log state commitment publication (`internal/tree`). -5. **`internal/server`**: C2SP HTTP read serving. - -Without a central orchestrator managing these components behind strict black-box interfaces, three critical failure modes emerge: -1. **Watermark Drift & Ambiguity**: If individual subsystems track state independently, cross-subsystem boundaries drift out of alignment. A crash between storage sync and Output Log publication leaves the node unable to determine where recovery begins. -2. **Moving-Goalpost Starvation**: In high-velocity logs appending thousands of entries per second, continuously polling the log head prevents the pipeline from ever finalizing a batch, causing synchronization starvation. -3. **Network-Dependent Crash Recovery**: If startup recovery relies on downloading historical tiles over the network, node restart times become unpredictable and fail completely during upstream network outages. - -### 1.2 Centralized Watermark Authority & Separation of Concerns -The Coordinator serves as the **single source of truth** for system lifecycle and watermark governance: -- Worker subsystems expose clean, black-box functional interfaces and do not inspect each other's internal metadata or storage engines. -- The Coordinator alone tracks the global progression chain, evaluates persistence boundaries, and calculates the authoritative `SafeWatermark` (`max(min(KV_CP.Size, MPT_Durable_Size), from)`) communicated to the tile cache reaper. - -### 1.3 Goals & Non-Goals -- **Goals**: - - Drive the steady-state batch processing loop across all pipeline stages in strictly monotonic sequence. - - Prevent moving-goalpost starvation by freezing target sync checkpoints prior to batch ingestion. - - Enforce the synchronous handoff: persist the KV checkpoint to disk, ratchet forward, predict the MPT change, and promote the state commitment entry to the Output Log. - - Guarantee deterministic, network-free Zero-WAL crash recovery (< 5ms clean start, < 500ms dirty crash recovery after `torchwood/mpt` journal replay completes on open). -- **Non-Goals**: - - **No Distributed Consensus**: Operates strictly within a single process; does not run Raft, Paxos, or distributed leader election. - - **No Direct In-Memory Mutation**: The Coordinator does not touch trie nodes or LSM chunk bytes directly; it coordinates subsystem boundaries. - - **No Arbitrary Runtime Mode Switching**: Operates Normal Serving Mode for all serving logs; Genesis Backfill is used strictly during unserved initial bootstrap when the Output Log is empty (`Output_CP.Size == 0`). - -### 1.4 Requirements & Dependencies -- **Dependencies**: Orchestrates the ingestion fetcher and local tile cache, WASM mapping pipeline (`internal/wasmhost`), KV chunk store (`internal/kvstore`), MPT state trie (`internal/mpt`), Output Log publisher (`internal/tree`), and checkpoint policy verification (`internal/wire`). +### Non-Goals +- **No Distributed Consensus**: Operates strictly within a single process on a single host; high availability is achieved externally by running independent VIndex instances against the shared Input Log. +- **No Direct Storage or Trie Mutation**: Coordinates subsystem boundaries and checkpoints without manipulating LSM chunk records or MPT nodes directly. +- **No Arbitrary Runtime Mode Switching**: Runs Normal Serving Mode once the Output Log has at least one state commitment entry (`Output_CP.Size > 0`); Genesis Backfill is used strictly during initial bootstrap when the Output Log is empty (`Output_CP.Size == 0`). --- -## 2. Detailed Design +## 1. Design + +### 1.1 The Master Batch Loop -### 2.1 The Master Batch Loop -The coordinator executes batch synchronization through a structured, 6-step sequential lifecycle: +Each steady-state sync cycle executes six sequential steps across a frozen target range (`>90,000 entries/sec` sustained throughput): | Step | Phase | Action & Subsystem Transition | Durability Transition | | :- | :--- | :--- | :--- | -| **1** | **Poll & Freeze** | Polls upstream Input Log `/checkpoint`, verifies the Input Log checkpoint against `` and the configured Input Log checkpoint policy (e.g., C2SP `tlog-policy`), checks `Target_CP.Size >= max(prev_Target_CP.Size, from)` and Input Log Merkle consistency against `prev_Target_CP`, and writes `Target_CP` to Pebble metadata. | Freezes target sync boundary. | -| **2** | **Tile Ingestion** | Downloads missing 256-entry bundles across `[max(KV_CP.Size, from), Target_CP.Size)` into the local tile cache. | Durably stages verified entry bundles on disk. | +| **1** | **Poll & Freeze** | Polls upstream Input Log `/checkpoint`, verifies the checkpoint against `` and the configured Input Log checkpoint policy (such as [C2SP `tlog-policy`](https://c2sp.org/tlog-policy)), checks `Target_CP.Size >= max(prev_Target_CP.Size, from)` and Merkle consistency against `prev_Target_CP`, and persists `Target_CP` to Pebble metadata. | Freezes target sync boundary to prevent moving-goalpost starvation on high-velocity logs. | +| **2** | **Tile Ingestion** | Downloads missing 256-entry bundles across `[max(KV_CP.Size, from), Target_CP.Size)` into the local tile cache ([`internal/ingest`](../ingest/README.md)). | Durably stages verified entry bundles on disk. | | **3** | **Parallel Mapping** | Streams 256-entry bundles to WASM workers (`internal/wasmhost`), validates un-hashed canonical keys (`1..1024` bytes in `0x21..0x7E`), and resequences batches into multi-tile commit slabs. | In-memory key-index aggregation. | -| **4** | **Storage Commit** | Streams commit slabs into the KV store (committing intermediate slabs without disk sync while accumulating modified `(key, value_root)` mappings across the target window, and executing a blocking disk `fsync` barrier on the final slab reaching `Target_CP.Size`). | **Ratchets persisted KV storage watermark durably to disk and persists `KV_CP` at `Target_CP.Size`.** | -| **5** | **Commitment & Promotion** | Once `Target_CP.Size` is reached and synced in Pebble, predicts candidate `mpt_root` over accumulated modified `(key, value_root)` mappings, appends one 6-line state commitment entry (`c2sp.org/tlog-vindex-state@v1`: `input-origin`, `from`, `input-size`, `input-root`, and `mpt-root`) for `Target_CP` to the Output Log (while retaining the signed Input Log checkpoint durably out-of-band), collects witness cosignatures per Output Log `tlog-policy`, and ratchets `Serving_CP` under the exclusive tree lock (< 5ms). | **Promotes state to Output Log (`Output_CP`) & Serving (`Serving_CP`).** | -| **6** | **Pruning Notification** | Computes `SafeWatermark` (`max(min(KV_CP.Size, MPT_Durable_Size), from)`) and notifies the tile cache reaper. | Bounded disk cache garbage collection. | +| **4** | **Storage Commit** | Streams commit slabs into [`internal/kvstore`](../kvstore/README.md) (committing intermediate slabs without disk sync while accumulating modified `(key, value_root)` mappings across the target window, and executing a blocking disk `fsync` barrier on the final slab reaching `Target_CP.Size`). | **Advances persisted KV watermark and commits `KV_CP` at `Target_CP.Size` prior to Output Log publication.** | +| **5** | **Commitment & Promotion** | Predicts candidate `mpt_root` over accumulated modified `(key, value_root)` mappings, appends one 6-line `c2sp.org/tlog-vindex-state@v1` entry (`input-origin`, `from`, `input-size`, `input-root`, `mpt-root`) for `Target_CP` to the Output Log (retaining the signed Input Log checkpoint durably out-of-band), collects witness cosignatures per Output Log `tlog-policy`, and ratchets `Serving_CP` under the exclusive tree lock (< 5ms; see [`internal/tree`](../tree/README.md)). | **Promotes state to Output Log (`Output_CP`) and Serving (`Serving_CP`).** | +| **6** | **Pruning Notification** | Computes `SafeWatermark = max(min(KV_CP.Size, MPT_Durable_Size), from)` and notifies the tile cache reaper. | Bounded disk cache garbage collection. | -### 2.2 Checkpoint Governance & Progression Chain +### 1.2 Checkpoint Progression Chain & Safe Pruning Watermark -#### The 4 Authoritative Checkpoints +| Checkpoint | Log Type | Committed Size | Persistence | Role in Pipeline | +| :--- | :--- | :--- | :--- | :--- | +| **`Target_CP`** | Input Log | `Target_CP.Size` (`input_size`) | Durable (Pebble metadata) | Frozen upper goalpost for the active sync cycle. | +| **`KV_CP`** | Input Log | `KV_CP.Size` | Durable (Pebble LSM) | Binds synced `'c'` chunk records to the verified Input Log state they cover. | +| **`Output_CP`** | Output Log | `Output_CP.Size` (`output_size`), committing `Output_CP.InputSize` | Durable (Tessera Output Log) | Witnessed state commitment (`input-origin`, `from`, `input-size`, `input-root`, `mpt-root`), with signed Input Log checkpoint retained out-of-band. | +| **`Serving_CP`** | Output Log Entry | `Serving_CP.InputSize` (`input_size`) | Volatile (atomic pointer) | Active snapshot exposed to HTTP lookup readers. | -| Checkpoint | Log Type | Committed Size | Persistence | Advancement Mechanism | Role in Pipeline | -| :--- | :--- | :--- | :--- | :--- | :--- | -| **`Target_CP`** | Input Log | `Target_CP.Size` (`input_size`) | Durable (synced in Pebble metadata) | Upstream poll, Input Log checkpoint policy & Merkle consistency check (`>= max(prev_Target_CP.Size, from)`) | Upper goalpost for current sync cycle; prevents moving-goalpost starvation. | -| **`KV_CP`** | Input Log | `KV_CP.Size` | Durable (synced in Pebble LSM) | Atomic flush of `'c'` chunk records | Durably binds indexed `'c'` chunk records to the verified Input Log state they cover. | -| **`Output_CP`** | Output Log | `Output_CP.Size` (`output_size`), committing `Output_CP.InputSize` (`input_size` in state commitment entry) | Durable (Tessera Output Log storage) | Append state commitment entry & witness cosigning | Cryptographically commits to index state (`input-origin`, `from`, `input-size`, `input-root`, and `mpt-root`), with the signed Input Log checkpoint retained out-of-band. | -| **`Serving_CP`** | Output Log Entry | `Serving_CP.InputSize` (`input_size`) | Volatile (In-memory atomic snapshot pointer) | Pointer swap via atomic state promotion under tree lock (< 5ms) | Active state exposed to client HTTP readers. | +The checkpoints obey a strict monotonic progression invariant across all execution and crash states: -**Monotonic Checkpoint Progression Invariant**: ```text -Target_CP.Size >= KV_CP.Size >= Output_CP.InputSize >= Serving_CP.InputSize +Target_CP.Size >= KV_CP.Size >= Output_CP.InputSize >= Serving_CP.InputSize >= MPT_Durable_Size ``` -Every relation is `>=`: -- `Target_CP.Size >= KV_CP.Size`: Batch aggregation advances toward the target checkpoint in discrete commit slices. -- `KV_CP.Size >= Output_CP.InputSize`: KV storage persistence strictly precedes Output Log publication. -- `Output_CP.InputSize >= Serving_CP.InputSize`: Output Log publication and witness collection execute outside reader locking; `Output_CP` advances before the in-memory `Serving_CP` pointer is swapped via atomic state promotion under the tree lock. - -#### Intermediate Buffers & Trailing Durability -- `Target_CP -> KV_CP`: Ingest stages 256-entry bundles on disk and through worker mapping queues; the persisted KV storage watermark accumulates across write batches until synced at `KV_CP.Size`. -- `KV_CP -> Output_CP`: Non-blocking MPT prediction calculates candidate `mpt_root`. -- `Output_CP -> Serving_CP`: In-memory atomic swap ratchets `Serving_CP` (< 5ms critical section). -- `Trailing Serving_CP`: `MPT_Durable_Size` `fsync`s to disk in the background, satisfying `Serving_CP.InputSize >= MPT_Durable_Size`. - -### 2.3 Authoritative Safe Pruning Watermark for Tile Cache Reclamation -To guarantee that the ingestion cache never deletes 256-entry bundles needed for crash recovery, the Coordinator calculates the authoritative safe pruning boundary `SafeWatermark = max(min(KV_CP.Size, MPT_Durable_Size), from)`. Because storage persistence strictly precedes Output Log publication and MPT persistence (`KV_CP.Size >= Output_CP.InputSize >= MPT_Durable_Size`), clamping the lower bound to `from` ensures that suffix indices (`from > 0`) never retain or replay pre-`from` tiles before their first durable MPT sync, while keeping all 256-entry bundles in `[max(MPT_Durable_Size, from), KV_CP.Size)` cached on disk until the MPT confirms a durable `fsync`. - -### 2.4 Moving-Goalpost Prevention -On high-traffic transparency logs, the log head advances continuously. If the coordinator polled unverified checkpoints on every iteration, the sync target would constantly move, starving downstream commitment. - -The Coordinator prevents this by writing the verified target checkpoint into Pebble DB metadata (`Target_CP`) prior to batch processing. The entire pipeline processes that fixed entry range to completion before the coordinator advances to a new target. - -### 2.5 Zero-WAL Startup Recovery Sequence & Uncommitted KV Writes -On daemon launch, the HTTP read server gates public lookup endpoints with `503 Service Unavailable` while opening the MPT replays any uncompacted `torchwood/mpt` journal frames on disk. Once journal replay finishes, the Coordinator computes `recoveryTarget = Output_CP.InputSize` when `Output_CP.Size > 0` (or `KV_CP.Size` when `Output_CP.Size == 0`) and executes a deterministic 3-phase recovery sequence: - -1. **Phase 1: Instant Warm Start (< 5ms after MPT open)**: - - Entered if and only if `Output_CP.Size > 0`, the MPT's persisted version equals `(Output_CP.InputSize, true)` (`exact == true` and `MPT_Durable_Size == Output_CP.InputSize`), and the trie root matches `mpt-root` in the latest Output Log state commitment entry. - - Clean shutdown is verified: activates `Serving_CP`, initializes watermarks, and **opens the HTTP Read Server immediately (< 5ms)**. -2. **Phase 2: Fast-Forward Tile Replay (< 500ms after MPT open)**: - - Entered whenever the persisted MPT lags `recoveryTarget` or is inexact (`version < recoveryTarget || !exact`): - - Streams missing historical 256-entry bundles across `[max(MPT_Durable_Size, from), recoveryTarget)` directly from the local disk tile cache. - - Maps replayed entries via `map_bundle` to collect distinct modified un-hashed canonical keys. - - Reconstructs value log roots (`value_root`) for modified keys at `recoveryTarget` from the KV store with **zero writes to the database**. - - Updates in-memory MPT nodes with sorted `(key, value_root)` pairs and flushes MPT persistence to disk. When `Output_CP.Size > 0`, asserts that the resulting `mpt_root` strictly matches the latest Output Log state commitment entry, activates `Serving_CP`, and **opens the Read Server (< 500ms)**; when `Output_CP.Size == 0`, public endpoints remain `503 Service Unavailable`. -3. **Phase 3: Background Catch-Up & Uncommitted KV Writes**: - - Resumes forward ingestion from `max(Serving_CP.InputSize, from)` when `Output_CP.Size > 0` (or from `max(KV_CP.Size, from)` in Genesis Backfill when `Output_CP.Size == 0`—including clean restarts mid-backfill where `version == KV_CP.Size && exact == true`) toward `Target_CP.Size`. - - **Handling Uncommitted KV Writes**: If a crash occurred in Normal Serving Mode after Pebble durably synced a batch to disk but before Output Log commitment (`KV_CP.Size > max(Serving_CP.InputSize, from)`), the next sync cycle streams starting from `max(Serving_CP.InputSize, from)`. When re-processing Input Log entries that were already written to disk, the KV store filters out already-persisted entry indices and reconstructs their `value_root` digests directly from Pebble without re-writing storage. This allows uncommitted batches to be safely published to the Output Log upon initial sync without duplicate entries or storage churn. - -### 2.6 In-Situ Invariants & Performance Optimizations - -- **[Correctness Invariant] Synchronous Stage Promotion**: - - *Rule*: Inverted chunk storage writes must complete a durable disk sync and commit `KV_CP` before Output Log publication is permitted to execute (`KV_CP.Size >= Output_CP.InputSize`). - - *Rationale*: Guarantees `KV_CP.Size >= Output_CP.InputSize` across all crash scenarios. - - *Consequence ("Or Else")*: A crash between Output Log publishing and storage sync would publish a state commitment entry referencing missing or rolled-back KV chunks, permanently breaking inclusion proofs for witnessed checkpoints. - -- **[Correctness Invariant] Zero-WAL Startup Durability Guarantee**: - - *Rule*: Because `KV_CP.Size >= Output_CP.InputSize` (and cached tiles cover `[max(MPT_Durable_Size, from), KV_CP.Size)`), crash recovery must resolve all lagging MPT state using the local tile cache and local Pebble DB records without issuing external network requests. - - *Rationale*: Protects node availability against upstream log outages and guarantees predictable cold-boot timing. - - *Consequence ("Or Else")*: A node recovering from a crash during an upstream network partition would fail to boot, causing extended service downtime. - -- **[Performance Optimization] Multi-Tile Slab Aggregation & Target-Level Witnessing**: - - *Mechanism*: Buffers mapped Input Log entries into multi-tile commit slabs without per-slab disk syncs on intermediate slabs, executing a single durable KV sync and Output Log witness cycle at `Target_CP.Size`. - - *Impact*: Amortizes disk write latency and external witness network RPCs across the sync window, sustaining high indexing throughput (>90,000 entries/sec). - -### 2.7 Coordination Contract & State Guarantees - -The coordinator acts as the single source of truth for pipeline progression and lifecycle management: - -- **Inputs**: - - Input Log fetcher and configured Input Log checkpoint policy. - - Configured `` and start index `from` (default `0`). - - Local tile cache. - - Mapping pipeline (`internal/wasmhost`). - - KV store (`internal/kvstore`). - - MPT state trie (`internal/mpt`). - - Output Log publisher (`internal/tree`) and configured Output Log checkpoint policy. -- **Recovery Contract**: - - Enforces 3-phase startup recovery: - 1. Verifies whether persisted trie state matches the committed Output Log watermark (`Output_CP.InputSize`). - 2. Replays uncommitted 256-entry bundles across `[max(MPT_Durable_Size, from), recoveryTarget)` from the local disk cache to align the MPT with zero database writes. - 3. Resumes ingestion without duplicating persisted writes. -- **Durability Invariant**: - - Storage durability strictly precedes public Output Log commitment (`KV_CP.Size >= Output_CP.InputSize`). -- **Concurrency**: - - The background polling loop runs synchronously per iteration; catch-up and steady-state ingestion are strictly single-threaded. - -### 2.8 Genesis Backfill Lifecycle (Unserved Bootstrap) - -When bootstrapping an unserved index from its configured start index `from` (`Output_CP.Size == 0`) against a large Input Log, accumulating every modified `(key, value_root)` pair across `[from, Target_CP.Size)` in memory before a single final commit would exhaust host RAM, while publishing and witnessing an Output Log state commitment entry on every intermediate batch would incur thousands of unnecessary external witness roundtrips before the index is ready to serve. - -When `Output_CP.Size == 0`, the Coordinator executes a synchronous **Genesis Backfill** flow: - -1. **Gating & Serving Plane Isolation (`Output_CP.Size == 0`)**: - - Activated automatically if and only if the Output Log has zero state commitment entries (`Output_CP.Size == 0`). - - Throughout Genesis Backfill, public `/keys` and `/values` endpoints and readiness probes remain `503 Service Unavailable` per [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md). - -2. **Bounded Coalescing & Synchronous Coarse Checkpointing (Suppressing Intermediate Output Log Appends)**: - - Multi-tile batches commit inverted `'c'` chunk updates to Pebble without per-batch `fsync` while coalescing modified `(key, value_root)` mappings in a bounded in-memory map and **suppressing intermediate Output Log appends and witness roundtrips**. - - Whenever the modified-key count or entry threshold is reached, the Coordinator executes a synchronous Pebble disk sync (advancing `KV_CP.Size`) followed immediately by a synchronous MPT commit and disk sync (advancing `MPT_Durable_Size` strictly after Pebble `fsync` so `KV_CP.Size >= MPT_Durable_Size` always holds even across power loss), clears the modified-key map, and updates `SafeWatermark` (`max(min(KV_CP.Size, MPT_Durable_Size), from)`) so the tile cache reaper prunes superseded tiles. - -3. **Genesis Finalization & Output Log `Entry 0` Publication**: - - Upon reaching `Target_CP.Size`, the Coordinator executes a final synchronous Pebble sync and MPT commit/sync, publishes Output Log `Entry 0` (`c2sp.org/tlog-vindex-state@v1` binding `input-origin`, `from`, `input-size`, `input-root`, and `mpt-root`), collects witness cosignatures, promotes `Serving_CP`, and permanently transitions to Normal Serving Mode (`Output_CP.Size > 0`). - -4. **Standard Phase 2 Crash Recovery**: - - If the process crashes during Genesis Backfill (`Output_CP.Size == 0` and `version < KV_CP.Size || !exact`), startup reuses standard Phase 2 recovery across `[max(MPT_Durable_Size, from), KV_CP.Size)`—replaying cached 256-entry bundles and reconstructing `value_root` digests from Pebble—with zero special-case backfill recovery machinery before resuming synchronous backfill from `KV_CP.Size` toward `Target_CP.Size`. + +- **Storage Precedes Commitment (`KV_CP.Size >= Output_CP.InputSize`)**: Pebble must complete a durable `fsync` before the Output Log entry is published, ensuring a crash never leaves a witnessed Output Log entry referencing missing KV chunks. +- **Authoritative Safe Pruning Watermark**: The Coordinator instructs the tile cache reaper to prune only bundles below `SafeWatermark = max(min(KV_CP.Size, MPT_Durable_Size), from)`. Clamping to `from` ensures suffix indices (`from > 0`) never retain or replay pre-`from` tiles, while keeping all 256-entry bundles in `[max(MPT_Durable_Size, from), KV_CP.Size)` cached on local disk until the MPT confirms a durable `fsync`. + +### 1.3 Zero-WAL Startup Recovery Sequence + +On daemon launch, the HTTP read server gates public lookup endpoints with `503 Service Unavailable` while opening the MPT replays any uncompacted `torchwood/mpt` journal frames on disk. Once journal replay finishes, the Coordinator computes `recoveryTarget = Output_CP.InputSize` when `Output_CP.Size > 0` (or `KV_CP.Size` when `Output_CP.Size == 0`) and executes a network-free 3-phase recovery sequence: + +1. **Phase 1: Instant Warm Start (< 5ms after MPT open)**: Entered if and only if `Output_CP.Size > 0`, the MPT's persisted version equals `(Output_CP.InputSize, true)` (`exact == true` and `MPT_Durable_Size == Output_CP.InputSize`), and the trie root matches `mpt-root` in the latest Output Log state commitment entry. Activates `Serving_CP`, initializes watermarks, and **opens the HTTP Read Server immediately (< 5ms)**. +2. **Phase 2: Fast-Forward Tile Replay (< 500ms after MPT open)**: Entered whenever the persisted MPT lags `recoveryTarget` or is inexact (`version < recoveryTarget || !exact`). Streams missing 256-entry bundles across `[max(MPT_Durable_Size, from), recoveryTarget)` from the local disk tile cache, maps them via `map_bundle` to collect distinct modified canonical keys, reconstructs point-in-time `value_root` digests at `recoveryTarget` from Pebble with **zero database writes**, commits sorted `(key, value_root)` updates to the MPT, and syncs the MPT to disk. When `Output_CP.Size > 0`, asserts that the resulting `mpt_root` matches the latest Output Log entry, activates `Serving_CP`, and **opens the Read Server (< 500ms)**; when `Output_CP.Size == 0`, public endpoints remain `503 Service Unavailable`. +3. **Phase 3: Background Catch-Up & Uncommitted KV Writes**: Resumes forward ingestion from `max(Serving_CP.InputSize, from)` when `Output_CP.Size > 0` (or from `max(KV_CP.Size, from)` in Genesis Backfill when `Output_CP.Size == 0`, including clean restarts mid-backfill where `version == KV_CP.Size && exact == true`) toward `Target_CP.Size`. If a crash occurred in Normal Serving Mode after Pebble synced a batch to disk but before Output Log publication (`KV_CP.Size > max(Serving_CP.InputSize, from)`), re-streaming from `max(Serving_CP.InputSize, from)` skips already-persisted entry indices in Pebble while reconstructing their `value_root` digests so uncommitted batches are published cleanly without duplicate offsets. + +### 1.4 Genesis Backfill Lifecycle (`Output_CP.Size == 0`) + +When bootstrapping an unserved index from `from` (`Output_CP.Size == 0`) against a large Input Log, accumulating every modified `(key, value_root)` pair across `[from, Target_CP.Size)` in memory before a single final commit would exhaust host RAM, while publishing and witnessing an Output Log entry on every intermediate batch would incur thousands of unnecessary external witness roundtrips before the index is ready to serve. When `Output_CP.Size == 0`, the Coordinator executes synchronous **Genesis Backfill**: + +1. **Serving Plane Gating**: Activated automatically if and only if `Output_CP.Size == 0`. Public `/keys` and `/values` endpoints and `/readyz` remain `503 Service Unavailable` throughout backfill. +2. **Bounded Coalescing & Coarse Local Checkpoints**: Multi-tile batches write inverted `'c'` chunk updates to Pebble without per-batch `fsync` while coalescing modified `(key, value_root)` mappings in a bounded in-memory map and **suppressing intermediate Output Log appends and witness roundtrips**. Whenever the modified-key count or entry threshold is reached (or on graceful shutdown), the Coordinator executes a synchronous Pebble disk sync (advancing `KV_CP.Size`) followed immediately by a synchronous MPT commit and disk sync (advancing `MPT_Durable_Size` strictly after Pebble `fsync` so `KV_CP.Size >= MPT_Durable_Size`), clears the modified-key map, and updates `SafeWatermark` so the tile cache reaper prunes superseded tiles. +3. **Genesis Finalization (`Entry 0`)**: Upon reaching `Target_CP.Size`, the Coordinator executes a final synchronous Pebble sync and MPT commit/sync, publishes Output Log `Entry 0` (`c2sp.org/tlog-vindex-state@v1`), collects witness cosignatures, promotes `Serving_CP`, and permanently transitions to Normal Serving Mode (`Output_CP.Size > 0`). +4. **Crash Recovery During Backfill**: If the process crashes during Genesis Backfill (`Output_CP.Size == 0` and `version < KV_CP.Size || !exact`), startup reuses standard Phase 2 recovery across `[max(MPT_Durable_Size, from), KV_CP.Size)` before resuming backfill from `KV_CP.Size` toward `Target_CP.Size`. --- -## 3. Alternatives Considered (or Tried) +## 2. Alternatives Considered -### 3.1 Decentralized Subsystem Watermarking vs. Centralized Coordinator Governance -- **Proposed**: Allowing each subsystem to independently determine when to advance watermarks and prune historical state. -- **Theoretical Rejection**: Decentralized state tracking creates race conditions during dirty crashes. Subsystems cannot establish whether sibling components successfully persisted data, leading to either unrecoverable data loss or redundant disk storage. -- **Chosen Design**: Centralized Coordinator governance with a single authoritative progression chain. +### 2.1 Decentralized Subsystem Watermarking vs. Centralized Coordinator Governance +- **Proposed**: Allowing each subsystem to advance watermarks and prune historical state independently. +- **Rejection**: Decentralized tracking creates recovery races on dirty crashes because individual subsystems cannot determine whether downstream components persisted a batch. +- **Chosen Design**: Centralized Coordinator governance with a single monotonic progression chain. -### 3.2 Distributed Consensus Orchestration (Raft/Paxos) vs. Single-Host Coordinator -- **Proposed**: Coordinating indexing across a distributed cluster using Raft consensus. -- **Theoretical & Architectural Rejection**: - - The MPT already requires single-host RAM/mmap locality; clustering the coordinator adds immense operational complexity without solving tree state synchronization. - - High availability is achieved externally: independent operators run standalone VIndex instances against the shared Input Log. +### 2.2 Distributed Consensus Orchestration (Raft/Paxos) vs. Single-Host Coordinator +- **Proposed**: Coordinating indexing across a multi-node cluster via Raft. +- **Rejection**: The MPT requires single-host RAM/mmap locality, and high availability is achieved externally by running independent VIndex operator instances against the shared Input Log. - **Chosen Design**: Single-process, single-host embedded coordinator. --- -## 4. Verification Gate - -All coordinator recovery, genesis backfill, and batch loop invariants are verified via: +## 3. Verification Gate ```bash go test -race ./vindex/internal/coordinator/... diff --git a/vindex/internal/hammer/README.md b/vindex/internal/hammer/README.md index 0c0a002..53720e3 100644 --- a/vindex/internal/hammer/README.md +++ b/vindex/internal/hammer/README.md @@ -1,120 +1,66 @@ # Hammer Load Testing & Verification Framework -This document defines the architecture, workload generation models, drip-feed scheduling protocols, cryptographic invariant verification rules, and operational telemetry for the internal **Hammer Integration Load-Testing & Verification Package** (`vindex/internal/hammer`). +Unit tests validate isolated package logic, but race conditions, LSM compaction stalls, writer lock contention, and 65,536-entry chunk boundary bugs only surface under sustained concurrent read-write load. `vindex/internal/hammer` is the end-to-end load-testing and verification harness for `vindexd`: it acts simultaneously as a synthetic upstream Input Log producer (serving signed checkpoints and tiled entry bundles across programmable release schedules) and a pool of concurrent lookup clients that cryptographically verify 100% of responses via [`vindex/client`](../../client/README.md). ---- - -## 1. Context & Objectives - -### 1.1 Problem Statement & Integration Testing Dilemma -Unit tests verify isolated package logic, but critical failure modes in a verifiable index only emerge under **sustained, concurrent read-write load**: -1. **LSM Compaction Stalls**: Heavy write ingestion triggers background Pebble LSM compactions that can spike read latency or stall batch commits. -2. **Lock Contention & Race Conditions**: High-frequency updates to hot keys stress in-memory MPT path compression and exclusive writer lock transitions, risking deadlocks or tail-latency regressions against concurrent HTTP lookups. -3. **Chunk Boundary Rollovers**: Inverted Value Log chunks roll over every 65,536 Input Log entries; verifying that multi-chunk backward pagination reconstructs complete, unbroken Value Logs requires driving individual hot keys across multiple 65,536-entry boundaries. - -Validating that `vindexd` preserves cryptographic integrity, zero data loss, and sub-millisecond read serving requires an internal integration load-testing and verification package (`vindex/internal/hammer`) that simultaneously acts as the upstream Input Log producer and a pool of concurrent lookup clients. - -### 1.2 Goals & Non-Goals -- **Goals**: - - Provide an internal integration load-testing and verification harness (`vindex/internal/hammer`) for stressing `vindexd` under multi-million-entry workloads with heavy Zipfian key skew. - - Act simultaneously as the upstream Input Log producer (serving signed Input Log checkpoints and `tlog-tiles` entry bundles under steady, burst, and pause/catch-up drip schedules) and a pool of concurrent lookup clients. - - Assert [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md) cryptographic invariants (Output Log checkpoint policy and inclusion proof verification, MPT point/prefix proofs, Value Log compact range roots, and monotonic history) on 100% of lookup queries. - - Render real-time terminal telemetry tracking ingestion throughput (entries/sec), read QPS, latency percentiles, and invariant checks. -- **Non-Goals**: - - **Not a Public SDK Package**: `vindex/internal/hammer` is strictly an internal test and benchmark harness, not an external library API. - - **Not a Production Log Sequencer**: Designed exclusively for integration testing, benchmarking, and chaos verification. - - **Not an In-Memory Mock**: Executes real HTTP requests over loopback sockets against real POSIX files. - -### 1.3 Requirements & Dependencies -- **Dependencies**: Integrates `tessera` (POSIX storage, appender, publication awaiter) and [`client/`](../../client/README.md) (`vindex/client`). +### Non-Goals +- **Not a Public SDK Package**: Strictly an internal load-testing and verification harness. +- **Not a Production Log Sequencer**: Generates synthetic workloads for stress testing, benchmarking, and chaos verification. +- **Not an In-Memory Mock**: Executes real HTTP requests over loopback sockets against real POSIX tiles, Pebble SSTables, and `mmap` MPT files. --- -## 2. Detailed Design +## 1. Design -### 2.1 Component Architecture & Pipeline Map +### 1.1 Component Architecture -| Subsystem Component | Core Responsibility | +| Component | Responsibility | | :--- | :--- | -| **Synthetic Entry Generator** | Emits structured synthetic Input Log entries across Zipfian/Pareto and uniform key distributions in configurable leaf formats (`raw`, `sumdb`, or `ct`), and provides disjoint non-inclusion probe keys for reader workers. | -| **Tessera Input Log Sequencer** | Writes generated Input Log entries into a local POSIX append-only log, computes Merkle roots, and signs Input Log checkpoints with a configurable or generated Ed25519 signer (printing the verifier key on startup for `vindexd`). | -| **Drip-Feed HTTP Server** | Serves standard C2SP `tlog-tiles` endpoints (`/checkpoint` and `/tile/...`) for the synthetic Input Log and schedules checkpoint visibility across steady, burst, and pause/catch-up modes. | -| **Verifying Concurrent Readers** | Queries `vindexd` (`GET /keys` and `GET /values`) via [`client/`](../../client/README.md) across hot, uniform, non-inclusion, and full-pagination query mixes, cryptographically verifying 100% of lookup responses. | -| **Metrics Analyzer & Dashboard** | Aggregates latency percentiles, read QPS, ingestion lag, and invariant assertions into periodic terminal summaries. | - ---- - -### 2.2 Synthetic Entry Generation & Leaf Formats -The entry generator synthesizes Input Log entries in raw key/value (`raw`), Go SumDB (`sumdb`, ` h1:`), or Certificate Transparency (`ct`, X.509/precertificate) formats across two write distributions: - -1. **Zipfian / Pareto Skew (`alpha > 1`)**: - - Models realistic registries where the hottest 1% of canonical keys account for over 80% of all Input Log entries. - - **Targeted Stress**: Drives individual hot keys across multiple 65,536-entry chunk boundaries, verifying that chunk rollovers (`^0 -> ^1`), Pebble SSTable compactions, and multi-page backward `/values` scans preserve unbroken Value Log continuity. -2. **Uniform Distribution**: - - Spreads Input Log entries evenly across a wide canonical keyspace (`[0, K)`) to stress MPT branch fanout and single-occurrence chunk creation. - ---- - -### 2.3 Tessera Input Log Sequencer & Drip-Feed HTTP Server -To evaluate `vindexd` under varied upstream conditions, the hammer controls Input Log checkpoint visibility: - -- **POSIX Storage & Signer**: Appends generated Input Log entries to a local directory using Tessera POSIX storage, signing Input Log checkpoints with a configurable or freshly generated Ed25519 key (and printing the corresponding verifier key at startup so `vindexd` can configure its Input Log checkpoint policy). -- **Drip-Feed Schedules**: - - **Steady Drip**: Releases queued Input Log checkpoints at a fixed rate, simulating steady upstream log integration. - - **Burst Mode**: Buffers generated checkpoints and releases them in sudden bursts, simulating upstream batch publications. - - **Pause & Catch-Up Mode**: Holds checkpoint publication for a configurable window so `vindexd` idles, then exposes a deep backlog of 256-entry bundles to benchmark catch-up throughput. - ---- - -### 2.4 Verifying Concurrent Readers -A pool of concurrent reader workers issues continuous point and prefix lookups against `vindexd` (`GET /keys?key=`, `GET /keys?prefix=`, and `GET /values?key=&before=`), validating every response through [`client/`](../../client/README.md) configured with the expected ``, ``, and Output Log checkpoint policy. - -#### Reader Query Mix: -Each reader query selects a target key according to a weighted workload mix: -- **Hot Keys (~60%)**: Samples heavily skewed Zipfian keys and verifies the newest `/values` page (single-page break). +| **Synthetic Entry Generator** | Emits synthetic Input Log entries across Zipfian/Pareto (`alpha > 1`) and uniform key distributions in configurable leaf formats (`raw`, `sumdb`, or `ct`), plus disjoint non-inclusion probe keys (`nonexistent/`). | +| **Tessera Input Log Sequencer** | Appends generated entries into a local POSIX log, computes Merkle roots, and signs Input Log checkpoints with a configurable or freshly generated Ed25519 key (printing the verifier key at startup for `vindexd`). | +| **Drip-Feed HTTP Server** | Serves standard [C2SP `tlog-tiles`](https://c2sp.org/tlog-tiles) endpoints (`/checkpoint` and `/tile/...`) and schedules checkpoint visibility across steady, burst, and pause/catch-up modes. | +| **Verifying Concurrent Readers** | Queries `vindexd` concurrently via [`vindex/client`](../../client/README.md) across hot, uniform, non-inclusion, and full-pagination query mixes, asserting cryptographic proof validity and monotonic history invariants. | +| **Metrics Analyzer & Dashboard** | Aggregates latency percentiles, read QPS, ingestion lag, operational status counts (`503`/`413`), and invariant checks into periodic terminal summaries. | + +### 1.2 Workload Distributions & Drip-Feed Schedules + +- **Synthetic Entry Distributions & Formats**: Generates entries in raw key/value (`raw`), Go SumDB (`sumdb`, ` h1:`), or Certificate Transparency (`ct`, X.509/precertificate) formats: + - **Zipfian / Pareto Skew (`alpha > 1`)**: Concentrates >80% of entries onto the hottest 1% of keys, driving individual hot keys across multiple 65,536-entry chunk boundaries (`^0 -> ^1`) to stress chunk rollovers, Pebble compactions, and multi-page backward `/values` scans. + - **Uniform Distribution**: Spreads entries evenly across `[0, K)` to stress MPT branch fanout and single-occurrence chunk creation. +- **Drip-Feed Checkpoint Schedules**: + - **Steady Drip**: Releases Input Log checkpoints at a fixed rate to test steady-state ingestion and read concurrency. + - **Burst Mode**: Buffers checkpoints and releases them in sudden bursts to simulate batch upstream publications. + - **Pause & Catch-Up Mode**: Holds checkpoint publication so `vindexd` idles, then exposes a deep backlog of 256-entry bundles to benchmark catch-up throughput. + +### 1.3 Verifying Concurrent Readers & Fault Discrimination + +Concurrent reader workers query `vindexd` (`LookupKey`, `LookupPrefix`, and `MatchedKey.Values(ctx)`) using [`vindex/client`](../../client/README.md) configured with the expected ``, ``, and Output Log checkpoint policy across a weighted query mix: +- **Hot Keys (~60%)**: Samples skewed Zipfian keys and verifies the newest `/values` page (single-page `break`). - **Uniform Keys (~25%)**: Samples uniformly across the active keyspace and verifies the newest `/values` page. -- **Non-Inclusion Probes (~10%)**: Samples synthetic keys (`nonexistent/`) never appended to the Input Log, exercising cryptographic MPT non-inclusion and empty-prefix proofs (`0` matched key lines in `c2sp.org/tlog-vindex-keys@v1`). -- **Full Backward Pagination (~5%)**: Samples a hot key and walks `MatchedKey.Values(ctx)` all the way back to `preceding == 0` across multiple 65,536-entry chunks. +- **Non-Inclusion Probes (~10%)**: Queries synthetic `nonexistent/` keys never appended to the Input Log, exercising MPT non-inclusion and empty-prefix proofs. +- **Full Backward Pagination (~5%)**: Walks a hot key's `MatchedKey.Values(ctx)` iterator all the way back to `preceding == 0` across multiple 65,536-entry chunks. -#### Query Verification Sequence: -1. **Output Log Proof & State Commitment**: `Client.LookupKey` / `Client.LookupPrefix` verifies `` and `c2sp.org/tlog-proof@v1` against `ExpectedOutputLogOrigin` and `OutputLogPolicy` (asserting `output_size >= 1` and canonical decimal `index == output_size - 1`) and parses the 6-line `c2sp.org/tlog-vindex-state@v1` record (`input-origin == ExpectedInputLogOrigin` and `0 <= from <= input_size`). -2. **MPT Proof Verification**: Verifies the `c2sp.org/tlog-vindex-keys@v1` binary MPT inclusion, non-inclusion, or two-sided prefix proof against `mpt_root`, returning an immutable `*client.LookupResult` snapshot. -3. **Value Log Compact Range Verification**: Iterates `MatchedKey.Values(ctx)` (either stopping after the newest page on normal hot/uniform probes or walking to `preceding == 0` on full-pagination probes), reconstructing and verifying the RFC 6962 Value Log root from `preceding` compact range hashes and returned Input Log entry indices on every fetched page. -4. **Monotonic History & Parameter Assertions**: Asserts that `LookupResult.From` never regresses across queries. Whenever a query observes the **complete** Value Log history for canonical key `K` (either a single-page result with `preceding == 0` or a full backward pagination walk to `preceding == 0`), the reader records the chronologically ordered entry index slice `I_new` at `input_size_new` and asserts that for any prior complete observation `I_old` at `input_size_old <= input_size_new`, `I_new` is a prefix-preserving superset of `I_old` (`len(I_new) >= len(I_old)` and `I_new[:len(I_old)] == I_old`). +#### Verification & Monotonic History Assertions +1. **Cryptographic Proof Verification (`vindex/client`)**: Every query verifies the Output Log checkpoint policy and inclusion proof at `index == output_size - 1`, the 6-line `c2sp.org/tlog-vindex-state@v1` commitment, the binary MPT inclusion/non-inclusion/prefix proof against `mpt_root`, and the RFC 6962 Value Log compact range root on every fetched `/values` page. +2. **Monotonic History Superset & `From` Non-Regression**: Readers assert that `LookupResult.From` never regresses across queries. Whenever a query observes the **complete** Value Log history for canonical key `K` (either a single-page result with `preceding == 0` or a full backward walk to `preceding == 0`), the reader records the chronologically ordered entry index slice `I_new` at `input_size_new` and asserts that for any prior complete observation `I_old` at `input_size_old <= input_size_new`, `I_new` is a prefix-preserving superset of `I_old` (`len(I_new) >= len(I_old)` and `I_new[:len(I_old)] == I_old`). +3. **Fault Discrimination**: Any verification error (`client.ErrVerificationFailed`) or reader-side consistency failure (`From` regression or monotonic history superset violation) immediately records a fatal invariant violation and fails the hammer run, whereas transient operational status codes (`client.ErrUnavailable` / `HTTP 503` during cold startup or `client.ErrTooManyKeys` / `HTTP 413` on broad prefix limits) are tracked in telemetry without failing the run. --- -### 2.5 In-Situ Invariants & Performance Assertions +## 2. Alternatives Considered -- **[Correctness Invariant] Verification & Consistency Fault Discrimination**: - - *Rule*: Any verification error from `vindex/client` (`ErrVerificationFailed`) or reader-side consistency failure (`From` regression or monotonic history superset violation) immediately records a fatal invariant violation and fails the hammer run, whereas transient operational status codes (`503 Service Unavailable` during startup/backfill or `413 Content Too Large` on broad prefix limits) are tracked in telemetry without failing the run. - - *Rationale*: A verifiable index must never serve forged, torn, or unverifiable responses under any read/write concurrency stress, while legitimately returning `503` before its first Output Log checkpoint is published. - - *Consequence ("Or Else")*: Tolerating cryptographic proof errors would mask race conditions between MPT snapshots, Output Log witnessing, and Pebble chunk flushes, while treating initial `503` responses as cryptographic faults would cause false failures during cold startup. - -- **[Correctness Invariant] Monotonic History Superset on Complete Scans**: - - *Rule*: For repeated complete Value Log observations (`preceding == 0` reached) on canonical key `K` where `input_size_new >= input_size_old`, `I_new` must be a superset of `I_old` with an identical prefix (`I_new[:len(I_old)] == I_old`). - - *Rationale*: Input Logs and per-key Value Logs are strictly append-only; committed Input Log entry indices can never vanish, shrink, or reorder. - - *Consequence ("Or Else")*: Any violation proves chunk rollover corruption, out-of-order batch commits, or broken snapshot isolation. - ---- - -## 3. Alternatives Considered (or Tried) - -### 3.1 In-Process Go Unit Test Mocks vs. Out-of-Process End-to-End Harness -- **Proposed**: Stress-testing `vindexd` purely in-memory using mocked storage and channel harnesses. -- **Practical Rejection**: In-memory mocks miss real concurrency and I/O dynamics—loopback HTTP connection pooling, OS scheduler preemption, `mmap` page flushing, and Pebble LSM compaction stalls. +### 2.1 In-Process Unit Test Mocks vs. End-to-End Loopback Harness +- **Proposed**: Stress-testing `vindexd` purely in memory using mocked storage and channel harnesses. +- **Rejection**: In-memory mocks miss real concurrency and I/O dynamics: loopback HTTP connection pooling, OS scheduler preemption, `mmap` page flushing, and Pebble LSM compaction stalls. - **Chosen Design**: Real HTTP client/server execution backed by POSIX Tessera tiles, Pebble SSTables, and `mmap` MPT files. -### 3.2 Static Pre-Recorded Log Dumps vs. Dynamic Drip-Feed Sequencer +### 2.2 Static Pre-Recorded Log Dumps vs. Dynamic Drip-Feed Sequencer - **Proposed**: Replaying a fixed snapshot of an existing public transparency log. -- **Practical Rejection**: Static dumps cannot modulate write rates on demand, dial exact Zipfian skew to force 65,536-entry chunk rollovers, inject targeted non-inclusion probes, or orchestrate pause/burst checkpoint schedules. +- **Rejection**: Static dumps cannot modulate write rates on demand, dial exact Zipfian skew to force 65,536-entry chunk rollovers, inject targeted non-inclusion probes, or orchestrate pause/burst checkpoint schedules. - **Chosen Design**: Deterministic synthetic entry generator paired with a programmable drip-feed `tlog-tiles` sequencer. --- -## 4. Verification Gate - -All internal hammer generator, sequencer, and concurrent client verification tests are verified via: +## 3. Verification Gate ```bash go test -race ./vindex/internal/hammer/... diff --git a/vindex/internal/ingest/README.md b/vindex/internal/ingest/README.md index 6029db2..c0185e8 100644 --- a/vindex/internal/ingest/README.md +++ b/vindex/internal/ingest/README.md @@ -1,111 +1,39 @@ # Ingestion Pipeline & Local Tile Cache -This document defines the architecture, data structures, concurrency model, load-bearing invariants, and verified performance optimizations for the **Ingestion Subsystem** (`vindex/internal/ingest`). +`vindex/internal/ingest` is the front-end pipeline that turns a remote append-only Input Log into a verified, strictly ordered stream of `(canonicalKey, entryIdx)` batches for downstream storage. It verifies upstream checkpoints and Merkle consistency against the configured Input Log policy, downloads and authenticates 256-entry bundles across parallel fetchers, caches verified bundles on local disk so crash recovery completes without network egress, maps bundles in parallel across sandboxed WebAssembly workers, and resequences out-of-order worker completions back into contiguous entry index order. ---- - -## 1. Context & Objectives - -### 1.1 Problem Statement & Real-World Friction -The ingestion engine sits at the front of the VIndex pipeline. It must continuously poll remote append-only transparency logs over HTTP, verify checkpoint policies and Merkle consistency, download 256-entry bundles, execute untrusted mapping logic inside WebAssembly sandboxes (`internal/wasmhost`), and feed sorted batches to downstream storage. - -If ingestion is naive, four severe operational failure modes emerge: -1. **Host-Sandbox Boundary Overhead**: Invoking WebAssembly guest functions per entry incurs severe boundary-crossing penalties. Every transition across the Foreign Function Interface (FFI)—switching execution contexts and copying memory between the Go host process and the WebAssembly sandbox—costs CPU cycles. When executed entry-by-entry, FFI boundary overhead alone consumes ~23% of total host CPU time. -2. **Out-of-Order Corruption**: Executing mapping across parallel worker threads causes batches to finish at different times. If out-of-order batches reach the storage engine, the database commits non-contiguous entry index sequences, permanently corrupting the RFC 6962 compact range hashes of affected keys' value logs. -3. **Upstream Rate Limiting vs. Catch-Up Bandwidth**: Ingesting high-velocity logs from genesis (tens of millions of entries) requires saturating network bandwidth, but upstream CDNs and log endpoints enforce strict rate limits (HTTP 429 Too Many Requests). Ingestion must maximize parallel download concurrency while gracefully backing off to avoid upstream bans. -4. **Network-Free Restart Guarantee (Cache Durability)**: Network egress is expensive and external logs can be transiently unreachable. Ingestion stages consumed tiles locally in a bounded staging buffer until downstream storage and trie persistence confirm they are no longer needed, ensuring daemon restarts and crash recoveries complete with zero network roundtrips. - -### 1.2 Goals & Non-Goals -- **Goals**: - - Ingest upstream Input Logs in native 256-entry bundles matching the underlying tile structure. - - Maximize upstream download throughput with adaptive concurrency while respecting HTTP 429 rate limits. - - Parallelize CPU-heavy parsing across worker pools while emitting strictly contiguous Input Log entry sequences. - - Maintain a bounded local staging buffer and crash-recovery cache, enabling restarts and Zero-WAL recovery with zero network egress (not an immutable log of record). - - Bound local disk growth by having the background tile reaper actively prune cached tiles older than `SafeWatermark`. -- **Non-Goals**: - - **No Upstream Checkpoint Creation**: The ingestion engine does not publish checkpoints or modify upstream log state; it is strictly a read-only consumer. - - **No In-Guest Storage Writes**: The WASM sandbox is forbidden from performing disk or network I/O; guest modules solely transform entry bytes into canonical keys. - - **No Cross-Subsystem Watermark Calculation**: The ingestion cache does not inspect the internal state of Pebble DB or the MPT; watermark authority resides strictly with the Coordinator. - -### 1.3 Requirements, Dependencies & Constraints -- **Upstream Log Contract**: Supports standard tile-based logs ([C2SP tlog-tiles](https://c2sp.org/tlog-tiles) `/tile/entries/...` and [C2SP static-ct-api](https://c2sp.org/static-ct-api) `/tile/data/...`) as well as custom leaf-by-leaf log adapters that group entries into 256-entry bundles, with checkpoints verified against configured checkpoint verification policies (such as C2SP [tlog-policy](https://c2sp.org/tlog-policy)). -- **WASM Host Runtime**: Pure Go WebAssembly sandbox execution via `internal/wasmhost` backed by `github.com/tetratelabs/wazero` (zero CGO dependencies). -- **Resequencer Straggler Buffering**: If a single WASM worker stalls on an anomalous entry bundle, subsequent workers buffer finished batches in the in-memory min-heap until the straggler bundle completes. +### Non-Goals +- **No Upstream Log Mutation**: Strictly a read-only consumer of the upstream Input Log. +- **No In-Guest Storage Writes**: Sandboxed mapping workers transform raw entry bytes into canonical keys in memory and perform zero disk or network I/O. +- **No Independent Pruning Authority**: The local tile cache never inspects downstream database or trie state directly; it prunes cached bundles strictly up to the `SafeWatermark` dictated by [`internal/coordinator`](../coordinator/README.md). --- -## 2. Detailed Design - -### 2.1 The Assembly Line Sorter Architecture -Checkpoint polling and `Target_CP` persistence are driven by the Coordinator using the upstream log fetcher; once a verified `Target_CP` is established, the Coordinator streams `[max(persisted_watermark, from), Target_CP.Size)` through the three-stage ingestion pipeline: - -1. **Tile Fetcher**: Queries the upstream Input Log's `/checkpoint` endpoint and verifies checkpoints against `` and the configured Input Log checkpoint policy. Downloads 256-entry bundles (`/tile/entries/[.p/]` for `tlog-tiles`, `/tile/data/[.p/]` for `static-ct-api` using partial tile width `.p/` on unsealed tail tiles, or synthesized 256-entry bundles via a leaf-by-leaf adapter) and authenticates each bundle against its Level-0 tree tile and upper-level tree tiles up to `Target_CP.Root` across `[max(persisted_watermark, from), Target_CP.Size)`. -2. **Local Tile Cache**: Writes downloaded, Merkle-authenticated 256-entry bundles to the local filesystem via atomic temp-file rename. When `Target_CP.Size` advances within an unsealed 256-entry tail tile across consecutive sync cycles, any cached partial bundle narrower than required (`bundleEnd < Target_CP.Size` and `len < 256`) is treated as a cache miss, re-fetched at the wider width, and atomically overwritten on disk. -3. **WASM Worker Pool**: A configurable pool of parallel Wazero sandboxes (`internal/wasmhost`, scaling across available host CPU capacity) running the compiled `.wasm` plugin. Each worker slices boundary bundles to `[max(bundleStart, start), min(bundleEnd, Target_CP.Size))`—excluding entries `< from`, entries already persisted in a partially indexed tail bundle, and entries `>= Target_CP.Size`—and maps the slice in a single `map_bundle` invocation via the in-guest SDK (`mapfn/sdk`). -4. **Canonical Key Validation & Per-Entry Deduplication**: Upon unpacking the WASM output slab, the host validates every emitted key against the canonical key bounds in [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md) (`1..1024` printable ASCII bytes in `0x21..0x7E`, containing no `NUL` or whitespace bytes) and sorts and deduplicates canonical keys per entry (`entryIdx`). If the WASM module traps, overflows its linear memory slabs, or emits any key violating canonical key bounds (`1..1024` bytes in `0x21..0x7E`), the host returns a fatal validation error that immediately **`HALT`s** ingestion—never silently dropping invalid keys—preserving bit-for-bit `mpt_root` equivalence with [`internal/auditor`](../auditor/README.md) and [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md). Valid deduplicated un-hashed canonical keys are passed downstream to storage and the MPT. -5. **Resequencer (Min-Heap)**: Receives completed batches out of order from workers and buffers them in a priority queue. Emits strictly ascending entry batches across `[max(persisted_watermark, from), input_size)` to the downstream storage channel. -6. **Tile Reaper**: Periodically receives the authorized `SafeWatermark` from the Coordinator (`max(min(KV_CP.Size, MPT_Durable_Size), from)`) and actively prunes cached tiles fully covered by that watermark (`(tileIndex + 1) * 256 <= SafeWatermark`) to bound local disk usage. +## 1. Design -### 2.2 In-Situ Invariants & Performance Optimizations +### 1.1 The Assembly Line Sorter Pipeline -- **[Correctness Invariant] Target Checkpoint Policy & Consistency Verification**: - - *Rule*: Upstream Input Log checkpoints must be cryptographically validated against `` and the configured Input Log checkpoint policy (enforcing primary log signature and witness cosignature quorum), must satisfy `Target_CP.Size >= max(prev_Target_CP.Size, from)`, and (after the first verified checkpoint) must verify Merkle consistency against `prev_Target_CP` before the size advancement is accepted. - - *Rationale*: Guarantees that the indexer only ingests authentic, append-only entries committed by the legitimate log authority and witnesses at or above `from`. - - *Consequence ("Or Else")*: If an unauthenticated, regressed, or split-view checkpoint were accepted, an adversary could feed forged or rolled-back entries, causing the indexer to commit bogus state and permanently diverge from independent auditors. +Once the Coordinator establishes a verified `Target_CP` (`Target_CP.Size >= max(prev_Target_CP.Size, from)` and Merkle-consistent with `prev_Target_CP`), it streams `[max(persisted_watermark, from), Target_CP.Size)` through six stages: -- **[Correctness Invariant] Fatal `HALT` on WASM Trap or Invalid Canonical Key**: - - *Rule*: Every key emitted by the WASM guest must satisfy canonical key bounds (`1..1024` bytes in `0x21..0x7E`). If the WASM module traps or emits an invalid canonical key, the host must return a fatal validation error that immediately **`HALT`s** ingestion rather than dropping or sanitizing the offending key. - - *Rationale*: Preserves bit-for-bit root equivalence with [`internal/auditor`](../auditor/README.md) and the auditing rules in [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md). - - *Consequence ("Or Else")*: Silently dropping invalid keys would allow a defective mapping module to publish Output Log state commitments that spec-compliant auditors reject. - -- **[Correctness Invariant] Strictly Monotonic Resequencing**: - - *Rule*: Batches pushed to downstream storage must strictly follow ascending Input Log entry index (`entryIdx`) order without gaps, where each emitted batch's starting entry index equals the preceding batch's ending entry index. - - *Rationale*: Inverted chunks maintain cumulative RFC 6962 compact range hashes over value log leaves, which require strictly increasing Input Log entry indices. - - *Consequence ("Or Else")*: Committing out-of-order batches breaks the Merkle compact range tree structure, computing invalid `value_root` digests that fail client inclusion proofs. - -- **[Correctness Invariant] Tile Reaper Authority & Pruning Bounds**: - - *Rule*: The background tile reaper must strictly prune only those tile files whose highest entry index is covered by `SafeWatermark` (`(tileIndex + 1) * 256 <= SafeWatermark`), where `SafeWatermark` is authoritative input dictated by the Coordinator. - - *Rationale*: Preserves un-synced historical 256-entry bundles on disk so that Zero-WAL crash recovery can fast-forward lagging MPT state without network requests. - - *Consequence ("Or Else")*: Pruning ahead of the durable MPT watermark causes dirty crash recovery to fail, forcing the daemon to re-download historical tiles over the network and violating the sub-500ms time-to-first-serve SLO. - -- **[Performance Optimization] Native 256-Entry Tile Bundling**: - - *Mechanism*: Fetches and caches data in native 256-entry bundles rather than issuing individual HTTP requests per entry. - - *Impact*: Reduces upstream network requests by 256x compared to per-entry REST APIs. - -- **[Performance Optimization] Bundled WebAssembly Invocation (`map_bundle`)**: - - *Mechanism*: Transfers an entire 256-entry bundle across the WASM boundary in a single FFI call. - - *Impact*: Reduces FFI boundary crossings from 768 to 2–3 per tile, dropping FFI CPU overhead from ~23% of host CPU time to < 1%. - -### 2.3 Ingestion Pipeline Contract - -The ingestion subsystem manages upstream data retrieval, sandboxed mapping, and local tile caching: - -- **Inputs**: - - Upstream Input Log HTTP endpoint, expected ``, and Input Log checkpoint policy. - - Local tile cache directory. - - Sandboxed WASM mapping runner (`internal/wasmhost`). - - Worker concurrency. -- **Outputs**: - - Stream of mapped `(canonicalKey, entryIdx)` batches ordered monotonically by Input Log entry index. -- **Pruning Contract**: - - Local tile cache retains 256-entry bundles strictly within the active lag window; tiles covered by `SafeWatermark` are deleted by the background reaper. +1. **Tile Fetcher & Merkle Authentication**: Supports standard tiled logs ([C2SP `tlog-tiles`](https://c2sp.org/tlog-tiles) `/tile/entries/[.p/]` and [C2SP `static-ct-api`](https://c2sp.org/static-ct-api) `/tile/data/[.p/]` using partial tile width `.p/` on unsealed tail tiles) as well as leaf-by-leaf log adapters that group entries into 256-entry bundles. Checkpoint verification follows the configured Input Log policy (such as [C2SP `tlog-policy`](https://c2sp.org/tlog-policy)). Fetching in 256-entry bundles cuts upstream HTTP requests by 256x, and each bundle is authenticated against its Level-0 tree tile and in-memory cached upper-level tree tiles (so shared parent tiles are fetched once per checkpoint rather than re-fetched across consecutive bundles) up to `Target_CP.Root` before use. +2. **Local Tile Cache**: Writes Merkle-authenticated 256-entry bundles to sharded subdirectories (`1,000` bundles per shard, preventing flat-directory inode contention) on local disk via atomic temp-file rename. When `Target_CP.Size` advances within an unsealed 256-entry tail tile across consecutive sync cycles, any cached partial bundle narrower than required (`bundleEnd < Target_CP.Size` and `len < 256`) is treated as a cache miss, re-fetched at the wider width, and atomically overwritten on disk. +3. **WASM Worker Pool**: A pool of parallel Wazero sandboxes (`internal/wasmhost`) running the compiled `.wasm` plugin. Each worker slices boundary bundles to `[max(bundleStart, start), min(bundleEnd, Target_CP.Size))` (excluding entries `< from`, entries already persisted in a partially indexed tail bundle, and entries `>= Target_CP.Size`) and maps the slice in a single `map_bundle` FFI call via [`mapfn/sdk`](../../mapfn/README.md) (reducing FFI crossings from 768 to 2–3 per bundle and FFI CPU overhead from ~23% to < 1%). +4. **Canonical Key Validation & Per-Entry Deduplication**: Upon unpacking the WASM output slab, the host validates every emitted key against the canonical key bounds in [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md) (`1..1024` printable ASCII bytes in `0x21..0x7E`, containing no `NUL` or whitespace bytes) and sorts and deduplicates canonical keys per entry (`entryIdx`). If the WASM module traps, overflows its linear memory slabs, or emits any invalid canonical key, the host returns a fatal validation error that immediately **`HALT`s** ingestion rather than silently dropping the key, preserving bit-for-bit `mpt_root` equivalence with [`internal/auditor`](../auditor/README.md). +5. **Resequencer (Min-Heap)**: Parallel workers finish bundles out of chronological order, and committing out-of-order entry indices would corrupt downstream RFC 6962 Value Log compact ranges. An in-memory min-heap buffers completed worker batches (holding subsequent completions if a straggler worker stalls on a large bundle) and emits strictly contiguous, ascending `(canonicalKey, entryIdx)` batches across `[max(persisted_watermark, from), Target_CP.Size)`. +6. **Tile Reaper**: Periodically receives the Coordinator's `SafeWatermark` (`max(min(KV_CP.Size, MPT_Durable_Size), from)`) and deletes only cached tile files fully covered by that watermark (`(tileIndex + 1) * 256 <= SafeWatermark`), preserving un-synced bundles on disk for Zero-WAL crash recovery while bounding local disk usage. --- -## 3. Alternatives Considered (or Tried) +## 2. Alternatives Considered -### 3.1 Ephemeral In-Memory Fetching vs. Persistent Local Tile Cache -- **Proposed**: Streaming 256-entry bundles directly from HTTP responses into the WASM worker pool in memory without persisting verified bundles on local disk. -- **Theoretical & Operational Rejection**: - - Tightly couples upstream HTTP jitter to downstream mapping and storage throughput. - - Any daemon restart or dirty crash where `MPT_Durable_Size` lags behind `KV_CP.Size` or `Output_CP.InputSize` would force the node to re-download historical 256-entry bundles over the network—risking upstream HTTP 429 rate limits, stalling startup completely during upstream outages, and violating the sub-500ms Zero-WAL recovery SLO. -- **Chosen Design**: Persist Merkle-authenticated 256-entry bundles on local disk up to `SafeWatermark` (`max(min(KV_CP.Size, MPT_Durable_Size), from)`), enabling offline Zero-WAL crash replay while bounding disk usage via the background tile reaper. +### 2.1 Ephemeral In-Memory Fetching vs. Persistent Local Tile Cache +- **Proposed**: Streaming 256-entry bundles directly from HTTP responses into the WASM worker pool in memory without caching verified bundles on local disk. +- **Rejection**: Tightly couples upstream HTTP jitter to downstream mapping throughput, and forces any restart or dirty crash where `MPT_Durable_Size` lags `KV_CP.Size` or `Output_CP.InputSize` to re-download historical bundles over the network (risking `HTTP 429` rate limits, failing during upstream outages, and violating the sub-500ms Zero-WAL recovery SLO). +- **Chosen Design**: Persist Merkle-authenticated 256-entry bundles on local disk up to `SafeWatermark` (`max(min(KV_CP.Size, MPT_Durable_Size), from)`), enabling offline Zero-WAL crash replay while bounding disk usage via the tile reaper. --- -## 4. Verification Gate - -All tile fetching, cryptographic tile verification, cache durability, reaper, and WASM pipeline tests are verified via: +## 3. Verification Gate ```bash go test -race ./vindex/internal/ingest/... diff --git a/vindex/internal/kvstore/README.md b/vindex/internal/kvstore/README.md index 7198262..d91e1e4 100644 --- a/vindex/internal/kvstore/README.md +++ b/vindex/internal/kvstore/README.md @@ -1,65 +1,29 @@ # Inverted Chunk Storage Engine -This document defines the storage architecture, LSM-tree key-value layout, binary serialization schemas, load-bearing invariants, verified performance optimizations, and empirical benchmarks for the **Storage Subsystem** (`vindex/internal/kvstore`). +`vindex/internal/kvstore` is the embedded [Pebble](https://github.com/cockroachdb/pebble) storage engine that persists per-key Value Logs: the chronologically ordered sequences of Input Log entry indices at which each canonical key appeared, together with their RFC 6962 Merkle compact ranges. Transparency logs have extreme key cardinality skew (most keys appear once or twice, while hot keys accumulate hundreds of thousands of entries). By partitioning occurrences into global 65,536-entry windows and ordering chunks newest-first in the LSM keyspace, the store supports single-seek appends, lock-free backward pagination with zero read-path SHA-256 hashing, and point-in-time `value_root` reconstruction for crash recovery. ---- - -## 1. Context & Objectives - -### 1.1 Problem Statement & Skewed Cardinality Dilemma -The storage engine persists the per-key value logs mapping canonical keys (`1..1024` bytes in `0x21..0x7E`) to chronologically ordered sequences of Input Log entry indices (`entryIdx`), indexing physical Pebble chunks internally by 32-byte `KeyHash = SHA-256(key)` digests and returning modified `(canonicalKey, value_root)` pairs for MPT state commitment. - -Transparency logs exhibit extreme key cardinality skew: -- **Long-tail keys**: The vast majority of domains or packages appear only once or twice in the entire log. -- **Hot keys**: Popular keys (e.g. major cloud provider domains, popular open-source packages) accumulate hundreds of thousands of entries over years. - -In an LSM-tree database (such as Pebble), storing this mapping naively leads to two severe failure modes: -1. **O(N^2) Write Amplification**: Appending to flat lists per key requires rewriting the entire historical list on every new entry. -2. **Scan Latency on Deep Keys**: If chunks are stored in chronological forward order (`chunk_0, chunk_1, ...`), locating the active chunk to append a new entry requires seeking past years of historical SSTables, introducing a 7.5x latency penalty. - -### 1.2 Goals & Non-Goals -- **Goals**: - - Provide single-seek active chunk location for batch appends regardless of key depth or historical occurrence count. - - Eliminate disk I/O on missing keys via full-table 33-byte prefix Bloom filters. - - Maximize storage density by encoding occurrence entry indices as 16-bit relative offsets within 65,536-entry chunks. - - Guarantee zero-overhead point-in-time `value_root` calculation to support non-blocking MPT prediction and Zero-WAL crash recovery. - - Enforce synchronous durability before state commitment handoff to guarantee `KV_CP.Size >= Output_CP.InputSize`. -- **Non-Goals**: - - **No Distributed Storage**: Operates strictly as a single-host embedded database; distributed clustering and remote RPC storage backends are explicitly out of scope. - - **No Cross-Key Range Scans**: Supports point queries by exact 32-byte `KeyHash`; lexical prefix scanning across distinct canonical keys inside Pebble is out of scope (all prefix discovery is served by un-hashed canonical keys in the MPT). - - **No Deletions or Tombstones**: Data is strictly append-only; records are never deleted or overwritten once sealed. - -### 1.3 Requirements & Dependencies -- **Storage Engine**: Embedded [Pebble](https://github.com/cockroachdb/pebble) LSM-tree with a custom 33-byte prefix splitter and 10-bit Bloom filter policy. -- **Merkle Tree Utilities**: RFC 6962 value log hashing and compact range hashes via `internal/wire` (`vindex/internal/wire`) and `github.com/transparency-dev/merkle`. +### Non-Goals +- **No Distributed Storage**: Operates strictly as a single-host embedded database; remote RPC storage backends are out of scope. +- **No Lexical Prefix Scans in Pebble**: Indexes chunks internally by fixed-width 32-byte key hashes for exact key lookup; lexical prefix discovery across canonical keys is served by the in-memory MPT ([`internal/tree`](../tree/README.md)). +- **No Deletions or Tombstones**: Strictly append-only; sealed historical chunks are immutable. --- -## 2. Detailed Design +## 1. Design -### 2.1 Logical Model & Inverted Chunk Key Layout -To eliminate write amplification and support direct window-based seeks, the Input Log entry index space is partitioned into fixed global windows of 65,536 entries. Each occurrence at Input Log entry index `entryIdx` maps deterministically to: -- **Chunk Number**: `chunkNum = entryIdx / 65536` -- **Relative Offset**: `relOffset = uint16(entryIdx % 65536)` +### 1.1 Inverted 65,536-Entry Chunk Key Layout -While the MPT stores un-hashed variable-length canonical keys (`1..1024` bytes in `0x21..0x7E`), Pebble allocates chunks only for windows in which a canonical key actually occurs, keyed by fixed-width 41-byte records: `'c'` (1 byte) + `KeyHash` (32 bytes, `SHA-256(key)`) + `^BigEndian(chunkNum)` (8 bytes, bitwise inverted so newer chunks sort first): +The Input Log entry index space is partitioned into fixed global windows of 65,536 entries (`chunkNum = entryIdx / 65536`, `relOffset = uint16(entryIdx % 65536)`). While the MPT stores un-hashed canonical keys (`1..1024` bytes in `0x21..0x7E`), Pebble allocates chunks only for windows in which a key occurs, keyed by fixed 41-byte records using internal `KeyHash = SHA-256(key)` digests and bitwise-inverted chunk numbers (`^BigEndian(chunkNum) = 0xFFFFFFFFFFFFFFFF - chunkNum`): ```text Key (41 bytes) = [Prefix 'c' (1B)] + [KeyHash (32B)] + [^BigEndian(chunkNum) (8B)] ``` -where `^BigEndian(chunkNum) = 0xFFFFFFFFFFFFFFFF - chunkNum`. -#### The Inverted Ordering Mechanism: -Because chunk numbers are bitwise-inverted, the **highest (newest) chunk has the lexicographically smallest key** among all chunks for that key prefix. -- Pebble's custom key comparer splits keys at byte 33 (`'c' + KeyHash`). -- Full-table 10-bit Bloom filters are indexed on this 33-byte prefix across all SSTable levels. -- Performing a prefix-bounded seek on `'c' + KeyHash` achieves two simultaneous benefits: - 1. If the key does not exist, the Bloom filter prunes all SSTables, returning empty with **zero disk I/O**. - 2. If the key exists, the iterator lands **directly on the latest active chunk**, completely skipping all older historical chunks. +- **Inverted Ordering & 33-Byte Prefix Bloom Filter**: Pebble's custom comparer splits keys at byte 33 (`'c' + KeyHash`) and builds full-table 10-bit Bloom filters on that 33-byte prefix across all SSTable levels. A prefix-bounded seek on `'c' + KeyHash` prunes non-existent keys with **zero disk I/O**, and for existing keys lands **directly on the newest active chunk** (eliminating the 7.5x scan latency penalty of forward-ordered chunks on deep keys). -### 2.2 Delimitless Binary Chunk Serialization & Compact Ranges +### 1.2 Delimitless Binary Chunk Values & Compact Ranges -Values store a cumulative RFC 6962 compact range covering all occurrences of that canonical key across prior chunks (`chunk < chunkNum`), plus a dense array of 16-bit relative offsets for occurrences within the current 65,536-entry window (`chunkNum`): +Each chunk value stores a cumulative RFC 6962 compact range covering all occurrences of that key across prior chunks (`chunk < chunkNum`), followed by dense 16-bit relative offsets for occurrences within `chunkNum` (saving 75% storage compared to 64-bit indices): ```text Preceding uint64 (8 bytes, BigEndian) || CompactRange (32 bytes * popcount(Preceding)) || RelativeIndices (uint16, 2 bytes BigEndian * N) @@ -67,112 +31,49 @@ Preceding uint64 (8 bytes, BigEndian) || CompactRange (32 bytes * popcount(Prece | Byte Offset | Field Name | Data Type | Description | | :--- | :--- | :--- | :--- | -| `0` | `Preceding` | `uint64` (8 bytes, BigEndian) | Cumulative number of occurrences (value log leaves) for this key across all prior chunks. | -| `8` | `CompactRange` | `[K][32]byte` | Compact range hashes committing to the `Preceding` older value log leaves of this key, where `K = popcount(Preceding)` (32 bytes * K). | -| `8 + 32*K` | `RelativeIndices` | `[]uint16` (2 bytes BigEndian * N) | Dense 2-byte offsets: `uint16(entryIdx % 65536)` for each occurrence of this key within global 65,536-entry window `chunkNum` (`entryIdx = chunkNum*65536 + uint64(relOffset)`). | - -#### Per-Entry Key Deduplication, Crash-Replay Idempotency & Value Log Leaf Hashing: -Each leaf in a key's RFC 6962 value log commits to a distinct Input Log entry index (`entryIdx`) where the canonical key occurred: -- **Per-Entry Deduplication**: While the WASM host already sorts and deduplicates canonical keys per entry on output slab unpacking, batch indexing also defensively sorts and deduplicates `(canonicalKey, entryIdx)` pairs (hashing `canonicalKey` internally to `KeyHash = SHA-256(key)` for Pebble storage). Each `entryIdx` is appended at most once to a key's value log, ensuring strictly increasing entry indices within every chunk (`relOffset_0 < relOffset_1 < ...`). -- **Crash-Replay Idempotency (`entryIdx < KV_CP.Size`)**: Every batch write atomically records the batch's ending entry watermark alongside the modified `'c'` chunk records in Pebble (with the full `KV_CP` checkpoint note and durable `fsync` committed on the final slab reaching `Target_CP.Size`). During startup crash recovery or batch replay across `[Output_CP.InputSize, KV_CP.Size)`, any `entryIdx` below the persisted KV watermark—or whose `(chunkNum, relOffset)` is `<=` the active chunk's highest persisted offset—is skipped. When all occurrences of a key in a replayed batch are already persisted in Pebble, the storage engine skips chunk mutation and reconstructs the key's point-in-time `value_root` as of the target size so the MPT receives the exact modified `(canonicalKey, value_root)` mapping without duplicating chunk offsets. -- **Value Log Leaf Hashing**: Each absolute Input Log entry index (`entryIdx = chunkNum*65536 + uint64(relOffset)`) is encoded as an 8-byte big-endian integer and hashed with RFC 6962 leaf domain separator `0x00` (`value_log_leaf_hash(entryIdx) = SHA-256(0x00 || uint64_be(entryIdx))`). -- **Interior Nodes & Chunk Rollover**: Value log interior nodes are hashed with RFC 6962 node domain separator `0x01` (`value_log_node_hash(left, right) = SHA-256(0x01 || left || right)`). When a new occurrence for a key falls into a higher global 65,536-entry window, the previous chunk is sealed and a new chunk is initialized with `Preceding` and `CompactRange` snapshotted from the key's running compact range hashes across all prior chunks. - -#### Delimitless Deserialization: -Because the number of compact range hashes is mathematically determined by the population count (number of set bits) of `Preceding` (`K = popcount(Preceding)`), the byte offset where `RelativeIndices` begins is always `8 + 32*K`. This layout eliminates field delimiters, length prefixes, and serialization padding, achieving optimal binary packing density. - -### 2.3 Two-Generational Active Chunk Cache & Running Compact Range -To avoid repeatedly reading active chunk descriptors from Pebble or re-hashing historical offsets within the active 65,536-entry window during sequential batching, the store maintains a bounded in-memory two-generational chunk cache: -- **Cached Descriptor & Running Compact Range**: Each cache entry holds both the active chunk record (`Preceding`, `CompactRange`, `RelativeIndices`) and its cumulative in-memory running compact range across all occurrences up to the latest batch. Incremental batch appends update the running compact range directly in O(1) amortized leaf hashes per new occurrence instead of re-hashing prior `RelativeIndices` from index 0; on cache eviction or node restart, the running compact range is lazily reconstructed once when the active chunk is loaded from Pebble. -- **Current & Previous Generations**: The current generation stores chunk descriptors accessed or modified during the active batch, while the previous generation retains descriptors from the immediately preceding batch. -- **Eviction**: When the current generation reaches capacity (scaling to hundreds of thousands of hot keys), the previous generation is replaced by the current generation, and a fresh current generation map is allocated. This absorbs repeated writes to hot keys without LRU lock contention, eliminating >90% of Pebble block cache read I/O during high-throughput ingestion. - -### 2.4 In-Situ Invariants & Performance Optimizations - -- **[Correctness Invariant] Synchronous Persistence Barrier**: - - *Rule*: Within a target sync cycle, intermediate commit slabs may write to Pebble without an immediate `fsync`, but the final slab reaching `Target_CP.Size` (or an explicit flush barrier) must execute a durable disk sync to flush all modified `'c'` records and metadata (`KV_CP`) to disk before returning to the Coordinator for Output Log publication. - - *Rationale*: Enforces `KV_CP.Size >= Output_CP.InputSize` across all crash and power-loss scenarios while amortizing disk `fsync` overhead across multi-slab sync cycles. - - *Consequence ("Or Else")*: If the node crashed after appending a state commitment entry to the Output Log but before Pebble synced to disk, the published root would reference missing KV chunks. The node would be permanently unable to produce inclusion proofs for its own witnessed checkpoints. - -- **[Correctness Invariant] Per-Entry Key Deduplication**: - - *Rule*: The KV store must deduplicate identical `(canonicalKey, entryIdx)` emissions before appending relative offsets or updating the key's running compact range. - - *Rationale*: Guarantees that a key's value log contains each matching Input Log entry index at most once in strictly ascending order. - - *Consequence ("Or Else")*: Duplicate relative offsets for the same Input Log entry would be appended to chunk records, corrupting the value log compact range hashes and violating pagination cursor monotonicity (`before=`). - -- **[Correctness Invariant] Point-in-Time Value Log Root Isolation**: - - *Rule*: When computing or reconstructing `value_root` digests for target Input Log size `input_size`, the storage engine strictly evaluates records with `entryIdx < input_size`, ignoring any chunks or entry indices written ahead of `input_size`. - - *Rationale*: Allows startup recovery and reader lookups to safely inspect the database even if the KV store has synced batches ahead of the active serving state. - - *Consequence ("Or Else")*: Query responses would leak unwitnessed or in-flight entry indices, violating reader snapshot isolation. - -- **[Performance Optimization] Bitwise Key Inversion (`^chunkNum`) & 33-Byte Bloom Filters**: - - *Mechanism*: Bitwise inversion places the active chunk first under the prefix; the 33-byte prefix Bloom filter prunes non-existent keys. - - *Impact*: Delivers single-seek active chunk lookups, eliminating the 7.5x append latency penalty observed on deep keys in forward-ordered layouts. - -- **[Performance Optimization] 16-Bit Relative Offset Encoding**: - - *Mechanism*: Stores entry index offsets as `uint16(entryIdx % 65536)` within each global 65,536-entry window instead of 8-byte integers. - - *Impact*: Saves 75% disk storage on entry index occurrence lists within each 65,536-entry chunk. - -- **[Performance Optimization] Two-Generational Active Chunk Cache**: - - *Mechanism*: Caches active chunk descriptors across sequential batches. - - *Impact*: Cuts Pebble block read I/O by >90% during sustained high-concurrency ingestion. - -- **[Performance Optimization] Chunk-Aligned Backward Lookup & Zero-Hashing Compact Range Reuse**: - - *Mechanism*: Seeks directly to the highest valid chunk (`(before - 1) / 65536`), aligns page boundaries to whole 65,536-entry chunks until `totalMatched >= targetPageSize` (default `100`) or `chunksVisited >= maxChunksPerRequest` (default `16`), and copies the `c2sp.org/tlog-vindex-values@v1` `preceding` count and compact range hashes directly from the oldest visited chunk's stored `Preceding` and `CompactRange` with zero dynamic SHA-256 hashing on the read path. - - *Impact*: Bounds SSTable chunk reads (`maxChunksPerRequest`) per request while eliminating intra-chunk splitting and dynamic Merkle re-hashing across deep keys. - -- **[Performance Optimization] Key-Partitioned In-Memory Value Log Hashing**: - - *Mechanism*: Partitions distinct keys within each incoming mapped batch across a worker pool. Workers independently compute value log compact range updates, `value_root` digests (`internal/wire`), and serialized chunk records in parallel in memory, while the committer sequentially applies the precomputed writes into an atomic Pebble batch. - - *Impact*: Eliminates single-thread CPU bottlenecks on RFC 6962 value log Merkle hashing during high-throughput ingestion, preventing upstream pipeline backpressure on mapping workers while keeping database writes strictly sequential. - -### 2.5 Storage Engine Contract - -The storage subsystem manages physical chunk persistence, value log compact range hashes, and point lookups: - -- **Physical Format & Key Layout**: - - Chunks are partitioned by global 65,536-entry Input Log windows (`chunkNum = entryIdx / 65536`) and keyed by 41 bytes: `'c'` (1 byte) + `KeyHash` (32 bytes, `SHA-256(key)`) + `^BigEndian(chunkNum)` (8 bytes bitwise inverted). - - Bitwise inversion places the newest active chunk first under the key prefix, enabling single-seek active chunk lookups with 33-byte prefix Bloom filtering. - - Chunk values use a delimitless binary serialization format: an 8-byte big-endian `Preceding` count (total occurrences for that key across prior chunks), followed by RFC 6962 `CompactRange` hashes committing to those `Preceding` prior value log leaves, followed by dense 2-byte relative entry index offsets (`uint16(entryIdx % 65536)`). -- **Write Contract & Rollover**: - - Monotonic batch writes accept `(canonicalKey, entryIdx)` pairs, compute `KeyHash = SHA-256(key)` internally, deduplicate per Input Log entry, accumulate occurrences across multi-tile commit slabs, return the modified `(canonicalKey, value_root)` mappings for MPT commitment, and execute a durable disk sync at the target checkpoint boundary before Output Log commitment. - - When an occurrence for a key crosses into a new global 65,536-entry window, the prior chunk is sealed as immutable historical state and a new active chunk is allocated with the cumulative `Preceding` and `CompactRange` of all prior occurrences. -- **Point-in-Time `value_root` Reconstruction**: - - To reconstruct a modified key's 32-byte `value_root` as of any target `input_size` (used during crash recovery and uncommitted batch staging for keys with `>= 1` occurrences in `[from, input_size)`), the store seeks to `'c' + KeyHash + ^BigEndian((input_size - 1) / 65536)`, skips any chunk where `chunkNum * 65536 >= input_size`, clones that chunk's `Preceding` and `CompactRange`, appends `SHA-256(0x00 || uint64_be(entryIdx))` into the running compact range for each relative offset where `chunkNum * 65536 + relOffset < input_size`, and folds the compact range right-to-left into the 32-byte `value_root` (keys with zero total occurrences in `[from, input_size)`—where `Preceding` plus matching offsets in the chunk equals `0`—are never inserted into the MPT per [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md)). -- **Lookup Contract & Chunk-Aligned Backward Traversal**: - - The HTTP serving layer rejects `before > Serving_CP.InputSize` with `503 Service Unavailable` prior to storage lookup; once `before <= Serving_CP.InputSize` is established, point lookups accept `canonicalKey` (hashed internally to `KeyHash = SHA-256(key)`) and retrieve Input Log entry indices in `[from, before)` (ignoring any uncommitted chunks written ahead of `Serving_CP.InputSize`) using server-governed chunk-aligned page boundaries per [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md) (`targetPageSize` default `100`, `maxChunksPerRequest` default `16`). - - **Direct Seek**: Computes the target window (`(before - 1) / 65536`) and seeks directly to that inverted chunk key under `'c' + KeyHash`, landing on the newest candidate chunk without scanning future or non-matching SSTables. - - **Whole-Chunk Reverse Traversal**: Iterates forward through inverted keys (`^chunkNum`), moving backward chronologically. Any chunk where `chunkNum * 65536 >= before` or where zero offsets fall in `[from, before)` is skipped without incrementing `chunksVisited` or copying `Preceding`/`CompactRange`. Each visited chunk with `>= 1` occurrences in `[from, before)` emits **all** matching offsets in that 65,536-entry window without intra-chunk splitting, halting once `totalMatched >= targetPageSize` or `chunksVisited >= maxChunksPerRequest` (default `16`). Consequently, a single dense chunk may return up to `65,536` entry indices while sparse chunks return fewer than `targetPageSize`, returning empty (`404 Not Found` at the HTTP layer) if zero matching occurrences exist in `[from, before)`. When `preceding > 0` on the returned page, clients paginate backward using the first (smallest) returned entry index (`entry_idx_0`) as the next `before` cursor. - - **Zero-Hashing Compact Range Reuse**: Because page boundaries always align to whole chunks, when `Preceding > 0` on the oldest visited matching chunk, `preceding` and the compact range hashes are copied directly from that chunk's precomputed `Preceding` and `CompactRange` with zero dynamic SHA-256 hashing on the read path. +| `0` | `Preceding` | `uint64` (8 bytes, BigEndian) | Cumulative number of occurrences (Value Log leaves) for this key across all prior chunks. | +| `8` | `CompactRange` | `[K][32]byte` | Compact range hashes committing to the `Preceding` older Value Log leaves, where `K = popcount(Preceding)` (`32 * K` bytes). | +| `8 + 32*K` | `RelativeIndices` | `[]uint16` (`2 * N` bytes, BigEndian) | Strictly increasing 2-byte offsets `uint16(entryIdx % 65536)` within window `chunkNum` (`entryIdx = chunkNum*65536 + uint64(relOffset)`). | + +Because `K = popcount(Preceding)` determines the exact length of `CompactRange`, `RelativeIndices` always starts at byte offset `8 + 32*K` without delimiters or length prefixes. + +### 1.3 Write Pipeline, Two-Generational Cache & Point-in-Time Reconstruction + +- **Per-Entry Deduplication & Parallel Value Log Hashing**: Batch writes defensively sort and deduplicate `(canonicalKey, entryIdx)` pairs so each `entryIdx` is appended at most once per key (`relOffset_0 < relOffset_1 < ...`). Distinct modified keys in a batch are partitioned across worker goroutines that compute RFC 6962 leaf hashes (`SHA-256(0x00 || uint64_be(entryIdx))`), interior node merges (`SHA-256(0x01 || left || right)`), `value_root` digests (`internal/wire`), and serialized chunk values in parallel in memory before committing sequentially in a single atomic Pebble batch, returning modified `(canonicalKey, value_root)` mappings for MPT commitment. When an occurrence crosses into a higher 65,536-entry window, the prior chunk is sealed and a new active chunk is initialized with the snapshotted `Preceding` and `CompactRange`. +- **Two-Generational Active Chunk Cache**: Each cache entry holds the active chunk descriptor (`Preceding`, `CompactRange`, `RelativeIndices`) alongside its in-memory running compact range, updating incrementally in O(1) amortized leaf hashes per new occurrence instead of re-hashing prior `RelativeIndices` from index `0` (reconstructed lazily once on cache miss or restart). Maintaining current and previous batch generations absorbs repeated writes to hot keys without LRU lock contention and cuts Pebble block cache read I/O by >90%. +- **Synchronous Persistence Barrier & Crash-Replay Idempotency**: Intermediate commit slabs within a sync cycle may write to Pebble without an immediate `fsync`, but the final slab reaching `Target_CP.Size` (or an explicit flush barrier) executes a durable disk sync committing `KV_CP` before Output Log publication (`KV_CP.Size >= Output_CP.InputSize`). Every batch write atomically records the batch's ending entry watermark alongside modified `'c'` records; during startup recovery or batch replay across `[Output_CP.InputSize, KV_CP.Size)`, any `entryIdx` below the persisted KV watermark (or `<=` the active chunk's highest persisted offset) is skipped. +- **Point-in-Time `value_root` Reconstruction**: To reconstruct a key's 32-byte `value_root` as of any target `input_size` (used during crash recovery and batch replay for keys with `>= 1` occurrences in `[from, input_size)`), the store seeks to `'c' + KeyHash + ^BigEndian((input_size - 1) / 65536)`, skips any chunk with `chunkNum * 65536 >= input_size`, clones that chunk's `Preceding` and `CompactRange`, appends `SHA-256(0x00 || uint64_be(entryIdx))` for each relative offset with `chunkNum * 65536 + relOffset < input_size`, and folds the compact range right-to-left into `value_root`. Keys with zero total occurrences in `[from, input_size)` (`Preceding` plus matching offsets equals `0`) are omitted from the MPT per [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md). + +### 1.4 Chunk-Aligned Backward Lookup & Zero-Hashing Compact Range Reuse + +Once the HTTP layer establishes `before <= Serving_CP.InputSize`, point lookups hash `canonicalKey` to `KeyHash = SHA-256(key)` and retrieve entry indices in `[from, before)` (ignoring any uncommitted chunks written ahead of `Serving_CP.InputSize`) using chunk-aligned page boundaries (`targetPageSize` default `100`, `maxChunksPerRequest` default `16`): +1. **Direct Seek**: Seeks directly to `'c' + KeyHash + ^BigEndian((before - 1) / 65536)`, landing on the newest candidate chunk without scanning newer or unrelated SSTables. +2. **Whole-Chunk Reverse Traversal**: Iterates forward through inverted keys (`^chunkNum`), moving backward chronologically. Any chunk where `chunkNum * 65536 >= before` or where zero offsets fall in `[from, before)` is skipped without incrementing `chunksVisited`. Each visited chunk with `>= 1` occurrences in `[from, before)` emits **all** matching offsets in that 65,536-entry window without intra-chunk splitting, halting once `totalMatched >= targetPageSize` or `chunksVisited >= maxChunksPerRequest` (returning up to `65,536` indices for a dense chunk, fewer than `targetPageSize` for sparse chunks, or empty / `HTTP 404` if zero occurrences exist in `[from, before)`). +3. **Zero-Hashing Compact Range Reuse**: Because page boundaries always align to whole chunks, when `Preceding > 0` on the oldest visited matching chunk, the `/values` response copies `preceding` and the compact range hashes directly from that chunk's stored `Preceding` and `CompactRange` with zero dynamic SHA-256 hashing on the read path. --- -## 3. Alternatives Considered (or Tried) +## 2. Alternatives Considered -### 3.1 Evaluated Storage Layouts & Chunk Sizing -1. **Flat Lists per Key**: Storing an unbounded flat list of 8-byte integers under `KeyHash` caused `O(N^2)` write amplification on hot keys as full lists were rewritten on every batch. -2. **One Record per Occurrence**: Appending a discrete key-value pair per occurrence (`KeyHash + BigEndian(entryIdx)`) inflated SSTable metadata and slowed range scans. -3. **Chunk Size (`65,536` vs. `256` / `1,024`)**: Small windows (`256` or `1,024`) caused frequent chunk rollovers and inflated Pebble's block index; `65,536` caps uncompressed chunk values at ~130 KB and enables 16-bit relative offset encoding. -4. **Inverted vs. Forward Chunk Ordering**: Forward ordering (`chunk_0, chunk_1, ...`) required scanning past historical chunks to locate the active tail, whereas bitwise inversion (`^chunkNum`) places the newest chunk first under `'c' + KeyHash` for single-seek access. +### 2.1 Storage Layouts & Chunk Sizing +- **Flat Lists per Key**: Rewriting a single flat list of 8-byte indices per key caused O(N^2) write amplification on hot keys. +- **One Record per Occurrence**: Storing one KV record per occurrence (`KeyHash + BigEndian(entryIdx)`) inflated SSTable metadata and slowed pagination scans. +- **Window Size (`65,536` vs. `256` / `1,024`)**: Small windows caused frequent chunk rollovers and bloated Pebble's block index; `65,536` caps uncompressed chunk values at ~130 KB and enables exact 16-bit relative offset encoding. +- **Inverted vs. Forward Chunk Ordering**: Forward ordering (`chunk_0, chunk_1, ...`) required scanning past historical SSTables to reach the active tail (7.5x slower appends), whereas bitwise inversion (`^chunkNum`) places the newest chunk first under `'c' + KeyHash`. -### 3.2 Distributed NoSQL (Cloud Bigtable, Spanner, Cassandra) vs. Embedded Pebble LSM -- **Proposed**: Backing the inverted chunk store with a distributed NoSQL engine. -- **Theoretical & Architectural Rejection**: - - Remote RPC roundtrips introduce 5–20ms network latency per batch seek, destroying ingestion throughput. - - The authenticated MPT already requires single-host RAM/mmap locality; placing the storage engine on a distributed cluster introduces operational complexity without solving tree state replication. +### 2.2 Distributed NoSQL vs. Embedded Pebble LSM +- **Proposed**: Backing the chunk store with a distributed NoSQL database (Bigtable, Spanner, or Cassandra). +- **Rejection**: Remote RPCs add 5–20ms network latency per batch seek, and the MPT already requires single-host RAM/mmap locality. - **Chosen Design**: Single-host embedded Pebble LSM engine. -### 3.3 Concurrent Database Iterators vs. Key-Partitioned In-Memory Hashing -- **Proposed (Concurrent Database Iterators)**: Partition keys across worker goroutines where each worker manages an independent database iterator to read, mutate, and write chunks concurrently to disk. -- **Empirical Rejection**: Concurrent Pebble iterators introduced lock contention, allocator overhead, and cache thrashing, yielding only an isolated ~14% speedup on 4 workers while degrading performance beyond 8 workers. Furthermore, concurrent disk I/O competed directly with background LSM compactions. -- **Adopted Design (Key-Partitioned In-Memory Hashing)**: Decouples CPU-bound Merkle calculations from I/O-bound database mutations: - 1. *Parallel In-Memory Hashing*: Distinct keys in each batch are distributed across worker goroutines. Each worker computes compact range appends and serializes chunk entries independently in pure memory with zero shared state. - 2. *Sequential Database Commit*: The precomputed key-value writes are merged into a single atomic Pebble batch and committed sequentially. -- **Impact**: Delivers linear multi-core scaling for CPU-intensive value log hashing on high-throughput workloads, eliminating pipeline stalls while preserving zero-contention, deterministic single-threaded database durability. +### 2.3 Concurrent Database Iterators vs. Key-Partitioned In-Memory Hashing +- **Proposed**: Partitioning keys across worker goroutines that each hold an independent Pebble iterator to read, mutate, and write chunks concurrently. +- **Rejection**: Concurrent Pebble iterators suffered lock contention and cache thrashing (~14% speedup on 4 workers, degrading beyond 8 workers) while competing with background LSM compactions. +- **Chosen Design**: Parallel key-partitioned Value Log hashing in pure memory followed by a single sequential atomic Pebble batch commit. --- -## 4. Verification Gate - -All inverted chunk storage, chunk-aligned paging, point-in-time root reconstruction, and batch staging tests are verified via: +## 3. Verification Gate ```bash go test -race ./vindex/internal/kvstore/... diff --git a/vindex/internal/server/README.md b/vindex/internal/server/README.md index d8be974..ebb5605 100644 --- a/vindex/internal/server/README.md +++ b/vindex/internal/server/README.md @@ -1,97 +1,46 @@ # Read Serving & C2SP Query Protocol -This document defines the HTTP serving architecture, endpoint admission rules, reader snapshot isolation, load-bearing invariants, and verified performance optimizations for the **Serving Subsystem** (`vindex/internal/server`). For the full `tlog-vindex` wire format specifications (`c2sp.org/tlog-vindex-keys@v1`, `c2sp.org/tlog-vindex-values@v1`, `c2sp.org/tlog-vindex-state@v1`) and client-side cryptographic verification rules, see [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md) and [`client/README.md`](../../client/README.md). +`vindex/internal/server` is the HTTP serving layer that exposes a Verifiable Index over the public [C2SP `tlog-vindex`](../../docs/tlog-vindex.md) lookup protocol. Unlike a traditional database query server, every response must carry cryptographic proofs showing that no matching entries were omitted, while bounding memory and I/O when hot keys match hundreds of thousands of entries. The server splits lookups into two endpoints: key/prefix proof generation against the latest witnessed Output Log commitment, and lock-free backward Value Log pagination aligned to 65,536-entry storage chunks. ---- - -## 1. Context & Objectives - -### 1.1 Problem Statement & Verifiable Pagination Dilemma -The read serving engine exposes HTTP endpoints for point and prefix lookups over variable-length canonical keys (`1..1024` printable ASCII bytes in `0x21..0x7E`). In a verifiable index, serving search results is fundamentally different from a traditional database: -1. **Omission Resistance**: Lookup clients cannot trust the index operator. Every query response must deliver cryptographic inclusion, non-inclusion, or two-sided prefix proofs, proving that no matching Input Log entries were omitted. -2. **Cardinality Skew & Pagination**: When a canonical key accumulates tens of thousands of occurrences (e.g. popular domains in Certificate Transparency), returning all Input Log entry indices in a single HTTP response causes memory exhaustion, serialization stalls, and denial-of-service amplification. -3. **Backward Pagination over Compact Ranges**: If the server paginated forward in time (`start=X`), proving that an intermediate page did not omit records would require computing Merkle consistency proofs across arbitrary Value Log prefixes or scanning from entry `0`. By serving Value Log pages backward from `input_size` (`before=`) aligned to whole 65,536-entry Pebble chunks, each page embeds the oldest visited chunk's precomputed `preceding` count and compact range hashes (`c2sp.org/tlog-vindex-values@v1`) with zero dynamic SHA-256 hashing on the read path. - -### 1.2 Goals & Non-Goals -- **Goals**: - - Implement the two-endpoint C2SP `tlog-vindex` HTTP read protocol separating key/prefix proof generation (`GET /keys?key=` or `GET /keys?prefix=`) from lock-free Pebble Value Log pagination (`GET /values?key=&before=`). - - Return verifiable point inclusion proofs, non-inclusion proofs, and two-sided prefix proofs (`c2sp.org/tlog-vindex-keys@v1`) paired with the latest Output Log inclusion proof (`c2sp.org/tlog-proof@v1`). - - Remain completely application-agnostic while enforcing an optional caller-supplied prefix policy callback (`400 Bad Request`, e.g. CT/MTC eTLD+1 policies configured by the binary) alongside a configurable maximum prefix match limit (`413 Content Too Large`, default `10,000`). - - Enforce strict reader snapshot isolation: lookup clients observe state strictly bounded by witnessed Output Log checkpoints (`Serving_CP.InputSize`). - - Stream responses using direct plaintext formatting (`internal/wire`) without JSON reflection or intermediate heap allocations. -- **Non-Goals**: - - **No Unbounded Full-Tree Scans**: Prefix scans are guarded by the optional prefix policy callback and maximum prefix match limit. - - **No Uncommitted / Read-Uncommitted Mode**: Lookups never observe unwitnessed or in-flight Input Log entries ahead of `Serving_CP.InputSize`. - - **No In-Server Client Verification**: The server generates proofs; cryptographic verification is executed exclusively by the lookup client ([`client/`](../../client/README.md)). - -### 1.3 Requirements, Dependencies & Constraints -- **Protocol Standards**: Implements [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md) and [C2SP `tlog-tiles`](https://c2sp.org/tlog-tiles@v1.0.0). -- **Dependencies**: Reads from the Pebble inverted chunk store ([`internal/kvstore`](../kvstore/README.md)), the in-memory MPT ([`internal/tree`](../tree/README.md)), and the active serving snapshot (`Serving_CP`). -- **Sequential Roundtrips for Deep Keys**: Because each `/values` continuation cursor (`before = Indices[0]`) is derived from the oldest entry index returned on the preceding page, callers fetching the full history of a key spanning multiple 65,536-entry chunks must issue `/values` requests sequentially rather than in parallel. +### Non-Goals +- **No Unbounded Tree Scans**: Prefix lookups are bounded by an optional application prefix policy callback and a configurable maximum prefix match limit. +- **No Read-Uncommitted Serving**: Lookup responses never expose unwitnessed or in-flight Input Log entries ahead of the active witnessed serving snapshot. +- **No Server-Side Client Verification**: The server generates and formats proofs; cryptographic verification is performed by the lookup client ([`client/`](../../client/README.md)). --- -## 2. Detailed Design +## 1. Design -### 2.1 HTTP Endpoints -The public read server exposes the spec-defined C2SP `tlog-vindex` lookup endpoints under `` (optionally mounted under a configured URL path prefix without `3xx` redirects) alongside `tlog-tiles` Output Log endpoints when the Output Log directory is co-hosted on the same mux, while operational probes and telemetry are isolated onto an internal administrative/metrics listener: +### 1.1 Public & Operational HTTP Endpoints -#### A. Public C2SP `tlog-vindex` & Co-Hosted `tlog-tiles` Endpoints +The public read server exposes the C2SP `tlog-vindex` lookup endpoints under `` (optionally mounted under a configured URL path prefix without `3xx` redirects) alongside `tlog-tiles` Output Log endpoints when co-hosted on the same mux. All responses are formatted directly into contiguous plaintext buffers (`internal/wire`) without JSON reflection (< 1 ms P50 read latency): | Endpoint | Method | Purpose, Status Codes & Caching | | :--- | :--- | :--- | -| `/keys` | `GET` | Exact key or key prefix lookup over canonical keys (`1..1024` printable ASCII bytes in `0x21..0x7E`).
Query params: either `key=` (exact lookup) or `prefix=` (prefix lookup), mutually exclusive with duplicate or unknown params rejected, parsed via strict RFC 3986 percent-decoding (never converting literal `+` to space `0x20`).
On `prefix=`, validates `` against the optional prefix policy callback and scans up to `maxPrefixKeys + 1` matching `(key, value_root)` pairs **before** writing `200 OK` headers (returning `413 Content Too Large` if a `(maxPrefixKeys + 1)`-th match exists), then writes `200 OK` (`text/plain; charset=utf-8`, `Cache-Control: no-store`) with `c2sp.org/tlog-vindex-keys@v1` followed by `c2sp.org/tlog-proof@v1` (see [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md)).
Returns `400 Bad Request` if neither/both/duplicate/unknown params are present, if percent-decoding fails or the unescaped key/prefix is not a valid canonical key, or if the configured prefix policy rejects ``; `413 Content Too Large` if matched keys exceed the configured prefix limit; or `503 Service Unavailable` (with `Retry-After: 30`) if the server has not yet published its first Output Log checkpoint. | -| `/values` | `GET` | Lock-free Value Log pagination for `key`.
Query params: `key` (**required**, parsed via strict RFC 3986 percent-decoding without `+`-to-space conversion) and `before` (**required** canonical positive ASCII decimal integer `1 <= before < 2^64` with no leading zeros or `+`). Page boundaries are server-governed via whole 65,536-entry chunk alignment (`targetPageSize` default `100`, `maxChunksPerRequest` default `16`), returning at least `1` entry index whenever occurrences in `[from, before)` exist.
Acquires **zero MPT locks**; returns `200 OK` (`text/plain; charset=utf-8`, `Cache-Control: public, max-age=31536000, immutable`) with `c2sp.org/tlog-vindex-values@v1` (see [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md)).
Returns `400 Bad Request` on missing/unknown/duplicate/invalid params, non-canonical or non-positive `before`, or malformed `%`-escapes; `503 Service Unavailable` (with `Retry-After: 30`) if uninitialized or `before > Serving_CP.InputSize`; or `404 Not Found` once `before <= Serving_CP.InputSize` and zero occurrences exist in `[from, before)`. | -| `/checkpoint` | `GET` | Co-hosted `tlog-tiles` endpoint returning the latest signed, witnessed Output Log checkpoint note from `Serving_CP` (`503 Service Unavailable` with `Retry-After: 30` if uninitialized). | +| `/keys` | `GET` | Exact key (`key=`) or prefix (`prefix=`) lookup over canonical keys (`1..1024` printable ASCII bytes in `0x21..0x7E`), mutually exclusive with duplicate or unknown query params rejected, parsed via strict RFC 3986 percent-decoding (never converting literal `+` to space `0x20`).
On `prefix=`, validates `` against the optional prefix policy callback and collects at most `maxPrefixKeys + 1` matching `(key, value_root)` pairs **before** writing `200 OK` headers (returning `413 Content Too Large` cleanly if a `(maxPrefixKeys + 1)`-th match exists, default limit `10,000`, so a `200 OK` response is never truncated mid-stream), then writes `200 OK` (`text/plain; charset=utf-8`, `Cache-Control: no-store`) with `c2sp.org/tlog-vindex-keys@v1` followed by `c2sp.org/tlog-proof@v1`.
Returns `400 Bad Request` on invalid/missing/duplicate/unknown params, malformed `%`-escapes, non-canonical key syntax, or prefix policy rejection; `413 Content Too Large` when prefix matches exceed `maxPrefixKeys`; or `503 Service Unavailable` (`Retry-After: 30`) if the first Output Log checkpoint has not yet been published. | +| `/values` | `GET` | Lock-free backward Value Log pagination for `key` (**required**, strict RFC 3986 percent-decoding) and `before` (**required** canonical positive ASCII decimal integer `1 <= before < 2^64` with no leading zeros or `+`). Page boundaries align to whole 65,536-entry Pebble chunks (`targetPageSize` default `100`, `maxChunksPerRequest` default `16`), returning at least `1` entry index whenever occurrences in `[from, before)` exist and embedding the oldest visited chunk's precomputed `preceding` count and compact range hashes (`c2sp.org/tlog-vindex-values@v1`) with zero read-path SHA-256 hashing. Callers walking deep histories paginate sequentially using `before = Indices[0]` from the preceding page.
Acquires **zero MPT locks**; returns `200 OK` (`text/plain; charset=utf-8`, `Cache-Control: public, max-age=31536000, immutable`).
Returns `400 Bad Request` on missing/unknown/duplicate/invalid params, non-canonical or non-positive `before`, or malformed `%`-escapes; `503 Service Unavailable` (`Retry-After: 30`) if uninitialized or `before > Serving_CP.InputSize`; or `404 Not Found` once `before <= Serving_CP.InputSize` and zero occurrences exist in `[from, before)`. | +| `/checkpoint` | `GET` | Co-hosted [C2SP `tlog-tiles`](https://c2sp.org/tlog-tiles@v1.0.0) endpoint returning the latest signed, witnessed Output Log checkpoint note from `Serving_CP` (`503 Service Unavailable` with `Retry-After: 30` if uninitialized). | | `/tile/...` | `GET` | Co-hosted `tlog-tiles` internal hash tiles and `/tile/entries/...` Output Log entry bundles served from the local Tessera Output Log directory for auditors. | -The server implements only the spec-defined protocol paths above; other paths under `` (such as `/` and `/index.html` for an optional embedded single-page HTML explorer UI) are left open and are not part of the protocol. All C2SP `tlog-vindex` endpoints never serve `3xx` redirect responses. - -#### B. Non-Public Operational Endpoints (Admin / Metrics Listener Only) - -Operational liveness, readiness, Prometheus telemetry, and Go runtime profiling handlers are registered on a separate administrative mux and are never exposed on the public listener: +Other paths under `` (such as `/` and `/index.html` for an optional embedded single-page HTML explorer UI) are left open and are not part of the protocol. Operational endpoints are isolated onto a separate internal administrative/metrics listener and are never exposed on the public listener: | Endpoint | Method | Purpose | | :--- | :--- | :--- | -| `/healthz` | `GET` | Liveness probe. Returns **HTTP 200 OK** (`ok\n`) as long as the process and optional health callback report healthy. | -| `/readyz` | `GET` | Readiness probe. Returns **HTTP 200 OK** (`ok\n`) when `Serving_CP` is initialized and ready to serve, or **HTTP 503** (with `Retry-After: 30` and JSON status diagnostics) during MPT initialization, startup recovery, or Genesis Backfill. | -| `/metrics` | `GET` | Exposes Prometheus telemetry metrics. | +| `/healthz` | `GET` | Liveness probe (`200 OK` `ok\n` while the process and optional health callback report healthy). | +| `/readyz` | `GET` | Readiness probe (`200 OK` `ok\n` when `Serving_CP` is active, or `503` with `Retry-After: 30` and JSON diagnostics during MPT open, startup recovery, or Genesis Backfill). | +| `/metrics` | `GET` | Prometheus telemetry metrics. | | `/debug/pprof/*` | `GET` | Go runtime profiling endpoints. | -### 2.2 Reader Snapshot Isolation & Concurrency -To ensure lookup clients never observe uncommitted or unwitnessed records, the serving engine enforces strict watermark filtering (`Serving_CP.InputSize <= Output_CP.InputSize`): +### 1.2 Reader Snapshot Isolation & Concurrency -1. **Brief Shared Read Lock on `/keys`**: Every `/keys` query acquires a shared read lock on the in-memory MPT, reads the atomic `Serving_CP` snapshot under that lock, generates the binary MPT point or prefix proof (`0` bytes when `mpt_root == empty_mpt_hash`, single-leaf inclusion, or full-tree prefix match), and releases the read lock before formatting and writing the response. -2. **Zero-Lock Pebble Reads on `/values`**: Every `/values` query loads the atomic `Serving_CP` pointer without acquiring any MPT lock, rejects `before > Serving_CP.InputSize` with `503 Service Unavailable` (`Retry-After: 30`), hashes `` to `SHA-256(key)`, and performs a chunk-aligned backward lookup in Pebble restricted to `[from, before)`. -3. **Decoupled from Background Ingestion**: Because Output Log appends, witness network roundtrips, and MPT disk `fsync` calls execute outside the in-memory MPT lock (see [`internal/tree`](../tree/README.md)), concurrent `/keys` and `/values` requests never block on ingestion I/O. - -### 2.3 In-Situ Invariants & Performance Optimizations - -- **[Correctness Invariant] Serving Snapshot Isolation**: - - *Rule*: Query responses evaluate state strictly bounded by the latest witnessed Output Log commitment (`Serving_CP.InputSize <= Output_CP.InputSize`). - - *Rationale*: Guarantees that lookup clients only observe state committed in the Output Log and cosigned by witnesses. - - *Consequence ("Or Else")*: Lookup clients would observe uncommitted or unwitnessed entries, causing client-side `/keys` or `/values` proof verification to fail against witnessed checkpoints. - -- **[Correctness Invariant] Pre-Header Prefix Admission**: - - *Rule*: On `GET /keys?prefix=`, the server validates `` against the configured prefix policy and collects at most `maxPrefixKeys + 1` matching keys from the MPT **before** writing HTTP response headers, returning `400 Bad Request` or `413 Content Too Large` cleanly if rejected. - - *Rationale*: Prevents the server from committing a `200 OK` status header and then truncating a response stream mid-body if a broad prefix exceeds `maxPrefixKeys`. - - *Consequence ("Or Else")*: Mid-stream truncation after `200 OK` would deliver an incomplete key set that fails two-sided prefix proof verification on the client. - -- **[Performance Optimization] Zero-Reflection Plaintext Formatting**: - - *Mechanism*: Formats C2SP plaintext responses (`c2sp.org/tlog-vindex-keys@v1`, `c2sp.org/tlog-proof@v1`, and `c2sp.org/tlog-vindex-values@v1`) directly into contiguous byte buffers without JSON reflection. - - *Impact*: Delivers sub-millisecond P50 read latency (< 1 ms) and sustains high lookup throughput under concurrent ingestion. - -### 2.4 Serving Contract - -- **Non-Blocking Read Path**: `/keys` acquires only a brief shared read lock on the in-memory MPT (`Cache-Control: no-store`), while `/values` acquires zero MPT locks and reads immutable/append-only Pebble inverted chunks directly (`Cache-Control: public, max-age=31536000, immutable`). -- **Application-Agnostic Prefix Guardrails**: Contains no hardcoded domain or ecosystem rules; callers may supply an optional prefix policy callback (`400 Bad Request` on policy rejection) alongside a configurable maximum prefix key limit (`413 Content Too Large` when exceeded, default `10,000`). -- **Strict RFC 3986 Query Validation & Chunk-Aligned Paging**: Rejects malformed `%`-escapes, unknown parameters, or duplicate query parameters with `400 Bad Request`, never decodes literal `+` into space `0x20`, validates canonical key syntax (`1..1024` bytes in `0x21..0x7E`), and validates `before` as a canonical positive ASCII decimal integer (`1 <= before < 2^64`) without leading zeros or `+`. On `/values`, page boundaries align to whole 65,536-entry Pebble chunks (`targetPageSize` default `100`, `maxChunksPerRequest` default `16`). +Lookup clients only ever observe state bounded by the latest witnessed Output Log commitment (`Serving_CP.InputSize <= Output_CP.InputSize`): +1. **Brief Shared Read Lock on `/keys`**: Acquires a shared read lock on the in-memory MPT, reads the atomic `Serving_CP` snapshot under that lock, generates the binary MPT point or prefix proof (`0` bytes when `mpt_root == empty_mpt_hash`, single-leaf inclusion, or full-tree prefix match), and releases the read lock before formatting and writing the response. +2. **Zero-Lock Pebble Reads on `/values`**: Loads the atomic `Serving_CP` pointer without acquiring any MPT lock, rejects `before > Serving_CP.InputSize` with `503 Service Unavailable` (`Retry-After: 30`), hashes `` to `SHA-256(key)`, and performs a chunk-aligned backward lookup in [`internal/kvstore`](../kvstore/README.md) restricted to `[from, before)`. +3. **Decoupled from Ingestion I/O**: Because Output Log appends, witness network roundtrips, and MPT disk `fsync` calls execute outside the in-memory MPT lock (see [`internal/tree`](../tree/README.md)), concurrent `/keys` and `/values` requests never block on background ingestion. --- -## 3. Verification Gate - -All server endpoint, prefix policy, chunk-aligned paging, and C2SP wire response tests are verified via: +## 2. Verification Gate ```bash go test -race ./vindex/internal/server/... diff --git a/vindex/internal/tree/README.md b/vindex/internal/tree/README.md index 21164cc..6469273 100644 --- a/vindex/internal/tree/README.md +++ b/vindex/internal/tree/README.md @@ -1,166 +1,65 @@ # Authenticated State Commitment (MPT & Output Log) -This document defines the cryptographic commitment engine, tree synchronization protocols, concurrency model, load-bearing invariants, and verified performance optimizations for the **MPT Subsystem** (`vindex/internal/mpt`) and **Output Log Commitment Subsystem** (`vindex/internal/tree`). +The state commitment subsystem binds every present canonical key's Value Log root (`value_root`) into a path-compressed binary Merkle Patricia Trie (`mpt_root`), publishes state commitments to an append-only Output Log cosigned by external witnesses, and generates point inclusion, non-inclusion, and prefix proofs for HTTP lookup clients. By predicting candidate roots under a shared read lock before publishing and witnessing, and isolating background disk `fsync` calls from in-memory trie reads, concurrent HTTP queries never block on network witnessing or disk persistence. ---- +### Non-Goals +- **No Dynamic Rollbacks**: The trie and Output Log are strictly append-only; state transitions progress monotonically forward. +- **No Full In-Memory Trie Duplication**: Memory constraints require mutating a single in-memory trie under short-lived locks (< 5ms) rather than maintaining duplicate multi-gigabyte trie copies. +- **No Multi-Master Sequencing**: State commitment sequencing is driven by a single local Index Operator process. -## 1. Context & Objectives +--- -### 1.1 Problem Statement, Package Decoupling & Concurrency Dilemma -The commitment plane is split into two focused packages: -- **`internal/mpt`** (`vindex/internal/mpt`): Owns the `filippo.io/torchwood/mpt` state trie, canonical key validation, and binary MPT proof codecs—point inclusion/non-inclusion and two-sided prefix subtree proofs—with zero dependencies on `pebble` or `tessera` so it can be imported directly by [`client`](../../client/README.md) and server subsystems. On top of `torchwood/mpt`, `internal/mpt` enforces the strict `tlog-vindex` proof invariants: split-bit bounds `0 <= b <= 8192` on all path steps, canonical key (`1..1024` bytes in `0x21..0x7E`) and 32-byte value validation on terminal leaves in non-inclusion/empty-prefix proofs, and two-sided prefix verification (prefix match + strictly ascending key order + subtree root check). -- **`internal/tree`** (`vindex/internal/tree`): Owns Output Log publication, Tessera POSIX Output Log storage, and atomic `Serving_CP` promotion via the 3-step commitment lifecycle. +## 1. Design -The tree subsystem binds the search index into an authenticated 32-byte cryptographic root (`mpt_root`), anchors it in an append-only Output Log, and serves cryptographic inclusion, non-inclusion, and prefix proofs to HTTP readers. +### 1.1 Package Decoupling & Variable-Length MPT (`internal/mpt` & `internal/tree`) -Because readers must verify proofs against a coherent snapshot, all mutations to the in-memory MPT require taking an exclusive tree lock. This creates a severe operational dilemma: -1. **The Reader Starvation Trap**: A full commit requires computing tree updates, persisting MPT files to disk (5–20ms), appending a state commitment entry to the Output Log, and awaiting cosignatures from external witnesses over the network (50–200ms). If the writer lock is held across this entire sequence, concurrent HTTP readers are starved. Under heavy traffic, query availability collapses and lookups time out. -2. **The Equivocation Trap**: If the system writes a state commitment entry to the append-only Output Log and *subsequently* encounters an in-memory trie mutation error or calculation mismatch, the node has permanently committed an unprovable root to public witnesses, destroying operator trust. +The commitment plane is split into two internal packages: +- **`vindex/internal/mpt`**: Wraps [`filippo.io/torchwood/mpt`](https://pkg.go.dev/filippo.io/torchwood/mpt) with canonical key validation and binary MPT proof codecs (`uvarint(b) || sib_hash` path steps over split bits `0 <= b <= 8192`, bounded by at most `min(N - 1, 8193)` steps for `N` active keys) with zero dependencies on `pebble` or `tessera` so it can be imported directly by [`client`](../../client/README.md) and [`internal/server`](../server/README.md). On top of `torchwood/mpt`, `internal/mpt` enforces split-bit bounds `0 <= b <= 8192`, canonical key (`1..1024` bytes in `0x21..0x7E`) and 32-byte value validation on terminal no-match leaves, and [two-sided prefix verification](../../docs/tlog-vindex.md#two-sided-prefix-verification): checking `HasPrefix(key, prefix)` on every returned key ensures the server did not substitute a wider ancestor subtree containing keys outside `prefix`, while checking `b_1 < 8 * len(prefix)` on the first ancestor step, reconstructing the subtree root over the strictly ascending `(key, value_root)` pairs, and folding the ancestor path to `mpt_root` ensures no matching keys inside `prefix` were omitted. +- **`vindex/internal/tree`**: Coordinates candidate root prediction, Tessera POSIX Output Log publication, witness quorum collection, and atomic `Serving_CP` snapshot promotion. -### 1.2 Goals & Non-Goals -- **Goals**: - - Deliver cryptographic inclusion, non-inclusion, and two-sided prefix proofs with compact path sizes (`uvarint(b) || sib_hash` steps over split bits `0 <= b <= 8192`, bounded by at most `min(N - 1, 8193)` steps for `N` active keys). - - Minimize reader lock contention during continuous ingestion, keeping write lock duration under 5 milliseconds. - - Eliminate split-view equivocation by binding `mpt_root` commitments to `(input-origin, from, input-size, input-root)` inside witnessed Output Log entries (while retaining signed Input Log checkpoints out-of-band). - - Maintain an immutable on-disk MPT log for Zero-WAL crash recovery. -- **Non-Goals**: - - **No Dynamic Rollbacks**: The MPT is strictly append-only; state transitions progress monotonically forward. - - **No In-Memory Tree Duplication**: Does not duplicate full in-memory trie trees across buffers; memory constraints require mutating a single in-memory trie under short-lived locks. - - **No Multi-Master Consensus**: Tree sequencing is driven by a single local Index Operator process. +`torchwood/mpt` stores variable-length canonical keys in bitwise big-endian paths so keys sharing a byte prefix share a single subtrie. Inner trie nodes (`52 bytes/node`) live in anonymous virtual memory (~1–1.5 GB RAM for 10M keys, 100–150 GB for 1B keys) backed by alternating checksummed patch-frame journal files (`mpt.tree1`, `mpt.tree2`), while variable-length leaf nodes (`mpt_leaf_hash`) append to `mpt.disk`. Underlying file descriptors are wrapped with deferred-`fsync` handles so kernel `fsync(2)` calls never block in-memory reads; on process restart, opening the MPT replays uncompacted journal frames on disk before coordinator recovery begins (while the HTTP server returns `503 Service Unavailable`). -### 1.3 Requirements, Dependencies & Constraints -- **Dependencies**: MPT via `filippo.io/torchwood/mpt`, checkpoint policy verification (`internal/wire`), Tessera POSIX Output Log writer, and external witness clients. -- **RAM Sizing for Branch Locality**: Variable-length canonical keys scatter updates across the trie keyspace. `torchwood/mpt` holds inner trie nodes (`52 bytes/node`) in anonymous virtual memory backed by alternating checksummed patch-frame journal files (`mpt.tree1`, `mpt.tree2`) and an append-only leaf value file (`mpt.disk`), scaling from ~1–1.5 GB for 10M keys to 100–150 GB for 1B keys. -- **Cold-Start Journal Replay**: On process restart, opening the MPT replays uncompacted `torchwood/mpt` journal frames on disk before coordinator recovery begins (while the HTTP server returns `503 Service Unavailable` until replay and startup recovery complete). +### 1.2 The 3-Step Atomic Commitment Sequence & Split-Locking ---- +A commit cycle must satisfy two simultaneous invariants: +1. **Zero Unwitnessed State**: HTTP readers must never observe or obtain proofs against an `mpt_root` that has not yet been committed to the Output Log and cosigned by witnesses (`Serving_CP.InputSize <= Output_CP.InputSize`). +2. **Sub-Millisecond Read Latency**: Appending to the Output Log and collecting witness cosignatures takes 50–200ms+, and syncing journal/leaf files to disk takes 5–20ms. Holding an exclusive tree lock across network and disk I/O would starve HTTP readers. -## 2. Detailed Design - -### 2.1 Variable-Length MPT (`internal/mpt`) -The commitment plane maps un-hashed variable-length canonical keys (`1..1024` printable ASCII bytes in `0x21..0x7E`) to 32-byte value log roots (`value_root`): - -1. **Variable-Length Radix Structure & Proof Wrapper**: `filippo.io/torchwood/mpt` stores variable-length canonical keys (up to 1024 bytes) in bitwise big-endian paths. Keys sharing a common byte prefix share a single subtrie, enabling both point proofs and two-sided prefix subtree proofs, with `internal/mpt` enforcing `0 <= b <= 8192` split-bit bounds, canonical `path_key` / 32-byte `path_val` checks on terminal no-match leaves, and two-sided prefix completeness/soundness. -2. **Journaled Anonymous Memory & Leaf Storage**: `torchwood/mpt` stores inner trie nodes (`52 bytes/node`) in anonymous memory backed by two alternating checksummed patch-frame journal files (`mpt.tree1`, `mpt.tree2`) and stores variable-length MPT leaf nodes (`mpt_leaf_hash`) in an append-only leaf file (`mpt.disk`), wrapped with deferred `fsync` file descriptors so slow kernel `fsync(2)` calls never block in-memory trie reads. -3. **Value Log Roots (`value_root`)**: The 32-byte value committed at each MPT leaf node is the RFC 6962 value log root computed by the KV storage layer (`internal/wire`), authenticating all Input Log entry indices at which that canonical key occurred. - -### 2.2 The 3-Step Atomic Commitment Sequence -The Output Log publisher coordinates the progression from KV storage to public commitment through a structured 3-step sequence designed to satisfy two strict system constraints: - -1. **System Constraint 1 (Zero Unwitnessed State)**: Readers must never observe or obtain proofs against an in-memory root that has not yet been committed to the append-only Output Log and signed by a threshold of external witnesses. Mutating the live trie before publishing would leak unwitnessed state to readers. -2. **System Constraint 2 (Sub-Millisecond Read Latency)**: Readers must never block on external network I/O. Because appending to the Output Log and collecting witness cosignatures takes 50ms–200ms+, holding an exclusive tree write lock across network calls would stall concurrent client queries and violate read latency SLOs (< 15ms P99). -3. **Solution (Separate Prediction from Mutation)**: Pre-compute the deterministic candidate root under a shared read lock without mutating the live trie, release the read lock, and then append the state commitment entry to the Output Log and gather witness cosignatures while readers query unblocked. Once proofs and signatures are secured, acquire the exclusive tree lock briefly (< 5ms) to commit in-memory mutations, assert that the committed root equals the predicted root, and atomically ratchet the `Serving_CP` snapshot. -4. **Why Halt on Divergence**: Candidate root prediction and in-place mutation are deterministic functions over identical inputs. Any divergence indicates an internal invariant violation or memory corruption in the trie. Halting immediately prevents the node from serving proofs that contradict the witnessed Output Log. - -#### Batch Publication Locking & Concurrency: -Batch commits are serialized by the single-writer coordinator. Within each commit cycle, lock scope is minimized so readers never block on network or disk I/O: - -| Step | Phase | Operations | Lock State | Reader Impact | -| :--- | :--- | :--- | :--- | :--- | -| **1** | **Non-Blocking Prediction & Witnessing** | 1. Compute candidate `mpt_root` over modified `(key, value_root)` mappings.
2. Append state commitment entry to Output Log.
3. Obtain witness cosignatures satisfying the Output Log checkpoint policy (e.g., C2SP `tlog-policy`). | Shared read lock held only during in-memory prediction; **no lock held** during Output Log append and witness network calls. | **Zero blocking**; shared read locks allow concurrent HTTP proof generation to run in parallel. | -| **2** | **Atomic Mutation & Ratchet** | 1. Acquire exclusive persistence and tree locks.
2. Commit batch mutations and flush the in-memory journal patch buffer into the OS page cache (via deferred-`fsync` file wrappers).
3. Assert committed root equals predicted `mpt_root`.
4. Ratchet active `Serving_CP` pointer.
5. Release exclusive tree lock. | Exclusive persistence & tree locks held. | **< 5ms** critical section duration. | -| **3** | **Background Durability** | 1. Execute kernel `fsync(2)` on dirty journal/leaf files (`mpt.tree1`, `mpt.tree2`, `mpt.disk`).
2. Advance durable trie watermark (`MPT_Durable_Size`). | Exclusive persistence lock held; exclusive tree lock released. | **Zero blocking**; disk `fsync(2)` runs outside the exclusive tree lock without racing in-memory trie state. | - -#### Step Details: -1. **Non-Blocking Prediction & Witnessing**: - - The publisher acquires a shared read lock on the trie to compute candidate hashes over cloned branch nodes without mutating the live tree, then immediately releases the read lock. - - The canonical 6-line `c2sp.org/tlog-vindex-state@v1` state commitment entry is appended to the Output Log with zero tree or persistence locks held (while retaining the corresponding signed Input Log checkpoint durably out-of-band): - ```text - c2sp.org/tlog-vindex-state@v1 - input-origin - from - input-size - input-root - mpt-root - ``` - - The resulting Output Log checkpoint is submitted to external witnesses to gather cosignatures satisfying the configured Output Log checkpoint policy quorum (e.g., C2SP `tlog-policy`). - - **Throughout this entire phase, concurrent HTTP readers acquire shared read locks and generate proofs without blocking.** -2. **Atomic Mutation & Ratchet**: - - The publisher acquires the exclusive persistence lock and exclusive tree lock. - - Mutates in-memory MPT nodes and flushes `torchwood/mpt`'s in-memory header/journal patch buffer into the OS page cache (against deferred-`fsync` file wrappers) while holding the exclusive tree lock so header and patch-frame compaction state never race concurrent proof or scan readers. - - Asserts that the committed root equals the predicted root. If mismatched, the process terminates immediately. - - Ratchets the active `Serving_CP` pointer to the newly witnessed Output Log checkpoint and inclusion proof. - - Releases the exclusive tree lock (keeping the persistence lock for Step 3). - - **Total critical section duration: < 5 milliseconds.** -3. **Background Durability**: - - Still under the independent persistence lock—with zero tree lock held—`internal/mpt` executes blocking kernel `fsync(2)` calls on the dirty underlying file descriptors (`mpt.tree1`, `mpt.tree2`, `mpt.disk`). - - Once durable on disk, `MPT_Durable_Size` advances atomically, allowing the tile cache reaper to prune superseded 256-entry bundles. - -### 2.3 Concurrency Architecture & Split-Locking -The subsystem enforces strict isolation between in-memory proof serving and background disk persistence using two distinct locks: +To satisfy both, the subsystem separates in-memory trie access from disk persistence using two independent locks and executes each commit in three steps: | Conceptual Lock | Synchronization Model | Scope & Protected State | Typical Duration | | :--- | :--- | :--- | :--- | -| **In-Memory Tree Lock** | Read-write lock | Protects in-memory MPT nodes and the active `Serving_CP` snapshot. Acquired in shared read mode by HTTP proof readers and candidate root prediction; acquired in exclusive write mode only during the in-memory batch commit and `Serving_CP` ratchet. | **< 5ms** (exclusive write)
**< 50µs** (shared read) | -| **Disk Persistence Lock** | Exclusive mutex | Serializes in-memory trie mutations against background snapshot serialization and disk `fsync` operations. Never held during Output Log appends or witness network calls. | **5ms – 20ms** | - -Because disk persistence and network witnessing are decoupled from the exclusive tree lock, slow disk syncs and witness roundtrips never block HTTP readers. - -### 2.4 In-Situ Invariants & Performance Optimizations - -- **[Correctness Invariant] Fatal Halt on Root Prediction Divergence**: - - *Rule*: After the Output Log append succeeds, when the exclusive tree lock is acquired to apply the batch mutations in memory, the resulting trie root must equal the predicted root committed to the Output Log; any mismatch must immediately terminate the process. - - *Rationale*: The Output Log is immutable. If the committed in-memory root diverges from the predicted root that was already appended and witnessed, execution must halt immediately. - - *Consequence ("Or Else")*: Continuing execution would publish an equivocal state commitment entry to the Output Log that cannot be cryptographically proven by the node's internal state. - -- **[Correctness Invariant] Single-Timeline Equivocation Resistance**: - - *Rule*: Every published state commitment entry binds `input-origin`, `from`, `input-size`, `input-root`, and the exact `mpt_root` (`mpt-root`), while the operator retains the signed Input Log checkpoint out-of-band. - - *Rationale*: Independent witnesses cosign the Output Log checkpoint. The index operator cannot present conflicting views of index state to different clients without producing split-view signatures detectable by public monitors, and retaining the signed Input Log checkpoint proves upstream Input Log equivocation if a root conflict ever arises. - - *Consequence ("Or Else")*: An operator attempting to selectively hide certificates or packages would be mathematically detected by any monitor cross-checking witness cosignatures. +| **In-Memory Tree Lock** | Read-write lock | Protects in-memory MPT nodes and the active `Serving_CP` snapshot. Shared read mode for HTTP proof generation and Step 1 root prediction; exclusive write mode only during Step 2 in-memory mutation and `Serving_CP` ratchet. | **< 5ms** (exclusive write)
**< 50µs** (shared read) | +| **Disk Persistence Lock** | Exclusive mutex | Serializes Step 2 in-memory journal buffer flushes against Step 3 background disk `fsync(2)` calls. Never held during Output Log appends or witness network calls. | **5ms – 20ms** | -- **[Correctness Invariant] Serving Isolation & Atomic Ratcheting**: - - *Rule*: HTTP readers evaluate state strictly bounded by the active `Serving_CP` snapshot (`Serving_CP.InputSize <= Output_CP.InputSize`). - - *Rationale*: HTTP requests capture an atomic `Serving_CP` snapshot and filter all storage reads to Input Log entry indices strictly below the serving checkpoint's `input_size` (`InputSize`). - - *Consequence ("Or Else")*: Readers would observe uncommitted, unwitnessed Input Log entries ahead of the latest published Output Log entry, breaking client-side Merkle proof verification. - -- **[Performance Optimization] Non-Blocking MPT Prediction**: - - *Mechanism*: Computes candidate trie roots under a shared read lock and performs Output Log writes and witness network roundtrips with no tree locks held. - - *Impact*: Reduces exclusive writer lock duration from ~250ms to < 5ms per commit cycle. - -- **[Performance Optimization] Split-Locking Concurrency Engine**: - - *Mechanism*: Isolates background disk `fsync` operations from in-memory trie read locks. - - *Impact*: Sustains 678,000 read queries/sec during active disk commits, compared to 53,000 reads/sec under coarse global locking (12.8x throughput improvement). - -### 2.5 State Trie & Commitment Contract - -The state commitment subsystem provides authenticated indexing over canonical keys and coordinates public Output Log commitments: - -- **Inputs**: - - Batch of modified `(key, value_root)` mappings over un-hashed canonical keys (`1..1024` bytes in `0x21..0x7E`). - - Configured `` and start index `from` (`0 <= from <= input_size`). - - Verified Input Log checkpoint (`input_size` and `input_root`, validated against the Input Log checkpoint policy and retained out-of-band). -- **Outputs**: - - Witnessed Output Log state commitment entry (`c2sp.org/tlog-vindex-state@v1`). - - Verified Output Log Merkle inclusion proof (`c2sp.org/tlog-proof@v1` at `index == output_size - 1`). - - Atomically promoted `Serving_CP` snapshot (exposing `InputSize` and `OutputSize`). -- **Concurrency Contract**: - - Readers are wait-free and never block on disk `fsync` or network calls. - - State commits are strictly serialized (single-writer); state ratcheting promotes `Serving_CP` atomically. +| Step | Phase | Operations | Lock State & Reader Impact | +| :--- | :--- | :--- | :--- | +| **1** | **Non-Blocking Prediction & Witnessing** | Pre-sorts the batch's modified `(key, value_root)` mappings outside the lock (reused in Step 2), acquires a shared read lock on the trie to compute the candidate `mpt_root` over cloned branch nodes, and immediately releases the read lock. Appends the 6-line `c2sp.org/tlog-vindex-state@v1` entry (`input-origin`, `from`, `input-size`, `input-root`, `mpt-root`) to the Output Log (retaining the signed Input Log checkpoint durably out-of-band) and collects witness cosignatures satisfying the Output Log checkpoint policy (such as [C2SP `tlog-policy`](https://c2sp.org/tlog-policy)) with **no tree or persistence locks held**. | **Zero reader blocking**; concurrent HTTP `/keys` proof generation runs in parallel under shared read locks. | +| **2** | **Atomic Mutation & Ratchet** | Acquires the exclusive persistence lock and exclusive tree lock. Mutates in-memory MPT nodes and flushes `torchwood/mpt`'s in-memory header/journal patch buffer into the OS page cache (against deferred-`fsync` file wrappers) so header and patch-frame compaction state never race concurrent readers. **Asserts that the committed root equals the predicted `mpt_root`** (terminating the process immediately on any mismatch rather than serving proofs that contradict the witnessed Output Log), ratchets the active `Serving_CP` pointer to the newly witnessed Output Log checkpoint and inclusion proof (`index == output_size - 1`), and releases the exclusive tree lock. | **< 5ms** exclusive write critical section. | +| **3** | **Background Durability** | Still holding only the persistence lock (with the exclusive tree lock released), executes blocking kernel `fsync(2)` calls on the dirty underlying file descriptors (`mpt.tree1`, `mpt.tree2`, `mpt.disk`) and atomically advances `MPT_Durable_Size` so the tile cache reaper can prune superseded 256-entry bundles. | **Zero reader blocking**; disk `fsync(2)` runs outside the tree lock. | --- -## 3. Alternatives Considered (or Tried) +## 2. Alternatives Considered -### 3.1 Coarse Global Locking vs. Split-Locking with Prediction -- **Proposed**: Holding a single global lock across the entire commitment cycle (prediction, Output Log append, witness network calls, in-memory trie mutation, and disk `fsync`). -- **Empirical Rejection**: Benchmark profiling under heavy concurrent read traffic showed that coarse locking dropped HTTP read throughput to **53,000 queries/sec**, with P99 read latency ballooning past 250ms due to writer lock starvation. +### 2.1 Coarse Global Locking vs. Split-Locking with Prediction +- **Proposed**: Holding a single global lock across prediction, Output Log append, witness network calls, in-memory trie mutation, and disk `fsync`. +- **Rejection**: Under concurrent read load, coarse locking dropped HTTP read throughput to **53,000 queries/sec** with P99 read latency exceeding 250ms due to writer lock starvation. - **Chosen Design**: Split-locking with lock-free prediction holds the exclusive tree lock for < 5ms, sustaining **678,000 queries/sec** (a 12.8x speedup) with sub-millisecond P50 read latency. -### 3.2 Sparse Merkle Trees (SMT) vs. Merkle Patricia Trie (MPT) -- **Proposed**: Using a standard Sparse Merkle Tree (SMT) with fixed 256-level depth. -- **Theoretical & Empirical Rejection**: Standard SMTs require generating and traversing 256 levels of nodes per operation. Even with default empty node optimizations, SMTs incur massive memory footprint and excessive disk I/O during random-insert batch workloads. -- **Chosen Design**: Path-compressed binary MPT (`filippo.io/torchwood/mpt`), which collapses single-child paths over variable-length canonical keys, bounding tree depth to the active key set and enabling fast `mmap` persistence. +### 2.2 Sparse Merkle Trees (SMT) vs. Merkle Patricia Trie (MPT) +- **Proposed**: Using a fixed 256-level Sparse Merkle Tree (SMT). +- **Rejection**: Traversing 256 levels per key incurs large memory and disk I/O amplification during random-insert batch workloads. +- **Chosen Design**: Path-compressed binary MPT (`filippo.io/torchwood/mpt`) over variable-length canonical keys, collapsing single-child paths to bound depth to the active key set. -### 3.3 Post-Commit Output Log Publishing +### 2.3 Post-Commit Output Log Publishing - **Proposed**: Mutating and syncing the in-memory MPT to disk *first*, and then appending the state commitment entry to the Output Log. -- **Theoretical Rejection**: If the node crashes or power fails after the MPT disk sync but before the Output Log append completes, the local MPT has advanced to a state that was never committed or witnessed in the Output Log. On restart, the node would serve state ahead of the Output Log, violating the universal invariant `MPT_Durable_Size <= Output_CP.InputSize`. -- **Chosen Design**: Predict root lock-free -> append state commitment entry to Output Log -> commit to MPT under short lock. +- **Rejection**: A crash after MPT disk sync but before Output Log publication advances persisted trie state ahead of the Output Log (`MPT_Durable_Size > Output_CP.InputSize`), causing a restarted node to serve unwitnessed state. +- **Chosen Design**: Predict root under shared read lock -> publish and witness in the Output Log -> commit to MPT under short exclusive lock. --- -## 4. Verification Gate - -All MPT variable-length canonical key, prefix proof, and Output Log publisher tests are verified via: +## 3. Verification Gate ```bash go test -race ./vindex/internal/mpt/... ./vindex/internal/tree/... diff --git a/vindex/mapfn/README.md b/vindex/mapfn/README.md index 1503ce1..f8e66a1 100644 --- a/vindex/mapfn/README.md +++ b/vindex/mapfn/README.md @@ -1,52 +1,33 @@ # WASM Guest SDK & Plugin Interface -This document defines the WebAssembly guest ABI, developer contract, memory management protocols, load-bearing invariants, and verified performance optimizations across VIndex's mapping packages: -- **`vindex/mapfn/sdk`** *(public Go package)*: External zero-allocation Go guest SDK imported by WASM plugin authors. -- **`vindex/internal/wasmhost`** *(internal Go package)*: Internal WASI-free Wazero sandbox host used by `vindexd`, `vindex-auditor`, and `vindex-wasm`. +Every transparency ecosystem indexes different log entry formats (such as DER X.509 certificates in Certificate Transparency, module lines in Go SumDB, or JSON provenance records in Sigstore). VIndex isolates entry parsing and canonical key extraction inside a hermetic, zero-I/O WebAssembly sandbox so ecosystem authors can write custom mapping logic without risking host memory safety or causing state divergence across independent auditors compiled with different toolchains or operating systems. ---- +This document covers the public Go guest SDK (`vindex/mapfn/sdk`), the language-agnostic shared-memory bundle ABI (`input_buffer` / `map_bundle`), the internal Wazero sandbox host (`vindex/internal/wasmhost`), and the offline verification CLI (`vindex-wasm`). -## 1. Context & Objectives - -### 1.1 Architectural Separation: User Callback vs. Guest SDK vs. Host Runtime -Mapping in VIndex operates across three decoupled layers: -1. **User Logic (`MapFn`)**: The pure mapping logic (`func(entry []byte, emit func(key []byte))` in Go, or any language targeting the slab ABI) written by ecosystem developers. In Go, user logic has no knowledge of WebAssembly, linear memory pointers, or host runtimes. -2. **Public Guest SDK (`vindex/mapfn/sdk`, Go) / Language-Agnostic Slab ABI**: `vindex/mapfn/sdk` is the public Go guest harness compiled into `.wasm` alongside Go `MapFn` callbacks. Because the underlying `input_buffer` / `map_bundle` slab ABI (§2.2) uses flat little-endian linear memory, guest plugins can equally be written in Rust (`wasm32-unknown-unknown` / `wasm32-wasip1`), C/C++, Zig, or TinyGo without `mapfn/sdk`. -3. **Internal Host Sandbox Runtime (`vindex/internal/wasmhost` & `vindex/internal/ingest`)**: `internal/wasmhost` provides the internal, WASI-free Wazero sandbox pool that loads the compiled `.wasm` plugin, executes `map_bundle` over 256-entry bundles, strictly validates every emitted canonical key (`1..1024` bytes in `0x21..0x7E`, returning a fatal error that immediately `HALT`s the daemon on violation), sorts and deduplicates canonical keys per entry (`entry_idx`) immediately upon unpacking the output slab, and passes the deduplicated un-hashed canonical keys downstream. - -### 1.2 Problem Statement & Real-World Friction -Every transparency ecosystem indexes different payloads (e.g. DER X.509 certificates in Certificate Transparency, newline-delimited strings in Go SumDB, or JSON provenance records in Sigstore). VIndex must allow ecosystem authors to define custom extraction logic without compromising node security, crashing the host process, or causing state divergence across independent auditors. - -Relying on in-process native Go plugins or shared libraries introduces three severe failure modes: -1. **Toolchain Version Drift & Non-Determinism**: If an Index Operator and an Auditor compile their code with different Go versions (e.g. Go 1.23 vs. Go 1.25) or execute on different operating systems, standard library behavior can diverge. A subtle parsing change in ASN.1 DER certificate decoding or Unicode normalization across compiler versions on even **one entry out of 100 million** causes the computed `mpt_root` to diverge permanently, triggering false split-view alarms and breaking consensus. -2. **Host Process Fragility**: Untrusted or malformed entries in public logs could cause out-of-bounds memory accesses, nil pointer panics, or unbounded recursion. Running native code directly in the host engine risks corrupting host memory. -3. **Foreign Function Interface (FFI) Overhead**: Naively executing foreign functions across language boundaries on a per-entry basis introduces excessive context-switching and memory-copying overhead, consuming ~23% of total host CPU time purely in FFI crossings. - -### 1.3 Goals & Non-Goals -- **Goals**: - - Provide a high-level, idiomatic public Go developer contract (`func(entry []byte, emit func(key []byte))`) allowing ecosystem authors to write pure functions without managing raw WebAssembly linear memory pointers. - - Provide a zero-allocation public guest runtime SDK (`vindex/mapfn/sdk`) that binds the developer's function to the high-performance `map_bundle` ABI. - - Freeze compiler and standard library bytecode inside a hermetic `.wasm` binary, guaranteeing bit-for-bit identical execution across all host architectures, operating systems, and toolchain versions. - - Amortize boundary-crossing overhead by processing full 256-entry bundles in a single FFI call using contiguous shared-memory slabs. - - Provide an offline CLI (`vindex-wasm`) to inspect ABI exports and `.wasm` artifact SHA-256 digests, test canonical key extraction, and benchmark throughput prior to deployment. -- **Non-Goals**: - - **No Ingestion Concurrency in `mapfn/sdk`**: `vindex/mapfn/sdk` contains strictly in-guest ABI harnesses, while `vindex/internal/wasmhost` provides the internal zero-I/O Wazero module runner; multi-worker pool orchestration lives in the Ingest subsystem (`internal/ingest`). - - **No In-Place Dynamic Upgrades**: Once a compiled plugin binary is deployed for an Output Log, mapping logic is immutable for the lifetime of that Output Log. Fixing a parsing bug requires deploying a new index instance from its configured `from` start index (`0` or a suffix start index). - - **No Host I/O or State**: Guest modules are strictly pure functions; they have zero access to disk, network, environment variables, or host clocks. - -### 1.4 Requirements & Constraints -- **Host Runtime**: Pure Go WebAssembly runtime via `github.com/tetratelabs/wazero` (`GOOS=wasip1 GOARCH=wasm` guest binaries instantiated in `internal/wasmhost` with stubbed, zero-I/O WASI imports and zero CGO dependencies). -- **WASM Memory Constraints**: Guest modules execute within a bounded linear memory space (tens of megabytes by default, scalable via configuration), with pre-allocated multi-megabyte static input and output slabs. -- **Standard Go WASM Binary Size**: Standard Go (`GOOS=wasip1 GOARCH=wasm`) bundles the complete Go runtime and garbage collector into the binary (~1.9 MB to ~2.5 MB), whereas TinyGo produces ~200 KB binaries but lacks support for certain standard library packages (such as full `crypto/x509`). +### Non-Goals +- **No Worker Pool Orchestration in the Guest SDK**: The guest SDK provides only in-guest ABI bindings, while `internal/wasmhost` compiles the `.wasm` module once and provisions isolated per-worker module instances; multi-worker pipeline concurrency is managed by [`internal/ingest`](../internal/ingest/README.md). +- **No In-Place Mapping Upgrades**: A deployed mapping function is fixed for the lifetime of an Output Log. Changing extraction logic requires deploying a new index instance from its configured `from` start index. +- **No Host I/O or Cross-Call State**: Guest modules are pure, stateless functions with zero access to the filesystem, network, environment variables, or host clocks. --- -## 2. Detailed Design +## 1. Design + +### 1.1 Three-Layer Mapping Architecture + +Mapping is decoupled across three layers so plugin authors can write idiomatic Go (or Rust, C/C++, Zig, or TinyGo) while the host enforces strict sandboxing and canonical key validation: -### 2.1 The High-Level Developer Experience -Ecosystem developers write standard, idiomatic Go against `vindex/mapfn/sdk`. They do not write assembly, manage WebAssembly linear pointers, or handle memory layout serialization. +| Layer | Package / Component | Responsibility | +| :--- | :--- | :--- | +| **User Mapping Callback** | Ecosystem plugin (`MapFn`) | Pure function extracting zero or more canonical keys from a single raw Input Log entry. | +| **Guest SDK & Slab ABI** | `vindex/mapfn/sdk` (or non-Go slab target) | Binds user callbacks to the low-level `input_buffer` / `map_bundle` shared-memory ABI in flat little-endian WASM linear memory. | +| **Host Sandbox Runtime** | `vindex/internal/wasmhost` ([`wazero`](https://github.com/tetratelabs/wazero), zero CGO) | Instantiates `.wasm` modules with stubbed zero-I/O WASI imports, executes 256-entry bundles, validates canonical key bounds (`1..1024` bytes in `0x21..0x7E`), and sorts and deduplicates emitted keys per entry. | -The developer implements a pure mapping function using a streaming `emit` callback and registers it in `init()` (Go guest plugins compile as WASI reactors with `GOOS=wasip1 GOARCH=wasm -buildmode=c-shared`, isolating `//go:wasmexport` bindings in `mapfn/sdk` behind `//go:build wasm` so native `go test` runs cleanly on host OSes; the host instantiates modules without running `_start` / `main()` and invokes `_initialize` if exported so both WASI reactors and `wasm32-unknown-unknown` guests initialize deterministically): +Within `vindex`, the `.wasm` module is compiled once (for example with `go build -trimpath -buildvcs=false -buildmode=c-shared`) and pinned out-of-band by its `SHA-256(wasm_bytes)` digest (`vindex-wasm inspect`). At the `tlog-vindex` protocol layer, `c2sp.org/tlog-vindex-state@v1` omits an in-band `map-fn` commitment pre-1.0 (see [`../docs/tlog-vindex.md`](../docs/tlog-vindex.md#design-log)); independent auditors may either execute the pinned `.wasm` artifact or run a clean-room native implementation of the same mapping specification. + +### 1.2 Go Guest Developer Contract (`vindex/mapfn/sdk`) + +Go plugin authors implement a streaming `emit` callback and register it in `init()`: ```go type EmitMapFunc func(entry []byte, emit func(key []byte)) @@ -54,19 +35,13 @@ type EmitMapFunc func(entry []byte, emit func(key []byte)) func RegisterEmit(fn EmitMapFunc) ``` -Every emitted `key` MUST be a valid canonical key (`1..1024` printable ASCII bytes in `0x21..0x7E`, containing no `NUL` or whitespace bytes; see [`../docs/tlog-vindex.md`](../docs/tlog-vindex.md#canonical-keys) and [`../docs/APPLICATIONS.md`](../docs/APPLICATIONS.md) for ecosystem-specific canonicalization rules). Guest mappers MUST skip malformed or unindexable fields before calling `emit`: emitting an invalid canonical key (`len == 0`, `len > 1024`, or any byte outside `0x21..0x7E`) causes the host (`internal/wasmhost`) to return a fatal validation error that immediately **`HALT`s** the Index Operator (`vindexd`) or Auditor (`vindex-auditor`). +- **Build & Reactor Initialization**: Go guest plugins compile as WASI reactors (`GOOS=wasip1 GOARCH=wasm -buildmode=c-shared`, ~1.9 to 2.5 MB with the standard Go runtime or ~200 KB with TinyGo). `//go:wasmexport` bindings in `mapfn/sdk` are isolated behind `//go:build wasm` so native `go test` runs cleanly on host OSes. On startup, `internal/wasmhost` instantiates the module without running `_start` / `main()` and invokes `_initialize` if exported so both WASI reactors and `wasm32-unknown-unknown` guests initialize deterministically. +- **Canonical Key Validation & Fatal `HALT`**: Every emitted `key` MUST be a valid canonical key (`1..1024` printable ASCII bytes in `0x21..0x7E`, containing no `NUL` or whitespace bytes; see [`../docs/tlog-vindex.md`](../docs/tlog-vindex.md#canonical-keys) and [`../docs/APPLICATIONS.md`](../docs/APPLICATIONS.md)). Guest mappers must skip malformed or unindexable fields before calling `emit`: if the guest traps, overflows its linear memory slabs, or emits an invalid canonical key (`len == 0`, `len > 1024`, or any byte outside `0x21..0x7E`), `internal/wasmhost` returns a fatal validation error that immediately **`HALT`s** the indexer or auditor rather than silently dropping the key. +- **Zero-Allocation Streaming**: Each `emit(key)` call writes the length-prefixed canonical key directly into the pre-allocated output slab in-place, avoiding intermediate `[]string` or `[][]byte` slice allocations inside the guest. -The `sdk.RegisterEmit` harness automatically binds the user's pure function to the low-level `map_bundle` FFI export. Each call to `emit(key)` writes the length-prefixed canonical key directly into the pre-allocated output memory slab in-place. This eliminates intermediate slice allocations (`[]string` or `[][]byte`), dynamic array resizing, and garbage collector overhead inside the WebAssembly guest. +### 1.3 Language-Agnostic Slab ABI (`input_buffer` & `map_bundle`) -#### WASM Guest ABI Contract: -To exchange data across the FFI boundary while eliminating per-entry allocation overhead: -- **Memory Lifecycle**: The guest exports `input_buffer() uint32` returning the base pointer of a pre-allocated multi-megabyte input slab in linear memory. The host serializes the 256-entry bundle directly into this buffer, invokes `map_bundle`, and reads and validates the resulting canonical keys from the returned output slab pointer. -- **Statelessness Invariant**: Guest code must maintain zero mutable state across invocations. All mapping logic must be deterministic and pure; execution depends strictly on the bytes of each individual Input Log entry. - ---- - -### 2.2 Language-Agnostic Low-Level WASM ABI & The Pack-and-Wipe Memory Slab -To eliminate the 23% CPU overhead of per-entry FFI boundary crossings—and allow guest plugins to be authored in any WASM-capable systems language (Go via `mapfn/sdk`, Rust, C/C++, Zig, or TinyGo)—the host (`internal/wasmhost`) and guest communicate through contiguous pre-allocated linear memory slabs using the `input_buffer` and `map_bundle` exports: +Crossing the Go-to-WASM Foreign Function Interface (FFI) once per entry requires 768 transitions per 256-entry bundle and consumes ~23% of total host CPU time. To reduce FFI transitions to 2–3 per bundle (< 1% CPU overhead) and support guests written in any WASM-capable language, the host and guest exchange entire 256-entry bundles through pre-allocated contiguous linear memory slabs: ```text //go:wasmexport input_buffer @@ -76,99 +51,58 @@ func input_buffer() uint32 func map_bundle(inputPtr uint32, inputLen uint32) uint64 ``` -#### Memory Slab Lifecycle: -1. **Pack (Host)**: The Go host (`internal/wasmhost`) queries `input_buffer()` once on instance initialization (falling back to an exported `allocate(size)` function if `input_buffer` is not exported) and serializes all entries of a 256-entry bundle into the contiguous input slab in the guest's linear memory. If the framed input bundle exceeds the static input slab capacity, the host immediately returns a fatal error (`HALT`). -2. **Execute (FFI Call)**: The host invokes `map_bundle(inputPtr, inputLen)`. -3. **Stream (Guest SDK)**: Inside WASM, the SDK harness reads the `[Count+1]uint32` offset table (`entry := payload[offsets[i]:offsets[i+1]]`), iterates over all entries in a tight in-guest loop, and invokes `userMapFn(entry, emit)`. Each call to `emit` copies the length-prefixed canonical key directly into the pre-allocated output slab at the current write cursor, avoiding intermediate collection allocations. If the input buffer cannot be decoded or `emit` overflows the static output slab capacity, `map_bundle` aborts and returns `0` (`outputPtr = 0, outputLen = 0`). -4. **Return, Validate, Deduplicate & Unpack (Host)**: The guest returns a packed 64-bit integer `uint64(outputPtr)<<32 | uint64(outputLen)`. If the return value is `0` (indicating input decoding failure or output slab overflow) or falls outside guest linear memory bounds, the host returns a fatal error (`HALT`). Otherwise, the host reads all extracted canonical keys directly from guest linear memory, validates every key (`1 <= len(key) <= 1024`, all bytes in `0x21..0x7E`; any violation returns a fatal error that immediately **`HALT`s** the indexer or auditor), and **sorts and deduplicates the canonical keys emitted for each individual entry** before passing the batch downstream. -5. **Wipe (Guest/Host)**: Output slab write cursors are reset (`offset = 0`) at the start of the next `map_bundle` call, completely eliminating guest heap allocations and GC pauses. +1. **Pack (Host)**: `internal/wasmhost` queries `input_buffer()` once on instance initialization (falling back to an exported `allocate(size)` function if `input_buffer` is not exported) and serializes the 256-entry bundle into the guest's static input slab. If the framed bundle exceeds the input slab capacity, the host returns a fatal error (`HALT`). +2. **Execute & Stream (Guest)**: The host invokes `map_bundle(inputPtr, inputLen)`. The guest reads the `[Count+1]uint32` offset table (`entry := payload[offsets[i]:offsets[i+1]]`), invokes the mapping callback per entry, and appends length-prefixed keys into the static output slab. If input decoding fails or `emit` overflows the output slab capacity, `map_bundle` aborts and returns `0`. +3. **Validate, Deduplicate & Wipe (Host)**: The guest returns `uint64(outputPtr)<<32 | uint64(outputLen)`. If the return value is `0` or out of bounds, the host returns a fatal error (`HALT`). Otherwise, the host reads all emitted keys from guest memory, validates canonical key bounds (`1..1024` bytes in `0x21..0x7E`), and **sorts and deduplicates canonical keys per entry** before passing the batch downstream. Output slab cursors reset to `0` on the next `map_bundle` call with zero guest heap allocations. -#### Binary Slab Layouts (All `uint32` fields encoded in Little-Endian): +#### Binary Slab Layouts (All `uint32` fields encoded in Little-Endian) | Memory Buffer | Offset | Field Name | Data Type | Description | | :--- | :--- | :--- | :--- | :--- | -| **Input Slab** | `0` | `EntryCount` | `uint32` (4B, LE) | Number of entries in the 256-entry bundle (`Count`, 1 to 256). | -| | `4` | `OffsetTable` | `[Count+1]uint32` (LE) | `Count + 1` byte offsets relative to `PayloadBlock`: `OffsetTable[0] = 0`, followed by the end offset of each entry (`entry[i] = PayloadBlock[OffsetTable[i]:OffsetTable[i+1]]`). | +| **Input Slab** | `0` | `EntryCount` | `uint32` (4B, LE) | Number of entries in the bundle (`Count`, 1 to 256). | +| | `4` | `OffsetTable` | `[Count+1]uint32` (LE) | `Count + 1` byte offsets relative to `PayloadBlock`: `OffsetTable[0] = 0`, followed by each entry's end offset (`entry[i] = PayloadBlock[OffsetTable[i]:OffsetTable[i+1]]`). | | | `8 + 4*Count` | `PayloadBlock` | `[]byte` | Raw, contiguous Input Log entry bytes concatenated end-to-end. | | **Output Slab** | `0` | `EntryCount` | `uint32` (4B, LE) | Number of entries processed (`Count`). | | | `4` | `KeyCountTable` | `[Count]uint32` (LE) | Number of canonical keys emitted by each respective entry. | | | `4 + 4*Count` | `KeyBlock` | `[]byte` | Concatenated length-prefixed canonical keys (`uint32 length` (LE) + raw canonical key bytes). | ---- - -### 2.3 In-Situ Invariants & Performance Optimizations - -- **[Correctness Invariant] Pinned Artifact Determinism**: - - *Rule*: Within `vindex`, the `MapFn` `.wasm` module is compiled once (e.g. reproducibly via `go build -trimpath -buildvcs=false -buildmode=c-shared` or an equivalent Rust/Zig/C toolchain) and executed identically by `vindexd` and WASM-based `vindex-auditor` instances, pinned out-of-band by its `SHA-256(wasm_bytes)` artifact digest (`vindex-wasm inspect`). At the `tlog-vindex` protocol layer, `c2sp.org/tlog-vindex-state@v1` intentionally omits an in-band `map-fn` commitment pre-1.0 (see [`docs/tlog-vindex.md`](../docs/tlog-vindex.md#design-log)), and independent auditors may either verify and execute the pinned `.wasm` artifact or audit via a native clean-room implementation matching the mapping specification. - - *Rationale*: Running pinned `.wasm` bytecode across `vindex` nodes eliminates subtle compiler, standard library, or operating system drift (e.g. updates to ASN.1 parsers or Unicode tables), while keeping `tlog-vindex` open to multi-language guests and native non-WASM auditors. - - *Consequence ("Or Else")*: If Index Operators and WASM-based Auditors ran unpinned native builds compiled with drifting toolchains, edge-case parsing discrepancies on a single entry would produce conflicting `mpt_root` commitments and false split-view alarms. +### 1.4 Offline Verification CLI (`vindex-wasm`) -- **[Correctness Invariant] Hermetic Isolation, Canonical Key Validation & Fatal HALT Policy**: - - *Rule*: Guest WASM execution runs inside a hermetically sealed WASI-free sandbox (`internal/wasmhost`) with zero host filesystem access, zero network sockets, and deterministic frozen clocks. If the guest encounters an unhandled panic, memory fault, execution trap, or emits an invalid canonical key (`len == 0`, `len > 1024`, or any byte outside `0x21..0x7E`), `internal/wasmhost` returns a fatal validation error that immediately **`HALT`s** the indexer or auditor. - - *Rationale*: Ensures that execution is 100% deterministic, spec-compliant, and reproducible across all Index Operators and Auditors. - - *Consequence ("Or Else")*: Silently swallowing guest traps or dropping malformed canonical keys produces divergent key mappings, resulting in false non-inclusion proofs or `mpt-root` audit mismatches. - -- **[Performance Optimization] Bundled FFI Invocation (`map_bundle`)**: - - *Mechanism*: Packs an entire 256-entry bundle into the input slab and executes a single FFI call per bundle, rather than crossing the boundary once per entry. - - *Impact*: Reduces FFI transitions from 768 to 2–3 per 256-entry bundle, slashing FFI CPU overhead from ~23% of host CPU time to < 1%. - -- **[Performance Optimization] Pack-and-Wipe Slab Allocation**: - - *Mechanism*: Slabs are allocated once in guest linear memory and reused across sequential bundle invocations with simple pointer resets. - - *Impact*: Eliminates in-guest heap allocations and garbage collector pauses, keeping memory usage bounded and CPU dedicated to parsing. - ---- - -### 2.4 Offline Developer Verification Tooling (`vindex-wasm`) -The mapping subsystem provides a standalone CLI tool (`vindex/cmd/vindex-wasm`) so plugin authors can inspect, test, and benchmark their `.wasm` binaries before production deployment: - -- **`vindex-wasm inspect`**: Instantiates the module via `internal/wasmhost`, validates the required `input_buffer` and `map_bundle` ABI exports and linear memory bounds, and prints the 32-byte `.wasm` artifact SHA-256 digest in both canonical base64 and hex for out-of-band deployment and auditor pinning. -- **`vindex-wasm test`**: Executes the module against literal entry strings or binary entry files, validates every emitted canonical key against `1..1024` printable ASCII bounds (`0x21..0x7E`), and prints extracted canonical keys in plain text or JSON. -- **`vindex-wasm bench`**: Measures single-worker and multi-bundle mapping throughput (`entries/sec`) and per-bundle latency over repeated 256-entry bundles. +`vindex/cmd/vindex-wasm` allows plugin authors to inspect, test, and benchmark `.wasm` modules before deployment: +- **`vindex-wasm inspect`**: Instantiates the module via `internal/wasmhost`, validates `input_buffer` / `map_bundle` exports and memory bounds, and prints the 32-byte `.wasm` SHA-256 digest in base64 and hex for operator and auditor pinning. +- **`vindex-wasm test`**: Runs the module against literal entry strings or binary entry files, validates canonical key bounds (`1..1024` bytes in `0x21..0x7E`), and prints extracted keys in plain text or JSON. +- **`vindex-wasm bench`**: Benchmarks single-worker and multi-bundle mapping throughput (`entries/sec`) and per-bundle latency over repeated 256-entry bundles. ```bash -# Validate ABI exports, linear memory bounds, and print .wasm SHA-256 digest (base64 & hex) vindex-wasm inspect --wasm=map.wasm - -# Test canonical key extraction against a literal entry string or binary entry file vindex-wasm test --wasm=map.wasm --input="golang.org/x/crypto v0.1.0 h1:..." vindex-wasm test --wasm=map.wasm --input_file=entry.bin --format=json - -# Run performance and throughput benchmarks (entries/sec) vindex-wasm bench --wasm=map.wasm --input="golang.org/x/crypto v0.1.0 h1:..." --iterations=1000 ``` --- -## 3. Alternatives Considered (or Tried) +## 2. Alternatives Considered -### 3.1 Per-Entry Mapping (`map_leaf`) vs. Bundled Mapping (`map_bundle`) +### 2.1 Per-Entry Mapping (`map_leaf`) vs. Bundled Mapping (`map_bundle`) - **Proposed**: Invoking an exported `map_leaf(ptr, len)` guest function individually for every Input Log entry. -- **Empirical Rejection**: Generated 768 FFI calls per 256-entry bundle (memory allocation, entry mapping, and arena reset transitions x 256). Profiling revealed that ~23% of total host CPU time was spent purely on boundary-crossing overhead. -- **Chosen Design**: Bundled mapping (`map_bundle`) processes all 256 entries in a single invocation via shared memory slabs, reducing FFI overhead to < 1%. +- **Rejection**: Generated 768 FFI calls per 256-entry bundle (allocation, mapping, and reset x 256), spending ~23% of total host CPU time on boundary-crossing overhead. +- **Chosen Design**: Bundled mapping (`map_bundle`) processes up to 256 entries per FFI call via shared memory slabs, reducing FFI overhead to < 1%. -### 3.2 In-Process Go Plugins / CGO vs. WebAssembly Sandboxing -- **Proposed**: Dynamically loading Go plugins (`plugin.Open`) or shared C libraries (`cgo`). -- **Theoretical Rejection**: - - In-process native plugins do not guarantee determinism across toolchain versions (Go compiler upgrades can alter standard library parsing behavior). - - Memory corruptions or panics inside the plugin crash the main daemon. - - Go plugins have poor cross-platform support and require identical dependency version trees between the host and plugin. -- **Chosen Design**: Standardized on Wazero WebAssembly sandboxing (`internal/wasmhost`). +### 2.2 In-Process Go Plugins / CGO vs. WebAssembly Sandboxing +- **Proposed**: Dynamically loading native Go plugins (`plugin.Open`) or shared C libraries (`cgo`). +- **Rejection**: Native plugins do not guarantee bit-for-bit determinism across Go compiler/stdlib upgrades or host OSes, crash the entire daemon on a plugin panic or memory fault, and require identical host/plugin dependency trees. +- **Chosen Design**: Zero-CGO Wazero WebAssembly sandboxing (`internal/wasmhost`). --- ## Appendix: Guest Parsing Optimization Notes (Certificate Transparency) -While WebAssembly provides sandboxed memory safety and cross-platform determinism, compiling reflection-heavy standard library decoders (`crypto/x509`, `encoding/asn1`, `encoding/json`, `regexp`) into `wasip1/wasm` incurs high heap allocation churn and unnecessary cryptographic validation (such as public-key and signature decoding) irrelevant to canonical key extraction. - -- **[Performance Optimization] Zero-Allocation In-Guest Parsing**: - - *Mechanism*: Ecosystem plugins should prefer targeted, in-place byte scanners over reflection-heavy standard library decoders (for example, scanning raw ASN.1 TLV streams for Subject Common Name OID `2.5.4.3` and SAN `dNSName` OID `2.5.29.17` before applying canonical DNS label normalization per [`../docs/APPLICATIONS.md`](../docs/APPLICATIONS.md)). - - *Impact*: Eliminates in-guest AST heap allocations and extraneous cryptographic validation (~7.5x throughput improvement over `crypto/x509` in `wasip1/wasm`), preventing guest WASM execution from starving the downstream storage and trie pipeline. +Compiling reflection-heavy standard library decoders (`crypto/x509`, `encoding/asn1`, `encoding/json`, `regexp`) into `wasip1/wasm` incurs high heap allocation churn and unnecessary cryptographic validation (such as public-key and signature decoding) irrelevant to canonical key extraction. Ecosystem plugins should prefer targeted, in-place byte scanners (for example, scanning raw ASN.1 TLV streams for Subject Common Name OID `2.5.4.3` and SAN `dNSName` OID `2.5.29.17` before applying canonical DNS label normalization per [`../docs/APPLICATIONS.md`](../docs/APPLICATIONS.md)), which achieves a ~7.5x throughput improvement over `crypto/x509` in `wasip1/wasm`. --- -## 4. Verification Gate - -All public guest SDK (`mapfn/sdk`), internal sandbox host (`internal/wasmhost`), example mapper (`ct`, `mtc`, `sumdb`), and `vindex-wasm` CLI tests are verified via: +## 3. Verification Gate ```bash go test -race ./vindex/mapfn/... ./vindex/internal/wasmhost/... ./vindex/cmd/vindex-wasm/... From 5ab0897df9056597a1f48e7013cdb9b71b7ae7d5 Mon Sep 17 00:00:00 2001 From: Martin Hutchinson Date: Mon, 2 Nov 2026 11:52:56 +0000 Subject: [PATCH 3/3] docs(vindex): add Pebble LSM primer on prefix seeks and bloom filters --- vindex/internal/kvstore/README.md | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/vindex/internal/kvstore/README.md b/vindex/internal/kvstore/README.md index d91e1e4..9851a63 100644 --- a/vindex/internal/kvstore/README.md +++ b/vindex/internal/kvstore/README.md @@ -11,7 +11,15 @@ ## 1. Design -### 1.1 Inverted 65,536-Entry Chunk Key Layout +### 1.1 Pebble LSM Primer + +[Pebble](https://github.com/cockroachdb/pebble) is an embedded Log-Structured Merge-tree (LSM) key-value database (inspired by RocksDB) that stores keys in lexicographical byte order across an in-memory MemTable and immutable on-disk Sorted String Tables (SSTables). Three Pebble primitives underpin this design: + +- **Lower-Bound Range Seeks (`SeekGE` & `Next`)**: Unlike a hash table, an LSM iterator does not require an exact key match. Seeking to a target byte key binary-searches the SSTable block index for the **smallest existing key `>= target`** (jumping over arbitrarily large gaps of non-existent chunk numbers in a single lookup), and `Next()` steps forward to the next existing key in sorted order. +- **Split-Key Prefix Bloom Filters & `SeekPrefixGE`**: Pebble's [`Comparer.Split`](https://pkg.go.dev/github.com/cockroachdb/pebble#Comparer) divides every key into a fixed **prefix** (the first 33 bytes: `'c' + KeyHash`) and a **suffix** (the 8-byte inverted chunk number), and builds per-SSTable 10-bit [Bloom filters](https://pkg.go.dev/github.com/cockroachdb/pebble/bloom) over the 33-byte prefix rather than the full 41-byte key. Unless noted otherwise, **"seek"** in this document refers to [`Iterator.SeekPrefixGE`](https://pkg.go.dev/github.com/cockroachdb/pebble#Iterator.SeekPrefixGE): Pebble first checks each SSTable's Bloom filter against the 33-byte prefix (skipping SSTables that do not contain `KeyHash` with **zero disk I/O**, even when the caller's target suffix does not match an existing chunk), positions the iterator at the smallest existing key `>= target` within that prefix, and bounds `SeekPrefixGE` and `Next()` so iteration never crosses into an adjacent `KeyHash`. +- **Atomic Write Batches (`NoSync` vs. `Sync`)**: [`Batch.Commit`](https://pkg.go.dev/github.com/cockroachdb/pebble#Batch.Commit) applies a group of key mutations atomically. Committing with `NoSync` makes writes immediately visible in memory without blocking on disk I/O, while `Sync` enforces an OS `fsync` durability barrier before returning. + +### 1.2 Inverted 65,536-Entry Chunk Key Layout The Input Log entry index space is partitioned into fixed global windows of 65,536 entries (`chunkNum = entryIdx / 65536`, `relOffset = uint16(entryIdx % 65536)`). While the MPT stores un-hashed canonical keys (`1..1024` bytes in `0x21..0x7E`), Pebble allocates chunks only for windows in which a key occurs, keyed by fixed 41-byte records using internal `KeyHash = SHA-256(key)` digests and bitwise-inverted chunk numbers (`^BigEndian(chunkNum) = 0xFFFFFFFFFFFFFFFF - chunkNum`): @@ -19,9 +27,9 @@ The Input Log entry index space is partitioned into fixed global windows of 65,5 Key (41 bytes) = [Prefix 'c' (1B)] + [KeyHash (32B)] + [^BigEndian(chunkNum) (8B)] ``` -- **Inverted Ordering & 33-Byte Prefix Bloom Filter**: Pebble's custom comparer splits keys at byte 33 (`'c' + KeyHash`) and builds full-table 10-bit Bloom filters on that 33-byte prefix across all SSTable levels. A prefix-bounded seek on `'c' + KeyHash` prunes non-existent keys with **zero disk I/O**, and for existing keys lands **directly on the newest active chunk** (eliminating the 7.5x scan latency penalty of forward-ordered chunks on deep keys). +Because bitwise inversion reverses numerical order (`chunk_A < chunk_B` implies `^chunk_A > ^chunk_B`), chunks for a given `KeyHash` are stored **newest-first** in the LSM keyspace. Seeking to `'c' + KeyHash` lands directly on the newest active chunk (eliminating the 7.5x scan latency penalty of forward-ordered chunks on deep keys), while seeking to `'c' + KeyHash + ^BigEndian(targetChunk)` jumps directly to the newest existing chunk with `chunkNum <= targetChunk`. -### 1.2 Delimitless Binary Chunk Values & Compact Ranges +### 1.3 Delimitless Binary Chunk Values & Compact Ranges Each chunk value stores a cumulative RFC 6962 compact range covering all occurrences of that key across prior chunks (`chunk < chunkNum`), followed by dense 16-bit relative offsets for occurrences within `chunkNum` (saving 75% storage compared to 64-bit indices): @@ -37,18 +45,18 @@ Preceding uint64 (8 bytes, BigEndian) || CompactRange (32 bytes * popcount(Prece Because `K = popcount(Preceding)` determines the exact length of `CompactRange`, `RelativeIndices` always starts at byte offset `8 + 32*K` without delimiters or length prefixes. -### 1.3 Write Pipeline, Two-Generational Cache & Point-in-Time Reconstruction +### 1.4 Write Pipeline, Two-Generational Cache & Point-in-Time Reconstruction - **Per-Entry Deduplication & Parallel Value Log Hashing**: Batch writes defensively sort and deduplicate `(canonicalKey, entryIdx)` pairs so each `entryIdx` is appended at most once per key (`relOffset_0 < relOffset_1 < ...`). Distinct modified keys in a batch are partitioned across worker goroutines that compute RFC 6962 leaf hashes (`SHA-256(0x00 || uint64_be(entryIdx))`), interior node merges (`SHA-256(0x01 || left || right)`), `value_root` digests (`internal/wire`), and serialized chunk values in parallel in memory before committing sequentially in a single atomic Pebble batch, returning modified `(canonicalKey, value_root)` mappings for MPT commitment. When an occurrence crosses into a higher 65,536-entry window, the prior chunk is sealed and a new active chunk is initialized with the snapshotted `Preceding` and `CompactRange`. -- **Two-Generational Active Chunk Cache**: Each cache entry holds the active chunk descriptor (`Preceding`, `CompactRange`, `RelativeIndices`) alongside its in-memory running compact range, updating incrementally in O(1) amortized leaf hashes per new occurrence instead of re-hashing prior `RelativeIndices` from index `0` (reconstructed lazily once on cache miss or restart). Maintaining current and previous batch generations absorbs repeated writes to hot keys without LRU lock contention and cuts Pebble block cache read I/O by >90%. -- **Synchronous Persistence Barrier & Crash-Replay Idempotency**: Intermediate commit slabs within a sync cycle may write to Pebble without an immediate `fsync`, but the final slab reaching `Target_CP.Size` (or an explicit flush barrier) executes a durable disk sync committing `KV_CP` before Output Log publication (`KV_CP.Size >= Output_CP.InputSize`). Every batch write atomically records the batch's ending entry watermark alongside modified `'c'` records; during startup recovery or batch replay across `[Output_CP.InputSize, KV_CP.Size)`, any `entryIdx` below the persisted KV watermark (or `<=` the active chunk's highest persisted offset) is skipped. -- **Point-in-Time `value_root` Reconstruction**: To reconstruct a key's 32-byte `value_root` as of any target `input_size` (used during crash recovery and batch replay for keys with `>= 1` occurrences in `[from, input_size)`), the store seeks to `'c' + KeyHash + ^BigEndian((input_size - 1) / 65536)`, skips any chunk with `chunkNum * 65536 >= input_size`, clones that chunk's `Preceding` and `CompactRange`, appends `SHA-256(0x00 || uint64_be(entryIdx))` for each relative offset with `chunkNum * 65536 + relOffset < input_size`, and folds the compact range right-to-left into `value_root`. Keys with zero total occurrences in `[from, input_size)` (`Preceding` plus matching offsets equals `0`) are omitted from the MPT per [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md). +- **Two-Generational Active Chunk Cache**: Each cache entry holds the active chunk descriptor (`Preceding`, `CompactRange`, `RelativeIndices`) alongside its in-memory running compact range, updating incrementally in O(1) amortized leaf hashes per new occurrence instead of re-hashing prior `RelativeIndices` from index `0` (reconstructed lazily once via a prefix seek on cache miss or restart). Maintaining current and previous batch generations absorbs repeated writes to hot keys without LRU lock contention and cuts Pebble block cache read I/O by >90%. +- **Synchronous Persistence Barrier & Crash-Replay Idempotency**: Intermediate commit slabs within a sync cycle commit with `NoSync`, while the final slab reaching `Target_CP.Size` (or an explicit flush barrier) commits with `Sync` (`fsync`), advancing `KV_CP` before Output Log publication (`KV_CP.Size >= Output_CP.InputSize`). Every batch write atomically records the batch's ending entry watermark alongside modified `'c'` records; during startup recovery or batch replay across `[Output_CP.InputSize, KV_CP.Size)`, any `entryIdx` below the persisted KV watermark (or `<=` the active chunk's highest persisted offset) is skipped. +- **Point-in-Time `value_root` Reconstruction**: To reconstruct a key's 32-byte `value_root` as of any target `input_size` (used during crash recovery and batch replay for keys with `>= 1` occurrences in `[from, input_size)`), the store seeks to `'c' + KeyHash + ^BigEndian((input_size - 1) / 65536)` (landing on the newest existing chunk `<= (input_size - 1) / 65536`), clones that chunk's `Preceding` and `CompactRange`, appends `SHA-256(0x00 || uint64_be(entryIdx))` for each relative offset with `chunkNum * 65536 + relOffset < input_size`, and folds the compact range right-to-left into `value_root`. Keys with zero total occurrences in `[from, input_size)` (`Preceding` plus matching offsets equals `0`) are omitted from the MPT per [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md). -### 1.4 Chunk-Aligned Backward Lookup & Zero-Hashing Compact Range Reuse +### 1.5 Chunk-Aligned Backward Lookup & Zero-Hashing Compact Range Reuse Once the HTTP layer establishes `before <= Serving_CP.InputSize`, point lookups hash `canonicalKey` to `KeyHash = SHA-256(key)` and retrieve entry indices in `[from, before)` (ignoring any uncommitted chunks written ahead of `Serving_CP.InputSize`) using chunk-aligned page boundaries (`targetPageSize` default `100`, `maxChunksPerRequest` default `16`): -1. **Direct Seek**: Seeks directly to `'c' + KeyHash + ^BigEndian((before - 1) / 65536)`, landing on the newest candidate chunk without scanning newer or unrelated SSTables. -2. **Whole-Chunk Reverse Traversal**: Iterates forward through inverted keys (`^chunkNum`), moving backward chronologically. Any chunk where `chunkNum * 65536 >= before` or where zero offsets fall in `[from, before)` is skipped without incrementing `chunksVisited`. Each visited chunk with `>= 1` occurrences in `[from, before)` emits **all** matching offsets in that 65,536-entry window without intra-chunk splitting, halting once `totalMatched >= targetPageSize` or `chunksVisited >= maxChunksPerRequest` (returning up to `65,536` indices for a dense chunk, fewer than `targetPageSize` for sparse chunks, or empty / `HTTP 404` if zero occurrences exist in `[from, before)`). +1. **Lower-Bound Prefix Seek**: Calls `SeekPrefixGE` on `'c' + KeyHash + ^BigEndian((before - 1) / 65536)`, skipping any gap of empty 65,536-entry windows to land directly on the newest existing chunk with `chunkNum <= (before - 1) / 65536` without scanning newer or unrelated SSTables. +2. **Whole-Chunk Reverse Traversal**: Steps forward with `Next()` through inverted keys (`^chunkNum`), moving backward chronologically across the key's existing chunks. Any chunk where zero offsets fall in `[from, before)` is skipped without incrementing `chunksVisited`. Each visited chunk with `>= 1` occurrences in `[from, before)` emits **all** matching offsets in that 65,536-entry window without intra-chunk splitting, halting once `totalMatched >= targetPageSize` or `chunksVisited >= maxChunksPerRequest` (returning up to `65,536` indices for a dense chunk, fewer than `targetPageSize` for sparse chunks, or empty / `HTTP 404` if zero occurrences exist in `[from, before)`). 3. **Zero-Hashing Compact Range Reuse**: Because page boundaries always align to whole chunks, when `Preceding > 0` on the oldest visited matching chunk, the `/values` response copies `preceding` and the compact range hashes directly from that chunk's stored `Preceding` and `CompactRange` with zero dynamic SHA-256 hashing on the read path. ---