diff --git a/vindex/README.md b/vindex/README.md index 6bda417..5e6cc56 100644 --- a/vindex/README.md +++ b/vindex/README.md @@ -35,7 +35,7 @@ Earlier systems attempted to act as verifiable state machines that copied, aggre ## 3. The Map Sandwich Architecture -The VIndex topology bounds search indexing between two append-only logs—a structure known as the **"Map Sandwich"**: +The VIndex topology bounds search indexing between two append-only logs, a structure known as the **"Map Sandwich"**: | Component | Physical Role | Cryptographic Contract | | :--- | :--- | :--- | @@ -75,6 +75,18 @@ The VIndex topology bounds search indexing between two append-only logs—a stru ## 6. Documentation +### Core Specification & Architecture - [**C2SP `tlog-vindex` Specification**](./docs/tlog-vindex.md): Normative wire formats, MPT and value-log Merkle definitions, HTTP lookup endpoints, and client/auditor verification rules. -- [**Verifiable Index Applications**](./docs/APPLICATIONS.md): How VIndex applies across Certificate Transparency, Merkle Tree Certificates, Go SumDB, Sigstore, and Sigsum—who monitors each log, illustrative canonical key shapes, prefix queries, and epoch rotation (`from > 0`). +- [**Verifiable Index Applications**](./docs/APPLICATIONS.md): How VIndex applies across Certificate Transparency, Merkle Tree Certificates, Go SumDB, Sigstore, and Sigsum: who monitors each log, illustrative canonical key shapes, prefix queries, and epoch rotation (`from > 0`). - [**Verifiable Index Architecture**](./docs/ARCHITECTURE.md): High-level pipeline stages, checkpoint promotion invariants, concurrency model, Zero-WAL crash recovery, auditing boundaries, and architectural trade-offs. + +### Subcomponent Designs +- [**Client SDK & Query Proof Verification**](./client/README.md): Public Go lookup client API, backward value-log streaming, and offline cryptographic proof verification. +- [**WASM Guest SDK & Plugin Interface**](./mapfn/README.md): WebAssembly mapping function SDK, sandbox host interface, and verification tooling. +- [**Ingestion Pipeline & Bundle Cache**](./internal/ingest/README.md): Input Log checkpoint verification, local entry-bundle cache, worker pool orchestration, and chronological resequencing. +- [**Inverted Chunk Storage Engine**](./internal/kvstore/README.md): Embedded LSM value-log chunk layout, compact range persistence, and snapshot-bounded reads. +- [**Authenticated State Commitment**](./internal/tree/README.md): Variable-length MPT wrapper, non-blocking root prediction, Output Log publication, and serving-snapshot promotion. +- [**Read Serving & Query Endpoints**](./internal/server/README.md): `/keys` and `/values` HTTP handlers, prefix policy guardrails, and reader snapshot isolation. +- [**Coordinator & Lifecycle Engine**](./internal/coordinator/README.md): Sync-cycle orchestration, checkpoint ratcheting, Genesis Backfill, and Zero-WAL startup recovery. +- [**Independent Log Auditor**](./internal/auditor/README.md): Continuous Output Log tailing, independent Input Log replay, and divergence alerting. +- [**Hammer Load Testing & Verification**](./internal/hammer/README.md): Synthetic workload generation and continuous end-to-end lookup verification. diff --git a/vindex/client/README.md b/vindex/client/README.md new file mode 100644 index 0000000..d79b12f --- /dev/null +++ b/vindex/client/README.md @@ -0,0 +1,136 @@ +# Client SDK & Query Proof Verification + +`vindex/client` is a lightweight, concurrency-safe Go SDK for querying VIndex servers via their public API. It is safe by default: every lookup and paginated value response is cryptographically verified before data is returned to the caller. This allows Go applications to query a Verifiable Index without trusting the server operator, and serves as a reference model for client implementations in other languages. + +Clients must configure an appropriate [C2SP `tlog-policy`](https://c2sp.org/tlog-policy) (or programmatic verifier policy) for trusting the Output Log: lookup verification guarantees that results match a witnessed Output Log commitment, while end-to-end security relies on third-party auditors (such as [`internal/auditor`](../internal/auditor/README.md)) verifying that those commitments were honestly constructed from the Input Log (and, when a caller fetches matched raw entries from the Input Log, authenticating them against the committed `(InputSize, InputRoot)`). + +### Non-Goals +- **No Unbounded In-Memory Value Collection**: Hot keys can match millions of Input Log entries; the SDK streams verified entry indices lazily page-by-page rather than providing helpers that drain all historical matches into memory. +- **No Cross-Lookup State**: The client caches zero state across lookups; monitors tracking progress across polling intervals persist their own watermarks externally. +- **No Full-Log Auditing**: The client verifies individual query proofs against witnessed commitments; replaying the Input Log to verify index construction is performed by an auditor ([`internal/auditor`](../internal/auditor/README.md)). +- **No Direct Output Log Polling**: Lookup clients query only the VIndex server's lookup endpoints; they do not fetch Output Log `/checkpoint` or `/tile/...` endpoints directly. + +--- + +## 1. Design + +### 1.1 Public Architecture & Package Decoupling + +External Go callers import `vindex/client` exclusively. Shared wire and MPT proof primitives live in internal packages that evaluate binary MPT path steps and RFC 6962 compact range folds linearly over byte slices without allocating intermediate tree node graphs (~1 ms per query on commodity CPUs): + +| Responsibility | Package | Purpose | +| :--- | :--- | :--- | +| **HTTP Transport & Verification Client API** | `vindex/client` | Public snapshot-scoped lookup client (`Client`, `LookupResult`, `MatchedKey`) with zero dependencies on `pebble`, `wazero`, `tessera`, `internal/ingest`, or `internal/kvstore`. | +| **C2SP Plaintext Wire & Value Log Codecs** | `vindex/internal/wire` | C2SP `tlog-vindex` plaintext parsers/formatters, checkpoint policy verification, and RFC 6962 Value Log compact range folding. | +| **MPT Point & Prefix Proof Verification** | `vindex/internal/mpt` | Binary MPT point inclusion/non-inclusion and two-sided prefix proof verification (`b <= 8192` split-bit bounds and canonical terminal leaf validation on top of `filippo.io/torchwood/mpt`). | + +### 1.2 Snapshot-Scoped Client API (`Client`, `LookupResult`, `MatchedKey`) + +```go +type Config struct { + ExpectedOutputLogOrigin string + OutputLogPolicy CheckpointPolicy + ExpectedInputLogOrigin string +} + +func New(indexURL string, cfg Config, opts ...Option) (*Client, error) +func WithHTTPClient(hc *http.Client) Option + +func (c *Client) LookupKey(ctx context.Context, key string) (*LookupResult, error) +func (c *Client) LookupPrefix(ctx context.Context, prefix string) (*LookupResult, error) + +type LookupResult struct { + OutputSize uint64 + OutputRoot [32]byte + InputOrigin string + From uint64 + InputSize uint64 + InputRoot [32]byte + MPTRoot [32]byte + Keys []MatchedKey // 0..1 for LookupKey; 0..N (lexicographically sorted) for LookupPrefix +} + +type MatchedKey struct { + Key string + ValueRoot [32]byte + // unexported snapshot context: client, from, inputSize +} + +func (k MatchedKey) Values(ctx context.Context) iter.Seq2[uint64, error] + +var ( + ErrVerificationFailed = errors.New("vindex: verification failed") + ErrBadRequest = errors.New("vindex: bad request") + ErrTooManyKeys = errors.New("vindex: prefix matched too many keys") + ErrUnavailable = errors.New("vindex: server unavailable") +) +``` + +- **Configuration & Transport**: `New` requires non-empty `ExpectedOutputLogOrigin`, non-nil `OutputLogPolicy` (parsed from a [C2SP `tlog-policy`](https://c2sp.org/tlog-policy) witness quorum text file or constructed programmatically from a primary log verifier and witness threshold), and non-empty `ExpectedInputLogOrigin`. `WithHTTPClient` clones the caller-supplied `*http.Client` (preserving custom `Transport`, proxies, TLS settings, and timeouts) while overriding `CheckRedirect` to reject all `3xx` redirects per [`docs/tlog-vindex.md`](../docs/tlog-vindex.md). +- **Snapshot-Scoped Isolation**: `*Client`, `*LookupResult`, and `MatchedKey` hold zero mutable cross-lookup state. Pagination cursors `(before, expectedValueRoot)` exist strictly as stack-local variables inside each `MatchedKey.Values(ctx)` invocation, allowing a single `*Client` or `MatchedKey` to be used concurrently across goroutines and against load-balanced or short-TTL-cached server replicas where consecutive `/keys` responses may arrive slightly out of order. + +```go +res, err := cli.LookupPrefix(ctx, "com.google.") +if err != nil { + return err +} +if res.From > requiredHorizon { + return fmt.Errorf("index starts at %d, need <= %d", res.From, requiredHorizon) +} +for _, mk := range res.Keys { + if mk.Key != "com.google.mail." && mk.Key != "com.google.maps." { + continue // skip uninterested keys with zero /values requests + } + for entryIdx, err := range mk.Values(ctx) { + if err != nil { + return err + } + if entryIdx < lastProcessedSize { + break // newest-first stream: stops without fetching older /values pages + } + // Authenticate raw entry at entryIdx against (res.InputSize, res.InputRoot) + } +} +``` + +### 1.3 End-to-End HTTP & Cryptographic Verification Sequence + +`LookupKey` and `LookupPrefix` never return a `*LookupResult` until Steps 1–3 succeed, and `MatchedKey.Values` verifies each `/values` page's compact range root before yielding any `entry_idx` from that page: + +| Step | Phase | Operations | Cryptographic & Protocol Assertion | +| :--- | :--- | :--- | :--- | +| **1** | **Output Log Proof (`c2sp.org/tlog-proof@v1`)** | Validates local canonical key/prefix syntax (`1..1024` bytes in `0x21..0x7E`), percent-encodes per RFC 3986 (`unreserved` only, escaping `+` as `%2B`), fetches `GET /keys?...` (rejecting `3xx`, mapping `400`/`413`/`503`, bounding response body size), splits on `\n\n` into `c2sp.org/tlog-vindex-keys@v1`, `c2sp.org/tlog-proof@v1` (base64 `extra` `state_entry` and canonical decimal `index`), and ``, and computes `SHA-256(0x00 \|\| state_entry)`. | Verifies `` against `cfg.ExpectedOutputLogOrigin` and `cfg.OutputLogPolicy` to extract `output_size` and `output_root`, asserts `output_size >= 1` and `index == output_size - 1`, and verifies the Merkle inclusion proof for `SHA-256(0x00 \|\| state_entry)` at `index`. | +| **2** | **State Commitment (`c2sp.org/tlog-vindex-state@v1`)** | Parses `state_entry` as a strict 6-line `c2sp.org/tlog-vindex-state@v1` record (`input-origin`, `from`, `input-size`, `input-root`, `mpt-root`) with no extension lines or trailing data. | Asserts `input_origin == cfg.ExpectedInputLogOrigin` and `0 <= from <= input_size`. Because a lookup response only attests to Input Log entries in `[from, input_size)`, callers requiring coverage back to a specific entry index check `LookupResult.From` against that bound. | +| **3** | **MPT Key / Prefix Proof (`LookupKey` / `LookupPrefix`)** | Validates canonical key syntax (`1..1024` bytes in `0x21..0x7E`) on returned keys, and verifies the binary MPT proof (`internal/mpt`) against `mpt_root` (enforcing `b <= 8192` split-bit bounds and canonical `path_key` / 32-byte `path_val` checks on terminal leaves). | **Inclusion**: Single matched `(key, value_root)` with `key == target` hashes to `mpt_root`.
**Non-Inclusion**: `mpt_root == empty_mpt_hash` (`0`-byte proof) or terminal MPT leaf `mpt_leaf_hash(path_key, path_val)` with `path_key != target` hashes to `mpt_root` (`len(res.Keys) == 0`).
**Prefix ([two-sided](../docs/tlog-vindex.md#two-sided-prefix-verification))**: On `0` matches, `mpt_root == empty_mpt_hash` (`0`-byte proof) or terminal MPT leaf with `!HasPrefix(path_key, target)` hashes to `mpt_root`; on `n >= 1` matches, `HasPrefix(key, target)` on every returned key proves no keys outside `target` were included from a wider ancestor subtree, while strictly ascending key order + first ancestor split bit `b_1 < 8*len(target)` + reconstructed subtree root + ancestor path to `mpt_root` proves no keys inside `target` were omitted. | +| **4** | **Lazy Backward Value Pagination (`MatchedKey.Values`)** | Initializes local iterator state `(before = inputSize, expectedValueRoot = k.ValueRoot)`. Fetches `GET /values?key=&before=` and parses `c2sp.org/tlog-vindex-values@v1` (`preceding`, `popcount(preceding)` compact range hashes, and strictly increasing `Indices`). | Treats `HTTP 404` on an MPT-included key or promised continuation page (`preceding > 0`) as `ErrVerificationFailed`. Asserts `len(Indices) >= 1`, strictly increasing order in `[from, before)`, and `preceding <= Indices[0] - from`; appends `value_log_leaf_hash(entry_idx)` into the running compact range (merging interior nodes via `value_log_node_hash(left, right)` on trailing `1` bits of the running leaf count) and folds the resulting compact range right-to-left to assert equality with `expectedValueRoot` **before** yielding `Indices` in reverse (`len-1` down to `0`, newest-first). If the caller continues and `preceding > 0`, folds the page's initial `popcount(preceding)` compact range hashes right-to-left into `expectedValueRoot`, sets `before = Indices[0]`, and fetches the next backward page. | + +### 1.4 Failure Taxonomy & Client Handling + +| Failure Category | Root Cause | Client Action | +| :--- | :--- | :--- | +| **Untrusted Checkpoint / Quorum Failure** | Output Log proof or checkpoint fails verification against `ExpectedOutputLogOrigin` and `OutputLogPolicy`, or `input-origin` in `state_entry` does not match `ExpectedInputLogOrigin`. | Return `ErrVerificationFailed`; abort lookup without `*LookupResult`. | +| **Latest-Entry Proof Violation** | `c2sp.org/tlog-proof@v1` proves an entry at `index != output_size - 1` or `output_size == 0`. | Return `ErrVerificationFailed`; abort lookup (server attempted to serve a non-tip Output Log entry). | +| **MPT Proof Mismatch** | Binary MPT path steps or prefix subtree reconstruction do not evaluate to `mpt_root`, or returned keys violate canonical key / prefix bounds. | Return `ErrVerificationFailed`; abort lookup. | +| **Value Log Root or Range Mismatch** | Reconstructed RFC 6962 root from `preceding` compact range hashes and `Indices` does not equal `expectedValueRoot`, or `Indices` fall outside `[from, before)` or are not strictly increasing. | Yield `(0, ErrVerificationFailed)` and terminate iterator without yielding unverified page indices. | +| **Unexpected `/values` `HTTP 404` on Present Key** | `/values` returns `404 Not Found` for a key proven present in the MPT at `before = InputSize` (or when `preceding > 0` promised older entries). | Yield `(0, ErrVerificationFailed)` and terminate iteration (server state violates the witnessed MPT or Value Log commitment). | +| **Prefix Cardinality or Policy Rejection (`HTTP 413` / `HTTP 400`)** | `/keys?prefix=` matched more keys than the server's configured prefix limit (`413`), or violated the server's prefix policy (`400`). | Surface typed HTTP status error (`ErrTooManyKeys` / `ErrBadRequest`) so the caller can narrow the queried prefix. | +| **Upstream Server Unavailable (`HTTP 503`)** | Server has not yet published its first Output Log checkpoint or lags `before`. | Surface `ErrUnavailable` (preserving `Retry-After` metadata when present) so the caller can retry with backoff or fail over. | + +--- + +## 2. Alternatives Considered + +### 2.1 Stateless Snapshot Handles vs. Mutable Cross-Lookup Client State +- **Proposed**: Caching observed `(output_size, input_size)` watermarks inside `Client` across lookups or exposing an `AllValues` full-slice helper for value pagination. +- **Rejection**: + - **Cross-Lookup Watermark Races**: Caching watermarks inside `Client` verifies no Merkle consistency proof (witnesses and auditors provide that guarantee), causes false regression errors when concurrent goroutines or load-balanced/short-TTL-cached server replicas return checkpoints out of arrival order, and tangles state across unrelated key lookups. + - **Unbounded Slice Allocations on Hot Keys**: Hot keys can match millions of entries, triggering unbounded sequential HTTP pagination and large slice allocations, whereas callers needing a slice can collect from `iter.Seq2` explicitly with their own bounds. +- **Chosen Design**: `Client` is stateless across lookups and returns snapshot-scoped `*LookupResult` handles whose `MatchedKey.Values(ctx)` iterator streams verified entry indices newest-first with early `break` support. + +--- + +## 3. Verification Gate + +```bash +go test -race ./vindex/client/... +``` diff --git a/vindex/docs/ARCHITECTURE.md b/vindex/docs/ARCHITECTURE.md index 27b4a93..460cef3 100644 --- a/vindex/docs/ARCHITECTURE.md +++ b/vindex/docs/ARCHITECTURE.md @@ -20,16 +20,16 @@ VIndex operates as a **Map Sandwich** bounded between two append-only transparen Data flows sequentially through five stages governed by a central coordinator, verified externally by lookup clients and independent auditors: -| Stage / Role | Responsibility | -| :--------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **1. Ingest & Bundle Cache** | Polls Input Log checkpoints, verifies checkpoint policies and Merkle consistency, downloads and authenticates entry bundles into a local disk cache, and prunes cached bundles once downstream storage and MPT state are durable. | -| **2. Sandboxed Mapping** | Executes the index's deterministic mapping function inside a zero-I/O WebAssembly sandbox pool, validating emitted [canonical keys](./tlog-vindex.md#canonical-keys) and resequencing mapped batches into strict Input Log entry order. | -| **3. Value-Log Storage** | Partitions each key's matching Input Log entry indices into append-only chunks in an embedded LSM key-value store, precomputing RFC 6962 compact range hashes and syncing updates to disk at target checkpoint barriers. | -| **4. State Commitment** | Predicts the candidate `mpt_root` over un-hashed canonical keys under a shared read lock, appends the state commitment entry to the Output Log, collects witness cosignatures, and atomically swaps the active serving snapshot. | -| **5. Read Serving** | Serves `/keys` (exact key and bounded prefix proofs from the in-memory MPT) and `/values` (lock-free backward value-log pagination from the KV store), strictly isolated to witnessed snapshots. | -| **Coordinator** | Freezes target Input Log checkpoints, drives stages 1–4 in monotonic order, governs safe cache-pruning watermarks, and runs Zero-WAL startup recovery. | -| **Lookup Client** | Queries `/keys` and `/values` and verifies Output Log inclusion proofs, MPT inclusion/non-inclusion/prefix proofs, and value-log compact range roots offline against an Output Log checkpoint verification policy. | -| **Independent Auditor** | Tails the Output Log from entry `0`, verifies Input Log and Output Log checkpoints, independently replays Input Log entries from `from` through the mapping function, and halts and alerts if any published `mpt-root` diverges. | +| Stage / Role | Responsibility | +| :----------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [**1. Ingest & Bundle Cache**](../internal/ingest/README.md) | Polls Input Log checkpoints, verifies checkpoint policies and Merkle consistency, downloads and authenticates entry bundles into a local disk cache, and prunes cached bundles once downstream storage and MPT state are durable. | +| [**2. Sandboxed Mapping**](../mapfn/README.md) | Executes the index's deterministic mapping function inside a zero-I/O WebAssembly sandbox pool, validating emitted [canonical keys](./tlog-vindex.md#canonical-keys) and resequencing mapped batches into strict Input Log entry order. | +| [**3. Value-Log Storage**](../internal/kvstore/README.md) | Partitions each key's matching Input Log entry indices into append-only chunks in an embedded LSM key-value store, precomputing RFC 6962 compact range hashes and syncing updates to disk at target checkpoint barriers. | +| [**4. State Commitment**](../internal/tree/README.md) | Predicts the candidate `mpt_root` over un-hashed canonical keys under a shared read lock, appends the state commitment entry to the Output Log, collects witness cosignatures, and atomically swaps the active serving snapshot. | +| [**5. Read Serving**](../internal/server/README.md) | Serves `/keys` (exact key and bounded prefix proofs from the in-memory MPT) and `/values` (lock-free backward value-log pagination from the KV store), strictly isolated to witnessed snapshots. | +| [**Coordinator**](../internal/coordinator/README.md) | Freezes target Input Log checkpoints, drives stages 1–4 in monotonic order, governs safe cache-pruning watermarks, and runs Zero-WAL startup recovery. | +| [**Lookup Client**](../client/README.md) | Queries `/keys` and `/values` and verifies Output Log inclusion proofs, MPT inclusion/non-inclusion/prefix proofs, and value-log compact range roots offline against an Output Log checkpoint verification policy. | +| [**Independent Auditor**](../internal/auditor/README.md) | Tails the Output Log from entry `0`, verifies Input Log and Output Log checkpoints, independently replays Input Log entries from `from` through the mapping function, and halts and alerts if any published `mpt-root` diverges. | --- @@ -89,7 +89,7 @@ Ecosystem authors express mapping logic as a deterministic function from raw Inp VIndex separates trie structure from per-key occurrence lists using a **hybrid keying model**: -- The **MPT** indexes **un-hashed variable-length canonical keys** directly so keys sharing a string prefix form a contiguous subtrie for [two-sided prefix proofs](./tlog-vindex.md#verifying-a-keys-response). +- The **MPT** indexes **un-hashed variable-length canonical keys** directly so keys sharing a string prefix form a contiguous subtrie for [prefix subtree proofs with two-sided verification](./tlog-vindex.md#two-sided-prefix-verification). - The **KV store** indexes per-key value-log chunks by fixed-size **`SHA-256(key)` digests** computed on the host, keeping LSM keys uniform and decoupled from variable canonical key lengths. Within the KV store, each key's matching `entry_idx` sequence is partitioned into fixed-window chunks ordered newest-first under `SHA-256(key)`. Each chunk stores the RFC 6962 compact range hashes of all older occurrences preceding that chunk alongside compact relative offsets within the window: diff --git a/vindex/docs/tlog-vindex.md b/vindex/docs/tlog-vindex.md index 07425d3..45d53a2 100644 --- a/vindex/docs/tlog-vindex.md +++ b/vindex/docs/tlog-vindex.md @@ -243,8 +243,8 @@ example.com/outputlog 13 jR10gZzRRHNJbpwE1qNYfgk7jAv2Ta+5Ud8SpKcWZ6w= -- example.com/outputlog opLqBQsREYCgu6xQkYQwJr9fo45a62DN9EdmHXnZdXNqlcVGlCum2Wks+49/V6267UEjw6QUXTS5Rovnzv++qbSzm9Q= -- witness.example/w1 CMp+6LWBU0anHGH5aNDTJkH/gj79sG+T6+iP2ThYN5krrDJbR1HDnucjL39QsZTSvVjyQLrdk3DXDqI5G2HgLatVs0pWh6Up69HVOw== +— example.com/outputlog opLqBQsREYCgu6xQkYQwJr9fo45a62DN9EdmHXnZdXNqlcVGlCum2Wks+49/V6267UEjw6QUXTS5Rovnzv++qbSzm9Q= +— witness.example/w1 CMp+6LWBU0anHGH5aNDTJkH/gj79sG+T6+iP2ThYN5krrDJbR1HDnucjL39QsZTSvVjyQLrdk3DXDqI5G2HgLatVs0pWh6Up69HVOw== ``` #### Binary MPT proofs @@ -335,7 +335,7 @@ To authenticate the state commitment in a `keys` response: 3. Verify the `c2sp.org/tlog-proof@v1` block against that leaf hash, ``, and the client's output log checkpoint verification policy according to [tlog-proof][]. Let `output_size` be the verified output log checkpoint tree size. Check that `output_size >= 1` and that the proven entry `index` equals `output_size - 1`. -4. Parse `state_entry` as a `c2sp.org/tlog-vindex-state@v1` record (see [Output log entries](#output-log-entries)) to obtain `input_origin`, `from`, `input_size`, `input_root`, and `mpt_root`. Check that `input_origin` equals the client's configured `` and that `0 <= from <= input_size`. If `from = input_size`, check that `mpt_root` equals `empty_mpt_hash`. Because a lookup response only attests to input log entries in `[from, input_size)`, callers that require coverage back to a specific entry index MUST check that `from` does not exceed that bound. +4. Parse `state_entry` as a `c2sp.org/tlog-vindex-state@v1` record (see [Output log entries](#output-log-entries)) to obtain `input_origin`, `from`, `input_size`, `input_root`, and `mpt_root`. Check that `input_origin` equals the client's configured `` and that `0 <= from <= input_size`. Because a lookup response only attests to input log entries in `[from, input_size)`, callers that require coverage back to a specific entry index MUST check that `from` does not exceed that bound. If the client subsequently fetches matched entries from the input log, it MUST authenticate those entries against `(input_size, input_root)` according to the input log's specification, and unless relying solely on the witnessed output log and auditors for input log checkpoint validity, verify that `(input_size, input_root)` is Merkle-consistent with a signed input log [checkpoint][] verified under its input log policy. diff --git a/vindex/internal/auditor/README.md b/vindex/internal/auditor/README.md new file mode 100644 index 0000000..e8c3ed2 --- /dev/null +++ b/vindex/internal/auditor/README.md @@ -0,0 +1,83 @@ +# Independent Log Auditor + +Lookup clients ([`client/`](../../client/README.md)) verify that individual query responses match a witnessed Output Log commitment, but they cannot verify that the committed `mpt_root` itself was honestly constructed from the Input Log without omitting or fabricating unqueried keys. `vindex/internal/auditor` (`vindex-auditor`) closes that security loop: it independently replays the Input Log from `from` through the mapping function, recomputes every key's Value Log and the global MPT, and asserts that every state commitment published in the Output Log matches the reconstructed root. + +This reference auditor reuses the indexer's Pebble chunk store ([`internal/kvstore`](../kvstore/README.md)) and disk-backed MPT (`internal/mpt`) to share the high-throughput ingestion pipeline (90,000 to 240,000+ entries/sec, requiring equivalent RAM and NVMe sizing) and preserve inspectable on-disk forensic state immediately prior to any divergence. However, reusing the indexer's storage and trie code is not required by the protocol, and relying solely on a single codebase would only verify that the operator ran the same code rather than catching shared implementation bugs. Independent clean-room auditors (written in other languages or using different storage and mapping implementations conforming to [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md#auditing-the-index)) should also audit the Output Log to provide true implementation diversity. + +### Non-Goals & Future Work +- **No Automatic Retry / Self-Healing on Root Mismatch**: On any root mismatch or consistency violation, the auditor halts immediately and freezes local disk state to preserve forensic evidence. +- **No Checkpoint Signing or Publication**: Strictly an independent consumer and verifier of the Output Log and Input Log. +- **No HTTP Lookup Serving (Future Work)**: Although an auditor maintains a verified MPT and Pebble chunk store locally, `c2sp.org/tlog-vindex` does not yet specify read-mirror semantics (such as which Output Log checkpoint a lagging or halted auditor serves to satisfy the `index == output_size - 1` latest-entry proof check). Verified mirror serving is deferred to future work. + +--- + +## 1. Design + +### 1.1 The Continuous Audit Loop + +The auditor is configured with ``, the Input Log retrieval endpoint (requiring historical entry bundles across `[from, input_size)` to remain retrievable), expected `` and ``, checkpoint verification policies (such as [C2SP `tlog-policy`](https://c2sp.org/tlog-policy)) for both logs, the compiled WASM `MapFn` module, and the start index `from` (default `0`). + +Initialized with an empty local MPT and `prev_size = from` (or, on restart, after replaying local Phase 2 MPT catch-up across `[max(MPT_Durable_Size, from), audited_input_size)` if the MPT's persisted `(version, exact)` lags `audited_input_size`), the auditor executes the 4-step loop defined in [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md): + +| Step | Phase | Operations | Verification & Persistence Rules | +| :--- | :--- | :--- | :--- | +| **1** | **Poll Output & Input Checkpoints** | Fetches `/checkpoint` and a signed Input Log `/checkpoint` (`latest_input_size`, `latest_input_root`, re-fetched whenever a newer Output Log entry commits `input_size > latest_input_size`). | Verifies both checkpoints against their expected origins and checkpoint policies, and verifies Output Log Merkle consistency against the previously audited Output Log checkpoint before reading any entries. | +| **2** | **Stream Output Entries** | Fetches new Output Log entry bundles from `/tile/entries/...` from the last audited `output_size` up to the new `output_size`. | Authenticates bundles against the verified Output Log checkpoint root per `tlog-tiles`, parses each 6-line `c2sp.org/tlog-vindex-state@v1` entry (`input-origin`, `from`, `input-size`, `input-root`, `mpt-root`) in order, asserts `input_origin` and `from` match configured parameters and `prev_size <= input_size <= latest_input_size`, and verifies that `(input_size, input_root)` is Merkle-consistent with `(latest_input_size, latest_input_root)` (checking `input_root == latest_input_root` when `input_size == latest_input_size`). | +| **3** | **Fetch, Map & Calculate Local MPT Root** | Downloads Input Log entry bundles covering `[prev_size, input_size)` (slicing boundary 256-entry bundles to exclude entries `< from` on non-256-aligned initial bundles and entries `>= input_size` on partial tail bundles), maps entries via `MapFn`, updates modified Value Logs and `(canonicalKey, value_root)` mappings in the local MPT, and calculates the resulting local `mpt_root`. | Authenticates Input Log bundles against `input_root` (or `latest_input_root`), validates canonical key bounds (`1..1024` bytes in `0x21..0x7E`), deduplicates keys per `entry_idx`, appends `value_log_leaf_hash(entry_idx)` to each key's Value Log, and updates the local MPT to compute the root at `input_size`. On crash replay, any `entry_idx` already persisted in Pebble is skipped while reconstructing its point-in-time `(canonicalKey, value_root)` at `input_size`. | +| **4** | **Assert Root Equality** | Compares the calculated local `mpt_root` against `mpt_root` in the Output Log entry. | **Match**: Persists the updated chunk state, MPT snapshot, and `audited_output_size` / `audited_input_size` watermarks (`KV_CP >= MPT_Durable_Size`), and sets `prev_size = input_size`.
**Mismatch**: Executes the state-preserving halt protocol without advancing verified watermarks, freezing disk state and emitting alerts. | + +### 1.2 Root Mismatch Alerting, State-Preserving Halt & Runbook + +On any root mismatch or upstream consistency/policy violation, the auditor halts and freezes local database state on disk for forensic triage: + +1. **State Freeze & Telemetry**: Terminates the background audit loop without advancing verified watermarks, leaves local Pebble and MPT files intact on disk for inspection, sets the root mismatch gauge metric to `1`, and logs the divergent `output_index`, `input_size`, `local_root`, and `committed_root`. +2. **Operational Probes (Internal Admin Listener)**: + - `/healthz`: Remains **`200 OK`** while the process is alive so container orchestrators do not restart the halted auditor and disturb frozen disk state. + - `/readyz`: Returns **`503 Service Unavailable`** with structured JSON diagnostics: + ```json + { + "status": "degraded", + "error": "auditor root hash mismatch", + "output_index": 12480, + "input_size": 3194880, + "local_root": "a4f1...", + "committed_root": "b7c2..." + } + ``` +3. **Operator Runbook**: + - Do not wipe storage; inspect `/readyz` and structured logs for the divergent `(output_index, input_size, local_root, committed_root)`. + - Compare local Pebble `'c'` chunks against raw Input Log entries in `[prev_size, input_size)` and re-run the auditor in one-shot mode with an isolated scratch directory to confirm deterministic reproduction. + - If `MapFn` and Input Log entries are valid, publish the misbehavior proof bundle (witnessed Output Log checkpoint, inclusion proof, raw `c2sp.org/tlog-vindex-state@v1` entry, and signed Input Log checkpoint + consistency proof) to witnesses and monitoring channels. + +### 1.3 Adversarial Threat Matrix + +| Threat Vector | Adversary Action | Detection Mechanism | Auditor Action | +| :--- | :--- | :--- | :--- | +| **Operator Equivocation** | Serves conflicting Output Log checkpoints to different auditors or witnesses. | Output Log checkpoint policy verification and Merkle consistency proofs between Output Log checkpoints. | Halts audit loop; logs alert; freezes database. | +| **Tree Size Rollback** | Serves an Output Log checkpoint (or `input-size` commitment) smaller than previously audited watermarks. | Monotonic `output_size` and `prev_size <= input_size <= latest_input_size` assertions. | Halts audit loop; freezes watermarks; `/readyz` reports `503`. | +| **Forged MPT Root (Omission)** | Commits an invalid `mpt_root` in an Output Log entry to hide or fabricate records. | Local MPT root calculation from authenticated Input Log entries compared against `mpt_root`. | Freezes disk state; sets mismatch metric; `/readyz` reports `503`. | +| **Corrupted Input Entries** | Alters raw Input Log entry bundles in transit or on CDN. | Entry bundle Merkle authentication fails against verified `input_root` / `latest_input_root`. | Rejects bundle; retries transient fetch errors without advancing state. | +| **Unauthenticated Output Entries** | Serves Output Log entry bundles that do not match the Output Log checkpoint root. | `tlog-tiles` entry bundle Merkle authentication fails against Output Log checkpoint root. | Rejects bundle; terminates audit loop with authentication error. | +| **Parameter Drift (`input-origin` / `from`)** | Publishes an Output Log state commitment altering `input-origin` or `from`. | Checks `input-origin` and `from` against configured parameters on every Output Log entry. | Halts audit loop immediately; freezes state; emits alerts. | + +--- + +## 2. Alternatives Considered + +### 2.1 Lookup Client Verification Alone vs. Full-Log Independent Auditing +- **Proposed**: Relying solely on lookup clients ([`client/`](../../client/README.md)) to verify individual query responses without running a full-log auditor. +- **Rejection**: Lookup clients verify proofs against `mpt_root` only for queried keys and cannot detect whether `mpt_root` omitted or fabricated entries across unqueried keys. +- **Chosen Design**: Complementary verification: per-query proof verification in [`client/`](../../client/README.md) paired with continuous full-log replay auditing in `vindex/internal/auditor` (and independent clean-room auditors). + +### 2.2 Ephemeral In-Memory Auditing vs. State-Preserving Forensic Freeze +- **Proposed**: Storing auditor state purely in RAM or wiping local database files upon detecting a divergence. +- **Rejection**: Destroys the local state needed to isolate which Input Log entries were omitted or corrupted in `[prev_size, input_size)`. +- **Chosen Design**: State-preserving halt that freezes local Pebble and MPT files in-place without advancing verified watermarks. + +--- + +## 3. Verification Gate + +```bash +go test -race ./vindex/internal/auditor/... +``` diff --git a/vindex/internal/coordinator/README.md b/vindex/internal/coordinator/README.md new file mode 100644 index 0000000..f701b8f --- /dev/null +++ b/vindex/internal/coordinator/README.md @@ -0,0 +1,82 @@ +# Coordinator & Lifecycle Engine + +`vindex/internal/coordinator` is the single-threaded lifecycle orchestrator and watermark authority for `vindexd`. Worker subsystems (tile ingestion, WASM mapping, KV chunk storage, MPT state commitment, and HTTP serving) expose narrow interfaces and never inspect each other's internal state. The Coordinator drives the end-to-end indexing loop in monotonic order, freezes target checkpoints so high-velocity logs cannot starve batch completion, computes the safe pruning watermark for the local tile cache, and orchestrates network-free Zero-WAL startup recovery and unserved Genesis Backfill. + +### Non-Goals +- **No Distributed Consensus**: Operates strictly within a single process on a single host; high availability is achieved externally by running independent VIndex instances against the shared Input Log. +- **No Direct Storage or Trie Mutation**: Coordinates subsystem boundaries and checkpoints without manipulating LSM chunk records or MPT nodes directly. +- **No Arbitrary Runtime Mode Switching**: Runs Normal Serving Mode once the Output Log has at least one state commitment entry (`Output_CP.Size > 0`); Genesis Backfill is used strictly during initial bootstrap when the Output Log is empty (`Output_CP.Size == 0`). + +--- + +## 1. Design + +### 1.1 The Master Batch Loop + +Each steady-state sync cycle executes six sequential steps across a frozen target range (`>90,000 entries/sec` sustained throughput): + +| Step | Phase | Action & Subsystem Transition | Durability Transition | +| :- | :--- | :--- | :--- | +| **1** | **Poll & Freeze** | Polls upstream Input Log `/checkpoint`, verifies the checkpoint against `` and the configured Input Log checkpoint policy (such as [C2SP `tlog-policy`](https://c2sp.org/tlog-policy)), checks `Target_CP.Size >= max(prev_Target_CP.Size, from)` and Merkle consistency against `prev_Target_CP`, and persists `Target_CP` to Pebble metadata. | Freezes target sync boundary to prevent moving-goalpost starvation on high-velocity logs. | +| **2** | **Tile Ingestion** | Downloads missing 256-entry bundles across `[max(KV_CP.Size, from), Target_CP.Size)` into the local tile cache ([`internal/ingest`](../ingest/README.md)). | Durably stages verified entry bundles on disk. | +| **3** | **Parallel Mapping** | Streams 256-entry bundles to WASM workers (`internal/wasmhost`), validates un-hashed canonical keys (`1..1024` bytes in `0x21..0x7E`), and resequences batches into multi-tile commit slabs. | In-memory key-index aggregation. | +| **4** | **Storage Commit** | Streams commit slabs into [`internal/kvstore`](../kvstore/README.md) (committing intermediate slabs without disk sync while accumulating modified `(key, value_root)` mappings across the target window, and executing a blocking disk `fsync` barrier on the final slab reaching `Target_CP.Size`). | **Advances persisted KV watermark and commits `KV_CP` at `Target_CP.Size` prior to Output Log publication.** | +| **5** | **Commitment & Promotion** | Predicts candidate `mpt_root` over accumulated modified `(key, value_root)` mappings, appends one 6-line `c2sp.org/tlog-vindex-state@v1` entry (`input-origin`, `from`, `input-size`, `input-root`, `mpt-root`) for `Target_CP` to the Output Log (retaining the signed Input Log checkpoint durably out-of-band), collects witness cosignatures per Output Log `tlog-policy`, and ratchets `Serving_CP` under the exclusive tree lock (< 5ms; see [`internal/tree`](../tree/README.md)). | **Promotes state to Output Log (`Output_CP`) and Serving (`Serving_CP`).** | +| **6** | **Pruning Notification** | Computes `SafeWatermark = max(min(KV_CP.Size, MPT_Durable_Size), from)` and notifies the tile cache reaper. | Bounded disk cache garbage collection. | + +### 1.2 Checkpoint Progression Chain & Safe Pruning Watermark + +| Checkpoint | Log Type | Committed Size | Persistence | Role in Pipeline | +| :--- | :--- | :--- | :--- | :--- | +| **`Target_CP`** | Input Log | `Target_CP.Size` (`input_size`) | Durable (Pebble metadata) | Frozen upper goalpost for the active sync cycle. | +| **`KV_CP`** | Input Log | `KV_CP.Size` | Durable (Pebble LSM) | Binds synced `'c'` chunk records to the verified Input Log state they cover. | +| **`Output_CP`** | Output Log | `Output_CP.Size` (`output_size`), committing `Output_CP.InputSize` | Durable (Tessera Output Log) | Witnessed state commitment (`input-origin`, `from`, `input-size`, `input-root`, `mpt-root`), with signed Input Log checkpoint retained out-of-band. | +| **`Serving_CP`** | Output Log Entry | `Serving_CP.InputSize` (`input_size`) | Volatile (atomic pointer) | Active snapshot exposed to HTTP lookup readers. | + +The checkpoints obey a strict monotonic progression invariant across all execution and crash states: + +```text +Target_CP.Size >= KV_CP.Size >= Output_CP.InputSize >= Serving_CP.InputSize >= MPT_Durable_Size +``` + +- **Storage Precedes Commitment (`KV_CP.Size >= Output_CP.InputSize`)**: Pebble must complete a durable `fsync` before the Output Log entry is published, ensuring a crash never leaves a witnessed Output Log entry referencing missing KV chunks. +- **Authoritative Safe Pruning Watermark**: The Coordinator instructs the tile cache reaper to prune only bundles below `SafeWatermark = max(min(KV_CP.Size, MPT_Durable_Size), from)`. Clamping to `from` ensures suffix indices (`from > 0`) never retain or replay pre-`from` tiles, while keeping all 256-entry bundles in `[max(MPT_Durable_Size, from), KV_CP.Size)` cached on local disk until the MPT confirms a durable `fsync`. + +### 1.3 Zero-WAL Startup Recovery Sequence + +On daemon launch, the HTTP read server gates public lookup endpoints with `503 Service Unavailable` while opening the MPT replays any uncompacted `torchwood/mpt` journal frames on disk. Once journal replay finishes, the Coordinator computes `recoveryTarget = Output_CP.InputSize` when `Output_CP.Size > 0` (or `KV_CP.Size` when `Output_CP.Size == 0`) and executes a network-free 3-phase recovery sequence: + +1. **Phase 1: Instant Warm Start (< 5ms after MPT open)**: Entered if and only if `Output_CP.Size > 0`, the MPT's persisted version equals `(Output_CP.InputSize, true)` (`exact == true` and `MPT_Durable_Size == Output_CP.InputSize`), and the trie root matches `mpt-root` in the latest Output Log state commitment entry. Activates `Serving_CP`, initializes watermarks, and **opens the HTTP Read Server immediately (< 5ms)**. +2. **Phase 2: Fast-Forward Tile Replay (< 500ms after MPT open)**: Entered whenever the persisted MPT lags `recoveryTarget` or is inexact (`version < recoveryTarget || !exact`). Streams missing 256-entry bundles across `[max(MPT_Durable_Size, from), recoveryTarget)` from the local disk tile cache, maps them via `map_bundle` to collect distinct modified canonical keys, reconstructs point-in-time `value_root` digests at `recoveryTarget` from Pebble with **zero database writes**, commits sorted `(key, value_root)` updates to the MPT, and syncs the MPT to disk. When `Output_CP.Size > 0`, asserts that the resulting `mpt_root` matches the latest Output Log entry, activates `Serving_CP`, and **opens the Read Server (< 500ms)**; when `Output_CP.Size == 0`, public endpoints remain `503 Service Unavailable`. +3. **Phase 3: Background Catch-Up & Uncommitted KV Writes**: Resumes forward ingestion from `max(Serving_CP.InputSize, from)` when `Output_CP.Size > 0` (or from `max(KV_CP.Size, from)` in Genesis Backfill when `Output_CP.Size == 0`, including clean restarts mid-backfill where `version == KV_CP.Size && exact == true`) toward `Target_CP.Size`. If a crash occurred in Normal Serving Mode after Pebble synced a batch to disk but before Output Log publication (`KV_CP.Size > max(Serving_CP.InputSize, from)`), re-streaming from `max(Serving_CP.InputSize, from)` skips already-persisted entry indices in Pebble while reconstructing their `value_root` digests so uncommitted batches are published cleanly without duplicate offsets. + +### 1.4 Genesis Backfill Lifecycle (`Output_CP.Size == 0`) + +When bootstrapping an unserved index from `from` (`Output_CP.Size == 0`) against a large Input Log, accumulating every modified `(key, value_root)` pair across `[from, Target_CP.Size)` in memory before a single final commit would exhaust host RAM, while publishing and witnessing an Output Log entry on every intermediate batch would incur thousands of unnecessary external witness roundtrips before the index is ready to serve. When `Output_CP.Size == 0`, the Coordinator executes synchronous **Genesis Backfill**: + +1. **Serving Plane Gating**: Activated automatically if and only if `Output_CP.Size == 0`. Public `/keys` and `/values` endpoints and `/readyz` remain `503 Service Unavailable` throughout backfill. +2. **Bounded Coalescing & Coarse Local Checkpoints**: Multi-tile batches write inverted `'c'` chunk updates to Pebble without per-batch `fsync` while coalescing modified `(key, value_root)` mappings in a bounded in-memory map and **suppressing intermediate Output Log appends and witness roundtrips**. Whenever the modified-key count or entry threshold is reached (or on graceful shutdown), the Coordinator executes a synchronous Pebble disk sync (advancing `KV_CP.Size`) followed immediately by a synchronous MPT commit and disk sync (advancing `MPT_Durable_Size` strictly after Pebble `fsync` so `KV_CP.Size >= MPT_Durable_Size`), clears the modified-key map, and updates `SafeWatermark` so the tile cache reaper prunes superseded tiles. +3. **Genesis Finalization (`Entry 0`)**: Upon reaching `Target_CP.Size`, the Coordinator executes a final synchronous Pebble sync and MPT commit/sync, publishes Output Log `Entry 0` (`c2sp.org/tlog-vindex-state@v1`), collects witness cosignatures, promotes `Serving_CP`, and permanently transitions to Normal Serving Mode (`Output_CP.Size > 0`). +4. **Crash Recovery During Backfill**: If the process crashes during Genesis Backfill (`Output_CP.Size == 0` and `version < KV_CP.Size || !exact`), startup reuses standard Phase 2 recovery across `[max(MPT_Durable_Size, from), KV_CP.Size)` before resuming backfill from `KV_CP.Size` toward `Target_CP.Size`. + +--- + +## 2. Alternatives Considered + +### 2.1 Decentralized Subsystem Watermarking vs. Centralized Coordinator Governance +- **Proposed**: Allowing each subsystem to advance watermarks and prune historical state independently. +- **Rejection**: Decentralized tracking creates recovery races on dirty crashes because individual subsystems cannot determine whether downstream components persisted a batch. +- **Chosen Design**: Centralized Coordinator governance with a single monotonic progression chain. + +### 2.2 Distributed Consensus Orchestration (Raft/Paxos) vs. Single-Host Coordinator +- **Proposed**: Coordinating indexing across a multi-node cluster via Raft. +- **Rejection**: The MPT requires single-host RAM/mmap locality, and high availability is achieved externally by running independent VIndex operator instances against the shared Input Log. +- **Chosen Design**: Single-process, single-host embedded coordinator. + +--- + +## 3. Verification Gate + +```bash +go test -race ./vindex/internal/coordinator/... +``` diff --git a/vindex/internal/hammer/README.md b/vindex/internal/hammer/README.md new file mode 100644 index 0000000..53720e3 --- /dev/null +++ b/vindex/internal/hammer/README.md @@ -0,0 +1,67 @@ +# Hammer Load Testing & Verification Framework + +Unit tests validate isolated package logic, but race conditions, LSM compaction stalls, writer lock contention, and 65,536-entry chunk boundary bugs only surface under sustained concurrent read-write load. `vindex/internal/hammer` is the end-to-end load-testing and verification harness for `vindexd`: it acts simultaneously as a synthetic upstream Input Log producer (serving signed checkpoints and tiled entry bundles across programmable release schedules) and a pool of concurrent lookup clients that cryptographically verify 100% of responses via [`vindex/client`](../../client/README.md). + +### Non-Goals +- **Not a Public SDK Package**: Strictly an internal load-testing and verification harness. +- **Not a Production Log Sequencer**: Generates synthetic workloads for stress testing, benchmarking, and chaos verification. +- **Not an In-Memory Mock**: Executes real HTTP requests over loopback sockets against real POSIX tiles, Pebble SSTables, and `mmap` MPT files. + +--- + +## 1. Design + +### 1.1 Component Architecture + +| Component | Responsibility | +| :--- | :--- | +| **Synthetic Entry Generator** | Emits synthetic Input Log entries across Zipfian/Pareto (`alpha > 1`) and uniform key distributions in configurable leaf formats (`raw`, `sumdb`, or `ct`), plus disjoint non-inclusion probe keys (`nonexistent/`). | +| **Tessera Input Log Sequencer** | Appends generated entries into a local POSIX log, computes Merkle roots, and signs Input Log checkpoints with a configurable or freshly generated Ed25519 key (printing the verifier key at startup for `vindexd`). | +| **Drip-Feed HTTP Server** | Serves standard [C2SP `tlog-tiles`](https://c2sp.org/tlog-tiles) endpoints (`/checkpoint` and `/tile/...`) and schedules checkpoint visibility across steady, burst, and pause/catch-up modes. | +| **Verifying Concurrent Readers** | Queries `vindexd` concurrently via [`vindex/client`](../../client/README.md) across hot, uniform, non-inclusion, and full-pagination query mixes, asserting cryptographic proof validity and monotonic history invariants. | +| **Metrics Analyzer & Dashboard** | Aggregates latency percentiles, read QPS, ingestion lag, operational status counts (`503`/`413`), and invariant checks into periodic terminal summaries. | + +### 1.2 Workload Distributions & Drip-Feed Schedules + +- **Synthetic Entry Distributions & Formats**: Generates entries in raw key/value (`raw`), Go SumDB (`sumdb`, ` h1:`), or Certificate Transparency (`ct`, X.509/precertificate) formats: + - **Zipfian / Pareto Skew (`alpha > 1`)**: Concentrates >80% of entries onto the hottest 1% of keys, driving individual hot keys across multiple 65,536-entry chunk boundaries (`^0 -> ^1`) to stress chunk rollovers, Pebble compactions, and multi-page backward `/values` scans. + - **Uniform Distribution**: Spreads entries evenly across `[0, K)` to stress MPT branch fanout and single-occurrence chunk creation. +- **Drip-Feed Checkpoint Schedules**: + - **Steady Drip**: Releases Input Log checkpoints at a fixed rate to test steady-state ingestion and read concurrency. + - **Burst Mode**: Buffers checkpoints and releases them in sudden bursts to simulate batch upstream publications. + - **Pause & Catch-Up Mode**: Holds checkpoint publication so `vindexd` idles, then exposes a deep backlog of 256-entry bundles to benchmark catch-up throughput. + +### 1.3 Verifying Concurrent Readers & Fault Discrimination + +Concurrent reader workers query `vindexd` (`LookupKey`, `LookupPrefix`, and `MatchedKey.Values(ctx)`) using [`vindex/client`](../../client/README.md) configured with the expected ``, ``, and Output Log checkpoint policy across a weighted query mix: +- **Hot Keys (~60%)**: Samples skewed Zipfian keys and verifies the newest `/values` page (single-page `break`). +- **Uniform Keys (~25%)**: Samples uniformly across the active keyspace and verifies the newest `/values` page. +- **Non-Inclusion Probes (~10%)**: Queries synthetic `nonexistent/` keys never appended to the Input Log, exercising MPT non-inclusion and empty-prefix proofs. +- **Full Backward Pagination (~5%)**: Walks a hot key's `MatchedKey.Values(ctx)` iterator all the way back to `preceding == 0` across multiple 65,536-entry chunks. + +#### Verification & Monotonic History Assertions +1. **Cryptographic Proof Verification (`vindex/client`)**: Every query verifies the Output Log checkpoint policy and inclusion proof at `index == output_size - 1`, the 6-line `c2sp.org/tlog-vindex-state@v1` commitment, the binary MPT inclusion/non-inclusion/prefix proof against `mpt_root`, and the RFC 6962 Value Log compact range root on every fetched `/values` page. +2. **Monotonic History Superset & `From` Non-Regression**: Readers assert that `LookupResult.From` never regresses across queries. Whenever a query observes the **complete** Value Log history for canonical key `K` (either a single-page result with `preceding == 0` or a full backward walk to `preceding == 0`), the reader records the chronologically ordered entry index slice `I_new` at `input_size_new` and asserts that for any prior complete observation `I_old` at `input_size_old <= input_size_new`, `I_new` is a prefix-preserving superset of `I_old` (`len(I_new) >= len(I_old)` and `I_new[:len(I_old)] == I_old`). +3. **Fault Discrimination**: Any verification error (`client.ErrVerificationFailed`) or reader-side consistency failure (`From` regression or monotonic history superset violation) immediately records a fatal invariant violation and fails the hammer run, whereas transient operational status codes (`client.ErrUnavailable` / `HTTP 503` during cold startup or `client.ErrTooManyKeys` / `HTTP 413` on broad prefix limits) are tracked in telemetry without failing the run. + +--- + +## 2. Alternatives Considered + +### 2.1 In-Process Unit Test Mocks vs. End-to-End Loopback Harness +- **Proposed**: Stress-testing `vindexd` purely in memory using mocked storage and channel harnesses. +- **Rejection**: In-memory mocks miss real concurrency and I/O dynamics: loopback HTTP connection pooling, OS scheduler preemption, `mmap` page flushing, and Pebble LSM compaction stalls. +- **Chosen Design**: Real HTTP client/server execution backed by POSIX Tessera tiles, Pebble SSTables, and `mmap` MPT files. + +### 2.2 Static Pre-Recorded Log Dumps vs. Dynamic Drip-Feed Sequencer +- **Proposed**: Replaying a fixed snapshot of an existing public transparency log. +- **Rejection**: Static dumps cannot modulate write rates on demand, dial exact Zipfian skew to force 65,536-entry chunk rollovers, inject targeted non-inclusion probes, or orchestrate pause/burst checkpoint schedules. +- **Chosen Design**: Deterministic synthetic entry generator paired with a programmable drip-feed `tlog-tiles` sequencer. + +--- + +## 3. Verification Gate + +```bash +go test -race ./vindex/internal/hammer/... +``` diff --git a/vindex/internal/ingest/README.md b/vindex/internal/ingest/README.md new file mode 100644 index 0000000..c0185e8 --- /dev/null +++ b/vindex/internal/ingest/README.md @@ -0,0 +1,40 @@ +# Ingestion Pipeline & Local Tile Cache + +`vindex/internal/ingest` is the front-end pipeline that turns a remote append-only Input Log into a verified, strictly ordered stream of `(canonicalKey, entryIdx)` batches for downstream storage. It verifies upstream checkpoints and Merkle consistency against the configured Input Log policy, downloads and authenticates 256-entry bundles across parallel fetchers, caches verified bundles on local disk so crash recovery completes without network egress, maps bundles in parallel across sandboxed WebAssembly workers, and resequences out-of-order worker completions back into contiguous entry index order. + +### Non-Goals +- **No Upstream Log Mutation**: Strictly a read-only consumer of the upstream Input Log. +- **No In-Guest Storage Writes**: Sandboxed mapping workers transform raw entry bytes into canonical keys in memory and perform zero disk or network I/O. +- **No Independent Pruning Authority**: The local tile cache never inspects downstream database or trie state directly; it prunes cached bundles strictly up to the `SafeWatermark` dictated by [`internal/coordinator`](../coordinator/README.md). + +--- + +## 1. Design + +### 1.1 The Assembly Line Sorter Pipeline + +Once the Coordinator establishes a verified `Target_CP` (`Target_CP.Size >= max(prev_Target_CP.Size, from)` and Merkle-consistent with `prev_Target_CP`), it streams `[max(persisted_watermark, from), Target_CP.Size)` through six stages: + +1. **Tile Fetcher & Merkle Authentication**: Supports standard tiled logs ([C2SP `tlog-tiles`](https://c2sp.org/tlog-tiles) `/tile/entries/[.p/]` and [C2SP `static-ct-api`](https://c2sp.org/static-ct-api) `/tile/data/[.p/]` using partial tile width `.p/` on unsealed tail tiles) as well as leaf-by-leaf log adapters that group entries into 256-entry bundles. Checkpoint verification follows the configured Input Log policy (such as [C2SP `tlog-policy`](https://c2sp.org/tlog-policy)). Fetching in 256-entry bundles cuts upstream HTTP requests by 256x, and each bundle is authenticated against its Level-0 tree tile and in-memory cached upper-level tree tiles (so shared parent tiles are fetched once per checkpoint rather than re-fetched across consecutive bundles) up to `Target_CP.Root` before use. +2. **Local Tile Cache**: Writes Merkle-authenticated 256-entry bundles to sharded subdirectories (`1,000` bundles per shard, preventing flat-directory inode contention) on local disk via atomic temp-file rename. When `Target_CP.Size` advances within an unsealed 256-entry tail tile across consecutive sync cycles, any cached partial bundle narrower than required (`bundleEnd < Target_CP.Size` and `len < 256`) is treated as a cache miss, re-fetched at the wider width, and atomically overwritten on disk. +3. **WASM Worker Pool**: A pool of parallel Wazero sandboxes (`internal/wasmhost`) running the compiled `.wasm` plugin. Each worker slices boundary bundles to `[max(bundleStart, start), min(bundleEnd, Target_CP.Size))` (excluding entries `< from`, entries already persisted in a partially indexed tail bundle, and entries `>= Target_CP.Size`) and maps the slice in a single `map_bundle` FFI call via [`mapfn/sdk`](../../mapfn/README.md) (reducing FFI crossings from 768 to 2–3 per bundle and FFI CPU overhead from ~23% to < 1%). +4. **Canonical Key Validation & Per-Entry Deduplication**: Upon unpacking the WASM output slab, the host validates every emitted key against the canonical key bounds in [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md) (`1..1024` printable ASCII bytes in `0x21..0x7E`, containing no `NUL` or whitespace bytes) and sorts and deduplicates canonical keys per entry (`entryIdx`). If the WASM module traps, overflows its linear memory slabs, or emits any invalid canonical key, the host returns a fatal validation error that immediately **`HALT`s** ingestion rather than silently dropping the key, preserving bit-for-bit `mpt_root` equivalence with [`internal/auditor`](../auditor/README.md). +5. **Resequencer (Min-Heap)**: Parallel workers finish bundles out of chronological order, and committing out-of-order entry indices would corrupt downstream RFC 6962 Value Log compact ranges. An in-memory min-heap buffers completed worker batches (holding subsequent completions if a straggler worker stalls on a large bundle) and emits strictly contiguous, ascending `(canonicalKey, entryIdx)` batches across `[max(persisted_watermark, from), Target_CP.Size)`. +6. **Tile Reaper**: Periodically receives the Coordinator's `SafeWatermark` (`max(min(KV_CP.Size, MPT_Durable_Size), from)`) and deletes only cached tile files fully covered by that watermark (`(tileIndex + 1) * 256 <= SafeWatermark`), preserving un-synced bundles on disk for Zero-WAL crash recovery while bounding local disk usage. + +--- + +## 2. Alternatives Considered + +### 2.1 Ephemeral In-Memory Fetching vs. Persistent Local Tile Cache +- **Proposed**: Streaming 256-entry bundles directly from HTTP responses into the WASM worker pool in memory without caching verified bundles on local disk. +- **Rejection**: Tightly couples upstream HTTP jitter to downstream mapping throughput, and forces any restart or dirty crash where `MPT_Durable_Size` lags `KV_CP.Size` or `Output_CP.InputSize` to re-download historical bundles over the network (risking `HTTP 429` rate limits, failing during upstream outages, and violating the sub-500ms Zero-WAL recovery SLO). +- **Chosen Design**: Persist Merkle-authenticated 256-entry bundles on local disk up to `SafeWatermark` (`max(min(KV_CP.Size, MPT_Durable_Size), from)`), enabling offline Zero-WAL crash replay while bounding disk usage via the tile reaper. + +--- + +## 3. Verification Gate + +```bash +go test -race ./vindex/internal/ingest/... +``` diff --git a/vindex/internal/kvstore/README.md b/vindex/internal/kvstore/README.md new file mode 100644 index 0000000..9851a63 --- /dev/null +++ b/vindex/internal/kvstore/README.md @@ -0,0 +1,88 @@ +# Inverted Chunk Storage Engine + +`vindex/internal/kvstore` is the embedded [Pebble](https://github.com/cockroachdb/pebble) storage engine that persists per-key Value Logs: the chronologically ordered sequences of Input Log entry indices at which each canonical key appeared, together with their RFC 6962 Merkle compact ranges. Transparency logs have extreme key cardinality skew (most keys appear once or twice, while hot keys accumulate hundreds of thousands of entries). By partitioning occurrences into global 65,536-entry windows and ordering chunks newest-first in the LSM keyspace, the store supports single-seek appends, lock-free backward pagination with zero read-path SHA-256 hashing, and point-in-time `value_root` reconstruction for crash recovery. + +### Non-Goals +- **No Distributed Storage**: Operates strictly as a single-host embedded database; remote RPC storage backends are out of scope. +- **No Lexical Prefix Scans in Pebble**: Indexes chunks internally by fixed-width 32-byte key hashes for exact key lookup; lexical prefix discovery across canonical keys is served by the in-memory MPT ([`internal/tree`](../tree/README.md)). +- **No Deletions or Tombstones**: Strictly append-only; sealed historical chunks are immutable. + +--- + +## 1. Design + +### 1.1 Pebble LSM Primer + +[Pebble](https://github.com/cockroachdb/pebble) is an embedded Log-Structured Merge-tree (LSM) key-value database (inspired by RocksDB) that stores keys in lexicographical byte order across an in-memory MemTable and immutable on-disk Sorted String Tables (SSTables). Three Pebble primitives underpin this design: + +- **Lower-Bound Range Seeks (`SeekGE` & `Next`)**: Unlike a hash table, an LSM iterator does not require an exact key match. Seeking to a target byte key binary-searches the SSTable block index for the **smallest existing key `>= target`** (jumping over arbitrarily large gaps of non-existent chunk numbers in a single lookup), and `Next()` steps forward to the next existing key in sorted order. +- **Split-Key Prefix Bloom Filters & `SeekPrefixGE`**: Pebble's [`Comparer.Split`](https://pkg.go.dev/github.com/cockroachdb/pebble#Comparer) divides every key into a fixed **prefix** (the first 33 bytes: `'c' + KeyHash`) and a **suffix** (the 8-byte inverted chunk number), and builds per-SSTable 10-bit [Bloom filters](https://pkg.go.dev/github.com/cockroachdb/pebble/bloom) over the 33-byte prefix rather than the full 41-byte key. Unless noted otherwise, **"seek"** in this document refers to [`Iterator.SeekPrefixGE`](https://pkg.go.dev/github.com/cockroachdb/pebble#Iterator.SeekPrefixGE): Pebble first checks each SSTable's Bloom filter against the 33-byte prefix (skipping SSTables that do not contain `KeyHash` with **zero disk I/O**, even when the caller's target suffix does not match an existing chunk), positions the iterator at the smallest existing key `>= target` within that prefix, and bounds `SeekPrefixGE` and `Next()` so iteration never crosses into an adjacent `KeyHash`. +- **Atomic Write Batches (`NoSync` vs. `Sync`)**: [`Batch.Commit`](https://pkg.go.dev/github.com/cockroachdb/pebble#Batch.Commit) applies a group of key mutations atomically. Committing with `NoSync` makes writes immediately visible in memory without blocking on disk I/O, while `Sync` enforces an OS `fsync` durability barrier before returning. + +### 1.2 Inverted 65,536-Entry Chunk Key Layout + +The Input Log entry index space is partitioned into fixed global windows of 65,536 entries (`chunkNum = entryIdx / 65536`, `relOffset = uint16(entryIdx % 65536)`). While the MPT stores un-hashed canonical keys (`1..1024` bytes in `0x21..0x7E`), Pebble allocates chunks only for windows in which a key occurs, keyed by fixed 41-byte records using internal `KeyHash = SHA-256(key)` digests and bitwise-inverted chunk numbers (`^BigEndian(chunkNum) = 0xFFFFFFFFFFFFFFFF - chunkNum`): + +```text +Key (41 bytes) = [Prefix 'c' (1B)] + [KeyHash (32B)] + [^BigEndian(chunkNum) (8B)] +``` + +Because bitwise inversion reverses numerical order (`chunk_A < chunk_B` implies `^chunk_A > ^chunk_B`), chunks for a given `KeyHash` are stored **newest-first** in the LSM keyspace. Seeking to `'c' + KeyHash` lands directly on the newest active chunk (eliminating the 7.5x scan latency penalty of forward-ordered chunks on deep keys), while seeking to `'c' + KeyHash + ^BigEndian(targetChunk)` jumps directly to the newest existing chunk with `chunkNum <= targetChunk`. + +### 1.3 Delimitless Binary Chunk Values & Compact Ranges + +Each chunk value stores a cumulative RFC 6962 compact range covering all occurrences of that key across prior chunks (`chunk < chunkNum`), followed by dense 16-bit relative offsets for occurrences within `chunkNum` (saving 75% storage compared to 64-bit indices): + +```text +Preceding uint64 (8 bytes, BigEndian) || CompactRange (32 bytes * popcount(Preceding)) || RelativeIndices (uint16, 2 bytes BigEndian * N) +``` + +| Byte Offset | Field Name | Data Type | Description | +| :--- | :--- | :--- | :--- | +| `0` | `Preceding` | `uint64` (8 bytes, BigEndian) | Cumulative number of occurrences (Value Log leaves) for this key across all prior chunks. | +| `8` | `CompactRange` | `[K][32]byte` | Compact range hashes committing to the `Preceding` older Value Log leaves, where `K = popcount(Preceding)` (`32 * K` bytes). | +| `8 + 32*K` | `RelativeIndices` | `[]uint16` (`2 * N` bytes, BigEndian) | Strictly increasing 2-byte offsets `uint16(entryIdx % 65536)` within window `chunkNum` (`entryIdx = chunkNum*65536 + uint64(relOffset)`). | + +Because `K = popcount(Preceding)` determines the exact length of `CompactRange`, `RelativeIndices` always starts at byte offset `8 + 32*K` without delimiters or length prefixes. + +### 1.4 Write Pipeline, Two-Generational Cache & Point-in-Time Reconstruction + +- **Per-Entry Deduplication & Parallel Value Log Hashing**: Batch writes defensively sort and deduplicate `(canonicalKey, entryIdx)` pairs so each `entryIdx` is appended at most once per key (`relOffset_0 < relOffset_1 < ...`). Distinct modified keys in a batch are partitioned across worker goroutines that compute RFC 6962 leaf hashes (`SHA-256(0x00 || uint64_be(entryIdx))`), interior node merges (`SHA-256(0x01 || left || right)`), `value_root` digests (`internal/wire`), and serialized chunk values in parallel in memory before committing sequentially in a single atomic Pebble batch, returning modified `(canonicalKey, value_root)` mappings for MPT commitment. When an occurrence crosses into a higher 65,536-entry window, the prior chunk is sealed and a new active chunk is initialized with the snapshotted `Preceding` and `CompactRange`. +- **Two-Generational Active Chunk Cache**: Each cache entry holds the active chunk descriptor (`Preceding`, `CompactRange`, `RelativeIndices`) alongside its in-memory running compact range, updating incrementally in O(1) amortized leaf hashes per new occurrence instead of re-hashing prior `RelativeIndices` from index `0` (reconstructed lazily once via a prefix seek on cache miss or restart). Maintaining current and previous batch generations absorbs repeated writes to hot keys without LRU lock contention and cuts Pebble block cache read I/O by >90%. +- **Synchronous Persistence Barrier & Crash-Replay Idempotency**: Intermediate commit slabs within a sync cycle commit with `NoSync`, while the final slab reaching `Target_CP.Size` (or an explicit flush barrier) commits with `Sync` (`fsync`), advancing `KV_CP` before Output Log publication (`KV_CP.Size >= Output_CP.InputSize`). Every batch write atomically records the batch's ending entry watermark alongside modified `'c'` records; during startup recovery or batch replay across `[Output_CP.InputSize, KV_CP.Size)`, any `entryIdx` below the persisted KV watermark (or `<=` the active chunk's highest persisted offset) is skipped. +- **Point-in-Time `value_root` Reconstruction**: To reconstruct a key's 32-byte `value_root` as of any target `input_size` (used during crash recovery and batch replay for keys with `>= 1` occurrences in `[from, input_size)`), the store seeks to `'c' + KeyHash + ^BigEndian((input_size - 1) / 65536)` (landing on the newest existing chunk `<= (input_size - 1) / 65536`), clones that chunk's `Preceding` and `CompactRange`, appends `SHA-256(0x00 || uint64_be(entryIdx))` for each relative offset with `chunkNum * 65536 + relOffset < input_size`, and folds the compact range right-to-left into `value_root`. Keys with zero total occurrences in `[from, input_size)` (`Preceding` plus matching offsets equals `0`) are omitted from the MPT per [`docs/tlog-vindex.md`](../../docs/tlog-vindex.md). + +### 1.5 Chunk-Aligned Backward Lookup & Zero-Hashing Compact Range Reuse + +Once the HTTP layer establishes `before <= Serving_CP.InputSize`, point lookups hash `canonicalKey` to `KeyHash = SHA-256(key)` and retrieve entry indices in `[from, before)` (ignoring any uncommitted chunks written ahead of `Serving_CP.InputSize`) using chunk-aligned page boundaries (`targetPageSize` default `100`, `maxChunksPerRequest` default `16`): +1. **Lower-Bound Prefix Seek**: Calls `SeekPrefixGE` on `'c' + KeyHash + ^BigEndian((before - 1) / 65536)`, skipping any gap of empty 65,536-entry windows to land directly on the newest existing chunk with `chunkNum <= (before - 1) / 65536` without scanning newer or unrelated SSTables. +2. **Whole-Chunk Reverse Traversal**: Steps forward with `Next()` through inverted keys (`^chunkNum`), moving backward chronologically across the key's existing chunks. Any chunk where zero offsets fall in `[from, before)` is skipped without incrementing `chunksVisited`. Each visited chunk with `>= 1` occurrences in `[from, before)` emits **all** matching offsets in that 65,536-entry window without intra-chunk splitting, halting once `totalMatched >= targetPageSize` or `chunksVisited >= maxChunksPerRequest` (returning up to `65,536` indices for a dense chunk, fewer than `targetPageSize` for sparse chunks, or empty / `HTTP 404` if zero occurrences exist in `[from, before)`). +3. **Zero-Hashing Compact Range Reuse**: Because page boundaries always align to whole chunks, when `Preceding > 0` on the oldest visited matching chunk, the `/values` response copies `preceding` and the compact range hashes directly from that chunk's stored `Preceding` and `CompactRange` with zero dynamic SHA-256 hashing on the read path. + +--- + +## 2. Alternatives Considered + +### 2.1 Storage Layouts & Chunk Sizing +- **Flat Lists per Key**: Rewriting a single flat list of 8-byte indices per key caused O(N^2) write amplification on hot keys. +- **One Record per Occurrence**: Storing one KV record per occurrence (`KeyHash + BigEndian(entryIdx)`) inflated SSTable metadata and slowed pagination scans. +- **Window Size (`65,536` vs. `256` / `1,024`)**: Small windows caused frequent chunk rollovers and bloated Pebble's block index; `65,536` caps uncompressed chunk values at ~130 KB and enables exact 16-bit relative offset encoding. +- **Inverted vs. Forward Chunk Ordering**: Forward ordering (`chunk_0, chunk_1, ...`) required scanning past historical SSTables to reach the active tail (7.5x slower appends), whereas bitwise inversion (`^chunkNum`) places the newest chunk first under `'c' + KeyHash`. + +### 2.2 Distributed NoSQL vs. Embedded Pebble LSM +- **Proposed**: Backing the chunk store with a distributed NoSQL database (Bigtable, Spanner, or Cassandra). +- **Rejection**: Remote RPCs add 5–20ms network latency per batch seek, and the MPT already requires single-host RAM/mmap locality. +- **Chosen Design**: Single-host embedded Pebble LSM engine. + +### 2.3 Concurrent Database Iterators vs. Key-Partitioned In-Memory Hashing +- **Proposed**: Partitioning keys across worker goroutines that each hold an independent Pebble iterator to read, mutate, and write chunks concurrently. +- **Rejection**: Concurrent Pebble iterators suffered lock contention and cache thrashing (~14% speedup on 4 workers, degrading beyond 8 workers) while competing with background LSM compactions. +- **Chosen Design**: Parallel key-partitioned Value Log hashing in pure memory followed by a single sequential atomic Pebble batch commit. + +--- + +## 3. Verification Gate + +```bash +go test -race ./vindex/internal/kvstore/... +``` diff --git a/vindex/internal/server/README.md b/vindex/internal/server/README.md new file mode 100644 index 0000000..ebb5605 --- /dev/null +++ b/vindex/internal/server/README.md @@ -0,0 +1,47 @@ +# Read Serving & C2SP Query Protocol + +`vindex/internal/server` is the HTTP serving layer that exposes a Verifiable Index over the public [C2SP `tlog-vindex`](../../docs/tlog-vindex.md) lookup protocol. Unlike a traditional database query server, every response must carry cryptographic proofs showing that no matching entries were omitted, while bounding memory and I/O when hot keys match hundreds of thousands of entries. The server splits lookups into two endpoints: key/prefix proof generation against the latest witnessed Output Log commitment, and lock-free backward Value Log pagination aligned to 65,536-entry storage chunks. + +### Non-Goals +- **No Unbounded Tree Scans**: Prefix lookups are bounded by an optional application prefix policy callback and a configurable maximum prefix match limit. +- **No Read-Uncommitted Serving**: Lookup responses never expose unwitnessed or in-flight Input Log entries ahead of the active witnessed serving snapshot. +- **No Server-Side Client Verification**: The server generates and formats proofs; cryptographic verification is performed by the lookup client ([`client/`](../../client/README.md)). + +--- + +## 1. Design + +### 1.1 Public & Operational HTTP Endpoints + +The public read server exposes the C2SP `tlog-vindex` lookup endpoints under `` (optionally mounted under a configured URL path prefix without `3xx` redirects) alongside `tlog-tiles` Output Log endpoints when co-hosted on the same mux. All responses are formatted directly into contiguous plaintext buffers (`internal/wire`) without JSON reflection (< 1 ms P50 read latency): + +| Endpoint | Method | Purpose, Status Codes & Caching | +| :--- | :--- | :--- | +| `/keys` | `GET` | Exact key (`key=`) or prefix (`prefix=`) lookup over canonical keys (`1..1024` printable ASCII bytes in `0x21..0x7E`), mutually exclusive with duplicate or unknown query params rejected, parsed via strict RFC 3986 percent-decoding (never converting literal `+` to space `0x20`).
On `prefix=`, validates `` against the optional prefix policy callback and collects at most `maxPrefixKeys + 1` matching `(key, value_root)` pairs **before** writing `200 OK` headers (returning `413 Content Too Large` cleanly if a `(maxPrefixKeys + 1)`-th match exists, default limit `10,000`, so a `200 OK` response is never truncated mid-stream), then writes `200 OK` (`text/plain; charset=utf-8`, `Cache-Control: no-store`) with `c2sp.org/tlog-vindex-keys@v1` followed by `c2sp.org/tlog-proof@v1`.
Returns `400 Bad Request` on invalid/missing/duplicate/unknown params, malformed `%`-escapes, non-canonical key syntax, or prefix policy rejection; `413 Content Too Large` when prefix matches exceed `maxPrefixKeys`; or `503 Service Unavailable` (`Retry-After: 30`) if the first Output Log checkpoint has not yet been published. | +| `/values` | `GET` | Lock-free backward Value Log pagination for `key` (**required**, strict RFC 3986 percent-decoding) and `before` (**required** canonical positive ASCII decimal integer `1 <= before < 2^64` with no leading zeros or `+`). Page boundaries align to whole 65,536-entry Pebble chunks (`targetPageSize` default `100`, `maxChunksPerRequest` default `16`), returning at least `1` entry index whenever occurrences in `[from, before)` exist and embedding the oldest visited chunk's precomputed `preceding` count and compact range hashes (`c2sp.org/tlog-vindex-values@v1`) with zero read-path SHA-256 hashing. Callers walking deep histories paginate sequentially using `before = Indices[0]` from the preceding page.
Acquires **zero MPT locks**; returns `200 OK` (`text/plain; charset=utf-8`, `Cache-Control: public, max-age=31536000, immutable`).
Returns `400 Bad Request` on missing/unknown/duplicate/invalid params, non-canonical or non-positive `before`, or malformed `%`-escapes; `503 Service Unavailable` (`Retry-After: 30`) if uninitialized or `before > Serving_CP.InputSize`; or `404 Not Found` once `before <= Serving_CP.InputSize` and zero occurrences exist in `[from, before)`. | +| `/checkpoint` | `GET` | Co-hosted [C2SP `tlog-tiles`](https://c2sp.org/tlog-tiles@v1.0.0) endpoint returning the latest signed, witnessed Output Log checkpoint note from `Serving_CP` (`503 Service Unavailable` with `Retry-After: 30` if uninitialized). | +| `/tile/...` | `GET` | Co-hosted `tlog-tiles` internal hash tiles and `/tile/entries/...` Output Log entry bundles served from the local Tessera Output Log directory for auditors. | + +Other paths under `` (such as `/` and `/index.html` for an optional embedded single-page HTML explorer UI) are left open and are not part of the protocol. Operational endpoints are isolated onto a separate internal administrative/metrics listener and are never exposed on the public listener: + +| Endpoint | Method | Purpose | +| :--- | :--- | :--- | +| `/healthz` | `GET` | Liveness probe (`200 OK` `ok\n` while the process and optional health callback report healthy). | +| `/readyz` | `GET` | Readiness probe (`200 OK` `ok\n` when `Serving_CP` is active, or `503` with `Retry-After: 30` and JSON diagnostics during MPT open, startup recovery, or Genesis Backfill). | +| `/metrics` | `GET` | Prometheus telemetry metrics. | +| `/debug/pprof/*` | `GET` | Go runtime profiling endpoints. | + +### 1.2 Reader Snapshot Isolation & Concurrency + +Lookup clients only ever observe state bounded by the latest witnessed Output Log commitment (`Serving_CP.InputSize <= Output_CP.InputSize`): +1. **Brief Shared Read Lock on `/keys`**: Acquires a shared read lock on the in-memory MPT, reads the atomic `Serving_CP` snapshot under that lock, generates the binary MPT point or prefix proof (`0` bytes when `mpt_root == empty_mpt_hash`, single-leaf inclusion, or full-tree prefix match), and releases the read lock before formatting and writing the response. +2. **Zero-Lock Pebble Reads on `/values`**: Loads the atomic `Serving_CP` pointer without acquiring any MPT lock, rejects `before > Serving_CP.InputSize` with `503 Service Unavailable` (`Retry-After: 30`), hashes `` to `SHA-256(key)`, and performs a chunk-aligned backward lookup in [`internal/kvstore`](../kvstore/README.md) restricted to `[from, before)`. +3. **Decoupled from Ingestion I/O**: Because Output Log appends, witness network roundtrips, and MPT disk `fsync` calls execute outside the in-memory MPT lock (see [`internal/tree`](../tree/README.md)), concurrent `/keys` and `/values` requests never block on background ingestion. + +--- + +## 2. Verification Gate + +```bash +go test -race ./vindex/internal/server/... +``` diff --git a/vindex/internal/tree/README.md b/vindex/internal/tree/README.md new file mode 100644 index 0000000..6469273 --- /dev/null +++ b/vindex/internal/tree/README.md @@ -0,0 +1,66 @@ +# Authenticated State Commitment (MPT & Output Log) + +The state commitment subsystem binds every present canonical key's Value Log root (`value_root`) into a path-compressed binary Merkle Patricia Trie (`mpt_root`), publishes state commitments to an append-only Output Log cosigned by external witnesses, and generates point inclusion, non-inclusion, and prefix proofs for HTTP lookup clients. By predicting candidate roots under a shared read lock before publishing and witnessing, and isolating background disk `fsync` calls from in-memory trie reads, concurrent HTTP queries never block on network witnessing or disk persistence. + +### Non-Goals +- **No Dynamic Rollbacks**: The trie and Output Log are strictly append-only; state transitions progress monotonically forward. +- **No Full In-Memory Trie Duplication**: Memory constraints require mutating a single in-memory trie under short-lived locks (< 5ms) rather than maintaining duplicate multi-gigabyte trie copies. +- **No Multi-Master Sequencing**: State commitment sequencing is driven by a single local Index Operator process. + +--- + +## 1. Design + +### 1.1 Package Decoupling & Variable-Length MPT (`internal/mpt` & `internal/tree`) + +The commitment plane is split into two internal packages: +- **`vindex/internal/mpt`**: Wraps [`filippo.io/torchwood/mpt`](https://pkg.go.dev/filippo.io/torchwood/mpt) with canonical key validation and binary MPT proof codecs (`uvarint(b) || sib_hash` path steps over split bits `0 <= b <= 8192`, bounded by at most `min(N - 1, 8193)` steps for `N` active keys) with zero dependencies on `pebble` or `tessera` so it can be imported directly by [`client`](../../client/README.md) and [`internal/server`](../server/README.md). On top of `torchwood/mpt`, `internal/mpt` enforces split-bit bounds `0 <= b <= 8192`, canonical key (`1..1024` bytes in `0x21..0x7E`) and 32-byte value validation on terminal no-match leaves, and [two-sided prefix verification](../../docs/tlog-vindex.md#two-sided-prefix-verification): checking `HasPrefix(key, prefix)` on every returned key ensures the server did not substitute a wider ancestor subtree containing keys outside `prefix`, while checking `b_1 < 8 * len(prefix)` on the first ancestor step, reconstructing the subtree root over the strictly ascending `(key, value_root)` pairs, and folding the ancestor path to `mpt_root` ensures no matching keys inside `prefix` were omitted. +- **`vindex/internal/tree`**: Coordinates candidate root prediction, Tessera POSIX Output Log publication, witness quorum collection, and atomic `Serving_CP` snapshot promotion. + +`torchwood/mpt` stores variable-length canonical keys in bitwise big-endian paths so keys sharing a byte prefix share a single subtrie. Inner trie nodes (`52 bytes/node`) live in anonymous virtual memory (~1–1.5 GB RAM for 10M keys, 100–150 GB for 1B keys) backed by alternating checksummed patch-frame journal files (`mpt.tree1`, `mpt.tree2`), while variable-length leaf nodes (`mpt_leaf_hash`) append to `mpt.disk`. Underlying file descriptors are wrapped with deferred-`fsync` handles so kernel `fsync(2)` calls never block in-memory reads; on process restart, opening the MPT replays uncompacted journal frames on disk before coordinator recovery begins (while the HTTP server returns `503 Service Unavailable`). + +### 1.2 The 3-Step Atomic Commitment Sequence & Split-Locking + +A commit cycle must satisfy two simultaneous invariants: +1. **Zero Unwitnessed State**: HTTP readers must never observe or obtain proofs against an `mpt_root` that has not yet been committed to the Output Log and cosigned by witnesses (`Serving_CP.InputSize <= Output_CP.InputSize`). +2. **Sub-Millisecond Read Latency**: Appending to the Output Log and collecting witness cosignatures takes 50–200ms+, and syncing journal/leaf files to disk takes 5–20ms. Holding an exclusive tree lock across network and disk I/O would starve HTTP readers. + +To satisfy both, the subsystem separates in-memory trie access from disk persistence using two independent locks and executes each commit in three steps: + +| Conceptual Lock | Synchronization Model | Scope & Protected State | Typical Duration | +| :--- | :--- | :--- | :--- | +| **In-Memory Tree Lock** | Read-write lock | Protects in-memory MPT nodes and the active `Serving_CP` snapshot. Shared read mode for HTTP proof generation and Step 1 root prediction; exclusive write mode only during Step 2 in-memory mutation and `Serving_CP` ratchet. | **< 5ms** (exclusive write)
**< 50µs** (shared read) | +| **Disk Persistence Lock** | Exclusive mutex | Serializes Step 2 in-memory journal buffer flushes against Step 3 background disk `fsync(2)` calls. Never held during Output Log appends or witness network calls. | **5ms – 20ms** | + +| Step | Phase | Operations | Lock State & Reader Impact | +| :--- | :--- | :--- | :--- | +| **1** | **Non-Blocking Prediction & Witnessing** | Pre-sorts the batch's modified `(key, value_root)` mappings outside the lock (reused in Step 2), acquires a shared read lock on the trie to compute the candidate `mpt_root` over cloned branch nodes, and immediately releases the read lock. Appends the 6-line `c2sp.org/tlog-vindex-state@v1` entry (`input-origin`, `from`, `input-size`, `input-root`, `mpt-root`) to the Output Log (retaining the signed Input Log checkpoint durably out-of-band) and collects witness cosignatures satisfying the Output Log checkpoint policy (such as [C2SP `tlog-policy`](https://c2sp.org/tlog-policy)) with **no tree or persistence locks held**. | **Zero reader blocking**; concurrent HTTP `/keys` proof generation runs in parallel under shared read locks. | +| **2** | **Atomic Mutation & Ratchet** | Acquires the exclusive persistence lock and exclusive tree lock. Mutates in-memory MPT nodes and flushes `torchwood/mpt`'s in-memory header/journal patch buffer into the OS page cache (against deferred-`fsync` file wrappers) so header and patch-frame compaction state never race concurrent readers. **Asserts that the committed root equals the predicted `mpt_root`** (terminating the process immediately on any mismatch rather than serving proofs that contradict the witnessed Output Log), ratchets the active `Serving_CP` pointer to the newly witnessed Output Log checkpoint and inclusion proof (`index == output_size - 1`), and releases the exclusive tree lock. | **< 5ms** exclusive write critical section. | +| **3** | **Background Durability** | Still holding only the persistence lock (with the exclusive tree lock released), executes blocking kernel `fsync(2)` calls on the dirty underlying file descriptors (`mpt.tree1`, `mpt.tree2`, `mpt.disk`) and atomically advances `MPT_Durable_Size` so the tile cache reaper can prune superseded 256-entry bundles. | **Zero reader blocking**; disk `fsync(2)` runs outside the tree lock. | + +--- + +## 2. Alternatives Considered + +### 2.1 Coarse Global Locking vs. Split-Locking with Prediction +- **Proposed**: Holding a single global lock across prediction, Output Log append, witness network calls, in-memory trie mutation, and disk `fsync`. +- **Rejection**: Under concurrent read load, coarse locking dropped HTTP read throughput to **53,000 queries/sec** with P99 read latency exceeding 250ms due to writer lock starvation. +- **Chosen Design**: Split-locking with lock-free prediction holds the exclusive tree lock for < 5ms, sustaining **678,000 queries/sec** (a 12.8x speedup) with sub-millisecond P50 read latency. + +### 2.2 Sparse Merkle Trees (SMT) vs. Merkle Patricia Trie (MPT) +- **Proposed**: Using a fixed 256-level Sparse Merkle Tree (SMT). +- **Rejection**: Traversing 256 levels per key incurs large memory and disk I/O amplification during random-insert batch workloads. +- **Chosen Design**: Path-compressed binary MPT (`filippo.io/torchwood/mpt`) over variable-length canonical keys, collapsing single-child paths to bound depth to the active key set. + +### 2.3 Post-Commit Output Log Publishing +- **Proposed**: Mutating and syncing the in-memory MPT to disk *first*, and then appending the state commitment entry to the Output Log. +- **Rejection**: A crash after MPT disk sync but before Output Log publication advances persisted trie state ahead of the Output Log (`MPT_Durable_Size > Output_CP.InputSize`), causing a restarted node to serve unwitnessed state. +- **Chosen Design**: Predict root under shared read lock -> publish and witness in the Output Log -> commit to MPT under short exclusive lock. + +--- + +## 3. Verification Gate + +```bash +go test -race ./vindex/internal/mpt/... ./vindex/internal/tree/... +``` diff --git a/vindex/mapfn/README.md b/vindex/mapfn/README.md new file mode 100644 index 0000000..f8e66a1 --- /dev/null +++ b/vindex/mapfn/README.md @@ -0,0 +1,109 @@ +# WASM Guest SDK & Plugin Interface + +Every transparency ecosystem indexes different log entry formats (such as DER X.509 certificates in Certificate Transparency, module lines in Go SumDB, or JSON provenance records in Sigstore). VIndex isolates entry parsing and canonical key extraction inside a hermetic, zero-I/O WebAssembly sandbox so ecosystem authors can write custom mapping logic without risking host memory safety or causing state divergence across independent auditors compiled with different toolchains or operating systems. + +This document covers the public Go guest SDK (`vindex/mapfn/sdk`), the language-agnostic shared-memory bundle ABI (`input_buffer` / `map_bundle`), the internal Wazero sandbox host (`vindex/internal/wasmhost`), and the offline verification CLI (`vindex-wasm`). + +### Non-Goals +- **No Worker Pool Orchestration in the Guest SDK**: The guest SDK provides only in-guest ABI bindings, while `internal/wasmhost` compiles the `.wasm` module once and provisions isolated per-worker module instances; multi-worker pipeline concurrency is managed by [`internal/ingest`](../internal/ingest/README.md). +- **No In-Place Mapping Upgrades**: A deployed mapping function is fixed for the lifetime of an Output Log. Changing extraction logic requires deploying a new index instance from its configured `from` start index. +- **No Host I/O or Cross-Call State**: Guest modules are pure, stateless functions with zero access to the filesystem, network, environment variables, or host clocks. + +--- + +## 1. Design + +### 1.1 Three-Layer Mapping Architecture + +Mapping is decoupled across three layers so plugin authors can write idiomatic Go (or Rust, C/C++, Zig, or TinyGo) while the host enforces strict sandboxing and canonical key validation: + +| Layer | Package / Component | Responsibility | +| :--- | :--- | :--- | +| **User Mapping Callback** | Ecosystem plugin (`MapFn`) | Pure function extracting zero or more canonical keys from a single raw Input Log entry. | +| **Guest SDK & Slab ABI** | `vindex/mapfn/sdk` (or non-Go slab target) | Binds user callbacks to the low-level `input_buffer` / `map_bundle` shared-memory ABI in flat little-endian WASM linear memory. | +| **Host Sandbox Runtime** | `vindex/internal/wasmhost` ([`wazero`](https://github.com/tetratelabs/wazero), zero CGO) | Instantiates `.wasm` modules with stubbed zero-I/O WASI imports, executes 256-entry bundles, validates canonical key bounds (`1..1024` bytes in `0x21..0x7E`), and sorts and deduplicates emitted keys per entry. | + +Within `vindex`, the `.wasm` module is compiled once (for example with `go build -trimpath -buildvcs=false -buildmode=c-shared`) and pinned out-of-band by its `SHA-256(wasm_bytes)` digest (`vindex-wasm inspect`). At the `tlog-vindex` protocol layer, `c2sp.org/tlog-vindex-state@v1` omits an in-band `map-fn` commitment pre-1.0 (see [`../docs/tlog-vindex.md`](../docs/tlog-vindex.md#design-log)); independent auditors may either execute the pinned `.wasm` artifact or run a clean-room native implementation of the same mapping specification. + +### 1.2 Go Guest Developer Contract (`vindex/mapfn/sdk`) + +Go plugin authors implement a streaming `emit` callback and register it in `init()`: + +```go +type EmitMapFunc func(entry []byte, emit func(key []byte)) + +func RegisterEmit(fn EmitMapFunc) +``` + +- **Build & Reactor Initialization**: Go guest plugins compile as WASI reactors (`GOOS=wasip1 GOARCH=wasm -buildmode=c-shared`, ~1.9 to 2.5 MB with the standard Go runtime or ~200 KB with TinyGo). `//go:wasmexport` bindings in `mapfn/sdk` are isolated behind `//go:build wasm` so native `go test` runs cleanly on host OSes. On startup, `internal/wasmhost` instantiates the module without running `_start` / `main()` and invokes `_initialize` if exported so both WASI reactors and `wasm32-unknown-unknown` guests initialize deterministically. +- **Canonical Key Validation & Fatal `HALT`**: Every emitted `key` MUST be a valid canonical key (`1..1024` printable ASCII bytes in `0x21..0x7E`, containing no `NUL` or whitespace bytes; see [`../docs/tlog-vindex.md`](../docs/tlog-vindex.md#canonical-keys) and [`../docs/APPLICATIONS.md`](../docs/APPLICATIONS.md)). Guest mappers must skip malformed or unindexable fields before calling `emit`: if the guest traps, overflows its linear memory slabs, or emits an invalid canonical key (`len == 0`, `len > 1024`, or any byte outside `0x21..0x7E`), `internal/wasmhost` returns a fatal validation error that immediately **`HALT`s** the indexer or auditor rather than silently dropping the key. +- **Zero-Allocation Streaming**: Each `emit(key)` call writes the length-prefixed canonical key directly into the pre-allocated output slab in-place, avoiding intermediate `[]string` or `[][]byte` slice allocations inside the guest. + +### 1.3 Language-Agnostic Slab ABI (`input_buffer` & `map_bundle`) + +Crossing the Go-to-WASM Foreign Function Interface (FFI) once per entry requires 768 transitions per 256-entry bundle and consumes ~23% of total host CPU time. To reduce FFI transitions to 2–3 per bundle (< 1% CPU overhead) and support guests written in any WASM-capable language, the host and guest exchange entire 256-entry bundles through pre-allocated contiguous linear memory slabs: + +```text +//go:wasmexport input_buffer +func input_buffer() uint32 + +//go:wasmexport map_bundle +func map_bundle(inputPtr uint32, inputLen uint32) uint64 +``` + +1. **Pack (Host)**: `internal/wasmhost` queries `input_buffer()` once on instance initialization (falling back to an exported `allocate(size)` function if `input_buffer` is not exported) and serializes the 256-entry bundle into the guest's static input slab. If the framed bundle exceeds the input slab capacity, the host returns a fatal error (`HALT`). +2. **Execute & Stream (Guest)**: The host invokes `map_bundle(inputPtr, inputLen)`. The guest reads the `[Count+1]uint32` offset table (`entry := payload[offsets[i]:offsets[i+1]]`), invokes the mapping callback per entry, and appends length-prefixed keys into the static output slab. If input decoding fails or `emit` overflows the output slab capacity, `map_bundle` aborts and returns `0`. +3. **Validate, Deduplicate & Wipe (Host)**: The guest returns `uint64(outputPtr)<<32 | uint64(outputLen)`. If the return value is `0` or out of bounds, the host returns a fatal error (`HALT`). Otherwise, the host reads all emitted keys from guest memory, validates canonical key bounds (`1..1024` bytes in `0x21..0x7E`), and **sorts and deduplicates canonical keys per entry** before passing the batch downstream. Output slab cursors reset to `0` on the next `map_bundle` call with zero guest heap allocations. + +#### Binary Slab Layouts (All `uint32` fields encoded in Little-Endian) + +| Memory Buffer | Offset | Field Name | Data Type | Description | +| :--- | :--- | :--- | :--- | :--- | +| **Input Slab** | `0` | `EntryCount` | `uint32` (4B, LE) | Number of entries in the bundle (`Count`, 1 to 256). | +| | `4` | `OffsetTable` | `[Count+1]uint32` (LE) | `Count + 1` byte offsets relative to `PayloadBlock`: `OffsetTable[0] = 0`, followed by each entry's end offset (`entry[i] = PayloadBlock[OffsetTable[i]:OffsetTable[i+1]]`). | +| | `8 + 4*Count` | `PayloadBlock` | `[]byte` | Raw, contiguous Input Log entry bytes concatenated end-to-end. | +| **Output Slab** | `0` | `EntryCount` | `uint32` (4B, LE) | Number of entries processed (`Count`). | +| | `4` | `KeyCountTable` | `[Count]uint32` (LE) | Number of canonical keys emitted by each respective entry. | +| | `4 + 4*Count` | `KeyBlock` | `[]byte` | Concatenated length-prefixed canonical keys (`uint32 length` (LE) + raw canonical key bytes). | + +### 1.4 Offline Verification CLI (`vindex-wasm`) + +`vindex/cmd/vindex-wasm` allows plugin authors to inspect, test, and benchmark `.wasm` modules before deployment: +- **`vindex-wasm inspect`**: Instantiates the module via `internal/wasmhost`, validates `input_buffer` / `map_bundle` exports and memory bounds, and prints the 32-byte `.wasm` SHA-256 digest in base64 and hex for operator and auditor pinning. +- **`vindex-wasm test`**: Runs the module against literal entry strings or binary entry files, validates canonical key bounds (`1..1024` bytes in `0x21..0x7E`), and prints extracted keys in plain text or JSON. +- **`vindex-wasm bench`**: Benchmarks single-worker and multi-bundle mapping throughput (`entries/sec`) and per-bundle latency over repeated 256-entry bundles. + +```bash +vindex-wasm inspect --wasm=map.wasm +vindex-wasm test --wasm=map.wasm --input="golang.org/x/crypto v0.1.0 h1:..." +vindex-wasm test --wasm=map.wasm --input_file=entry.bin --format=json +vindex-wasm bench --wasm=map.wasm --input="golang.org/x/crypto v0.1.0 h1:..." --iterations=1000 +``` + +--- + +## 2. Alternatives Considered + +### 2.1 Per-Entry Mapping (`map_leaf`) vs. Bundled Mapping (`map_bundle`) +- **Proposed**: Invoking an exported `map_leaf(ptr, len)` guest function individually for every Input Log entry. +- **Rejection**: Generated 768 FFI calls per 256-entry bundle (allocation, mapping, and reset x 256), spending ~23% of total host CPU time on boundary-crossing overhead. +- **Chosen Design**: Bundled mapping (`map_bundle`) processes up to 256 entries per FFI call via shared memory slabs, reducing FFI overhead to < 1%. + +### 2.2 In-Process Go Plugins / CGO vs. WebAssembly Sandboxing +- **Proposed**: Dynamically loading native Go plugins (`plugin.Open`) or shared C libraries (`cgo`). +- **Rejection**: Native plugins do not guarantee bit-for-bit determinism across Go compiler/stdlib upgrades or host OSes, crash the entire daemon on a plugin panic or memory fault, and require identical host/plugin dependency trees. +- **Chosen Design**: Zero-CGO Wazero WebAssembly sandboxing (`internal/wasmhost`). + +--- + +## Appendix: Guest Parsing Optimization Notes (Certificate Transparency) + +Compiling reflection-heavy standard library decoders (`crypto/x509`, `encoding/asn1`, `encoding/json`, `regexp`) into `wasip1/wasm` incurs high heap allocation churn and unnecessary cryptographic validation (such as public-key and signature decoding) irrelevant to canonical key extraction. Ecosystem plugins should prefer targeted, in-place byte scanners (for example, scanning raw ASN.1 TLV streams for Subject Common Name OID `2.5.4.3` and SAN `dNSName` OID `2.5.29.17` before applying canonical DNS label normalization per [`../docs/APPLICATIONS.md`](../docs/APPLICATIONS.md)), which achieves a ~7.5x throughput improvement over `crypto/x509` in `wasip1/wasm`. + +--- + +## 3. Verification Gate + +```bash +go test -race ./vindex/mapfn/... ./vindex/internal/wasmhost/... ./vindex/cmd/vindex-wasm/... +```