From feadad97da4187cd888e192cb37b5b15542b2a19 Mon Sep 17 00:00:00 2001 From: Martin Hutchinson Date: Tue, 27 Oct 2026 14:22:56 +0000 Subject: [PATCH] docs(vindex): update tlog-vindex spec and add applications and architecture docs - Simplify c2sp.org/tlog-vindex-state@v1 to 6 lines (omitting embedded input log checkpoint signatures and shelving in-band map-fn commitment pre-1.0) and clarify empty-range, MPT topology, and verification rules in docs/tlog-vindex.md. - Add docs/APPLICATIONS.md illustrating how CT, MTC, Go SumDB, Sigstore, and Sigsum map onto VIndex (who monitors each log, canonical key shapes, prefix queries, and epoch rotation). - Add docs/ARCHITECTURE.md defining the high-level 5-stage Map Sandwich pipeline, checkpoint promotion invariants, crash consistency, auditing boundaries, and architectural trade-offs. - Update vindex/README.md to align terminology and index the high-level documentation. --- vindex/README.md | 40 ++++---- vindex/docs/APPLICATIONS.md | 134 +++++++++++++++++++++++++++ vindex/docs/ARCHITECTURE.md | 176 ++++++++++++++++++++++++++++++++++++ vindex/docs/tlog-vindex.md | 100 ++++++++++---------- 4 files changed, 382 insertions(+), 68 deletions(-) create mode 100644 vindex/docs/APPLICATIONS.md create mode 100644 vindex/docs/ARCHITECTURE.md diff --git a/vindex/README.md b/vindex/README.md index b81b495..6bda417 100644 --- a/vindex/README.md +++ b/vindex/README.md @@ -1,6 +1,6 @@ # Verifiable Index (VIndex) -A **Verifiable Index (VIndex)** provides efficient, trustless, and cryptographically verifiable querying over append-only transparency logs (such as Certificate Transparency, Merkle Tree Certificates, and Go SumDB). +A **Verifiable Index (VIndex)** provides efficient, trustless, and cryptographically verifiable querying over append-only transparency logs (such as Certificate Transparency, Merkle Tree Certificates, Go SumDB, Sigstore, and Sigsum). > **Looking for the original WAL-based MVP?** The initial prototype that preceded this architecture has been retired on `main`; you can browse its source and docs at commit [`d76a0d4`](https://github.com/transparency-dev/incubator/tree/d76a0d4/vindex). @@ -8,7 +8,7 @@ A **Verifiable Index (VIndex)** provides efficient, trustless, and cryptographic ## 1. Context & Problem Statement -Append-only transparency logs provide tamper-evident discoverability: anyone can mathematically verify that an entry exists at a specific sequence number in the log. However, discovering records by content (e.g., domain name, module path, or public key hash) historically forced an impossible trade-off: +Append-only transparency logs provide tamper-evident discoverability: anyone can mathematically verify that an entry exists at a specific sequence number in the log. However, discovering records by content (e.g., domain name, module path, signing identity, or public key hash) historically forced an impossible trade-off: 1. **Massive Client Inefficiency**: Finding a subject required clients to download, deserialize, and scan gigabytes or terabytes of irrelevant log entries locally. 2. **Loss of Verifiability**: To avoid local downloads, users relied on centralized, third-party search APIs (such as `crt.sh`). However, these services lack cryptographic commitments; an untrusted or compromised operator can omit, reorder, or forge search results without detection. @@ -23,13 +23,13 @@ VIndex v1 represents the distillation of years of research, experimentation, and - [**Google Key Transparency**](https://github.com/google/key-transparency): Early pioneer in verifiable identity-to-key mapping, exploring sparse Merkle tree lookups and continuous auditability. - [**Trillian Batch Map**](https://github.com/google/trillian/tree/master/experimental/batchmap): Experimental map engine investigating batch-oriented Sparse Merkle Tree mutations and log integration. -- [**VIndex Prototype (v0 MVP)**](https://github.com/transparency-dev/incubator/tree/d76a0d4/vindex): The initial WAL-based prototype (preserved at commit [`d76a0d4`](https://github.com/transparency-dev/incubator/tree/d76a0d4/vindex)), which validated the core "Map Sandwich" concept, in-process Pebble storage, and demonstrated full-scale indexing on the Go Checksum Database (SumDB). +- [**VIndex Prototype (v0 MVP)**](https://github.com/transparency-dev/incubator/tree/d76a0d4/vindex): The initial WAL-based prototype (preserved at commit [`d76a0d4`](https://github.com/transparency-dev/incubator/tree/d76a0d4/vindex)), which validated the core "Map Sandwich" concept, in-process KV storage, and full-scale indexing on the Go Checksum Database (SumDB). ### The Key Evolution: Pure Indexing vs. State Summarization Earlier systems attempted to act as verifiable state machines that copied, aggregated, or summarized application data directly inside the map nodes. VIndex v1 makes a fundamental architectural departure: it focuses **purely on indexing** an existing log rather than summarizing its state: -- **Pointers, Not Copies**: The index stores lightweight leaf pointers (indices) referencing the authoritative Input Log, rather than replicating or transforming leaf payloads. -- **High Throughput & Bounded Footprint**: Storing compact relative index offsets eliminates data amplification, enabling sustained indexing speeds exceeding 240,000 leaves per second on commodity hardware. -- **Clear Security Boundaries**: The Input Log remains the sole source of truth. VIndex acts as an untrusted search overlay; if the indexer halts, corrupts, or falls behind, the underlying log's consensus and integrity are completely unaffected, and clients independently verify retrieved leaf data against the Input Log's signed checkpoints. +- **Pointers, Not Copies**: The index stores lightweight Input Log entry indices referencing the authoritative Input Log, rather than replicating or transforming entry payloads. +- **Bounded Storage Footprint**: Storing compact relative entry-index chunks rather than entry payloads avoids data amplification, keeping local storage proportional to occurrence counts rather than log payload volume. +- **Clear Security Boundaries**: The Input Log remains the sole source of truth. VIndex acts as an untrusted search overlay; if the indexer halts, corrupts, or falls behind, the underlying log's consensus and integrity are completely unaffected, and lookup clients independently verify retrieved entry data against the Input Log's signed checkpoints. --- @@ -39,24 +39,24 @@ The VIndex topology bounds search indexing between two append-only logs—a stru | Component | Physical Role | Cryptographic Contract | | :--- | :--- | :--- | -| **Input Log** | Authoritative source of truth (e.g. CT, MTC, Go SumDB, Sigstore). | Strictly append-only log of raw leaves and signed tree checkpoints. | -| **vindexd Daemon** | Single-host indexing and serving engine. | Ingests tiles, executes sandboxed WASM mapping, stores inverted chunks in Pebble DB, commits state in an in-memory MPT, and serves C2SP HTTP lookups. | -| **Output Log** | Cryptographic anchor committing to index state (`mpt-root` + Input Log checkpoint). | Append-only log of state commitments signed and monitored by an independent witness network. | +| **Input Log** | Authoritative source of truth (e.g. CT, MTC, Go SumDB, Sigstore, Sigsum). | Strictly append-only log of raw entries and signed tree checkpoints. | +| **Indexer & Serving Node** | Single-host indexing and serving process. | Ingests authenticated entry bundles, executes sandboxed WASM mapping, stores value-log chunks in an embedded KV store, commits state in an MPT, and serves C2SP HTTP lookups. | +| **Output Log** | Cryptographic anchor committing to index state (`input-origin`, `from`, `input-size`, `input-root`, and `mpt-root`). | Append-only log of index state commitments signed and monitored by an independent witness network. | ### Core System Guarantees -- **Point & Bounded Prefix Lookups**: Exact key lookups (`?key=`) and policy-bounded prefix scans (`?prefix=`) over variable-length canonical keys (`1..1024` bytes), returning verified value log roots and paginated lists of matching entry indices. -- **Two-Endpoint Read Protocol**: Separates state commitment & MPT proof generation (`GET /keys?key=` or `GET /keys?prefix=`) from lock-free Pebble value log pagination (`GET /values?key=&before=[&limit=]`). +- **Point & Bounded Prefix Lookups**: Exact key lookups (`?key=`) and policy-bounded prefix scans (`?prefix=`) over variable-length canonical keys (`1..1024` bytes), returning verified value-log roots and paginated lists of matching Input Log entry indices. +- **Two-Endpoint Read Protocol**: Separates state commitment and MPT proof generation (`GET /keys?key=` or `GET /keys?prefix=`) from lock-free value-log pagination (`GET /values?key=&before=`). - **Omission Resistance**: Every query response delivers cryptographic inclusion proofs, non-inclusion proofs, or two-sided prefix proofs against witnessed checkpoints. -- **Decoupled Security**: VIndex operates as a secondary search overlay. If an indexer halts, corrupts, or falls behind, the underlying Input Log's consensus and security remain completely unaffected. +- **Decoupled Security**: VIndex operates as a secondary search overlay. If an Index Operator halts, corrupts, or falls behind, the underlying Input Log's consensus and security remain completely unaffected. --- ## 4. Non-Negotiable System Scope -1. **Strictly Single-Machine Deployment**: VIndex operates strictly within a single process on a single host. It avoids internal clustering, distributed consensus (Raft/Paxos), or multi-node replication. High availability is achieved externally: independent monitors and mirrors run standalone VIndex instances against the shared Input Log. +1. **Strictly Single-Machine Deployment**: VIndex operates strictly within a single process on a single host. It avoids internal clustering, distributed consensus (Raft/Paxos), or multi-node replication. High availability is achieved externally: independent Index Operators run standalone VIndex instances against the shared Input Log. 2. **Canonical Key & Bounded Prefix Lookups**: The production serving plane provides point queries (`?key=`) and policy-guarded prefix queries (`?prefix=`) over canonical printable ASCII keys (`1..1024` bytes, `0x21..0x7E`). Unbounded full-tree scans, arbitrary substring matches, and regex searches are out of scope. 3. **Strictly Append-Only (Zero Tombstones)**: The index state progresses strictly forward. The system supports no deletions, key un-mappings, rollbacks, or tombstones. -4. **Hermetic, Deterministic Mapping**: The WebAssembly guest plugin operates in a restricted WASI-free sandbox (`mapfn/host`) with zero host I/O, zero network access, and deterministic execution. Any runtime trap, memory fault, or unhandled exception triggers an immediate `HALT` to prevent unverified state divergence across nodes. +4. **Hermetic, Deterministic Mapping**: WebAssembly mapping functions execute inside a zero-I/O sandbox with no filesystem, clock, or network access. Any guest trap, fault, or invalid canonical key emission halts indexing immediately to prevent state divergence across nodes. --- @@ -64,15 +64,17 @@ The VIndex topology bounds search indexing between two append-only logs—a stru - **Input Log**: The authoritative, append-only source transparency log containing the raw entries to be indexed. - **Canonical Key**: A normalized printable ASCII byte string (`1..1024` bytes in `0x21..0x7E`) identifying a subject in an Input Log entry (e.g. reversed-label domain `com.google.mail.` in CT/MTC or case-escaped module path `golang.org/x/!text` in Go SumDB). Corresponds to the normalized *Claim Subject* in [Claimant Model](https://github.com/google/trillian/blob/master/docs/claimantmodel/Maps.md) terminology. -- **MapFn**: A pure, deterministic mapping function (`map_fn(entry: bytes) -> list[CanonicalKey]`) that extracts zero or more Canonical Keys from a raw Input Log entry, identified in state commitments by a 32-byte SHA-256 digest (`map-fn`). -- **WASM** (*WebAssembly*): The hermetic, deterministic bytecode sandbox used to package and execute `MapFn` plugins with zero host I/O or network access, ensuring identical key extraction across operators and verifiers. +- **MapFn**: A pure, deterministic mapping function (`map_fn(entry: bytes) -> list[CanonicalKey]`) that extracts zero or more Canonical Keys from a raw Input Log entry, fixed for the lifetime of an Output Log (`tlog-vindex` itself is runtime-agnostic and omits an in-band `map-fn` commitment pre-1.0; see [`docs/tlog-vindex.md`](./docs/tlog-vindex.md#design-log)). +- **WASM** (*WebAssembly*): The hermetic, language-agnostic bytecode sandbox (targetable from Go, Rust, C/C++, Zig, or TinyGo) used by the indexer and independent auditor to execute `MapFn` plugins with zero host I/O or network access. - **Value Log**: A per-key, append-only RFC 6962 Merkle tree committing to the strictly ascending sequence of Input Log entry indices at which that Canonical Key appeared, summarized by a 32-byte `value_root`. - **MPT** (*Merkle Patricia Trie*): A path-compressed binary Merkle Patricia Trie mapping each present Canonical Key to its 32-byte Value Log root (`value_root`), supporting exact inclusion, non-inclusion, and two-sided prefix subtree proofs. -- **Witnessed Output Log**: A secondary append-only tiled transparency log (`tlog-tiles`) that records index state commitments (`mpt-root`, `map-fn`, `from`, and Input Log checkpoint) cosigned by independent witnesses. +- **Witnessed Output Log**: A secondary append-only tiled transparency log (`tlog-tiles`) that records 6-line index state commitments (`input-origin`, `from`, `input-size`, `input-root`, and `mpt-root`) cosigned by independent witnesses. - **Verifier of Correct Construction** (*Auditor*): An independent verifier that replays the Input Log through `MapFn` and recomputes Value Logs and the MPT to verify that every `mpt-root` published in the Output Log was faithfully constructed without omissions or fabrications. --- -## 6. Specification +## 6. Documentation -- [**C2SP tlog-vindex Specification**](./docs/tlog-vindex.md): Formal wire protocol, cryptographic tree definitions, and client/auditor verification specification. +- [**C2SP `tlog-vindex` Specification**](./docs/tlog-vindex.md): Normative wire formats, MPT and value-log Merkle definitions, HTTP lookup endpoints, and client/auditor verification rules. +- [**Verifiable Index Applications**](./docs/APPLICATIONS.md): How VIndex applies across Certificate Transparency, Merkle Tree Certificates, Go SumDB, Sigstore, and Sigsum—who monitors each log, illustrative canonical key shapes, prefix queries, and epoch rotation (`from > 0`). +- [**Verifiable Index Architecture**](./docs/ARCHITECTURE.md): High-level pipeline stages, checkpoint promotion invariants, concurrency model, Zero-WAL crash recovery, auditing boundaries, and architectural trade-offs. diff --git a/vindex/docs/APPLICATIONS.md b/vindex/docs/APPLICATIONS.md new file mode 100644 index 0000000..f16a86a --- /dev/null +++ b/vindex/docs/APPLICATIONS.md @@ -0,0 +1,134 @@ +# Verifiable Index Applications + +Append-only transparency logs make records tamper-evident once logged, but they only deliver accountability when the people affected by those records can actually find them. This document shows how real-world transparency ecosystems map onto the [C2SP `tlog-vindex`](./tlog-vindex.md) model: who monitors each log, what question a verifiable index answers for them, how log leaves map to canonical keys, and how prefix lookups and log rotation work in practice. + +--- + +## 1. Claim Subject Maps & Who Verifies the Log + +In the [Claimant Model](https://github.com/google/trillian/blob/master/docs/claimantmodel/Maps.md), an append-only transparency log records signed claims, while a verifiable index acts as a **Claim Subject Map (CSM)** (or **Map of Logs (MoL)**) over that _input log_: + +- **Claim Subjects (Canonical Keys)**: A deterministic _mapping function_ extracts zero or more printable ASCII _canonical keys_ from each input log entry in `[from, input_size)`, for example, a domain name, a Go module path, an OIDC signing identity, or a public key hash. +- **Value Logs**: Rather than copying or summarizing entry payloads inside the map, each present key in the Merkle Patricia Trie (MPT) points to the 32-byte [RFC 6962](https://www.rfc-editor.org/rfc/rfc6962.html) Merkle root (`value_root`) of an append-only _value log_ recording the ascending input log entry indices (`entry_idx`) where that key appeared. + +The security property a Claim Subject Map provides is **Efficient Discovery**. An artifact consumer who already holds a certificate, Go module, or signature bundle can verify a single inclusion proof directly against the input log. The harder problem belongs to **subject owners** (e.g. domain owners, module maintainers, and key holders) who need to verify that _no unexpected entries_ concerning their identity or namespace exist anywhere in the log, without downloading terabytes of unrelated entries or trusting an unauthenticated search service. Lookup clients query `/keys` and `/values` and verify completeness offline against witnessed _output log_ commitments. + +--- + +## 2. Mapping Principles & Ecosystem Overview + +Every ecosystem profile turns raw input log entries into canonical keys ([`tlog-vindex`](./tlog-vindex.md#index-structure), 1 to 1024 printable ASCII bytes without whitespace) under three shared rules: + +1. **Consistent Key Normalization**: Index operators, independent auditors, and lookup clients apply the exact same normalization from a real-world identifier to canonical key bytes. Because the MPT and `/keys` match raw canonical key bytes directly, querying `example.com` or `com.example` against an index keyed by `com.example.` walks a different trie path and returns a valid non-inclusion proof even when certificates exist for that domain. +2. **Skipping Unindexable Fields**: Real-world log entries contain fields that do not represent indexable subjects, such as human-readable X.509 Organization/Common Name strings containing spaces, output artifact hashes, or untagged commit pseudo-versions in Go SumDB. Mapping functions deterministically skip those fields so only valid canonical keys enter the trie. +3. **One Occurrence per Entry**: When a single log entry names the same subject multiple times (such as repeating a domain across Subject Common Name and Subject Alternative Name, or repeating a Go module path on both archive and `/go.mod` lines), duplicates within that entry are collapsed so `entry_idx` appears at most once in the key's value log. + +| Ecosystem | Who Monitors | Claim Subject | Illustrative Canonical Key | Keys per Entry | +| :--------------------------- | :-------------------- | :--------------------------- | :-------------------------------------------------------------------------------------------- | :------------- | +| **Certificate Transparency** | Domain owners | DNS name (CN / SAN) | `com.google.mail.`, `com.google.*.` | `0..N` | +| **Merkle Tree Certificates** | Domain owners | DNS name (subject assertion) | `com.google.mail.`, `com.google.*.` | `0..N` | +| **Go Checksum Database** | Module authors & orgs | Go module path | `github.com/!azure/azure-sdk-for-go` | `0..1` | +| **Sigstore (Rekor)** | Signers & CI owners | OIDC email or workload URI | `user@example.com`, `https://github.com/org/repo/.github/workflows/build.yml@refs/heads/main` | `0..N` | +| **Sigsum** | Key holders | Ed25519 public key hash | 64-character lowercase hex digest | `1` | + +--- + +## 3. Ecosystem Deployments + +### 3.1 Certificate Transparency (RFC 6962 & Static CT) + +- **References**: + - [RFC 6962: Certificate Transparency](https://www.rfc-editor.org/rfc/rfc6962.html) + - [C2SP Static CT API Specification](https://c2sp.org/static-ct-api) + - [Known Certificate Transparency Logs](https://certificate.transparency.dev/logs/) + +**Who verifies, and why**: Browsers check at TLS handshake time that a presented certificate was logged, which only forces an attacker or compromised CA to log a mis-issued certificate. **Domain owners** must monitor CT logs to verify that no unauthorized certificates cover their domains or subdomains. Without a verifiable index, monitors must either download and parse terabytes of certificate entries across dozens of logs or rely on centralized aggregators (such as `crt.sh`) that carry no cryptographic proof of completeness. + +**From leaf to canonical key**: CT log leaves (`x509_entry` certificates and `precert_entry` precertificates) contain ASN.1 DER X.509 `Certificate` / `TBSCertificate` structures carrying DNS identifiers in the Subject Common Name (CN) and Subject Alternative Name (SAN) `dNSName` extension. The mapping function extracts valid DNS names, converts internationalized labels to lowercase ASCII Punycode (IDNA2008), reverses the label order, and appends a trailing dot, turning `mail.google.com` into `com.google.mail.` and `*.google.com` into `com.google.*.`. Non-DNS Common Names (such as organization strings with spaces) and IP literals are skipped, and parent domain suffixes are not expanded at map time. + +**Querying the index**: A domain owner monitoring `google.com` queries `GET /keys?prefix=com.google.` (or `GET /keys?key=com.google.mail.` for a single host). The trailing dot aligns prefix matching on complete DNS label boundaries so `com.google.` never matches sibling domains like `com.googleapis.`, while server prefix policies reject bare public-suffix queries such as `com.` or `uk.co.`. The `/keys` response returns a verified list of every matching apex, subdomain, and wildcard key (`com.google.`, `com.google.*.`, `com.google.mail.`, ...) with its `value_root`. For each key of interest, the monitor queries `GET /values?key=&before=` to verify the matching log entry indices and fetches those certificates from the CT log. + +--- + +### 3.2 Merkle Tree Certificates (MTC) + +- **Reference**: + - [IETF PLANTS Working Group: Merkle Tree Certificates (`draft-ietf-plants-merkle-tree-certs`)](https://datatracker.ietf.org/doc/draft-ietf-plants-merkle-tree-certs/) + +**Who verifies, and why**: Under `draft-ietf-plants-merkle-tree-certs`, Merkle Tree Certificates embed public logging directly into issuance to avoid bulky per-certificate post-quantum signatures in TLS handshakes: CAs log subject assertions (binding DNS names, public keys, and short validity windows) into Merkle trees whose active landmark subtrees are pre-distributed to relying parties. Short certificate lifetimes (days to two weeks) and high issuance volume cause rapid log growth and rely on periodic pruning of expired entries, facing **domain owners** with the same discovery problem as CT at much higher entry velocity. + +**Key mapping & prefix lookups**: Identical to Certificate Transparency (§3.1). DNS identifiers in MTC subject assertions map to lowercase Punycode reversed-label keys with a trailing dot (`com.google.mail.`, `com.google.*.`), queried via `/keys` under the same trailing-dot and registrable-domain prefix guardrails. + +Two external `tlog-vindex` properties matter specifically for MTC deployments: + +- **Checkpoint Signature Decoupling**: Output log commitments (`c2sp.org/tlog-vindex-state@v1`) bind unsigned input log state (`input-origin`, `from`, `input-size`, `input-root`, and `mpt-root`). Domain monitors verifying `/keys` and `/values` responses only need the output log's checkpoint verification policy and are decoupled from MTC's upstream checkpoint and cosigned-subtree signature formats; only index operators, independent auditors, and clients verifying MTC checkpoints directly need an MTC checkpoint policy. +- **Log Pruning & Epoch Rotation (`from > 0`)**: Within a single VIndex instance, `from` is fixed and `[from, input_size)` is strictly append-only. When an MTC log prunes expired historical entries below a minimum retained index `P` (or when an operator sheds expired index history), the operator stands up a new VIndex instance and output log, starting at `from = P`. Once `[P, input_size)` covers the full lifetime of all unexpired certificates, monitors confirm `from` meets their coverage horizon and cut over cleanly before the older instance is retired. + +--- + +### 3.3 Go Checksum Database (SumDB) + +- **References**: + - [Go Modules Reference: Checksum Database](https://go.dev/ref/mod#checksum-database) + - [Proposal: Secure the Public Go Package Ecosystem](https://go.dev/design/25530-sumdb) + - Public log: [`sum.golang.org`](https://sum.golang.org) + +**Who verifies, and why**: When `go get` downloads a dependency, it checks `sum.golang.org` via `/lookup/module@version` to confirm the hash for a _known_ version string. What `sum.golang.org` cannot answer is enumeration: **module authors and security teams** have no verifiable way to ask _"what versions have ever been logged for my module?"_ (to spot an unauthorized release tag) or _"what modules exist under our organization's namespace?"_ without trusting external package registries or proxies. + +**From leaf to canonical key**: Each `sum.golang.org` leaf contains newline-delimited `go.sum` lines pairing a module archive and its `/go.mod` file with their hashes: + +```text +github.com/google/uuid v1.3.0 h1:tIOElFI0... +github.com/google/uuid v1.3.0/go.mod h1:Usq... +``` + +To keep the index focused on published releases rather than every ephemeral commit, the mapping function skips untagged Go pseudo-versions and converts tagged module paths into Go's case-encoded canonical form, encoding uppercase letters as `!` followed by lowercase (for example, `github.com/Azure/azure-sdk-for-go` becomes `github.com/!azure/azure-sdk-for-go`). Because both lines of a tagged entry name the same module, each standard release entry produces a single key (`0..1` keys per entry overall). + +**Querying the index**: Maintainers query an exact module (`GET /keys?key=github.com%2Fgoogle%2Fuuid`, percent-encoding `/` as `%2F` and `!` as `%21`) or an organizational namespace prefix (`GET /keys?prefix=github.com%2Fgoogle%2F`), page `/values` to obtain the verified log indices, and fetch those entries from `sum.golang.org` to inspect every recorded release tag. + +--- + +### 3.4 Sigstore (Rekor) + +- **References**: + - [Sigstore Documentation](https://docs.sigstore.dev/) + - [Rekor Transparency Log](https://github.com/sigstore/rekor) (`rekor.sigstore.dev`) + +**Who verifies, and why**: Artifact consumers verify a Rekor inclusion proof when installing a signed package or container, which only proves that _that_ signature was logged. Who verifies that an attacker did not briefly hijack a developer's OIDC session or a repository's GitHub Actions workflow to sign something else? **The signers themselves**; developers, release engineers, and CI identity owners must monitor Rekor for every signing event bound to their identity. Rekor's built-in search API is unauthenticated, so it cannot prove that a compromised signing event was not omitted from search results. + +**From leaf to canonical key**: Rekor entries (such as `hashedrekord`, `intoto`, and `dsse` records) embed short-lived Fulcio X.509 signing certificates that bind the signer's OpenID Connect (OIDC) identity in the Subject Alternative Name extension. Rather than indexing one-off artifact digests (which would create a new single-entry key for every build), the mapping function extracts the signer identity from the Fulcio SAN; either a lowercase email address (`user@example.com`) or a workload identity URI (`https://github.com/org/repo/.github/workflows/build.yml@refs/heads/main`). + +**Querying the index**: Credential owners query their exact email (`GET /keys?key=user%40example.com`) or their repository/organization URI prefix (`GET /keys?prefix=https%3A%2F%2Fgithub.com%2Forg%2Frepo%2F`), then page `/values` to audit every signature or attestation ever issued under their credentials. + +--- + +### 3.5 Sigsum + +- **References**: + - [Sigsum](https://www.sigsum.org/) + - [Sigsum Documentation & Specifications](https://www.sigsum.org/docs/) + +**Who verifies, and why**: Like Sigstore, Sigsum logs signed artifact checksums so downstream users only accept logged signatures, leaving **key holders** responsible for checking the log to verify that no unexpected signatures were ever produced with their Ed25519 signing key. + +**From leaf to canonical key & lookup**: Every Sigsum tree leaf is a fixed 128-byte binary record containing a 32-byte `checksum`, a 64-byte Ed25519 `signature`, and a 32-byte `key_hash` (`SHA-256` of the signer's Ed25519 public key). The mapping function hex-encodes `key_hash` into a 64-character lowercase hexadecimal string so it forms a printable ASCII canonical key (`1` key per leaf). Key holders query `GET /keys?key=<64-hex-key-hash>` and `/values` to verify the complete set of log entries attributed to their key. + +--- + +## 4. Prefix Lookups & Why Keys Remain Un-Hashed + +Storing un-hashed canonical keys directly in the MPT (where `0x00`-terminated bit ordering matches unsigned lexicographical byte order) means every byte prefix `` forms a contiguous subtree verifiable with a single two-sided prefix proof (`GET /keys?prefix=`; see [`tlog-vindex`](./tlog-vindex.md#verifying-a-keys-response)): + +1. **One Primitive Across Ecosystems**: The same subtree proof supports domain hierarchy monitoring (`prefix=com.example.`), Go module namespace discovery (`prefix=github.com%2Forg%2F`), and CI workload URI prefix auditing (`prefix=https%3A%2F%2Fgithub.com%2Forg%2F`). +2. **Why Query-Time Prefix Proofs Beat Map-Time Parent Expansion**: An alternative design for DNS hierarchies hashes MPT keys (`SHA-256(key)`) and expands every certificate domain into all registrable parent suffixes at map time (emitting `mail.sub.example.com`, `sub.example.com`, and `example.com` separately). Keeping un-hashed reversed-label keys and resolving prefixes at query time avoids three structural problems: + - _Key & Value Log Amplification_: Deep subdomains multiply the number of keys and value log leaves committed per certificate across every ancestor label. + - _Freezing the Mutable Public Suffix List (PSL) into Cryptographic State_: Preventing map-time expansion into bare public suffixes (`com`, `co.uk`) requires baking the ICANN Public Suffix List into the deterministic mapping function. Because the mapping function is fixed for the lifetime of an output log, every PSL amendment makes historical commitments stale unless the operator rebuilds a new index from `from`. With un-hashed reversed-label keys, registrable-domain (eTLD+1) checks live purely in a stateless HTTP prefix policy outside `mpt-root`. + - _Preserving Subdomain & Wildcard Granularity_: Collapsing child certificates into a single hashed parent key (`SHA-256("example.com")`) hides which subdomains or wildcards (`*.example.com`) exist unless monitors download every certificate ever issued under the parent domain, while hashing MPT keys eliminates prefix queries for module paths and workload URIs. +3. **Bounding Prefix Queries**: Because a prefix lookup returns all matched `(key, value_root)` pairs in one response so the client can reconstruct the prefix subtree hash, servers guard against overly broad scans using application-specific prefix policies (such as rejecting bare public-suffix prefixes like `com.` or `uk.co.`) and matched-key limits that prompt clients to query narrower prefixes. + +--- + +## 5. Operational Considerations + +- **Multi-Subject Fanout**: Unlike single-key logs (Go SumDB, Sigsum), a single CT or MTC certificate can carry dozens or hundreds of unique SAN `dNSName` identifiers, updating `N` distinct keys and value logs per input log entry. +- **Mapping Immutability & Re-Indexing**: Because `mpt-root` at any `input-size` is defined by replaying `[from, input_size)` through a fixed mapping function, changing how an ecosystem normalizes keys or selects indexable fields requires standing up a new VIndex instance (with a new output log) from `from = 0` or a chosen suffix start index `from`. +- **Hot Keys & Backward Pagination**: Open-admission logs can accumulate millions of entries under popular keys (such as shared apex domains or busy automated CI identities). Representing each key's occurrences as an append-only RFC 6962 value log and paginating backward (`GET /values?key=&before=`) lets monitors verify the newest page directly against `value_root` (via `preceding` compact range hashes) and stop as soon as they reach their previously inspected `entry_idx` watermark. diff --git a/vindex/docs/ARCHITECTURE.md b/vindex/docs/ARCHITECTURE.md new file mode 100644 index 0000000..27b4a93 --- /dev/null +++ b/vindex/docs/ARCHITECTURE.md @@ -0,0 +1,176 @@ +# Verifiable Index (VIndex) Architecture + +This document defines the high-level system architecture, checkpoint promotion invariants, crash consistency model, auditing boundaries, and architectural trade-offs for **VIndex**. + +For the problem statement and ecosystem use cases, see the [VIndex Overview](../README.md) and [Verifiable Index Applications](./APPLICATIONS.md). For the normative wire formats and cryptographic verification rules, see the [C2SP `tlog-vindex` Specification](./tlog-vindex.md). + +--- + +## 1. Architectural Overview + +### 1.1 The Map Sandwich + +VIndex operates as a **Map Sandwich** bounded between two append-only transparency logs: + +- **Input Log** (_source of truth_): The existing append-only transparency log whose entries are being indexed. +- **Indexer & Serving Node**: A single-host process that ingests verified Input Log entries, extracts canonical keys via a deterministic mapping function, maintains per-key value logs and an authenticated Merkle Patricia Trie (MPT), and serves HTTP lookup proofs. +- **Witnessed Output Log** (_state anchor_): A [tiled transparency log](https://c2sp.org/tlog-tiles) recording `c2sp.org/tlog-vindex-state@v1` index state commitments (`input-origin`, `from`, `input-size`, `input-root`, and `mpt-root`), cosigned by independent witnesses. + +### 1.2 Pipeline Stages & System Roles + +Data flows sequentially through five stages governed by a central coordinator, verified externally by lookup clients and independent auditors: + +| Stage / Role | Responsibility | +| :--------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **1. Ingest & Bundle Cache** | Polls Input Log checkpoints, verifies checkpoint policies and Merkle consistency, downloads and authenticates entry bundles into a local disk cache, and prunes cached bundles once downstream storage and MPT state are durable. | +| **2. Sandboxed Mapping** | Executes the index's deterministic mapping function inside a zero-I/O WebAssembly sandbox pool, validating emitted [canonical keys](./tlog-vindex.md#canonical-keys) and resequencing mapped batches into strict Input Log entry order. | +| **3. Value-Log Storage** | Partitions each key's matching Input Log entry indices into append-only chunks in an embedded LSM key-value store, precomputing RFC 6962 compact range hashes and syncing updates to disk at target checkpoint barriers. | +| **4. State Commitment** | Predicts the candidate `mpt_root` over un-hashed canonical keys under a shared read lock, appends the state commitment entry to the Output Log, collects witness cosignatures, and atomically swaps the active serving snapshot. | +| **5. Read Serving** | Serves `/keys` (exact key and bounded prefix proofs from the in-memory MPT) and `/values` (lock-free backward value-log pagination from the KV store), strictly isolated to witnessed snapshots. | +| **Coordinator** | Freezes target Input Log checkpoints, drives stages 1–4 in monotonic order, governs safe cache-pruning watermarks, and runs Zero-WAL startup recovery. | +| **Lookup Client** | Queries `/keys` and `/values` and verifies Output Log inclusion proofs, MPT inclusion/non-inclusion/prefix proofs, and value-log compact range roots offline against an Output Log checkpoint verification policy. | +| **Independent Auditor** | Tails the Output Log from entry `0`, verifies Input Log and Output Log checkpoints, independently replays Input Log entries from `from` through the mapping function, and halts and alerts if any published `mpt-root` diverges. | + +--- + +## 2. Checkpoint Promotion & Concurrency + +### 2.1 Checkpoint Promotion & Monotonic Ratcheting + +Earlier pipeline stages ingest and index new Input Log entries asynchronously while the HTTP serving path continues serving lookups against the most recently witnessed state. Progress across stages is governed by four checkpoints: + +| Checkpoint | Log Scope | Role | Advancement | Persistence | +| :--------------- | :---------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------------- | +| **`Target_CP`** | Input Log (`Target_CP.Size`) | Frozen upper bound (`>= from`) for the current sync cycle; prevents moving-goalpost starvation. | Upstream poll, Input Log checkpoint policy check, & Merkle consistency proof | Durable (KV metadata) | +| **`KV_CP`** | Input Log (`KV_CP.Size`) | Marks that all value-log chunks covering `[from, KV_CP.Size)` are durably persisted on disk. | Atomic watermark write and `fsync` once _all_ value-log entries for `Target_CP` are written (or at coarse periodic barriers in Genesis Backfill) | Durable (KV store `fsync`) | +| **`Output_CP`** | Output Log (`Output_CP.Size` entries, committing `Output_CP.InputSize`) | Publicly commits `(input-origin, from, input-size, input-root, mpt-root)`, while retaining the signed Input Log checkpoint out-of-band. | Append `c2sp.org/tlog-vindex-state@v1` entry & collect witness cosignatures | Durable (Output Log storage) | +| **`Serving_CP`** | Output Log Entry (`Serving_CP.InputSize`) | Snapshot exposed to HTTP lookup clients. | MPT commit & pointer swap under brief writer lock | Volatile (in-memory atomic pointer) | + +Note the two tree sizes on `Output_CP`: `Output_CP.Size` (`output_size`) is the number of state commitment entries in the Output Log itself, whereas `Output_CP.InputSize` (`input_size`) is the Input Log tree size committed inside the latest Output Log entry. + +All four checkpoints obey a strict monotonic invariant: + +```text +Target_CP.Size >= KV_CP.Size >= Output_CP.InputSize >= Serving_CP.InputSize +``` + +- `Target_CP.Size >= KV_CP.Size`: Intermediate batches write value-log chunk updates into the KV store without advancing `KV_CP` or forcing an `fsync`; in normal serving, `KV_CP` holds at the prior checkpoint until _all_ entries up to `Target_CP.Size` have been written, then ratchets atomically to `Target_CP.Size` (during Genesis Backfill, [§4.3](#43-genesis-backfill-unserved-bootstrap), `KV_CP` also advances at coarse intermediate barriers). +- `KV_CP.Size >= Output_CP.InputSize`: Value-log storage is synced to disk _before_ an Output Log commitment is published. +- `Output_CP.InputSize >= Serving_CP.InputSize`: Output Log publication and external witness cosigning complete before the serving pointer advances. + +### 2.2 Pipeline Concurrency & Backpressure + +VIndex balances three competing pressures on a single host: + +1. **Asymmetric Compute Allocation**: Entry-bundle mapping is embarrassingly parallel across worker sandboxes, whereas downstream KV batching and LSM memtable commits have a serial tail. Scaling both stages equally across all cores causes thread preemption and starves read serving and background compactions, so mapping scales wide while value-log indexing uses bounded concurrency. +2. **Bounded Inter-Stage Buffers**: Bounded channels absorb transient flush jitter between parallel mapping and sequential storage commits without letting queued batches accumulate into multi-gigabyte Go heap graphs. +3. **Self-Regulating Backpressure**: When disk flushes or compactions slow downstream commits, bounded channels fill and park upstream mapping workers, yielding CPU cores to storage and read serving until the commit queue drains. + +--- + +## 3. Pipeline Mechanics & Invariants + +### 3.1 Stage 1: Ingest & Bundle Cache + +The ingestion layer polls the Input Log for signed checkpoints, verifies them against the configured Input Log checkpoint policy and prior checkpoints, and downloads authenticated entry bundles into a local disk cache (supporting [C2SP `tlog-tiles`](https://c2sp.org/tlog-tiles), [C2SP `static-ct-api`](https://c2sp.org/static-ct-api), and user-defined log adapters). + +- **Target Checkpoint Verification**: Every polled Input Log checkpoint must satisfy the Input Log checkpoint verification policy, advance monotonically (`>= max(prev_Target_CP.Size, from)`), and pass Merkle consistency verification before `Target_CP` advances. Operators retain the signed Input Log checkpoint corresponding to each published Output Log entry out-of-band so Input Log equivocation can be proven if a conflict ever arises. +- **Recovery-Safe Cache Retention**: Verified entry bundles remain in the local disk cache until _both_ the KV store (`KV_CP.Size`) and the on-disk MPT (`MPT_Durable_Size`) have durably synced past them. An asynchronous reaper can be configured to prune bundles strictly below that safe watermark, bounding disk usage while guaranteeing local crash recovery without network fetches. + +### 3.2 Stage 2: Sandboxed Mapping + +Ecosystem authors express mapping logic as a deterministic function from raw Input Log entry bytes to canonical keys, compiled to WebAssembly and executed by a pool of zero-I/O sandboxes: + +- **Bundled Execution**: Rather than crossing the host-guest boundary once per entry, workers pass entire entry bundles in shared linear memory per invocation, amortizing boundary overhead while streaming emitted keys directly into pre-allocated slabs. +- **Deterministic Halt on Fault or Invalid Key**: Sandboxes run with no filesystem, network, or clock nondeterminism. Any guest trap, panic, or emitted key outside [`tlog-vindex` canonical key bounds](./tlog-vindex.md#canonical-keys) halts indexing immediately rather than silently dropping keys and diverging from auditors. +- **Chronological Resequencing**: Parallel workers finish bundles out of order; an in-memory resequencer restores strict ascending `entry_idx` order before batches reach storage. + +### 3.3 Stage 3: Value-Log Storage & Hybrid Keying + +VIndex separates trie structure from per-key occurrence lists using a **hybrid keying model**: + +- The **MPT** indexes **un-hashed variable-length canonical keys** directly so keys sharing a string prefix form a contiguous subtrie for [two-sided prefix proofs](./tlog-vindex.md#verifying-a-keys-response). +- The **KV store** indexes per-key value-log chunks by fixed-size **`SHA-256(key)` digests** computed on the host, keeping LSM keys uniform and decoupled from variable canonical key lengths. + +Within the KV store, each key's matching `entry_idx` sequence is partitioned into fixed-window chunks ordered newest-first under `SHA-256(key)`. Each chunk stores the RFC 6962 compact range hashes of all older occurrences preceding that chunk alongside compact relative offsets within the window: + +- **Synchronous Persistence Barrier**: Intermediate batches write chunk updates without advancing `KV_CP`; once all entries up to `Target_CP.Size` have been written, all modified chunks and `KV_CP = Target_CP` are synced to disk (`fsync`) together before Stage 4 publishes a new Output Log commitment (`KV_CP.Size >= Output_CP.InputSize`). +- **Snapshot-Bounded Reads**: Point-in-time `value_root` reconstruction and `/values` queries ignore any entries at or above the requested `before` / `Serving_CP.InputSize` bound, isolating readers and crash recovery from uncommitted batches ahead of the serving snapshot. + +### 3.4 Stage 4: State Commitment + +Once `KV_CP` is durable at `Target_CP.Size`, the commitment layer anchors the new `mpt_root` in the Output Log through a three-step handoff: + +1. **Non-Blocking Root Prediction**: Under a shared read lock on the MPT, the commitment layer predicts the candidate `mpt_root` from the sorted `(key, value_root)` changes without mutating the live trie or blocking concurrent `/keys` queries. +2. **Output Log Append & Witnessing**: The 6-line `c2sp.org/tlog-vindex-state@v1` entry (`input-origin`, `from`, `input-size`, `input-root`, and predicted `mpt-root`) is appended to the Output Log and cosigned by witnesses according to the Output Log checkpoint policy. +3. **Atomic State Ratchet & Background Persistence**: Under a brief exclusive tree lock, the commitment layer applies the `(key, value_root)` mutations to the in-memory MPT, asserts that the resulting root equals the predicted `mpt-root` (halting fatally on any mismatch), and swaps `Serving_CP`. Flushing updated MPT patch frames to disk (`fsync`) runs outside the exclusive tree lock so read serving never blocks on disk I/O. + +### 3.5 Stage 5: Read Serving + +The HTTP server exposes `/keys` and `/values` (and, when co-hosted, Output Log `/checkpoint` and `/tile/...` paths; non-protocol paths such as an optional human-facing browser UI remain outside the protocol): + +- **Key Discovery (`GET /keys`)**: Acquires a short shared read lock on the in-memory MPT, generates an exact key proof or bounded prefix subtree proof against `Serving_CP`, and pairs it with the `c2sp.org/tlog-proof@v1` inclusion proof to the witnessed Output Log checkpoint. Prefix queries are guarded by optional application-specific prefix policies (e.g., rejecting bare public suffixes) and matched-key limits. +- **Lock-Free Value Pagination (`GET /values`)**: Hashes `key` with `SHA-256` and reads newest-first chunks from the KV store with **zero MPT lock acquisition**, bounded by `before <= Serving_CP.InputSize`. Clients verify the newest page against `value_root` from `/keys` and inductively fold the returned compact range hashes to verify older pages backward without consistency proofs. + +--- + +## 4. Crash Consistency & Lifecycle + +### 4.1 Universal Crash Invariant + +Because KV storage syncs to disk before Output Log publication (`KV_CP.Size >= Output_CP.InputSize`), recovery never encounters a witnessed Output Log commitment whose value-log chunks are missing from local storage. If a crash occurs after `KV_CP` syncs but before Output Log publication (`KV_CP.Size > Output_CP.InputSize`), snapshot-bounded reads reconstruct `value_root` digests strictly at `Output_CP.InputSize` while the next sync cycle replays the cached window `[Output_CP.InputSize, KV_CP.Size)` to publish its state commitment without duplicating persisted KV entries. + +### 4.2 Zero-WAL Startup Recovery + +While the MPT opens and replays any uncompacted on-disk patch frames, lookup endpoints return `503 Service Unavailable`. Once open, the coordinator sets `recoveryTarget` to `Output_CP.InputSize` (or `KV_CP.Size` if crashing during initial bootstrap when `Output_CP.Size == 0`) and runs three phases: + +1. **Instant Warm Start**: If `Output_CP.Size > 0` and the persisted MPT snapshot is exact at `Output_CP.InputSize` with matching `mpt-root`, `Serving_CP` activates immediately. +2. **Local Bundle Replay**: If the on-disk MPT lags `recoveryTarget` or is inexact after an unclean crash, the coordinator streams `[max(MPT_Durable_Size, from), recoveryTarget)` from the **local bundle cache** (retained above the safe pruning watermark, requiring zero network egress), runs mapping to identify modified canonical keys, reconstructs their `value_root` digests at `recoveryTarget` from the KV store with zero KV writes, updates and syncs the MPT, verifies `mpt-root` against `Output_CP`, and opens read serving. +3. **Forward Catch-Up**: Resumes normal batch ingestion from `max(Serving_CP.InputSize, from)` toward `Target_CP.Size`. + +### 4.3 Genesis Backfill (Unserved Bootstrap) + +When bootstrapping a new index from `from` (`Output_CP.Size == 0`) against a large existing Input Log, accumulating all modified keys in RAM for a single commit would exhaust memory, while publishing and witnessing an Output Log entry on every intermediate batch would incur thousands of unnecessary witness round-trips before the index is ready to serve. + +While `Output_CP.Size == 0`, lookup endpoints remain `503 Service Unavailable` and the coordinator suppresses intermediate Output Log appends. Batches write chunks to the KV store and coalesce modified `(key, value_root)` pairs in bounded memory, periodically executing a synchronous `KV Sync -> MPT Commit & Sync` coarse checkpoint (maintaining `KV_CP.Size >= MPT_Durable_Size` and advancing the cache-pruning watermark). Once `Target_CP.Size` is reached, the coordinator publishes and witnesses Output Log `Entry 0`, activates `Serving_CP`, and permanently transitions to normal serving (`Output_CP.Size > 0`). + +--- + +## 5. Verification & Auditing Boundaries + +VIndex separates lightweight client query checks from full-log construction verification: + +1. **Lookup Client Verification**: Authenticates individual `/keys` and `/values` responses offline against ``, the client's Output Log checkpoint verification policy, ``, and the caller's required `from` coverage horizon, proving soundness and completeness relative to the witnessed `mpt-root` without trusting the HTTP server. +2. **Full-Log Independent Auditor**: Continuously tails the Output Log from entry `0`, verifies Output Log and Input Log checkpoint policies and Merkle consistency, downloads and authenticates Input Log entries starting from `from` (`[from, input_size)`), independently executes the mapping function, and verifies every published `mpt-root`. On any `mpt-root` mismatch or protocol violation, the auditor halts and alerts without advancing past its last verified commitment. + +--- + +## 6. Operational Characteristics + +- **Memory Residency & Storage Layout**: The MPT (`filippo.io/torchwood/mpt`) holds compact inner trie nodes in off-heap anonymous virtual memory (avoiding Go garbage-collector scan overhead) backed by alternating checksummed snapshot/patch-frame files and an on-disk leaf file, while per-key occurrence history lives on disk in the LSM KV store. Host RAM scales primarily with the number of distinct canonical keys in the MPT rather than total Input Log entries. +- **Disk I/O Isolation**: On high-throughput deployments, placing MPT snapshot/patch files and the LSM KV store (plus bundle cache) on separate NVMe volumes isolates sequential trie compaction writes from LSM memtable flushes and compactions. +- **Authoritative Output Log vs. Derived Index State**: The Output Log (and its signing key) is authoritative for witness continuity: because historical Output Log entries record runtime sync boundaries (`input-size`), losing the Output Log breaks Merkle consistency with previously witnessed checkpoints and requires standing up a new Output Log origin from scratch (so production deployments back the Output Log with durable storage). By contrast, the local KV store, MPT files, and bundle cache are purely derived state. Provided the Output Log survives, lost local index state can be rebuilt by re-indexing Input Log entries from `from` up to `Output_CP.InputSize`, verifying that the reconstructed `mpt_root` matches `Output_CP`, and resuming append-only operation. + +--- + +## 7. Architectural Alternatives Considered + +### 7.1 Dynamic Runtime Backfill Mode Switching + +- **Proposed**: Switching live serving nodes dynamically into an unserved bulk "backfill mode" whenever ingestion lag spiked. +- **Why Rejected**: Dropping HTTP read availability during catch-up starved lookup clients and complicated checkpoint and witness handoffs. Initial bootstrap uses Genesis Backfill strictly while `Output_CP.Size == 0`; once serving begins, nodes stay permanently in normal serving mode. + +### 7.2 Storage Write-Ahead Log Staging + +- **Proposed**: Writing mapped `(key, entry_idx)` pairs into a staging log inside the KV store before a background compactor folded them into value-log chunks. +- **Why Rejected**: Doubled disk write amplification, triggered heavy LSM compaction churn, and caused severe tail read-latency spikes compared to computing compact range updates in memory and writing inverted chunks directly. + +### 7.3 Sparse Merkle Trees & Verkle Trees + +- **Proposed**: Storing key-to-`value_root` state in a 256-level Sparse Merkle Tree (SMT) or polynomial-commitment Verkle tree. +- **Why Rejected**: Full-depth 256-bit SMTs incur high node fanout and hashing overhead (and hash keys, losing native byte-prefix subtrees), while Verkle polynomial commitments are CPU-prohibitive at high ingestion rates. Path-compressed binary MPTs keep expected proof depth proportional to actual key branching while preserving un-hashed prefix subtrees. + +### 7.4 Distributed Key-Value Databases + +- **Proposed**: Storing value-log chunks in an external distributed database (such as Spanner, Bigtable, or Cassandra). +- **Why Rejected**: The in-memory MPT already anchors each indexer to a single host; adding network RPC hops to storage degrades bulk ingestion throughput without providing multi-node MPT consensus. High availability is achieved by running independent VIndex instances over the shared Input Log. diff --git a/vindex/docs/tlog-vindex.md b/vindex/docs/tlog-vindex.md index 5387a39..07425d3 100644 --- a/vindex/docs/tlog-vindex.md +++ b/vindex/docs/tlog-vindex.md @@ -38,21 +38,21 @@ A verifiable index allows clients to look up which entries in an existing append The index operator applies a deterministic _mapping function_ to each entry in the input log to extract zero or more _canonical keys_. For each key, the index maintains an [RFC 6962][] Merkle tree (the key's _value log_) over the ascending sequence of input log _entry indices_ at which that key appeared. The 32-byte roots of all non-empty value logs (each a _value log root_, `value_root`) are stored in a path-compressed binary Merkle Patricia Trie (_MPT_). -Rather than signing standalone map checkpoints, the index operator publishes each _index state commitment_ (`c2sp.org/tlog-vindex-state@v1`)—binding the MPT's 32-byte root hash (`mpt-root`), a commitment to the mapping function (`map-fn`), the indexed _start index_ (`from`), and a signed input log [checkpoint][]—as an entry in a [tiled transparency log][tlog-tiles] (the _output log_) witnessed according to [tlog-witness][]. Lookup clients verify query responses offline against witnessed output log checkpoints, while monitors and auditors can deterministically replay the input log to verify every state commitment in the output log. +Rather than signing standalone map checkpoints, the index operator publishes each _index state commitment_ (`c2sp.org/tlog-vindex-state@v1`). This binds the input log origin (`input-origin`), the indexed entry range `[from, input_size)`, the input log root hash (`input-root`), and the resulting 32-byte MPT root hash (`mpt-root`) as an entry in a [tiled transparency log][tlog-tiles] (the _output log_) witnessed according to [tlog-witness][]. Lookup clients verify query responses offline against witnessed output log checkpoints, while monitors and auditors can deterministically replay the input log to verify every state commitment in the output log. ## Parameters A verifiable index is defined by: -- an _input log_, parameterized by its [checkpoint][] origin and [tlog-policy][], along with an entry retrieval mechanism that defines how clients and auditors fetch and authenticate input log entries against the input log checkpoint root hash (such as [tlog-tiles][] or [static-ct-api][]); +- an _input log_, parameterized by its [checkpoint][] origin (``). Auditors and clients that fetch entries from the input log additionally require the input log's checkpoint verification policy and entry retrieval mechanism (such as [tlog-tiles][] or [static-ct-api][]); - a _start index_ (`from`), the first input log entry index covered by the verifiable index (`0` when indexing from the start of the input log, or a positive integer when indexing a suffix of an existing input log); -- an _output log_, which is a [tiled transparency log][tlog-tiles] parameterized by its URL prefix (``) and its [tlog-policy][]; +- an _output log_, which is a [tiled transparency log][tlog-tiles] parameterized by its [checkpoint][] origin (``), its checkpoint verification policy, and its URL prefix (``); - an _index URL_ (``), the base URL serving the `/keys` and `/values` HTTP lookup endpoints defined in this document; and -- a deterministic _mapping function_ that maps a raw input log entry to a set of canonical keys, identified by a 32-byte SHA-256 commitment digest (`map-fn`) distributed out-of-band alongside the input and output log policies. +- a deterministic _mapping function_ that maps a raw input log entry to a set of canonical keys. An operator MAY use the same base URL for both `` and ``. @@ -113,7 +113,7 @@ The MPT over a set of `(key, value_root)` pairs is the unique path-compressed bi - If the set contains a single pair `(key, value_root)`, the trie is a single MPT leaf node. - If the set contains two or more keys, let `b` be the minimum `split_bit(x, y)` across all pairs of distinct keys in the set. The root of the subtree is an inner node with split bit `b`, whose left child is the MPT over all keys in the set with `bit(key, b) = 0` and whose right child is the MPT over all keys in the set with `bit(key, b) = 1`. -Because the topology is uniquely determined by the set of present keys and their value log roots, the root hash of the MPT (`mpt_root`) is independent of the order in which keys are inserted or updated. Moreover, when keys are sorted in lexicographical order, the minimum `split_bit(x, y)` in any key range is the minimum `split_bit` between adjacent keys, allowing the root hash of the MPT (or of any prefix subtree) to be computed in `O(n)` time in a single pass. +Because the topology is uniquely determined by the set of present keys and their value log roots, the root hash of the MPT (`mpt_root`) is independent of the order in which keys are inserted or updated. Moreover, when keys are sorted in lexicographical order, the minimum `split_bit(x, y)` in any key range is the minimum `split_bit` between adjacent keys, allowing the root hash of the MPT (or of any prefix subtree) to be computed in a single pass over the sorted keys. #### Node hashing @@ -141,45 +141,42 @@ mpt_inner_hash(b, left, right) = SHA-256(0x01 || uvarint(b) || left || right) ### Output log entries -Each entry in the output log is an _index state commitment_ over an input log entry range `[from, input_size)`: +Each entry in the output log is an _index state commitment_ over an input log entry range `[from, input_size)` consisting of exactly six newline-terminated lines: ``` c2sp.org/tlog-vindex-state@v1 -mpt-root -map-fn +input-origin from - - +input-size +input-root +mpt-root ``` The first line MUST be the string `c2sp.org/tlog-vindex-state@v1`. -The second line MUST start with the string `mpt-root` followed by a space (U+0020) and the base64 encoding of the 32-byte MPT root hash (`mpt_root`). +The second line MUST start with `input-origin` followed by a single space (U+0020) and the input log's [checkpoint][] origin string (``, a non-empty printable ASCII string in `0x20..0x7E` with no leading or trailing spaces; `` itself MAY contain internal spaces). The `input-origin` value MUST remain constant across all entries in a given output log. + +The third line MUST start with `from` followed by a single space (U+0020) and the starting input log entry index (inclusive), encoded as an ASCII decimal. The start index MUST be less than or equal to `input-size` (`0 <= from <= input_size`) and MUST remain constant across all entries in a given output log (typically `0` when indexing from the start of the input log). -The third line MUST start with the string `map-fn` followed by a space (U+0020) and the base64 encoding of the 32-byte SHA-256 commitment to the mapping function that produced `mpt_root`. The mapping function commitment MUST remain constant across all entries in a given output log. Lookup clients that do not pin a specific mapping function digest MUST validate the syntactic format of this line and MAY ignore its value. +The fourth line MUST start with `input-size` followed by a single space (U+0020) and the tree size (`input_size`) of the indexed input log [checkpoint][], encoded as an ASCII decimal. -> [!NOTE] -> **TODO (pre-1.0)**: Decide whether the `map-fn` preimage is the raw compiled `.wasm` module bytecode directly, or a canonical provenance/build manifest (`{source repo @ tag, build toolchain, output wasm hash}`) committing to both the source definition and compiled artifact. +The fifth line MUST start with `input-root` followed by a single space (U+0020) and the base64 encoding of the 32-byte Merkle Tree Hash (`input_root`) of the indexed input log [checkpoint][] at `input_size`. Any extension lines or signatures present on the input log checkpoint are omitted from `state_entry`. -The fourth line MUST start with the string `from` followed by a space (U+0020) and the starting input log entry index (inclusive), encoded as an ASCII decimal. The start index MUST be less than or equal to the tree size (`input_size`) of the embedded input log checkpoint and MUST remain constant across all entries in a given output log (typically `0` when indexing from the start of the input log). +The sixth line MUST start with `mpt-root` followed by a single space (U+0020) and the base64 encoding of the 32-byte MPT root hash (`mpt_root`) derived from the input log entries in `[from, input_size)`. -After the `from` line and an empty line, the signed input log [checkpoint][] is included verbatim, ending with a single trailing newline (U+000A). +Decoders MUST reject a `c2sp.org/tlog-vindex-state@v1` entry if it contains fewer or more than six lines, or any trailing data after the sixth newline (U+000A). -The output log entry (`state_entry`) is hashed into the output log's Merkle tree according to [RFC 6962, Section 2.1][] (`SHA-256(0x00 || state_entry)`). +The output log entry (`state_entry`) is hashed into the output log's Merkle tree according to [RFC 6962, Section 2.1][] (`SHA-256(0x00 || state_entry)`). Index operators SHOULD retain the signed input log [checkpoint][] corresponding to each published output log entry so they can demonstrate input log equivocation out-of-band if an auditor observes a conflicting input log root at `input_size`. Example output log entry: ``` c2sp.org/tlog-vindex-state@v1 -mpt-root UFxpN2pP0tW3wQ5xlaluI6r0iIjoc08s2nFK+nnL1uE= -map-fn kElQ2l5Fh62m98WyOGdrXVXnWRtByCf97CCrlK4XCQk= +input-origin example.com/inputlog from 0 - -example.com/inputlog -98765 -lxDZaupAJAxWNRnEK4esZKdB8d/Lq+JZGw+GBWCUK58= - -— example.com/inputlog Az3grlgtzPICa5OS8npVmf1Myq/5IZniMp+ZJurmRDeOoRDe4URYN7u5/Zhcyv2q1gGzGku9nTo+zyWE+xeMcTOAYQ8= +input-size 98765 +input-root lxDZaupAJAxWNRnEK4esZKdB8d/Lq+JZGw+GBWCUK58= +mpt-root UFxpN2pP0tW3wQ5xlaluI6r0iIjoc08s2nFK+nnL1uE= ``` ## HTTP Interface @@ -188,7 +185,7 @@ A verifiable index exposes two HTTP GET lookup endpoints under ``, al The endpoints defined in this section MUST NOT serve redirect (`3xx`) responses. If a server has not yet published its first output log checkpoint, or receives a `values` request with `before` greater than its currently indexed input log tree size, it MUST respond with a "503 Service Unavailable" HTTP status code. -`/keys` is mutable (returning the latest output log state commitment), so its HTTP headers SHOULD prevent caching beyond a few seconds. Once a server has indexed input log entries at least up to `before`, `/values?key=&before=[&limit=]` is deterministic for a fixed server page-boundary policy, so its caching headers MAY be long-lived. +`/keys` is mutable (returning the latest output log state commitment), so its HTTP headers SHOULD prevent caching beyond a few seconds. Once a server has indexed input log entries at least up to `before`, `/values?key=&before=` is deterministic for a fixed server page-boundary policy, so its caching headers MAY be long-lived. ### keys @@ -202,7 +199,7 @@ A request MUST specify either the `key` query parameter or the `prefix` query pa - `key` specifies an exact key lookup: the server proves whether `` exists in the MPT and, if present, returns its ` ` pair. - `prefix` specifies a key prefix lookup: the server proves and returns all `(key, value_root)` pairs in the MPT whose key has `` as a byte prefix. -The `` or `` parameter value MUST be percent-encoded per [RFC 3986][]. Clients MUST percent-encode any byte outside the RFC 3986 `unreserved` set (`ALPHA`, `DIGIT`, `-`, `.`, `_`, `~`—in particular query delimiters and `+` as `%2B`). The unescaped value MUST be a valid canonical key (`1` to `1024` bytes in `0x21..0x7E`, see [Canonical keys](#canonical-keys)). If neither or both of `key` and `prefix` are present, or if the unescaped value is not a valid canonical key, the server MUST respond with a "400 Bad Request" HTTP status code. +The `` or `` parameter value MUST be percent-encoded per [RFC 3986][]. Clients MUST percent-encode any byte outside the RFC 3986 `unreserved` set (`ALPHA`, `DIGIT`, `-`, `.`, `_`, `~`, in particular query delimiters and `+` as `%2B`). The unescaped value MUST be a valid canonical key (`1` to `1024` bytes in `0x21..0x7E`, see [Canonical keys](#canonical-keys)). If neither or both of `key` and `prefix` are present, or if the unescaped value is not a valid canonical key, the server MUST respond with a "400 Bad Request" HTTP status code. On a `prefix` query, a server MAY enforce an application-specific prefix policy (for example, rejecting bare TLD queries such as `com.` or `uk.co.` that do not contain a registrable domain). If `` violates the server's prefix policy, the server MUST respond with a "400 Bad Request" HTTP status code. A server SHOULD enforce a maximum matched key count on `prefix` queries; if the number of keys in the MPT starting with `` exceeds the server's limit, the server MUST abort the scan and respond with a "413 Content Too Large" HTTP status code. @@ -237,17 +234,17 @@ proof FQUrvVQc2/IXY3yZ4vmGm5Wdxo9B17djMlHbR6i5fTjP golang.org/x/crypto obw28IrVp+Sx1FhKBsu5p2Lv2JwyXAFseatVCHqtUsE= c2sp.org/tlog-proof@v1 -extra YzJzcC5vcmcvdGxvZy12aW5kZXgtc3RhdGVAdjEKbXB0LXJvb3QgVUZ4cE4ycFAwdFczd1E1eGxhbHVJNnIwaUlqb2MwOHMybkZLK25uTDF1RT0KbWFwLWZuIGtFbFEybDVGaDYybTk4V3lPR2RyWFZYbldSdEJ5Q2Y5N0NDcmxLNFhDUWs9CmZyb20gMAoKZXhhbXBsZS5jb20vaW5wdXRsb2cKOTg3NjUKbHhEWmF1cEFKQXhXTlJuRUs0ZXNaS2RCOGQvTHErSlpHdytHQldDVUs1OD0KCuKAlCBleGFtcGxlLmNvbS9pbnB1dGxvZyBBejNncmxndHpQSUNhNU9TOG5wVm1mMU15cS81SVpuaU1wK1pKdXJtUkRlT29SRGU0VVJZTjd1NS9aaGN5djJxMWdHekdrdTluVG8renlXRSt4ZU1jVE9BWVE4PQo= +extra YzJzcC5vcmcvdGxvZy12aW5kZXgtc3RhdGVAdjEKaW5wdXQtb3JpZ2luIGV4YW1wbGUuY29tL2lucHV0bG9nCmZyb20gMAppbnB1dC1zaXplIDk4NzY1CmlucHV0LXJvb3QgbHhEWmF1cEFKQXhXTlJuRUs0ZXNaS2RCOGQvTHErSlpHdytHQldDVUs1OD0KbXB0LXJvb3QgVUZ4cE4ycFAwdFczd1E1eGxhbHVJNnIwaUlqb2MwOHMybkZLK25uTDF1RT0K index 12 RZP8gEWXTBy4gCcH9XPPkVhodGgAsHm+Z5ob/4sbqfM= JiuhJcYDxEsAsEN7LCirEe1u+Gwt0JHruBBQo0BqdNs= example.com/outputlog 13 -pdC++7DEwrjyf+ZbXonKydZY09m6c7UCd0UVVmA2o0s= +jR10gZzRRHNJbpwE1qNYfgk7jAv2Ta+5Ud8SpKcWZ6w= -— example.com/outputlog opLqBQsREYCgu6xQkYQwJr9fo45a62DN9EdmHXnZdXNqlcVGlCum2Wks+49/V6267UEjw6QUXTS5Rovnzv++qbSzm9Q= -— witness.example/w1 CMp+6LWBU0anHGH5aNDTJkH/gj79sG+T6+iP2ThYN5krrDJbR1HDnucjL39QsZTSvVjyQLrdk3DXDqI5G2HgLatVs0pWh6Up69HVOw== +- example.com/outputlog opLqBQsREYCgu6xQkYQwJr9fo45a62DN9EdmHXnZdXNqlcVGlCum2Wks+49/V6267UEjw6QUXTS5Rovnzv++qbSzm9Q= +- witness.example/w1 CMp+6LWBU0anHGH5aNDTJkH/gj79sG+T6+iP2ThYN5krrDJbR1HDnucjL39QsZTSvVjyQLrdk3DXDqI5G2HgLatVs0pWh6Up69HVOw== ``` #### Binary MPT proofs @@ -284,15 +281,15 @@ where `b` (`0 <= b <= 8192`) is the inner node's split bit position and `sib_has Value log entries (input log entry indices) for a key are served at - GET /values?key=&before=[&limit=] + GET /values?key=&before= -`` is a required query parameter containing the canonical key (see [Canonical keys](#canonical-keys)), percent-encoded per [RFC 3986][]. Clients MUST percent-encode any byte outside the RFC 3986 `unreserved` set (`ALPHA`, `DIGIT`, `-`, `.`, `_`, `~`—in particular query delimiters and `+` as `%2B`). +`` is a required query parameter containing the canonical key (see [Canonical keys](#canonical-keys)), percent-encoded per [RFC 3986][]. Clients MUST percent-encode any byte outside the RFC 3986 `unreserved` set (`ALPHA`, `DIGIT`, `-`, `.`, `_`, `~`). -`before` is a required query parameter giving the exclusive upper bound on input log entry indices, encoded as a positive ASCII decimal integer (`1 <= before < 2^64`). On the initial page for a key discovered via `keys`, the client MUST set `before` to the input log checkpoint tree size (`input_size`) from the `c2sp.org/tlog-vindex-state@v1` payload of the `keys` response, pinning the query to the exact snapshot committed by `value_root`. On continuation pages, the client MUST set `before` to the first (smallest) input log entry index returned on the preceding page. +`before` is a required query parameter giving the exclusive upper bound on input log entry indices, encoded as a positive ASCII decimal integer (`1 <= before < 2^64`). On the initial page for a key discovered via `keys`, the client MUST set `before` to the input log tree size (`input-size`) from the `c2sp.org/tlog-vindex-state@v1` payload of the `keys` response, pinning the query to the exact snapshot committed by `value_root`. On continuation pages, the client MUST set `before` to the first (smallest) input log entry index returned on the preceding page. -`limit` is an optional query parameter giving an advisory maximum number of entry indices to return, encoded as a positive ASCII decimal integer (`1 <= limit < 2^64`). The server MAY return fewer or more entry indices (for example, to align page boundaries with precomputed storage chunks) provided at least one entry index is returned whenever matching occurrences strictly less than `before` exist. +The server MAY align page boundaries with precomputed storage chunks, provided at least one entry index is returned whenever matching occurrences in `[from, before)` exist. -If `` is missing or is not a valid canonical key, `before` is missing, or `before` or `limit` is not a positive ASCII decimal integer (`1 <= x < 2^64`), the server MUST respond with a "400 Bad Request" HTTP status code. Once the server has indexed input log entries at least up to `before`, if `` has zero occurrences strictly less than `before`, the server MUST respond with a "404 Not Found" HTTP status code. +If `` is missing or is not a valid canonical key, or if `before` is missing or is not a positive ASCII decimal integer (`1 <= before < 2^64`), the server MUST respond with a "400 Bad Request" HTTP status code. Once the server has indexed input log entries at least up to `before`, if `` has zero occurrences in `[from, before)`, the server MUST respond with a "404 Not Found" HTTP status code. ``` c2sp.org/tlog-vindex-values@v1 @@ -336,13 +333,11 @@ To authenticate the state commitment in a `keys` response: 2. Extract the base64-encoded `extra` field from the `c2sp.org/tlog-proof@v1` block, decode `state_entry`, and compute the output log leaf hash `SHA-256(0x00 || state_entry)`. -3. Verify the `c2sp.org/tlog-proof@v1` block against that leaf hash and the client's configured output log [tlog-policy][] according to [tlog-proof][]. Let `output_size` be the verified output log checkpoint tree size. Check that `output_size >= 1` and that the proven entry `index` equals `output_size - 1`. - -4. Parse `state_entry` as a `c2sp.org/tlog-vindex-state@v1` record (see [Output log entries](#output-log-entries)) to obtain `mpt_root`, the `map-fn` digest, `from`, and the embedded ``. If the client is configured with an expected `map-fn` commitment, check that the entry's `map-fn` matches; otherwise, validate its syntactic encoding and ignore its value. +3. Verify the `c2sp.org/tlog-proof@v1` block against that leaf hash, ``, and the client's output log checkpoint verification policy according to [tlog-proof][]. Let `output_size` be the verified output log checkpoint tree size. Check that `output_size >= 1` and that the proven entry `index` equals `output_size - 1`. -5. Verify the embedded `` against the client's configured input log [tlog-policy][]. Let `input_size` be the verified input log checkpoint tree size, and check that `0 <= from <= input_size`. +4. Parse `state_entry` as a `c2sp.org/tlog-vindex-state@v1` record (see [Output log entries](#output-log-entries)) to obtain `input_origin`, `from`, `input_size`, `input_root`, and `mpt_root`. Check that `input_origin` equals the client's configured `` and that `0 <= from <= input_size`. If `from = input_size`, check that `mpt_root` equals `empty_mpt_hash`. Because a lookup response only attests to input log entries in `[from, input_size)`, callers that require coverage back to a specific entry index MUST check that `from` does not exceed that bound. -6. A stateful client that caches previously observed checkpoints MUST reject the response if `output_size` or `input_size` is smaller than previously verified sizes, if a checkpoint at an equal tree size has a different root hash, or if `map-fn` or `from` differs from previously verified output log entries. + If the client subsequently fetches matched entries from the input log, it MUST authenticate those entries against `(input_size, input_root)` according to the input log's specification, and unless relying solely on the witnessed output log and auditors for input log checkpoint validity, verify that `(input_size, input_root)` is Merkle-consistent with a signed input log [checkpoint][] verified under its input log policy. ### Verifying an MPT path @@ -372,7 +367,7 @@ To verify an exact key lookup (`GET /keys?key=`, with `target = Given a canonical key `key`, an expected 32-byte value log root `value_root`, the start index `from`, and cursor `before` (initialized to `input_size` from the `keys` response): -1. Fetch `GET /values?key=&before=[&limit=]`. +1. Fetch `GET /values?key=&before=`. 2. Check that the first line is `c2sp.org/tlog-vindex-values@v1`, parse `preceding` from the second line, and decode the remaining lines before the empty line as 32-byte compact range hashes (checking that the number of hashes equals the number of `1` bits in the binary representation of `preceding`). @@ -388,18 +383,22 @@ Given a canonical key `key`, an expected 32-byte value log root `value_root`, th Because every state transition of a verifiable index is committed to the output log ([tlog-tiles][]), any third party can audit the index operator without trusting the `keys` or `values` HTTP endpoints: -1. Fetch and verify the output log checkpoint at `/checkpoint` against the output log [tlog-policy][], verifying Merkle consistency with previously audited output log checkpoints. Fetch new output log entry bundles from `/tile/entries/...` and authenticate them against the output log checkpoint root hash according to [tlog-tiles][]. +1. Fetch and verify the output log [checkpoint][] at `/checkpoint` against `` and the auditor's output log checkpoint verification policy, verifying Merkle consistency with previously audited output log checkpoints. Fetch new output log entry bundles from `/tile/entries/...` and authenticate them against the output log checkpoint root hash according to [tlog-tiles][]. -2. Initialize the auditor's local MPT to the empty trie and set `prev_size` to the index's configured `from` parameter. +2. Fetch and verify a signed input log [checkpoint][] against the auditor's input log verification policy to obtain an authenticated input log tree size `latest_input_size` and root hash `latest_input_root` (re-fetching as needed when auditing newer output log entries). Initialize the auditor's local MPT to the empty trie and set `prev_size` to the index's configured `from` parameter. 3. For each output log entry in sequential order: - - Parse the `c2sp.org/tlog-vindex-state@v1` payload to obtain `mpt_root`, the `map-fn` digest, `from`, and the embedded input log checkpoint. Check that `map-fn` and `from` match the index's configured parameters. - - Verify the input log checkpoint against the input log [tlog-policy][], let `input_size` be its tree size, check that `prev_size <= input_size`, and (after the first output log entry) verify that the input log checkpoint is Merkle-consistent with the previous entry's input log checkpoint. - - Fetch the input log entries in `[prev_size, input_size)` and authenticate them against the input log checkpoint root hash according to the input log's specification. For each input log entry at `entry_idx` in sequential order, execute the mapping function to obtain the deduplicated set of canonical keys, check that each emitted key is a valid canonical key (see [Canonical keys](#canonical-keys)), append `value_log_leaf_hash(entry_idx)` to the key's value log, and update `(key, value_root)` in the local MPT. + - Parse the `c2sp.org/tlog-vindex-state@v1` payload to obtain `input_origin`, `from`, `input_size`, `input_root`, and `mpt_root`. Check that `input_origin` matches ``, `from` matches the index's configured start index, and `prev_size <= input_size <= latest_input_size`. + - Verify that `(input_size, input_root)` is Merkle-consistent with `(latest_input_size, latest_input_root)` (checking `input_root = latest_input_root` when `input_size = latest_input_size`). + - Fetch the input log entries in `[prev_size, input_size)` and authenticate them against `input_root` (or `latest_input_root`) according to the input log's specification. For each input log entry at `entry_idx` in sequential order, execute the mapping function to obtain the deduplicated set of canonical keys, check that each emitted key is a valid canonical key (see [Canonical keys](#canonical-keys)), append `value_log_leaf_hash(entry_idx)` to the key's value log, and update `(key, value_root)` in the local MPT. - Check that the local MPT root hash equals `mpt_root`, and set `prev_size = input_size`. Any mismatch proves that the index operator published an invalid state commitment to the output log. ## Design Rationale +### Unsigned input log state in output log entries + +Output log entries commit to `input-origin`, `input-size`, and `input-root` directly rather than embedding a signed input log [checkpoint][]. Embedding input log signatures and checkpoint extension lines inside `state_entry` would freeze a single set of input log witness cosignatures at publish time (preventing clients with different witness policies from verifying the entry), couple `/keys` lookup clients to input log signature formats (such as [static-ct-api][] ECDSA notes), and allow optional checkpoint extension lines to cause two indexers at the same `input_size` to produce different `state_entry` bytes. Instead, auditors verify Merkle consistency of `(input_size, input_root)` against signed input log checkpoints, and index operators retain the signed input log checkpoints out-of-band to prove input log equivocation if a conflict ever arises. + ### Unpaginated prefix proofs Verifying a non-empty prefix proof requires reconstructing the Merkle hash of the matched MPT subtree from all `(key, value_root)` pairs under ``. Because the subtree hash cannot be verified from a partial subset of keys without introducing interior subtree proofs, `keys?prefix=` returns the complete matched key set in a single response. Servers bound response sizes by enforcing domain or prefix policies (`400 Bad Request`) and maximum matched-key limits (`413 Content Too Large`). @@ -410,7 +409,11 @@ When verifying a non-empty prefix proof, the MPT path verification step only che ### Backward pagination of value log entries -A single canonical key may appear millions of times in the input log, whereas most lookup clients only need the most recent occurrences. Representing each key's occurrences as an append-only [RFC 6962][] value log and paginating backward from `before` allows the server to summarize all `preceding` older occurrences using the compact range hashes of the `preceding` older leaves. The client verifies the newest page directly against `value_root` and, if it wishes to inspect older pages, folds those same compact range hashes to obtain the exact expected `value_root` for the next backward page—requiring no Merkle consistency proofs and no additional round-trips. +A single canonical key may appear millions of times in the input log, whereas most lookup clients only need the most recent occurrences. Representing each key's occurrences as an append-only [RFC 6962][] value log and paginating backward from `before` allows the server to summarize all `preceding` older occurrences using the compact range hashes of the `preceding` older leaves. The client verifies the newest page directly against `value_root` and, if it wishes to inspect older pages, folds those same compact range hashes to obtain the exact expected `value_root` for the next backward page, requiring no Merkle consistency proofs and no additional round-trips. + +## Design Log + +- **Mapping function commitment (`map-fn`) shelved pre-1.0**: An in-band commitment to the mapping function inside `c2sp.org/tlog-vindex-state@v1` was considered and removed for now to keep the specification minimal, due to open questions around both **(a) encoding** (whether to commit to raw compiled bytecode such as `.wasm`, a reproducible build/provenance manifest, or a language-agnostic prose profile identifier) and **(b) verification** (lookup clients cannot verify that an opaque digest matches expected mapping semantics, while independent clean-room auditors in other languages verify prose specifications rather than executing shared bytecode). An in-band commitment can be re-added prior to 1.0 following ecosystem discussion. ## Acknowledgements @@ -421,7 +424,6 @@ The MPT structure and binary proof encoding are based on the Merkle Patricia Tri [RFC 6962, Section 2.1]: https://www.rfc-editor.org/rfc/rfc6962.html#section-2.1 [checkpoint]: https://c2sp.org/tlog-checkpoint@v1.1.0 [static-ct-api]: https://c2sp.org/static-ct-api@v1.1.0 -[tlog-policy]: https://c2sp.org/tlog-policy [tlog-proof]: https://c2sp.org/tlog-proof [tlog-tiles]: https://c2sp.org/tlog-tiles@v1.0.0 [tlog-witness]: https://c2sp.org/tlog-witness@v1.1.0