diff --git a/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md new file mode 100644 index 000000000..33a20ee88 --- /dev/null +++ b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md @@ -0,0 +1,278 @@ +--- +status: accepted +date: 2026-10-08 +relates-to: ADR-0001; stack-encrypt ADR-0003, ADR-0007 +--- + +# One value model, three encodings in vitaminc, and EQL v4 separated by producer + +> **Amended 2026-10-11** by +> [`docs/plans/2026-10-11-order-and-equality-terms-layering.md`](../plans/2026-10-11-order-and-equality-terms-layering.md). +> Where the two disagree, the plan wins. In short: vitaminc sits at the +> bottom of the stack and the ORE schemes implement `vitaminc-ore`'s trait; +> equality and order terms take their own per-domain transforms instead of +> one shared canonical form; `vitaminc-prf` no longer depends on order +> encodings; and `orderable-bytes` is frozen. The plan lists every statement +> below that it replaces. + +Every value Stack Encrypt handles is encoded three times: as a **ciphertext** +(reversible and self-describing), as an **equality term** (a keyed hash that +must be unambiguous) and as an **order term** (bytes whose order is the +value's order). This ADR decides where each encoding lives, which kinds of +value exist, the canonical form each kind takes, and how EQL keeps columns +written by Stack Encrypt apart from columns written by cipherstash-client. + +## The problem + +In October 2026 Stack Encrypt could produce one of EQL's 51 types, `TextEq`. +The number, date, timestamp and boolean families were blocked on encoding, +not code: + +- **The three encodings disagreed about which types exist.** `i16` had an + equality domain in `vitaminc-prf` and an order encoding, but no ciphertext + tag, so it could not be a kind. Date, timestamp and decimal had order + encodings only. +- **The mapping from a kind to a Rust type lived in Stack Encrypt.** + `dynamic::term::Scalar` mirrored `FfiValue`'s variants one for one to + dispatch to the term crates, and `dynamic::Value` wrapped `FfiValue` only to + add `Clone`. Both restated vitaminc's vocabulary in a downstream crate. +- **Ordering was implemented twice.** `orderable-bytes` defines a canonical, + order-preserving encoding per type. cllw-ore uses it for chrono and decimal + but hand-rolls integers and floats, and differs on `-0.0`. Block ORE was not + derivable at all, so `TextOrdOre` and `TextSearchOre` were refused. +- **The existing writer's encodings are inconsistent.** cipherstash-client + hashes a timestamp's milliseconds but orders by its nanoseconds, and hashes + a decimal with its scale while ordering ignores it. Its block ORE text is + ASCII-only, lowercased, maps every digit to one symbol and truncates to six + blocks. cllw-ore's text decomposes to NFD and strips accents. +- **Two producers wrote the same EQL domains.** A Stack Encrypt query term + never matches a cipherstash-client term for the same value, and the + `eql_v3_*` domains could not tell them apart. A query from one against a + column written by the other returned no rows, silently. + +## Options considered + +**Where the value model lives.** + +1. **Move `vitaminc-aead-value` into Stack Encrypt.** It looks like FFI + plumbing. But it is the value model for vitaminc's own `Cipher` traits: + `aead-napi` and vitaminc's Go binding encrypt with `Aes256Cipher` through + it, without Stack Encrypt. Moving it makes vitaminc Rust-only, makes + vitaminc depend on Stack Encrypt (which depends on six vitaminc crates), + or forks the frozen tag table into two copies. +2. **Keep it in vitaminc, and move term derivation there too.** All three + encodings then live in one repository, and one exhaustive `match` on the + value type per layer makes "a kind exists in every layer or in none" a + compile error rather than a convention. Chosen. + +**How EQL separates the two producers.** The SQL for both is identical; only +the producer of the terms differs. + +1. **Name only.** Stack Encrypt payloads go in the `eql_v3_*` domains, + distinguished by the `stack-encrypt:1:` ciphertext prefix. Nothing in the + database stops a cross-producer comparison. +2. **A producer tag in every payload and term, checked by every operator.** + A runtime check on the hottest SQL paths, across about 24k lines of + hand-written SQL. +3. **One SQL source, emitted under two names.** The build writes the same + source out as `eql_v3` (cipherstash-client terms) and `eql_v4` (Stack + Encrypt terms). A column's domain names its producer, and Postgres refuses + to compare an `eql_v4_*` query term with an `eql_v3_*` column at plan time. + Chosen. + +**How v3 and v4 ship.** Bumping `main` to v4 and patching v3 from a branch +needs a maintenance release path for five lockstep artefacts that does not +exist, and the EQL publish script puts every stable release on `latest`, so a +3.x patch would move `latest` backwards. A second package doubles the +trusted-publishing and release surface. One package carrying both bundles +needs neither. Chosen. + +## Decision + +### Kinds and the ciphertext + +- vitaminc 0.6.0 adds the kinds `Int8`, `UInt8`, `Int16`, `UInt16`, `Int128`, + `UInt128`, `Date`, `Timestamp` and `Decimal`, each with a `ValueKind` name, + a `Value` variant and a ciphertext tag, in one breaking release. Rust `i128` + and `u128` get `Encrypt` and `Decrypt` impls. +- `FfiValue` is renamed `Value`, with `#[deprecated] pub type FfiValue = Value` + for one release. `Value` and `ValueKind` become `#[non_exhaustive]`, so later + kinds are additive. +- `Value` implements `Clone` as a deep copy that rebuilds every leaf into a + fresh `Protected`. A clone is under the same custody as its original. +- The sealed leaf tag table (`tags.rs`) stays frozen and contiguous. The + transport codec's framing tags move from `0x10`–`0x12` to `0xF0`–`0xF2`; + transport carries no compatibility commitment and every user of it updates + together. The new leaves: + + | Tag | Kind | Payload | + |---|---|---| + | `0x0C`–`0x11` | `Int8`, `UInt8`, `Int16`, `UInt16`, `Int128`, `UInt128` | 1, 1, 2, 2, 16, 16 bytes; two's complement for signed; little-endian | + | `0x12` | `Date` | `i32` days counted from 0001-01-01 (`num_days_from_ce`), little-endian | + | `0x13` | `Timestamp` | `i64` Unix seconds then `u32` nanoseconds, little-endian, UTC | + | `0x14` | `Decimal` | rust_decimal's 16-byte `serialize()`, which keeps the scale | + +- `transport` stays a module of `vitaminc-aead-value`. + +### Canonical forms + +The ciphertext keeps the value exactly as given: `1.50` decrypts as `1.50`, +and a timestamp keeps its nanoseconds. Equality and order terms are computed +from one canonical form per kind, and both layers use the same one, so +equality and ordering agree by construction. + +| Kind | Canonical form for terms | +|---|---| +| `Timestamp` | truncated to microseconds, Postgres's precision | +| `Decimal` | scale normalised (`1`, `1.0` and `1.00` are equal). NaN and ±Infinity are refused at encode time; rust_decimal cannot represent them | +| `Float32`, `Float64` | `-0.0` folded into `+0.0`; every NaN replaced by one positive quiet NaN, which sorts above +Infinity. This matches Postgres | +| text, equality | NFC | +| text, order | NFC, then NFD, combining marks removed, Unicode default case folding | + +Text normalisation is pinned. The Unicode version is part of the encoding's +domain label (for example `text-nfc/unicode-16/v1`), the +`unicode-normalization` crate is pinned to it, and strings containing +unassigned code points are refused. Order terms fold accent and case because +code-point order puts `é` after `z`; a fixed, pinned fold approximates the +first level of Unicode collation without depending on ICU, whose sort keys +change between versions. Equality is not folded: a folded order term only +produces ties, while a folded equality term produces false matches. A +case-insensitive equality is its own domain. + +Truncation and alphabet packing are settings of an EQL domain, named in its +label, never part of the shared encoding. + +### Which kinds get which terms + +- **Every scalar kind has a ciphertext.** Containers and null have no terms. +- **Equality:** every scalar kind except `Bool`, including floats over their + canonical bits. +- **Order:** every scalar kind except `Bool`, under all three schemes. +- **`Bool` has a ciphertext only.** A keyed hash or an order term over a + domain of two values hides nothing: it splits the rows into two groups, and + an order term also says which group is `true`. The existing CLLW order terms + for `Bool` are removed. + +These exceptions are an explicit, documented list kept next to vitaminc's +conformance test, which fails for any kind that is missing from a layer +without an entry. + +### Where each encoding lives + +- **Ciphertext:** `vitaminc-aead-value`. +- **Equality:** `vitaminc-prf`, which gains domains for every new kind, floats + and `Decimal`, over the canonical bytes. +- **Order:** a new `vitaminc-ore` crate. + - The plaintext layer is `orderable-bytes`, which stays its own crate in + ore.rs and gains a variable-length encoding for text and bytes. Its + existing fixed-length output does not change. + - A `Scheme` trait, with block ORE (over ore-rs), CLLW ORE and CLLW OPE + (over cllw-ore). A scheme only encrypts the bytes the plaintext layer + produces, so every orderable kind works under every scheme. + - Block ORE uses each kind's natural width. The 8-byte padding is a + cipherstash-client detail kept for its stored terms. + - cllw-ore keeps its typed impls, unchanged, behind a cargo feature that + cipherstash-client enables, and gains a bytes-level entry point that + `vitaminc-ore` calls. +- **Term derivation** takes a `&Value` in `vitaminc-prf` and `vitaminc-ore`, + with one exhaustive match per layer that keeps leaves inside `Protected`. A + pairing a layer does not support returns a typed error from vitaminc. + +### Stack Encrypt + +- `dynamic::Value` and `dynamic::term::Scalar` are deleted. `dynamic::term` + takes a `&vitaminc_aead_value::Value`. `admits` keeps only the rules that + belong to Stack Encrypt, such as `Match` taking text only. +- Order terms go through `vitaminc-ore`, and a block ORE term kind joins CLLW + ORE and OPE. Stack Encrypt no longer depends on cllw-ore directly. +- `TextEq` normalises to NFC and moves to EQL v4. The Stack Encrypt targets on + `eql_v3_*` domains are deleted. The `stack-encrypt:1:` ciphertext prefix + stays: decryption does not pass through Postgres types, and the prefix is + what lets it reject the other producer's payload. + +### Host languages + +- **Go.** `int8`, `int16`, `uint8` and `uint16` map to their own kinds instead + of widening to 32 bits; this lands before the Go SDK ships, so no stored + data uses the old mapping. `Int128` and `Uint128` are SDK value types. + `time.Time` means `Timestamp`, and `encrypt.Date{Year, Month, Day}` is a + date. A field whose EQL target is a date family accepts `time.Time`, + truncated to its UTC calendar day. +- **JavaScript.** A `BigInt` maps to the smallest kind that holds it, up to + 128 bits, and decodes as a `BigInt`. `Date` means `Timestamp`; a date has + its own wrapper. +- **Every binding** runs vitaminc's shared test vectors: a host value, the + kind it must map to, and the exact `[tag] ++ payload` leaf bytes, including + normalisation cases. The leaf is deterministic even though the AEAD is not. + +### EQL v4 + +- **v4 is the v3 SQL with Stack Encrypt terms.** One SQL source is emitted + under two names. The hand-written SQL takes the schema as a build-time + placeholder, as eql-codegen's templates already do. Consistent with + ADR-0001, the data-bearing domains (`eql_v3_*`, `eql_v4_*`) live in + `public` and survive reinstall, and the implementation schemas + (`eql_v3`, `eql_v4` and their `_internal` schemas) stay disposable. The + second name separates producers; it is not a versioned upgrade mechanism of + the kind ADR-0001 rejects. +- **A v4 payload's envelope carries `"v": 4`**, and the `eql_v4_*` domains' + check constraints test it, as the `eql_v3_*` ones test `3`. The version is + one more build-time placeholder. A payload written to the other producer's + domain then fails on insert, rather than only matching nothing when it is + queried. +- **New EQL targets are v4-only.** In v3 they stay refused, with a reason + that points to v4. +- **`@cipherstash/eql` 4.x ships both bundles** from `main`. A SQL fix lands in + both names in one release. +- **`stash eql install --eql-version 3|4|all`** chooses the bundle and defaults + to 3, which is today's behaviour. The default changes to 4, announced in + advance, when the TypeScript stack has moved to Stack Encrypt. +- **During the overlap**, expected to last a quarter or more, cipherstash-client + takes fixes only. New kinds and domains are produced through Stack Encrypt + and v4. An exception is a decision written down in its issue. + +### Review + +Using the canonical order bytes as the PRF input is the simplest way to make +equality and ordering agree, and it is frozen once data is stored under it. +Dan Draper signs it off in writing before the `vitaminc-prf` change lands. + +## Consequences + +- **The release order is fixed.** ore.rs (`orderable-bytes`) and cllw-ore + first, then vitaminc 0.6.0, then the Stack Encrypt breaking release. Stack + Encrypt must reach crates.io before `eql-bindings` uses its new API, since + `cargo publish` builds `eql-bindings` against the registry. The EQL v4 + bundle and the Go SDK changes follow. +- **Adding a kind later is additive**, because `Value` and `ValueKind` are + `#[non_exhaustive]`. It still takes a tag, a PRF domain, an order encoding, + a codec in every binding, and test vectors, or an entry in the exceptions + list. +- **Stack Encrypt's terms change** for `-0.0` and negative-sign NaN floats, + for non-NFC text equality, and for `Bool` order terms, which are removed. + None is deployed, so the breaking release carries them without migration. +- **cipherstash-client's terms do not change.** Its fixed-length + `orderable-bytes` output, cllw-ore's typed impls and its block ORE padding + are all kept, so every stored v3 payload stays queryable. +- **Moving a column from v3 to v4 means re-encrypting it.** The ciphertext and + every term change. How that migration runs belongs to the decision that + moves the TypeScript stack onto Stack Encrypt. +- **The CLI surface changes** (`--eql-version`), so `skills/stash-cli`, + `skills/stash-indexing` and `skills/stash-postgres` change in the same PR. +- **This replaces a definition in a plan.** `docs/plans/2026-10-04-plan-builder.md` + calls "EQL v4" a name for a Stack Encrypt payload in the v3 envelope and + the `eql_v3_*` domains. That plan is updated to this ADR. + +## Deferred + +- **Block ORE for text** waits for ore-rs's chained, variable-length block ORE + to be reviewed and released, then arrives in a minor release of + `vitaminc-ore`. Until then it is an entry in the exceptions list. +- **Block ORE for `Bool`.** Block ORE stored as right ciphertexts only is + fully randomised and semantically secure, so a two-value domain leaks + nothing through it. `Bool` could support that scheme alone, enforced by a + marker trait on schemes. Not built until needed. +- **Locale-aware collation**, as its own domain with the collation version in + its name. +- **ASCII-packed text domains**, until a customer's column sizes make the + case. diff --git a/docs/plans/2026-10-04-plan-builder.md b/docs/plans/2026-10-04-plan-builder.md index 88dd628fb..e70ea3b11 100644 --- a/docs/plans/2026-10-04-plan-builder.md +++ b/docs/plans/2026-10-04-plan-builder.md @@ -662,9 +662,11 @@ In the `Text` family, the three `Ord` suffixes carry equality too. Each type has a query type, with `Query` after its name: `TextEqQuery`. The value of `encrypt_into` is the Go type name. -An EQL value has the EQL v3 envelope: its version field is `3`, and Postgres stores it in an `eql_v3` domain. +An EQL value from the engine is an EQL v4 value: its version field is `4`, and Postgres stores it in an `eql_v4_*` domain. The ciphertext inside it is a Stack Encrypt ciphertext, which starts with `stack-encrypt:1:`. -"EQL v4" in this plan is the name of that form, and not a new envelope. +EQL v4 is the v3 SQL, emitted from the same source under a second name, so that a column written by Stack Encrypt and one written by cipherstash-client are different Postgres types ([ADR-0002](../adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md)). +This replaces this plan's first definition, under which "EQL v4" named a Stack Encrypt payload in the v3 envelope and an `eql_v3` domain; Postgres could not tell the two producers apart, and a query from one against a column written by the other matched nothing. +The `TextEq` described below is still produced into an `eql_v3` domain until that move lands. The engine produces one EQL type today: `TextEq`. Status (2026-10-06, #1062): `TextEq` is producible through the data plan's target form. @@ -1157,9 +1159,10 @@ model rather than a strain: it puts key material in the database. ## EQL v4 types as field targets -Naming: **EQL v4** is the EQL form of a stack-encrypt payload. **EQL v3** is -the existing SQL bundle and its `eql_v3_*` domains, which this section does -not change. Depends on #971 (`TextEq` / `TextEqQuery` through stack-encrypt +Naming: **EQL v4** is the EQL form of a stack-encrypt payload: the v3 SQL +emitted under a second name, with `eql_v4_*` domains and `"v": 4` envelopes +(ADR-0002). **EQL v3** is the same SQL holding cipherstash-client payloads in +its `eql_v3_*` domains, which this section does not change. Depends on #971 (`TextEq` / `TextEqQuery` through stack-encrypt in `eql-bindings`, and `Identifier` as a two-segment `Label`). **The engine returns an EQL type only when a plan names it as a target.** diff --git a/docs/plans/2026-10-11-order-and-equality-terms-layering.md b/docs/plans/2026-10-11-order-and-equality-terms-layering.md new file mode 100644 index 000000000..54f3d6947 --- /dev/null +++ b/docs/plans/2026-10-11-order-and-equality-terms-layering.md @@ -0,0 +1,331 @@ +# Order and equality terms: vitaminc at the bottom + +> **Plan, not specification.** This document is an indicative sketch of the +> steps required, written before the work was done. It is not kept in step +> with the implementation and must not be used as a formal specification or +> as a reference for reviewing what was actually built: the code, its rustdoc +> and the tests are the source of truth. Where the two disagree, the code wins +> and this document is simply out of date. + +**Status:** planned. Decisions settled 2026-10-11, except the one listed +under "Open decision". +**Date:** 2026-10-11 +**Amends:** ADR-0002 (`docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md`). +Where the two disagree, this document wins. +**Issues:** cipherstash/vitaminc#374 (`vitaminc-ore`), #1063 (index crates) + +## Why + +Starting on cipherstash/vitaminc#374 under ADR-0002 turned up four problems. + +- **Too many dependencies across repositories.** `vitaminc-ore` was to depend + on cllw-ore and ore-rs and adapt each to its `Scheme` trait, with + `orderable-bytes` underneath both. Order encodings, scheme traits and term + derivation ended up spread over three repositories, and every change needed + releases in all of them. +- **Raw bytes reach the schemes.** cllw-ore and ore-rs accept bytes that the + caller has already encoded. Bytes encoded by hand (little-endian integers, + an unflipped sign bit, `f64::to_bits`) encrypt without error and sort + wrong. Fixing that inside cllw-ore needed wrapper types in `orderable-bytes` + (cipherstash/ore.rs#103) and a feature split in cllw-ore + (cipherstash/cipherstash-suite#2308). Cargo features unify across a build, + so the feature split could not hold whenever anything else in the build + turned the feature on. +- **PRF depends on ordering.** `vitaminc-prf` reaches `orderable-bytes` + through `vitaminc-aead-value`'s `canonical` module, which reuses order + encodings as equality input. Equality needs a canonical form, not an order. +- **One canonical form for both term kinds is too rigid.** ADR-0002 builds + equality and order terms from the same canonical form. A leakage-tuned + domain needs them to differ, for example an exact timestamp for equality + and only the day for order. + +## Decisions + +### vitaminc is the bottom of the stack + +- `vitaminc-ore` owns: + - the order encodings; + - an `OrderScheme` trait; + - the derivation of order terms from `&Value`, generic over the scheme. + + It depends on no scheme crate. +- A scheme's input is a sealed type, `OrderPlaintext`, that only + `vitaminc-ore` can construct. It comes in fixed-length and variable-length + forms. A scheme can't be given bytes that skipped the encoding, so no + wrapper types or feature flags are needed to keep raw input out. +- `OrderScheme` returns a scheme-defined term, roughly + `type Term: Ord + AsRef<[u8]>`, whose length is only known at runtime. Each + scheme keeps its own comparison (CLLW ORE and block ORE don't compare byte + by byte) and its own stored bytes. Fixed-width types such as + `OreCllw8V1` stay as inherent APIs on the scheme crates, because stable + Rust can't name their width generically. +- The trait also covers the key type, domain separation (the term's domain + label goes in as the salt) and typed errors. +- cllw-ore and ore-rs depend on `vitaminc-ore` and implement `OrderScheme` + behind an optional `vitaminc` feature, so cipherstash-client doesn't take a + vitaminc dependency. Their existing typed APIs stay as they are, so + cipherstash-client's stored terms don't change. +- `vitaminc-ore` releases in lockstep with the other vitaminc crates. On each + vitaminc minor release, cllw-ore and ore-rs bump their optional + `vitaminc-ore` dependency before stack can upgrade. That is acceptable + while stack is the only user of the trait path. +- `vitaminc-ore`'s tests use a mock scheme defined inside the crate. A + dev-dependency on cllw-ore would form a cycle across repositories and pull + a second `vitaminc-ore` in from crates.io. + +### Equality and order are tuned per domain + +- There is no shared canonical form. Each kind has a **semantic baseline**, + which makes values that are equal under `Value`'s own semantics encode the + same: decimal scale (`1.0` = `1.00`), signed zero, NaN, NFC. +- On top of the baseline, each term's domain chooses its own **lossy + transforms**, such as timestamp resolution (exact, microsecond, day), + text folding (removing combining marks, case folding, with the Unicode + version pinned) and prefix truncation. +- Each transform is named in its term's domain label, for example + `…/timestamp/day/v1`, so terms built with different transforms can never + be compared. +- Equality and order can be **independent** (an exact timestamp for + equality, the day for order) or **locked** (the same transform value + passed to both). Stack's index spec (#1063) carries the choice per column. +- The semantic baseline lives in each term crate: `vitaminc-prf` and + `vitaminc-ore` each apply it. A conformance test checks that values equal + under `Value` semantics produce equal terms in both. There is no shared + canonicalisation crate. +- A column's leakage is the combined leakage of all its terms. Truncating the + order term to the day hides the order of events within a day, but an exact + equality term on the same column still reveals which rows share a + timestamp. Domain design must account for the combination. + +### PRF has no ordering dependency + +- `vitaminc-prf` depends only on `vitaminc-aead-value` and applies its own + baseline and transforms. +- `vitaminc-aead-value` does no term normalisation. Its `canonical` module and + its dependency on `orderable-bytes` are removed. +- The value-derived PRF domains (cipherstash/vitaminc#380) haven't been + released: they landed on 2026-10-08, after vitaminc 0.5.1. So `vitaminc-prf` + can choose its own canonical bytes for dates, timestamps and decimals + without affecting any stored term. + +### `orderable-bytes` is frozen + +- `orderable-bytes` stays as it is for cipherstash-client's stored terms. It + is not part of the target design. +- There is no 0.3. cipherstash/ore.rs#103 (binary wrapper types) is closed. +- 0.2.0 is not yanked: ore-rs 0.8.4 depends on it and nothing new needs it + gone. + +## Open decision: exhaustive matching over `Value` + +Once `canonical.rs` leaves `vitaminc-aead-value`, `vitaminc-prf` and +`vitaminc-ore` must each handle every kind of `Value`, and adding a kind must +fail to compile until each one does. Two approaches are on the table. + +### Prerequisite for either approach: variants that exist in every build + +`Value::Date` and `Value::Timestamp` exist only with `vitaminc-aead-value`'s +`chrono` feature, and `Value::Decimal` only with its `rust_decimal` feature. +Because features unify across a build, an exhaustive match outside +`vitaminc-aead-value` breaks: + +1. `vitaminc-prf` matches `Value` exhaustively, with its `Date` arm behind its + own `chrono` feature. +2. In a build where another crate turns on `vitaminc-aead-value/chrono` and + `vitaminc-prf`'s `chrono` is off, `Value::Date` exists and `vitaminc-prf` + has no arm for it. `vitaminc-prf` fails to compile in that build. + +A visitor fails the same way: a required `visit_date` that exists only with +`chrono` is a method an implementor can't know whether to write. A crate can +check its own features, never a dependency's. Today's design avoids this only +because the match sits inside `vitaminc-aead-value`, which checks its own +features. + +The fix is to make every variant exist in every build, using vitaminc's own +representation, with conversions to and from chrono and rust_decimal behind +the features: + +| Variant | Representation | +|---|---| +| `Date` | days from the start of the common era, `i32` | +| `Timestamp` | Unix seconds `i64`, nanoseconds `u32` | +| `Decimal` | mantissa and scale | + +The ciphertext wire format already uses these payloads (tags `0x12`–`0x14` +are independent of chrono), so stored ciphertexts don't change. The Rust +types do, which is a breaking change to `Value`. + +### The two approaches + +**A: drop `#[non_exhaustive]`.** `vitaminc-prf`, `vitaminc-ore` and the +bindings match `Value` exhaustively in their own crates. + +**B: a visitor trait, keeping `#[non_exhaustive]`.** `vitaminc-aead-value` +defines a `ValueVisitor` with one required method per kind and does the +dispatch. Crates that must handle every kind implement it. Other crates keep +matching with a `_ =>` arm. + +| | A: drop `#[non_exhaustive]` | B: visitor, keep `#[non_exhaustive]` | +|---|---|---| +| Who gets exhaustiveness | Every crate that matches `Value` | Crates that implement the visitor | +| Adding a kind | Breaking for every crate that matches | Breaking for visitor implementors, additive for others | +| Code to maintain | Nothing new | One trait, one dispatch function, a method per kind in each implementor | +| Grouping | Each match writes out every integer width | The trait can offer grouped hooks, such as one integer method that the per-width methods default to | +| Changing `Value`'s representation | Breaks every match | Hidden behind the method signatures | +| Main risk | Low | A default on a per-kind method lets a new kind compile without being handled. Per-kind methods need no defaults; only grouping hooks may have them. | + +**Evidence:** +- Outside `vitaminc-aead-value`, `Value` is matched in `vitaminc-prf` (through + `canonical`), in `aead-napi`'s `convert.rs` and `scalar.rs`, and in the hmac + tests. +- `aead-napi` already has `_ => Err("unsupported value kind")` + (`convert.rs`). That is the silent refusal `#[non_exhaustive]` invites. +- ADR-0002 requires every binding to handle every kind, so the bindings want + exhaustiveness as well. +- Under A they get it automatically. Under B each binding opts in by + implementing the visitor. + +**Leaning:** B. The decision waits on a closer look at the `Value` enum's +design, which may change the representation in ways that favour one +approach. + +## Current state + +Arrows point from a crate to what it depends on. cipherstash-client and +`vitaminc-protected` are left out. + +```mermaid +flowchart TB + subgraph stack + SE[stack-encrypt] + end + subgraph suite["cipherstash-suite"] + CLLW["cllw-ore 0.5
typed impls, own encodings"] + end + subgraph vitaminc + PRF[vitaminc-prf] + AV["vitaminc-aead-value
Value, canonical module"] + end + subgraph orers["ore.rs"] + ORERS["ore-rs 0.8"] + OB["orderable-bytes"] + end + SE --> PRF + SE --> AV + SE --> CLLW + PRF -- "canonical feature" --> AV + AV -- "git pin, pre-0.2" --> OB + CLLW -- "0.1, chrono and decimal" --> OB + ORERS -- "0.2" --> OB +``` + +- stack-encrypt calls cllw-ore directly for order terms. There is no block + ORE path. +- Equality bytes for dates, timestamps and decimals come from order + encodings, through `vitaminc-aead-value`. +- `orderable-bytes` is reached three ways, at three different versions. + +## Target state + +```mermaid +flowchart TB + subgraph stack + EQI["Equality index
picks transforms"] + ORI["Ore / Ope index
picks scheme and transforms"] + end + subgraph schemes["scheme crates"] + ORERS["ore-rs
block ORE"] + CLLW["cllw-ore
CLLW ORE and OPE"] + end + subgraph vitaminc + PRF["vitaminc-prf
equality terms, own baseline and transforms"] + VORE["vitaminc-ore
order encodings, transforms,
OrderScheme, sealed OrderPlaintext"] + AV["vitaminc-aead-value
Value"] + end + EQI --> PRF + ORI --> VORE + ORI --> ORERS + ORI --> CLLW + ORERS -- "impl OrderScheme" --> VORE + CLLW -- "impl OrderScheme" --> VORE + PRF --> AV + VORE --> AV +``` + +- vitaminc depends on nothing in the other repositories. +- The scheme crates implement `vitaminc-ore`'s trait. The index crates choose + a concrete scheme and the transforms for each column. +- `orderable-bytes` doesn't appear. It stays frozen for cipherstash-client. + +## Changes to ADR-0002 + +| ADR-0002 says | Now | +|---|---| +| "Equality and order terms are computed from one canonical form per kind, and both layers use the same one" | Each kind has a semantic baseline; lossy transforms are chosen per domain, independently or locked | +| Canonical forms table: timestamp truncated to microseconds, text order folded | These become per-domain transforms, named in each domain's label | +| Order: "The plaintext layer is `orderable-bytes`, which … gains a variable-length encoding" | `vitaminc-ore` owns the order encodings; `orderable-bytes` is frozen | +| "A `Scheme` trait, with block ORE (over ore-rs), CLLW ORE and CLLW OPE (over cllw-ore)" | `OrderScheme` lives in `vitaminc-ore`; ore-rs and cllw-ore implement it | +| "cllw-ore keeps its typed impls … behind a cargo feature that cipherstash-client enables, and gains a bytes-level entry point" | cllw-ore's typed API is unchanged and not feature-gated; it gains an `OrderScheme` impl behind an optional `vitaminc` feature | +| Review: "Using the canonical order bytes as the PRF input … Dan Draper signs it off" | PRF no longer uses order bytes; the review item applies only to domains that lock equality and order together | +| `Value` and `ValueKind` become `#[non_exhaustive]` | Open: see "Open decision" | +| Release order: "ore.rs (`orderable-bytes`) and cllw-ore first, then vitaminc 0.6.0" | vitaminc first, then cllw-ore and ore-rs, then stack | + +## Steps + +### Phase 1: tidy up work in flight + +1. Close cipherstash/ore.rs#103. Don't yank `orderable-bytes` 0.2.0. +2. Cut cipherstash/cipherstash-suite#2308 down to its first two commits: + known-answer tests that pin every cllw-ore encrypt path, and wiping the + plaintext bit buffer in `encrypt_ope_bits`. It becomes a cllw-ore patch + release. The known-answer tests then protect cipherstash-client's terms + through the rest of this work. + +### Phase 2: vitaminc + +3. Settle the open decision. If the `Value` changes are accepted, make every + variant exist in every build. +4. `vitaminc-aead-value` and `vitaminc-prf`: + - add the chosen exhaustiveness mechanism; + - remove `canonical.rs` and the `orderable-bytes` dependency from + `vitaminc-aead-value`; + - move the equality baseline and transforms into `vitaminc-prf`, with each + transform named in its domain label. +5. New crate `vitaminc-ore` (rewrite cipherstash/vitaminc#374 to match): + - `OrderScheme` and the sealed `OrderPlaintext`; + - order encodings ported from `orderable-bytes`, kept in `Protected`, + pinned by golden vectors, and checked against `orderable-bytes` where + they match byte for byte; + - per-domain transforms: timestamp resolution, text folding with a pinned + Unicode version, room for prefix truncation; + - order-term derivation from `&Value`, generic over the scheme; + - a mock scheme for tests; + - docs that state the combined-leakage rule. +6. A conformance test across `vitaminc-prf` and `vitaminc-ore`: every kind is + handled or listed as an exception, and values equal under `Value` + semantics produce equal terms. +7. Release vitaminc 0.6.0. + +### Phase 3: scheme crates + +8. cllw-ore: implement `OrderScheme` behind an optional `vitaminc` feature, + with known-answer tests for the trait path. This is an additive release. +9. ore-rs: the same. Block ORE uses each kind's natural width; + cipherstash-client keeps its 8-byte padding through the existing API. + Block ORE for text waits for cipherstash/ore.rs#95 and goes on the + exceptions list. +10. Release both. + +### Phase 4: stack + +11. Index crates (#1063): each index spec carries its scheme and transforms. + The Equality index calls `vitaminc-prf`; the Ore and Ope index builds the + concrete scheme and calls `vitaminc-ore`. stack-encrypt drops its direct + use of cllw-ore. +12. EQL v4 domain labels name their transforms. + +### Later + +- Block ORE for text, once cipherstash/ore.rs#95 is released. +- Retire cllw-ore's typed impls and `orderable-bytes` once cipherstash-client + is retired.