From 2bab28801108c3c83e8afd71728cf8934becc673 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Thu, 8 Oct 2026 18:08:48 +1100 Subject: [PATCH 1/2] docs(adr): one value model, three encodings in vitaminc, EQL v4 by producer Stack Encrypt can produce one EQL type because its ciphertext, equality and order encodings disagree about which kinds exist, and the mapping from a kind to a Rust type lives in stack-encrypt (Scalar, dynamic::Value) instead of vitaminc. EQL also cannot tell a Stack Encrypt column from a cipherstash-client one, so a query from one producer against the other's column matches nothing, silently. ADR-0002 records the decisions from the value-encodings RFC review: the new kinds and their frozen tags, one canonical form per kind for terms, term derivation moving into vitaminc-prf and a new vitaminc-ore, and EQL v4 as the v3 SQL emitted under a second name with "v": 4 envelopes, so Postgres keeps the two producers apart. The plan-builder plan defined "EQL v4" as a name for a Stack Encrypt payload in the v3 envelope and eql_v3 domains; it now points at the ADR. --- ...-three-encodings-and-eql-v4-by-producer.md | 269 ++++++++++++++++++ docs/plans/2026-10-04-plan-builder.md | 13 +- 2 files changed, 277 insertions(+), 5 deletions(-) create mode 100644 docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md diff --git a/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md new file mode 100644 index 000000000..b5f858b10 --- /dev/null +++ b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md @@ -0,0 +1,269 @@ +--- +status: accepted +date: 2026-10-08 +relates-to: ADR-0001; stack-encrypt ADR-0003, ADR-0007 +--- + +# One value model, three encodings in vitaminc, and EQL v4 separated by producer + +Every value Stack Encrypt handles is encoded three times: as a **ciphertext** +(reversible and self-describing), as an **equality term** (a keyed hash that +must be unambiguous) and as an **order term** (bytes whose order is the +value's order). This ADR decides where each encoding lives, which kinds of +value exist, the canonical form each kind takes, and how EQL keeps columns +written by Stack Encrypt apart from columns written by cipherstash-client. + +## The problem + +In October 2026 Stack Encrypt could produce one of EQL's 51 types, `TextEq`. +The number, date, timestamp and boolean families were blocked on encoding, +not code: + +- **The three encodings disagreed about which types exist.** `i16` had an + equality domain in `vitaminc-prf` and an order encoding, but no ciphertext + tag, so it could not be a kind. Date, timestamp and decimal had order + encodings only. +- **The mapping from a kind to a Rust type lived in Stack Encrypt.** + `dynamic::term::Scalar` mirrored `FfiValue`'s variants one for one to + dispatch to the term crates, and `dynamic::Value` wrapped `FfiValue` only to + add `Clone`. Both restated vitaminc's vocabulary in a downstream crate. +- **Ordering was implemented twice.** `orderable-bytes` defines a canonical, + order-preserving encoding per type. cllw-ore uses it for chrono and decimal + but hand-rolls integers and floats, and differs on `-0.0`. Block ORE was not + derivable at all, so `TextOrdOre` and `TextSearchOre` were refused. +- **The existing writer's encodings are inconsistent.** cipherstash-client + hashes a timestamp's milliseconds but orders by its nanoseconds, and hashes + a decimal with its scale while ordering ignores it. Its block ORE text is + ASCII-only, lowercased, maps every digit to one symbol and truncates to six + blocks. cllw-ore's text decomposes to NFD and strips accents. +- **Two producers wrote the same EQL domains.** A Stack Encrypt query term + never matches a cipherstash-client term for the same value, and the + `eql_v3_*` domains could not tell them apart. A query from one against a + column written by the other returned no rows, silently. + +## Options considered + +**Where the value model lives.** + +1. **Move `vitaminc-aead-value` into Stack Encrypt.** It looks like FFI + plumbing. But it is the value model for vitaminc's own `Cipher` traits: + `aead-napi` and vitaminc's Go binding encrypt with `Aes256Cipher` through + it, without Stack Encrypt. Moving it makes vitaminc Rust-only, makes + vitaminc depend on Stack Encrypt (which depends on six vitaminc crates), + or forks the frozen tag table into two copies. +2. **Keep it in vitaminc, and move term derivation there too.** All three + encodings then live in one repository, and one exhaustive `match` on the + value type per layer makes "a kind exists in every layer or in none" a + compile error rather than a convention. Chosen. + +**How EQL separates the two producers.** The SQL for both is identical; only +the producer of the terms differs. + +1. **Name only.** Stack Encrypt payloads go in the `eql_v3_*` domains, + distinguished by the `stack-encrypt:1:` ciphertext prefix. Nothing in the + database stops a cross-producer comparison. +2. **A producer tag in every payload and term, checked by every operator.** + A runtime check on the hottest SQL paths, across about 24k lines of + hand-written SQL. +3. **One SQL source, emitted under two names.** The build writes the same + source out as `eql_v3` (cipherstash-client terms) and `eql_v4` (Stack + Encrypt terms). A column's domain names its producer, and Postgres refuses + to compare an `eql_v4_*` query term with an `eql_v3_*` column at plan time. + Chosen. + +**How v3 and v4 ship.** Bumping `main` to v4 and patching v3 from a branch +needs a maintenance release path for five lockstep artefacts that does not +exist, and the EQL publish script puts every stable release on `latest`, so a +3.x patch would move `latest` backwards. A second package doubles the +trusted-publishing and release surface. One package carrying both bundles +needs neither. Chosen. + +## Decision + +### Kinds and the ciphertext + +- vitaminc 0.6.0 adds the kinds `Int8`, `UInt8`, `Int16`, `UInt16`, `Int128`, + `UInt128`, `Date`, `Timestamp` and `Decimal`, each with a `ValueKind` name, + a `Value` variant and a ciphertext tag, in one breaking release. Rust `i128` + and `u128` get `Encrypt` and `Decrypt` impls. +- `FfiValue` is renamed `Value`, with `#[deprecated] pub type FfiValue = Value` + for one release. `Value` and `ValueKind` become `#[non_exhaustive]`, so later + kinds are additive. +- `Value` implements `Clone` as a deep copy that rebuilds every leaf into a + fresh `Protected`. A clone is under the same custody as its original. +- The sealed leaf tag table (`tags.rs`) stays frozen and contiguous. The + transport codec's framing tags move from `0x10`–`0x12` to `0xF0`–`0xF2`; + transport carries no compatibility commitment and every user of it updates + together. The new leaves: + + | Tag | Kind | Payload | + |---|---|---| + | `0x0C`–`0x11` | `Int8`, `UInt8`, `Int16`, `UInt16`, `Int128`, `UInt128` | 1, 1, 2, 2, 16, 16 bytes; two's complement for signed; little-endian | + | `0x12` | `Date` | `i32` days counted from 0001-01-01 (`num_days_from_ce`), little-endian | + | `0x13` | `Timestamp` | `i64` Unix seconds then `u32` nanoseconds, little-endian, UTC | + | `0x14` | `Decimal` | rust_decimal's 16-byte `serialize()`, which keeps the scale | + +- `transport` stays a module of `vitaminc-aead-value`. + +### Canonical forms + +The ciphertext keeps the value exactly as given: `1.50` decrypts as `1.50`, +and a timestamp keeps its nanoseconds. Equality and order terms are computed +from one canonical form per kind, and both layers use the same one, so +equality and ordering agree by construction. + +| Kind | Canonical form for terms | +|---|---| +| `Timestamp` | truncated to microseconds, Postgres's precision | +| `Decimal` | scale normalised (`1`, `1.0` and `1.00` are equal). NaN and ±Infinity are refused at encode time; rust_decimal cannot represent them | +| `Float32`, `Float64` | `-0.0` folded into `+0.0`; every NaN replaced by one positive quiet NaN, which sorts above +Infinity. This matches Postgres | +| text, equality | NFC | +| text, order | NFC, then NFD, combining marks removed, Unicode default case folding | + +Text normalisation is pinned. The Unicode version is part of the encoding's +domain label (for example `text-nfc/unicode-16/v1`), the +`unicode-normalization` crate is pinned to it, and strings containing +unassigned code points are refused. Order terms fold accent and case because +code-point order puts `é` after `z`; a fixed, pinned fold approximates the +first level of Unicode collation without depending on ICU, whose sort keys +change between versions. Equality is not folded: a folded order term only +produces ties, while a folded equality term produces false matches. A +case-insensitive equality is its own domain. + +Truncation and alphabet packing are settings of an EQL domain, named in its +label, never part of the shared encoding. + +### Which kinds get which terms + +- **Every scalar kind has a ciphertext.** Containers and null have no terms. +- **Equality:** every scalar kind except `Bool`, including floats over their + canonical bits. +- **Order:** every scalar kind except `Bool`, under all three schemes. +- **`Bool` has a ciphertext only.** A keyed hash or an order term over a + domain of two values hides nothing: it splits the rows into two groups, and + an order term also says which group is `true`. The existing CLLW order terms + for `Bool` are removed. + +These exceptions are an explicit, documented list kept next to vitaminc's +conformance test, which fails for any kind that is missing from a layer +without an entry. + +### Where each encoding lives + +- **Ciphertext:** `vitaminc-aead-value`. +- **Equality:** `vitaminc-prf`, which gains domains for every new kind, floats + and `Decimal`, over the canonical bytes. +- **Order:** a new `vitaminc-ore` crate. + - The plaintext layer is `orderable-bytes`, which stays its own crate in + ore.rs and gains a variable-length encoding for text and bytes. Its + existing fixed-length output does not change. + - A `Scheme` trait, with block ORE (over ore-rs), CLLW ORE and CLLW OPE + (over cllw-ore). A scheme only encrypts the bytes the plaintext layer + produces, so every orderable kind works under every scheme. + - Block ORE uses each kind's natural width. The 8-byte padding is a + cipherstash-client detail kept for its stored terms. + - cllw-ore keeps its typed impls, unchanged, behind a cargo feature that + cipherstash-client enables, and gains a bytes-level entry point that + `vitaminc-ore` calls. +- **Term derivation** takes a `&Value` in `vitaminc-prf` and `vitaminc-ore`, + with one exhaustive match per layer that keeps leaves inside `Protected`. A + pairing a layer does not support returns a typed error from vitaminc. + +### Stack Encrypt + +- `dynamic::Value` and `dynamic::term::Scalar` are deleted. `dynamic::term` + takes a `&vitaminc_aead_value::Value`. `admits` keeps only the rules that + belong to Stack Encrypt, such as `Match` taking text only. +- Order terms go through `vitaminc-ore`, and a block ORE term kind joins CLLW + ORE and OPE. Stack Encrypt no longer depends on cllw-ore directly. +- `TextEq` normalises to NFC and moves to EQL v4. The Stack Encrypt targets on + `eql_v3_*` domains are deleted. The `stack-encrypt:1:` ciphertext prefix + stays: decryption does not pass through Postgres types, and the prefix is + what lets it reject the other producer's payload. + +### Host languages + +- **Go.** `int8`, `int16`, `uint8` and `uint16` map to their own kinds instead + of widening to 32 bits; this lands before the Go SDK ships, so no stored + data uses the old mapping. `Int128` and `Uint128` are SDK value types. + `time.Time` means `Timestamp`, and `encrypt.Date{Year, Month, Day}` is a + date. A field whose EQL target is a date family accepts `time.Time`, + truncated to its UTC calendar day. +- **JavaScript.** A `BigInt` maps to the smallest kind that holds it, up to + 128 bits, and decodes as a `BigInt`. `Date` means `Timestamp`; a date has + its own wrapper. +- **Every binding** runs vitaminc's shared test vectors: a host value, the + kind it must map to, and the exact `[tag] ++ payload` leaf bytes, including + normalisation cases. The leaf is deterministic even though the AEAD is not. + +### EQL v4 + +- **v4 is the v3 SQL with Stack Encrypt terms.** One SQL source is emitted + under two names. The hand-written SQL takes the schema as a build-time + placeholder, as eql-codegen's templates already do. Consistent with + ADR-0001, the data-bearing domains (`eql_v3_*`, `eql_v4_*`) live in + `public` and survive reinstall, and the implementation schemas + (`eql_v3`, `eql_v4` and their `_internal` schemas) stay disposable. The + second name separates producers; it is not a versioned upgrade mechanism of + the kind ADR-0001 rejects. +- **A v4 payload's envelope carries `"v": 4`**, and the `eql_v4_*` domains' + check constraints test it, as the `eql_v3_*` ones test `3`. The version is + one more build-time placeholder. A payload written to the other producer's + domain then fails on insert, rather than only matching nothing when it is + queried. +- **New EQL targets are v4-only.** In v3 they stay refused, with a reason + that points to v4. +- **`@cipherstash/eql` 4.x ships both bundles** from `main`. A SQL fix lands in + both names in one release. +- **`stash eql install --eql-version 3|4|all`** chooses the bundle and defaults + to 3, which is today's behaviour. The default changes to 4, announced in + advance, when the TypeScript stack has moved to Stack Encrypt. +- **During the overlap**, expected to last a quarter or more, cipherstash-client + takes fixes only. New kinds and domains are produced through Stack Encrypt + and v4. An exception is a decision written down in its issue. + +### Review + +Using the canonical order bytes as the PRF input is the simplest way to make +equality and ordering agree, and it is frozen once data is stored under it. +Dan Draper signs it off in writing before the `vitaminc-prf` change lands. + +## Consequences + +- **The release order is fixed.** ore.rs (`orderable-bytes`) and cllw-ore + first, then vitaminc 0.6.0, then the Stack Encrypt breaking release. Stack + Encrypt must reach crates.io before `eql-bindings` uses its new API, since + `cargo publish` builds `eql-bindings` against the registry. The EQL v4 + bundle and the Go SDK changes follow. +- **Adding a kind later is additive**, because `Value` and `ValueKind` are + `#[non_exhaustive]`. It still takes a tag, a PRF domain, an order encoding, + a codec in every binding, and test vectors, or an entry in the exceptions + list. +- **Stack Encrypt's terms change** for `-0.0` and negative-sign NaN floats, + for non-NFC text equality, and for `Bool` order terms, which are removed. + None is deployed, so the breaking release carries them without migration. +- **cipherstash-client's terms do not change.** Its fixed-length + `orderable-bytes` output, cllw-ore's typed impls and its block ORE padding + are all kept, so every stored v3 payload stays queryable. +- **Moving a column from v3 to v4 means re-encrypting it.** The ciphertext and + every term change. How that migration runs belongs to the decision that + moves the TypeScript stack onto Stack Encrypt. +- **The CLI surface changes** (`--eql-version`), so `skills/stash-cli`, + `skills/stash-indexing` and `skills/stash-postgres` change in the same PR. +- **This replaces a definition in a plan.** `docs/plans/2026-10-04-plan-builder.md` + calls "EQL v4" a name for a Stack Encrypt payload in the v3 envelope and + the `eql_v3_*` domains. That plan is updated to this ADR. + +## Deferred + +- **Block ORE for text** waits for ore-rs's chained, variable-length block ORE + to be reviewed and released, then arrives in a minor release of + `vitaminc-ore`. Until then it is an entry in the exceptions list. +- **Block ORE for `Bool`.** Block ORE stored as right ciphertexts only is + fully randomised and semantically secure, so a two-value domain leaks + nothing through it. `Bool` could support that scheme alone, enforced by a + marker trait on schemes. Not built until needed. +- **Locale-aware collation**, as its own domain with the collation version in + its name. +- **ASCII-packed text domains**, until a customer's column sizes make the + case. diff --git a/docs/plans/2026-10-04-plan-builder.md b/docs/plans/2026-10-04-plan-builder.md index 88dd628fb..e70ea3b11 100644 --- a/docs/plans/2026-10-04-plan-builder.md +++ b/docs/plans/2026-10-04-plan-builder.md @@ -662,9 +662,11 @@ In the `Text` family, the three `Ord` suffixes carry equality too. Each type has a query type, with `Query` after its name: `TextEqQuery`. The value of `encrypt_into` is the Go type name. -An EQL value has the EQL v3 envelope: its version field is `3`, and Postgres stores it in an `eql_v3` domain. +An EQL value from the engine is an EQL v4 value: its version field is `4`, and Postgres stores it in an `eql_v4_*` domain. The ciphertext inside it is a Stack Encrypt ciphertext, which starts with `stack-encrypt:1:`. -"EQL v4" in this plan is the name of that form, and not a new envelope. +EQL v4 is the v3 SQL, emitted from the same source under a second name, so that a column written by Stack Encrypt and one written by cipherstash-client are different Postgres types ([ADR-0002](../adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md)). +This replaces this plan's first definition, under which "EQL v4" named a Stack Encrypt payload in the v3 envelope and an `eql_v3` domain; Postgres could not tell the two producers apart, and a query from one against a column written by the other matched nothing. +The `TextEq` described below is still produced into an `eql_v3` domain until that move lands. The engine produces one EQL type today: `TextEq`. Status (2026-10-06, #1062): `TextEq` is producible through the data plan's target form. @@ -1157,9 +1159,10 @@ model rather than a strain: it puts key material in the database. ## EQL v4 types as field targets -Naming: **EQL v4** is the EQL form of a stack-encrypt payload. **EQL v3** is -the existing SQL bundle and its `eql_v3_*` domains, which this section does -not change. Depends on #971 (`TextEq` / `TextEqQuery` through stack-encrypt +Naming: **EQL v4** is the EQL form of a stack-encrypt payload: the v3 SQL +emitted under a second name, with `eql_v4_*` domains and `"v": 4` envelopes +(ADR-0002). **EQL v3** is the same SQL holding cipherstash-client payloads in +its `eql_v3_*` domains, which this section does not change. Depends on #971 (`TextEq` / `TextEqQuery` through stack-encrypt in `eql-bindings`, and `Identifier` as a two-segment `Label`). **The engine returns an EQL type only when a plan names it as a target.** From 2fb728bff66a9a8dc028033cb5f1e95baca70d86 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 11 Oct 2026 17:01:10 +1100 Subject: [PATCH 2/2] docs(plans): put vitaminc at the bottom of the order and equality term stack Starting vitaminc-ore under ADR-0002 surfaced four problems: order encodings, scheme traits and term derivation spread over three repositories; scheme crates that accept raw, hand-encoded bytes; vitaminc-prf reaching orderable-bytes through aead-value's canonical module; and one canonical form shared by equality and order, which rules out leakage-tuned domains such as an exact timestamp for equality with day resolution for order. The plan settles the replacement design: vitaminc-ore owns the order encodings and an OrderScheme trait with a sealed input, and ore-rs and cllw-ore implement it; each term crate applies a semantic baseline and per-domain transforms named in its labels; PRF has no ordering dependency; orderable-bytes is frozen. Exhaustive matching over Value is left open, with both approaches and their shared prerequisite laid out. It includes current and target dependency diagrams and the steps, and ADR-0002 now points to it. Claude-Session: https://claude.ai/code/session_012zHhtTEo968WazExKxA5oT --- ...-three-encodings-and-eql-v4-by-producer.md | 9 + ...10-11-order-and-equality-terms-layering.md | 331 ++++++++++++++++++ 2 files changed, 340 insertions(+) create mode 100644 docs/plans/2026-10-11-order-and-equality-terms-layering.md diff --git a/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md index b5f858b10..33a20ee88 100644 --- a/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md +++ b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md @@ -6,6 +6,15 @@ relates-to: ADR-0001; stack-encrypt ADR-0003, ADR-0007 # One value model, three encodings in vitaminc, and EQL v4 separated by producer +> **Amended 2026-10-11** by +> [`docs/plans/2026-10-11-order-and-equality-terms-layering.md`](../plans/2026-10-11-order-and-equality-terms-layering.md). +> Where the two disagree, the plan wins. In short: vitaminc sits at the +> bottom of the stack and the ORE schemes implement `vitaminc-ore`'s trait; +> equality and order terms take their own per-domain transforms instead of +> one shared canonical form; `vitaminc-prf` no longer depends on order +> encodings; and `orderable-bytes` is frozen. The plan lists every statement +> below that it replaces. + Every value Stack Encrypt handles is encoded three times: as a **ciphertext** (reversible and self-describing), as an **equality term** (a keyed hash that must be unambiguous) and as an **order term** (bytes whose order is the diff --git a/docs/plans/2026-10-11-order-and-equality-terms-layering.md b/docs/plans/2026-10-11-order-and-equality-terms-layering.md new file mode 100644 index 000000000..54f3d6947 --- /dev/null +++ b/docs/plans/2026-10-11-order-and-equality-terms-layering.md @@ -0,0 +1,331 @@ +# Order and equality terms: vitaminc at the bottom + +> **Plan, not specification.** This document is an indicative sketch of the +> steps required, written before the work was done. It is not kept in step +> with the implementation and must not be used as a formal specification or +> as a reference for reviewing what was actually built: the code, its rustdoc +> and the tests are the source of truth. Where the two disagree, the code wins +> and this document is simply out of date. + +**Status:** planned. Decisions settled 2026-10-11, except the one listed +under "Open decision". +**Date:** 2026-10-11 +**Amends:** ADR-0002 (`docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md`). +Where the two disagree, this document wins. +**Issues:** cipherstash/vitaminc#374 (`vitaminc-ore`), #1063 (index crates) + +## Why + +Starting on cipherstash/vitaminc#374 under ADR-0002 turned up four problems. + +- **Too many dependencies across repositories.** `vitaminc-ore` was to depend + on cllw-ore and ore-rs and adapt each to its `Scheme` trait, with + `orderable-bytes` underneath both. Order encodings, scheme traits and term + derivation ended up spread over three repositories, and every change needed + releases in all of them. +- **Raw bytes reach the schemes.** cllw-ore and ore-rs accept bytes that the + caller has already encoded. Bytes encoded by hand (little-endian integers, + an unflipped sign bit, `f64::to_bits`) encrypt without error and sort + wrong. Fixing that inside cllw-ore needed wrapper types in `orderable-bytes` + (cipherstash/ore.rs#103) and a feature split in cllw-ore + (cipherstash/cipherstash-suite#2308). Cargo features unify across a build, + so the feature split could not hold whenever anything else in the build + turned the feature on. +- **PRF depends on ordering.** `vitaminc-prf` reaches `orderable-bytes` + through `vitaminc-aead-value`'s `canonical` module, which reuses order + encodings as equality input. Equality needs a canonical form, not an order. +- **One canonical form for both term kinds is too rigid.** ADR-0002 builds + equality and order terms from the same canonical form. A leakage-tuned + domain needs them to differ, for example an exact timestamp for equality + and only the day for order. + +## Decisions + +### vitaminc is the bottom of the stack + +- `vitaminc-ore` owns: + - the order encodings; + - an `OrderScheme` trait; + - the derivation of order terms from `&Value`, generic over the scheme. + + It depends on no scheme crate. +- A scheme's input is a sealed type, `OrderPlaintext`, that only + `vitaminc-ore` can construct. It comes in fixed-length and variable-length + forms. A scheme can't be given bytes that skipped the encoding, so no + wrapper types or feature flags are needed to keep raw input out. +- `OrderScheme` returns a scheme-defined term, roughly + `type Term: Ord + AsRef<[u8]>`, whose length is only known at runtime. Each + scheme keeps its own comparison (CLLW ORE and block ORE don't compare byte + by byte) and its own stored bytes. Fixed-width types such as + `OreCllw8V1` stay as inherent APIs on the scheme crates, because stable + Rust can't name their width generically. +- The trait also covers the key type, domain separation (the term's domain + label goes in as the salt) and typed errors. +- cllw-ore and ore-rs depend on `vitaminc-ore` and implement `OrderScheme` + behind an optional `vitaminc` feature, so cipherstash-client doesn't take a + vitaminc dependency. Their existing typed APIs stay as they are, so + cipherstash-client's stored terms don't change. +- `vitaminc-ore` releases in lockstep with the other vitaminc crates. On each + vitaminc minor release, cllw-ore and ore-rs bump their optional + `vitaminc-ore` dependency before stack can upgrade. That is acceptable + while stack is the only user of the trait path. +- `vitaminc-ore`'s tests use a mock scheme defined inside the crate. A + dev-dependency on cllw-ore would form a cycle across repositories and pull + a second `vitaminc-ore` in from crates.io. + +### Equality and order are tuned per domain + +- There is no shared canonical form. Each kind has a **semantic baseline**, + which makes values that are equal under `Value`'s own semantics encode the + same: decimal scale (`1.0` = `1.00`), signed zero, NaN, NFC. +- On top of the baseline, each term's domain chooses its own **lossy + transforms**, such as timestamp resolution (exact, microsecond, day), + text folding (removing combining marks, case folding, with the Unicode + version pinned) and prefix truncation. +- Each transform is named in its term's domain label, for example + `…/timestamp/day/v1`, so terms built with different transforms can never + be compared. +- Equality and order can be **independent** (an exact timestamp for + equality, the day for order) or **locked** (the same transform value + passed to both). Stack's index spec (#1063) carries the choice per column. +- The semantic baseline lives in each term crate: `vitaminc-prf` and + `vitaminc-ore` each apply it. A conformance test checks that values equal + under `Value` semantics produce equal terms in both. There is no shared + canonicalisation crate. +- A column's leakage is the combined leakage of all its terms. Truncating the + order term to the day hides the order of events within a day, but an exact + equality term on the same column still reveals which rows share a + timestamp. Domain design must account for the combination. + +### PRF has no ordering dependency + +- `vitaminc-prf` depends only on `vitaminc-aead-value` and applies its own + baseline and transforms. +- `vitaminc-aead-value` does no term normalisation. Its `canonical` module and + its dependency on `orderable-bytes` are removed. +- The value-derived PRF domains (cipherstash/vitaminc#380) haven't been + released: they landed on 2026-10-08, after vitaminc 0.5.1. So `vitaminc-prf` + can choose its own canonical bytes for dates, timestamps and decimals + without affecting any stored term. + +### `orderable-bytes` is frozen + +- `orderable-bytes` stays as it is for cipherstash-client's stored terms. It + is not part of the target design. +- There is no 0.3. cipherstash/ore.rs#103 (binary wrapper types) is closed. +- 0.2.0 is not yanked: ore-rs 0.8.4 depends on it and nothing new needs it + gone. + +## Open decision: exhaustive matching over `Value` + +Once `canonical.rs` leaves `vitaminc-aead-value`, `vitaminc-prf` and +`vitaminc-ore` must each handle every kind of `Value`, and adding a kind must +fail to compile until each one does. Two approaches are on the table. + +### Prerequisite for either approach: variants that exist in every build + +`Value::Date` and `Value::Timestamp` exist only with `vitaminc-aead-value`'s +`chrono` feature, and `Value::Decimal` only with its `rust_decimal` feature. +Because features unify across a build, an exhaustive match outside +`vitaminc-aead-value` breaks: + +1. `vitaminc-prf` matches `Value` exhaustively, with its `Date` arm behind its + own `chrono` feature. +2. In a build where another crate turns on `vitaminc-aead-value/chrono` and + `vitaminc-prf`'s `chrono` is off, `Value::Date` exists and `vitaminc-prf` + has no arm for it. `vitaminc-prf` fails to compile in that build. + +A visitor fails the same way: a required `visit_date` that exists only with +`chrono` is a method an implementor can't know whether to write. A crate can +check its own features, never a dependency's. Today's design avoids this only +because the match sits inside `vitaminc-aead-value`, which checks its own +features. + +The fix is to make every variant exist in every build, using vitaminc's own +representation, with conversions to and from chrono and rust_decimal behind +the features: + +| Variant | Representation | +|---|---| +| `Date` | days from the start of the common era, `i32` | +| `Timestamp` | Unix seconds `i64`, nanoseconds `u32` | +| `Decimal` | mantissa and scale | + +The ciphertext wire format already uses these payloads (tags `0x12`–`0x14` +are independent of chrono), so stored ciphertexts don't change. The Rust +types do, which is a breaking change to `Value`. + +### The two approaches + +**A: drop `#[non_exhaustive]`.** `vitaminc-prf`, `vitaminc-ore` and the +bindings match `Value` exhaustively in their own crates. + +**B: a visitor trait, keeping `#[non_exhaustive]`.** `vitaminc-aead-value` +defines a `ValueVisitor` with one required method per kind and does the +dispatch. Crates that must handle every kind implement it. Other crates keep +matching with a `_ =>` arm. + +| | A: drop `#[non_exhaustive]` | B: visitor, keep `#[non_exhaustive]` | +|---|---|---| +| Who gets exhaustiveness | Every crate that matches `Value` | Crates that implement the visitor | +| Adding a kind | Breaking for every crate that matches | Breaking for visitor implementors, additive for others | +| Code to maintain | Nothing new | One trait, one dispatch function, a method per kind in each implementor | +| Grouping | Each match writes out every integer width | The trait can offer grouped hooks, such as one integer method that the per-width methods default to | +| Changing `Value`'s representation | Breaks every match | Hidden behind the method signatures | +| Main risk | Low | A default on a per-kind method lets a new kind compile without being handled. Per-kind methods need no defaults; only grouping hooks may have them. | + +**Evidence:** +- Outside `vitaminc-aead-value`, `Value` is matched in `vitaminc-prf` (through + `canonical`), in `aead-napi`'s `convert.rs` and `scalar.rs`, and in the hmac + tests. +- `aead-napi` already has `_ => Err("unsupported value kind")` + (`convert.rs`). That is the silent refusal `#[non_exhaustive]` invites. +- ADR-0002 requires every binding to handle every kind, so the bindings want + exhaustiveness as well. +- Under A they get it automatically. Under B each binding opts in by + implementing the visitor. + +**Leaning:** B. The decision waits on a closer look at the `Value` enum's +design, which may change the representation in ways that favour one +approach. + +## Current state + +Arrows point from a crate to what it depends on. cipherstash-client and +`vitaminc-protected` are left out. + +```mermaid +flowchart TB + subgraph stack + SE[stack-encrypt] + end + subgraph suite["cipherstash-suite"] + CLLW["cllw-ore 0.5
typed impls, own encodings"] + end + subgraph vitaminc + PRF[vitaminc-prf] + AV["vitaminc-aead-value
Value, canonical module"] + end + subgraph orers["ore.rs"] + ORERS["ore-rs 0.8"] + OB["orderable-bytes"] + end + SE --> PRF + SE --> AV + SE --> CLLW + PRF -- "canonical feature" --> AV + AV -- "git pin, pre-0.2" --> OB + CLLW -- "0.1, chrono and decimal" --> OB + ORERS -- "0.2" --> OB +``` + +- stack-encrypt calls cllw-ore directly for order terms. There is no block + ORE path. +- Equality bytes for dates, timestamps and decimals come from order + encodings, through `vitaminc-aead-value`. +- `orderable-bytes` is reached three ways, at three different versions. + +## Target state + +```mermaid +flowchart TB + subgraph stack + EQI["Equality index
picks transforms"] + ORI["Ore / Ope index
picks scheme and transforms"] + end + subgraph schemes["scheme crates"] + ORERS["ore-rs
block ORE"] + CLLW["cllw-ore
CLLW ORE and OPE"] + end + subgraph vitaminc + PRF["vitaminc-prf
equality terms, own baseline and transforms"] + VORE["vitaminc-ore
order encodings, transforms,
OrderScheme, sealed OrderPlaintext"] + AV["vitaminc-aead-value
Value"] + end + EQI --> PRF + ORI --> VORE + ORI --> ORERS + ORI --> CLLW + ORERS -- "impl OrderScheme" --> VORE + CLLW -- "impl OrderScheme" --> VORE + PRF --> AV + VORE --> AV +``` + +- vitaminc depends on nothing in the other repositories. +- The scheme crates implement `vitaminc-ore`'s trait. The index crates choose + a concrete scheme and the transforms for each column. +- `orderable-bytes` doesn't appear. It stays frozen for cipherstash-client. + +## Changes to ADR-0002 + +| ADR-0002 says | Now | +|---|---| +| "Equality and order terms are computed from one canonical form per kind, and both layers use the same one" | Each kind has a semantic baseline; lossy transforms are chosen per domain, independently or locked | +| Canonical forms table: timestamp truncated to microseconds, text order folded | These become per-domain transforms, named in each domain's label | +| Order: "The plaintext layer is `orderable-bytes`, which … gains a variable-length encoding" | `vitaminc-ore` owns the order encodings; `orderable-bytes` is frozen | +| "A `Scheme` trait, with block ORE (over ore-rs), CLLW ORE and CLLW OPE (over cllw-ore)" | `OrderScheme` lives in `vitaminc-ore`; ore-rs and cllw-ore implement it | +| "cllw-ore keeps its typed impls … behind a cargo feature that cipherstash-client enables, and gains a bytes-level entry point" | cllw-ore's typed API is unchanged and not feature-gated; it gains an `OrderScheme` impl behind an optional `vitaminc` feature | +| Review: "Using the canonical order bytes as the PRF input … Dan Draper signs it off" | PRF no longer uses order bytes; the review item applies only to domains that lock equality and order together | +| `Value` and `ValueKind` become `#[non_exhaustive]` | Open: see "Open decision" | +| Release order: "ore.rs (`orderable-bytes`) and cllw-ore first, then vitaminc 0.6.0" | vitaminc first, then cllw-ore and ore-rs, then stack | + +## Steps + +### Phase 1: tidy up work in flight + +1. Close cipherstash/ore.rs#103. Don't yank `orderable-bytes` 0.2.0. +2. Cut cipherstash/cipherstash-suite#2308 down to its first two commits: + known-answer tests that pin every cllw-ore encrypt path, and wiping the + plaintext bit buffer in `encrypt_ope_bits`. It becomes a cllw-ore patch + release. The known-answer tests then protect cipherstash-client's terms + through the rest of this work. + +### Phase 2: vitaminc + +3. Settle the open decision. If the `Value` changes are accepted, make every + variant exist in every build. +4. `vitaminc-aead-value` and `vitaminc-prf`: + - add the chosen exhaustiveness mechanism; + - remove `canonical.rs` and the `orderable-bytes` dependency from + `vitaminc-aead-value`; + - move the equality baseline and transforms into `vitaminc-prf`, with each + transform named in its domain label. +5. New crate `vitaminc-ore` (rewrite cipherstash/vitaminc#374 to match): + - `OrderScheme` and the sealed `OrderPlaintext`; + - order encodings ported from `orderable-bytes`, kept in `Protected`, + pinned by golden vectors, and checked against `orderable-bytes` where + they match byte for byte; + - per-domain transforms: timestamp resolution, text folding with a pinned + Unicode version, room for prefix truncation; + - order-term derivation from `&Value`, generic over the scheme; + - a mock scheme for tests; + - docs that state the combined-leakage rule. +6. A conformance test across `vitaminc-prf` and `vitaminc-ore`: every kind is + handled or listed as an exception, and values equal under `Value` + semantics produce equal terms. +7. Release vitaminc 0.6.0. + +### Phase 3: scheme crates + +8. cllw-ore: implement `OrderScheme` behind an optional `vitaminc` feature, + with known-answer tests for the trait path. This is an additive release. +9. ore-rs: the same. Block ORE uses each kind's natural width; + cipherstash-client keeps its 8-byte padding through the existing API. + Block ORE for text waits for cipherstash/ore.rs#95 and goes on the + exceptions list. +10. Release both. + +### Phase 4: stack + +11. Index crates (#1063): each index spec carries its scheme and transforms. + The Equality index calls `vitaminc-prf`; the Ore and Ope index builds the + concrete scheme and calls `vitaminc-ore`. stack-encrypt drops its direct + use of cllw-ore. +12. EQL v4 domain labels name their transforms. + +### Later + +- Block ORE for text, once cipherstash/ore.rs#95 is released. +- Retire cllw-ore's typed impls and `orderable-bytes` once cipherstash-client + is retired.