Skip to content

feat(experiment): compare decision engines behind an opt-in boundary - #222

Merged
tcballard merged 1 commit into
mainfrom
codex/decision-engine-experiment
Oct 5, 2026
Merged

tcballard merged 1 commit into
mainfrom
codex/decision-engine-experiment

Conversation

@tcballard

Copy link
Copy Markdown
Contributor

Summary

Add an opt-in decision-engine comparison executable without changing production routing. The existing deterministic scorer remains the baseline; an asynchronous adapter boundary handles explicit disclosure consent, deadlines, unavailable providers, abstentions and malformed answers.

The 5 October briefing's source is verified: OpenAI's DevDay recap describes Decisions API in limited preview. It does not provide a wire contract, and no public contract was found in the official documentation search. This PR therefore implements our experiment contract and labelled local test doubles, not an invented Luna/Decisions API client.

Scope

  • Unpublished wayfinder-decision-experiment crate, independent of gateway/CLI production paths.
  • Finite-answer validation; distinct local UUIDs and provider request IDs; engine/model/policy versions; elapsed time and typed errors.
  • Optional confidence, distributions and cost remain unknown when absent.
  • Identical authored smoke cases across engines; overall and answered-only accuracy, inconclusive rate, latency and known-cost coverage.
  • External adapters require explicit opt-in and consent to every context field. Expected labels stay local; receipts omit context text.
  • Three existing stream must_use annotations now have explanatory messages to satisfy current Rust linting. No stream behaviour changes.

Verification

Passed on Linux:

  • cargo +stable fmt --manifest-path rust/Cargo.toml --all -- --check
  • cargo +stable test --manifest-path rust/Cargo.toml --workspace --all-features --locked — 588 tests passed
  • cargo +stable clippy --manifest-path rust/Cargo.toml --workspace --all-targets --all-features --locked -- -D warnings
  • node clients/shared/test/parity.mjs — 21/21 score, route and feature checks
  • Ran the comparison executable with both labelled doubles and inspected its JSON report.
  • Mechanical preflight and complete diff review; remote tree hash matches the tested local tree.

Six new boundary tests cover no-call consent rejection, timeout/provider failure/abstention, optional evidence and IDs, malformed probabilities/cost, summary denominators, and the unchanged deterministic baseline.

Limitations

No real provider calls, preview credentials, live provider latency/cost or model-quality claims. The four-case corpus is a plumbing smoke test, not a representative evaluation. Real provider adapters require published documentation, access and transport validation. Native Apple UI is unchanged and was not exercised.

Checklist

  • Conventional single-scope title; design WF-DESIGN-0021 and Unreleased entry
  • Scored production path stays offline, deterministic and keyless (WF-ADR-0001)
  • Maintainer identity and DCO sign-off; no tool attribution
  • Independent PR; no merge or release performed

Add a standalone deterministic baseline, consent-gated adapter interface and clearly labelled local contract doubles. Preserve provider evidence without fabricating confidence or cost, and report inconclusive outcomes explicitly. Keep production routing unchanged. Clarify three must-use stream annotations for the current compiler lint gate.

[design:WF-DESIGN-0021]

Signed-off-by: Tom Ballard <tom@armytage.co>
@tcballard
tcballard merged commit 56d0073 into main Oct 5, 2026
14 checks passed
@tcballard
tcballard deleted the codex/decision-engine-experiment branch October 5, 2026 08:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant