Repository navigation
feat(experiment): compare decision engines behind an opt-in boundary - #222
Merged
Merged
Conversation
Add a standalone deterministic baseline, consent-gated adapter interface and clearly labelled local contract doubles. Preserve provider evidence without fabricating confidence or cost, and report inconclusive outcomes explicitly. Keep production routing unchanged. Clarify three must-use stream annotations for the current compiler lint gate. [design:WF-DESIGN-0021] Signed-off-by: Tom Ballard <tom@armytage.co>
11 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add an opt-in decision-engine comparison executable without changing production routing. The existing deterministic scorer remains the baseline; an asynchronous adapter boundary handles explicit disclosure consent, deadlines, unavailable providers, abstentions and malformed answers.
The 5 October briefing's source is verified: OpenAI's DevDay recap describes Decisions API in limited preview. It does not provide a wire contract, and no public contract was found in the official documentation search. This PR therefore implements our experiment contract and labelled local test doubles, not an invented Luna/Decisions API client.
Scope
wayfinder-decision-experimentcrate, independent of gateway/CLI production paths.must_useannotations now have explanatory messages to satisfy current Rust linting. No stream behaviour changes.Verification
Passed on Linux:
cargo +stable fmt --manifest-path rust/Cargo.toml --all -- --checkcargo +stable test --manifest-path rust/Cargo.toml --workspace --all-features --locked— 588 tests passedcargo +stable clippy --manifest-path rust/Cargo.toml --workspace --all-targets --all-features --locked -- -D warningsnode clients/shared/test/parity.mjs— 21/21 score, route and feature checksSix new boundary tests cover no-call consent rejection, timeout/provider failure/abstention, optional evidence and IDs, malformed probabilities/cost, summary denominators, and the unchanged deterministic baseline.
Limitations
No real provider calls, preview credentials, live provider latency/cost or model-quality claims. The four-case corpus is a plumbing smoke test, not a representative evaluation. Real provider adapters require published documentation, access and transport validation. Native Apple UI is unchanged and was not exercised.
Checklist