Skip to content

feat(evals): compare recorded routing thresholds - #886

Draft
cpakkamisaac-sae wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
cpakkamisaac-sae:feature/typesafe-threshold-sweep
Draft

cpakkamisaac-sae wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
cpakkamisaac-sae:feature/typesafe-threshold-sweep

Conversation

@cpakkamisaac-sae

Copy link
Copy Markdown
Contributor

Summary

  • Compare explicit routing thresholds offline against the same recorded decisions and outcomes.
  • Report fallback and target-selection counts, quality, cost, and regret for each threshold.
  • Preserve the existing single-threshold report; validate and sort comparison thresholds.

Stack

Depends on #883. This branch starts at its current head, so the draft diff includes that PR until it merges. After #883 merges, I will rebase this branch onto main and make this PR ready for review.

Closes #885.

Validation

  • Focused replay tests: 18 passed.
  • Available Python suite: 158 passed, 2 subtests passed.
  • Full Rust workspace tests, Clippy with warnings denied, formatting, Ruff, and mypy passed.
  • Docker integration tests were not run because the Docker daemon was unavailable.

Signed-off-by: Clement Pakkam Isaac <cpakkamisaac@nvidia.com>
Signed-off-by: Clement Pakkam Isaac <cpakkamisaac@nvidia.com>
Signed-off-by: Clement Pakkam Isaac <cpakkamisaac@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(evals): compare routing thresholds on recorded decisions

1 participant