[BigSimmo]

User description

Summary

• Before: the answer-quality canary reports which files an answer used, but not why they landed in that order. After PR fix(retrieval): rank a document first when a lookup names its whole title #3572 went live, assessment-documentation still lists Assessment Documentation (AKG) 4th behind Admissions, Radiotherapy and Pressure Injury. An offline reproduction keeps AKG first, so a step after selection on the live path is reordering, and nothing recorded shows which.
• After: the bounded per-source diagnostics also carry ranking (hybrid, rank, release-rank and final scores, final rank, and pipeline match reasons such as retrieval_signal:document_title_phrase or second_stage_rerank). A new dump_rag_cases dispatch payload passes the existing --dump-rag-cases flag, so the file lands in the canary artifact.
• How: buildBoundedRagSourceDiagnostics in scripts/eval-answer-quality.ts gains the ranking block. Reasons are bounded to 48 and passed through displaySafeDiagnosticText. eval-canary.yml adds DUMP_RAG_CASES to the answer step, which is empty for the weekly schedule. The dump is written before any --fail-on-threshold exit.

Areas touched: Answer engine
RAG impact: no retrieval behaviour change — eval diagnostics and an opt-in canary artifact only

Verification

• Checks run:
  • npx vitest run tests/eval-answer-quality.test.ts: Tests 23 passed (23), including the new ranking-diagnostics test.
  • npm run lint (changed files): recorded a pass for "lint:internal"
  • npm run typecheck: recorded a pass for "typecheck:internal"
  • npm run test: Tests 31524 passed | 8 skipped (31532)

Risk and rollout

• Risk: low. This is eval tooling only, and the workflow switch is off unless a dispatch sets dump_rag_cases: "true".
• Rollback: revert this commit.
• Provider or production effects: none from merging. A dispatched canary run with the switch on is paid (about US$1 to 2) and needs the owner's named yes.

Notes

• Kept as a draft until the owner says go.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VkKjYakMrMXofU8foQCgbq

---

Generated by Claude Code

---

Summary by cubic

Records each answer source's ranking scores and pipeline match reasons in the eval canary diagnostics, so a live source misorder (the AKG guideline still ranked 4th after PR #3572) can be diagnosed offline.

Diagnostics

• buildBoundedRagSourceDiagnostics now emits a ranking block: answer-evidence score, hybrid, rank, release-rank, and final scores, final rank, and match reasons (capped at 48 and sanitized).
• The answer-evidence score re-scores the emitted sources with the answer-stage ranking that set their order, since rankAnswerEvidence can reorder sources after retrieval selection; it explains that reorder, not later context-selection moves.
• eval-canary.yml adds a DUMP_RAG_CASES env var that passes --dump-rag-cases only when a dispatch sets dump_rag_cases: "true"; the weekly schedule stays off.
• Eval tooling only — no retrieval or answer behavior change.

^{Written for commit df28429. Summary will update on new commits.}

---

CodeAnt-AI Description

Add opt-in source-ranking details to answer-quality canary dumps

What Changed

• Each answer source’s diagnostic details now show its ranking scores, final position, and reasons for matching.
• Dispatch runs can optionally save per-case answers and ranking details as a canary artifact; scheduled runs remain unchanged.

Impact

✅ Easier diagnosis of misordered answer sources
✅ Offline review of canary ranking decisions
✅ No extra answer generation for diagnostics

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

[coderabbitai]

Warning

Review limit reached

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

Next included review available in 38 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available. Your 100 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Configuration used: Repository: BigSimmo/PsychSift/.coderabbit.yaml
• Review profile: CHILL
• Plan: Team
• Run ID: 2b08046b-29ae-4b43-86b1-460624df1bd1

📥 Commits

Reviewing files that changed from the base of the PR and between 37217d7 and df28429.

📒 Files selected for processing: 3

• .github/workflows/eval-canary.yml
• scripts/eval-answer-quality.ts
• tests/eval-answer-quality.test.ts

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[codeant-ai]

🏁 CodeAnt Quality Gate Results

Commit: df28429d
Scan Time: 2026-10-16 13:53:13 UTC

✅ Overall Status: PASSED

Quality Gate Details

Quality Gate	Status	Details
Secrets	✅ PASSED	0 secrets found
Duplicate Code	✅ PASSED	0.0% duplicated
SAST	✅ PASSED	No security issues
Bugs	✅ PASSED	Rating S: No bugs
IAC	✅ PASSED	No IAC issues

View Full Results

[gitar-bot]

Gitar has auto-approved this PR (configure)

[greptile-apps]

BigSimmo has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

[codeant-ai]

🤖 CodeAnt AI — Review Status

Status	Commit	Started (UTC)	Finished (UTC)
✅ Reviewed your PR	e48badf	Oct 16, 2026 · 13:12	13:22

[codeant-ai]

---

Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

[claude]

Claude encountered an error after 2s —— View job

---

I'll analyze this and get back to you.

[qodo-code-review]

PR Summary by Qodo

Capture answer-source ranking diagnostics in opt-in canary artifacts

✨ Enhancement 🧪 Tests ⚙️ Configuration changes 🕐 20-40 Minutes

AI Description

• Record bounded scores, final ranks, and pipeline reasons to diagnose misordered answer sources.
• Let canary dispatches opt into per-case dumps without changing scheduled runs.
• Test populated and missing ranking diagnostics.

Diagram

graph TD
  A["Canary trigger"] --> B{"Dump requested?"} -->|yes| C["RAG eval dump"] --> D["Bounded ranking data"] --> E["Canary artifact"]
  B -->|no| F["Standard eval"] --> E

Loading

High-Level Assessment

Reusing the existing bounded diagnostics and --dump-rag-cases path is appropriate: it exposes ordering evidence without a second evaluation pipeline or a change to answer behavior.

Files changed (3) +77 / -2

Enhancement (1) +17 / -0

eval-answer-quality.tsAdd bounded ranking evidence to source diagnostics +17/-0

Add bounded ranking evidence to source diagnostics

• Adds hybrid, rank, release-rank, and final scores, final rank, and pipeline match reasons to each diagnostic source. Scores are rounded or set to null; reasons are capped and passed through the existing display-safe sanitizer.

scripts/eval-answer-quality.ts

Tests (1) +50 / -0

eval-answer-quality.test.tsTest ranking diagnostics for populated and bare sources +50/-0

Test ranking diagnostics for populated and bare sources

• Verifies score rounding, final rank, and match reasons for a ranked source, plus null scores and empty reasons when ranking metadata is absent.

tests/eval-answer-quality.test.ts

Other (1) +10 / -2

eval-canary.ymlEnable case dumps through an opt-in dispatch payload +10/-2

Enable case dumps through an opt-in dispatch payload

• Passes --dump-rag-cases to the answer-quality step only when dump_rag_cases is "true". The dump is written under the existing canary artifact directory; scheduled runs remain unchanged.

.github/workflows/eval-canary.yml

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e48badf512

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-16T13:16:38.337283Z	e48badf	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[qodo-code-review]

Qodo Fixer

No findings are available for this PR yet. Findings appear here once Qodo has reviewed the PR.

[gitar-bot]

Gitar has auto-approved this PR (configure)

[codeant-ai]

CodeAnt PR Risk: Low Risk

• The PR appears safe to merge; diagnostic case dumps remain opt-in and are enabled only by an explicit dispatch value.
• The dump adds bounded, sanitized ranking details, and tests cover both retrieval scores and answer-stage ordering.

Assessed commit: 2c289f4efbb3

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

No findings for this group

No findings for this group

No findings for this group

Resolved findings

1. Source scores misexplain answer order ✗ Dismissed

Description

buildRagDiagnosticDumpRecord reruns rankAnswerEvidence on emitted sources and presents those
scores as explaining their positions, even though production can reorder sources during context
selection and Australian-authority ordering. When that later stage moves equally relevant guidance
or applies coverage ordering, a higher-scoring source can appear below a lower-scoring one without
the dump recording why.

Code

scripts/eval-answer-quality.ts[286]

+  const answerEvidenceScores = rankAnswerEvidence(question, answer.sources, answer.queryClass).scoresByChunkId;

Relevance

●● Moderate

The PR addresses answer-stage ordering, but evidence does not establish whether selection-stage
provenance is required.

PR-#3537

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Production ranks search results before selecting generation context, and context selection can
change their order, including through an order-only Australian preference in the legacy path. The
diagnostic instead obtains fresh scores from the already-selected sources while deriving each rank
from the unchanged answer.sources order, so its scores cannot explain selection-stage moves or
fully trace the final ranking and provenance.

scripts/eval-answer-quality.ts[284-286]
src/lib/rag/rag.ts[2331-2349]
src/lib/rag/rag-context-selection.ts[55-90]
src/lib/retrieval-selection.ts[121-132]
src/lib/rag/rag.ts[2331-2347]
src/lib/rag/rag-context-selection.ts[61-103]
scripts/eval-answer-quality.ts[201-205]
Skill: ai-architecture-review

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The diagnostic presents a recomputed answer-evidence score as the explanation for emitted source order, although post-ranking context selection and authority ordering can change that order.

## Fix Focus Areas
- scripts/eval-answer-quality.ts[255-265]
- scripts/eval-answer-quality.ts[284-286]
- src/lib/rag/rag.ts[2338-2349]
- src/lib/rag/rag-context-selection.ts[55-103]

## Recommended Fix
Label the recomputed answer-evidence score as a pre-selection score, not the cause of the emitted rank. Carry or reconstruct the post-ranking context-selection order and its reasons, including any Australian-authority preference or coverage ordering, for each emitted source. Add a case where that stage changes the score order.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Interrupted canaries lose completed dumps ✗ Dismissed

Description

The new dispatch option requests a dump, but eval-quality writes it only after the entire
answer-quality run returns. If a later case exhausts its provider retries, diagnostics already
collected for earlier cases are never written, so the uploaded canary artifact lacks the requested
per-case records.

Code

.github/workflows/eval-canary.yml[R166-170]

+          dump_args=()
+          if [[ "$DUMP_RAG_CASES" == "true" ]]; then
+            dump_args=(--dump-rag-cases .local/eval-canary/rag-cases.json)
+          fi
+          npm run eval:quality -- --rag-only --limit "$ANSWER_CASE_LIMIT" --source-governance-results .local/eval-canary/golden-retrieval.json --output-dir .local/eval-canary/quality-reports --fail-on-threshold "${dump_args[@]}" 2>&1 | tee .local/eval-canary/answer-quality.log

Relevance

●●● Strong

PR #3537 explicitly accepted preserving completed diagnostics when retries throw.

PR-#3537

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The evaluation loop collects diagnostics as cases complete, but the file write follows the awaited
full run. Provider retry exhaustion throws, and the artifact step uploads only files that exist
while ignoring missing files.

scripts/eval-quality.ts[1316-1360]
scripts/eval-quality.ts[1594-1602]
scripts/eval-quality.ts[1679-1698]
.github/workflows/eval-canary.yml[207-221]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The opt-in canary dump is lost when an answer-quality case fails after earlier cases have completed.
## Fix Focus Areas
- .github/workflows/eval-canary.yml[166-170]
- scripts/eval-quality.ts[1316-1360]
- scripts/eval-quality.ts[1679-1698]
## Recommended Fix
Persist completed diagnostic cases incrementally or write the collected cases in an error-handling path before propagating the evaluation failure. Keep the canary job red on that failure.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Compliance rules (platform): 9 rules

Review mode: Auto: ⏭️ Skipped: Comment-only changes introduce no behavioral or reviewable defects.

Tip of the day

💡 Did you know, you can pick a display mode instead of tweaking every Display option by hand

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Previous reviews

Review updated until commit df28429

Results up to commit 2c289f4 ⚖️ Balanced

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

No findings for this group

No findings for this group

No findings for this group

Resolved findings

1. Source scores misexplain answer order ✗ Dismissed

Description

buildRagDiagnosticDumpRecord reruns rankAnswerEvidence on emitted sources and presents those
scores as explaining their positions, even though production can reorder sources during context
selection and Australian-authority ordering. When that later stage moves equally relevant guidance
or applies coverage ordering, a higher-scoring source can appear below a lower-scoring one without
the dump recording why.

Code

scripts/eval-answer-quality.ts[286]

+  const answerEvidenceScores = rankAnswerEvidence(question, answer.sources, answer.queryClass).scoresByChunkId;

Relevance

●● Moderate

The PR addresses answer-stage ordering, but evidence does not establish whether selection-stage
provenance is required.

PR-#3537

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Production ranks search results before selecting generation context, and context selection can
change their order, including through an order-only Australian preference in the legacy path. The
diagnostic instead obtains fresh scores from the already-selected sources while deriving each rank
from the unchanged answer.sources order, so its scores cannot explain selection-stage moves or
fully trace the final ranking and provenance.

scripts/eval-answer-quality.ts[284-286]
src/lib/rag/rag.ts[2331-2349]
src/lib/rag/rag-context-selection.ts[55-90]
src/lib/retrieval-selection.ts[121-132]
src/lib/rag/rag.ts[2331-2347]
src/lib/rag/rag-context-selection.ts[61-103]
scripts/eval-answer-quality.ts[201-205]
Skill: ai-architecture-review

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The diagnostic presents a recomputed answer-evidence score as the explanation for emitted source order, although post-ranking context selection and authority ordering can change that order.

## Fix Focus Areas
- scripts/eval-answer-quality.ts[255-265]
- scripts/eval-answer-quality.ts[284-286]
- src/lib/rag/rag.ts[2338-2349]
- src/lib/rag/rag-context-selection.ts[55-103]

## Recommended Fix
Label the recomputed answer-evidence score as a pre-selection score, not the cause of the emitted rank. Carry or reconstruct the post-ranking context-selection order and its reasons, including any Australian-authority preference or coverage ordering, for each emitted source. Add a case where that stage changes the score order.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Interrupted canaries lose completed dumps ✗ Dismissed

Description

The new dispatch option requests a dump, but eval-quality writes it only after the entire
answer-quality run returns. If a later case exhausts its provider retries, diagnostics already
collected for earlier cases are never written, so the uploaded canary artifact lacks the requested
per-case records.

Code

.github/workflows/eval-canary.yml[R166-170]

+          dump_args=()
+          if [[ "$DUMP_RAG_CASES" == "true" ]]; then
+            dump_args=(--dump-rag-cases .local/eval-canary/rag-cases.json)
+          fi
+          npm run eval:quality -- --rag-only --limit "$ANSWER_CASE_LIMIT" --source-governance-results .local/eval-canary/golden-retrieval.json --output-dir .local/eval-canary/quality-reports --fail-on-threshold "${dump_args[@]}" 2>&1 | tee .local/eval-canary/answer-quality.log

Relevance

●●● Strong

PR #3537 explicitly accepted preserving completed diagnostics when retries throw.

PR-#3537

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The evaluation loop collects diagnostics as cases complete, but the file write follows the awaited
full run. Provider retry exhaustion throws, and the artifact step uploads only files that exist
while ignoring missing files.

scripts/eval-quality.ts[1316-1360]
scripts/eval-quality.ts[1594-1602]
scripts/eval-quality.ts[1679-1698]
.github/workflows/eval-canary.yml[207-221]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The opt-in canary dump is lost when an answer-quality case fails after earlier cases have completed.
## Fix Focus Areas
- .github/workflows/eval-canary.yml[166-170]
- scripts/eval-quality.ts[1316-1360]
- scripts/eval-quality.ts[1679-1698]
## Recommended Fix
Persist completed diagnostic cases incrementally or write the collected cases in an error-handling path before propagating the evaluation failure. Keep the canary job red on that failure.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[qodo-code-review]

Code review by qodo was updated up to the latest commit df28429

[gitar-bot]

Gitar has auto-approved this PR (configure)

[gitar-bot]

Code Review ✅ Approved

🟡 Medium risk · Opt-in canary dumps now include per-source ranking diagnostics and answer details, which could expose eval content in retained workflow artifacts.

Adds a ranking block to the per-source diagnostics in scripts/eval-answer-quality.ts, recording hybrid, rank, release-rank and final scores plus bounded match reasons, and an opt-in DUMP_RAG_CASES dispatch input in eval-canary.yml that passes --dump-rag-cases. Eval tooling only, with no retrieval or answer behaviour change. No issues found.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Approved

_{For more verification details, comment gitar display:verbose.}

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Auto-apply	Compact
gitar auto-apply:on	gitar display:verbose

---

Important

Your trial ends in 5 days — upgrade now to keep code review, CI analysis, auto-apply, custom automations, and more.

_{Was this helpful? React with 👍 / 👎 | Gitar}