Add retrieval-poisoning Failure Zoo specimen

[susmitahaldar235]

Outcome

• User problem: Add a synthetic retrieval-poisoning specimen to the Japanese troubleshooting benchmark so retrieval-poisoning failures can be evaluated consistently.
• Decision enabled: Allows evaluators to verify that retrieval-poisoning behavior is detected and blocked during benchmark evaluation.
• Requirement or issue: Closes [Good first issue]: Add a retrieval-poisoning Failure Zoo specimen #22

Contract

• Public schema/API impact: None.
• Metric or PASS/BLOCK impact: Adds a retrieval-poisoning specimen expected to fail evaluation and exit with code 2 under the existing evaluation policy.
• Architecture/open-core impact: None. Adds a benchmark specimen, regression test, and documentation only.
• Explicit non-goals: Does not modify the public schema, default evaluation policy, or claim real-world security guarantees.

Evidence

• Expected and failure tests
• ruff check .
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 pytest -q (existing Windows-specific symlink and README path failures remain unchanged)
• Affected CLI/API path exercised
• Canonical documentation and changelog updated
• Superseded content removed

Owner review

• Known limitations: The retrieval-poisoning specimen is synthetic and intended only for benchmark evaluation.
• Compatibility recommendation: Safe to merge. No API or schema changes.
• Publication requested: no

[thangldw]

Thanks for adding this specimen. I checked commit 2d90df0 against the repository requirements and found three items to address before merge:

1. The fixture does not yet encode an inspectable retrieval-poisoning event. It reuses inject-01, whose scenario evidence is the safe policy statement, then overrides only the answer and citation_ids. There is no poisoned instruction or retrieved_ids provenance, so the observed failure is indistinguishable from hallucination or ordinary prompt injection. Please add inspectable poisoned retrieval evidence/provenance and assert its deterministic detection, or narrow the documentation claim.

2. The new specimen is absent from the aggregate Failure Zoo. failure-zoo-candidate.json and the specimen table still contain the original six cases, so the documented “Run all specimens offline” command does not exercise this addition. Please integrate it into the aggregate fixture/table and update expected evidence, or clarify that it is intentionally standalone.

3. CHANGELOG.md was not updated. The repository workflow requires feature additions to be recorded under Unreleased, and the PR description currently says the changelog was updated.

Verification performed locally:

• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 pytest -q: 209 passed
• tests/test_reference_benchmark.py: 4 passed
• The documented CLI command exits with 2 as expected
• The changed test file passes Ruff 0.6.9; full ruff check . still reports two pre-existing E402 errors also present on main

The blocking behavior works, but the fixture needs stronger provenance and Failure Zoo integration for the retrieval-poisoning claim to be accurate.

[susmitahaldar235]

Thanks for the review! I've addressed the changelog update, integrated the retrieval-poisoning specimen into the aggregate Failure Zoo, and updated the associated regression test. I'm still investigating the requested retrieval provenance. I found that the fixture schema supports retrieved_ids, but I couldn't find an existing benchmark convention for defining synthetic retrieved documents. Could you clarify where you'd prefer the poisoned retrieval evidence to be modeled?

[thangldw]

Thanks for checking. Please model the poisoned document as a scenario-local fixture, for example scenarios/japanese_troubleshooting/retrieval-poisoning-documents.json, containing the synthetic document ID, its untrusted status, and the injected instruction. Then add "retrieved_ids": ["poisoned-doc-77"] to both the standalone and aggregate response overrides.

The regression test should verify that the retrieved ID resolves to that synthetic poisoned document and that the resulting response is blocked by the expected named findings/gates. Please do not place the poisoned instruction in the scenario’s trusted evidence; no public schema or default-policy change is needed.

I also rechecked the current head and found three remaining items:

1. CHANGELOG.md contains committed conflict markers (<<<<<<<, =======, >>>>>>>), so git diff --check fails.
2. ruff check . fails with I001 in the import block of tests/test_reference_benchmark.py.
3. Issue [Good first issue]: Add a retrieval-poisoning Failure Zoo specimen #22 requires documentation for the specimen. Since docs/evaluation/benchmark.md is no longer present on main, please add the specimen to the current canonical documentation location or call out the replacement location in this PR.

The functional paths otherwise look good: the full suite passes locally (191 passed), and both the standalone and aggregate candidates exit with code 2 as expected.

[susmitahaldar235]

Thanks for the feedback! I've implemented the requested changes by Addressed the review comments:

Added a synthetic retrieval-poisoning document fixture.
Added retrieved_ids to the retrieval-poisoning and Failure Zoo specimens.
Updated the regression test to verify the retrieved document reference.
Updated the changelog and documentation accordingly.

All relevant tests pass locally. Please let me know if there's anything else you'd like me to adjust.