You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
planning eval case wayfind-prototype-files-on-dossier-folder-and-safe-push returns INVALID #16579
The plugin eval case plugins/planning/evals/wayfind-prototype-files-on-dossier-folder-and-safe-push returned verdict: INVALID (2 permission denials in the traces) on two local claude plugin eval runs on 2026-11-16 (3 runs per arm each, branch of #16551 after it took main).
What happens
In the without-plugin arm, runs 1 and 3, the first tool call is a Read of the plugin copy's planning/skills/wayfind/SKILL.md, which the sandbox denies. The denials void the comparison, so the case has no valid result.
Likely cause
The case has only prompt.md. Its harness note tells the agent to "Read any skill's or plugin's files with the Read tool at their exact path", and the prompt is a /planning:wayfind work ... slash command the without arm cannot resolve, so that arm goes looking for the skill file. Where it learns the copy's path is not recorded in the trace.
Done when
The case returns VALID on a local run, with the without arm making no denied call.
Other cases in plugins/planning/evals that share the harness note are checked for the same verdict.
Found while verifying #16551; not caused by that change (the prompt text is the same on main).
Size: Small
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: the planning eval case wayfind-prototype-files-on-dossier-folder-and-safe-push returns INVALID because the without-plugin arm tries to Read the plugin copy's wayfind/SKILL.md and the sandbox denies it. The denials void the comparison. The done-when is a VALID local run with no denied call in the without arm, plus a check of the other planning eval cases that share the harness note.
What is missing: the trace doesn't record where the without arm learned the plugin copy's path. Reading the case's harness note and the run traces should settle that before choosing a fix, such as changing the harness note or the case setup.
No duplicate found among open issues. Related planning eval issues are #16304 and #14878, but they cover different failures. No merged pull request in the recent list appears to fix this.
Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed - [ ] criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: The without-plugin arm no longer makes a denied Read of the plugin copy's wayfind SKILL.md, so the case returns VALID, and the other wayfind cases that share the harness note are checked for the same verdict.
Which fix is wanted: reword the harness note in all 44 wayfind prompt.md files, or change only this case (for example by adding a case.yaml or setup that makes the skill path resolvable)?
RECOMMENDED: Reword the shared harness note, since the same Read-the-skill sentence sits in 44 wayfind prompts and the done-when asks to check the others. Do not tell the without arm to Read skill files. Basis: plugins/planning/evals/wayfind-prototype-files-on-dossier-folder-and-safe-push/prompt.md:13
The done-when needs a VALID local claude plugin eval run, which an unattended lane may not be able to perform. Is that run a person-run check, with the lane's criteria limited to file contents?
RECOMMENDED: Yes. Treat the VALID local run and the check of the other 43 cases as checks a person runs, and keep the prompt edits as the lane's part. Basis: judgment
Where does the without arm learn the plugin copy's path? The trace does not record it, and the fix depends on it: if it comes from the harness note's wording, a rewording fixes it, but if the runner leaks the path, the runner must change.
RECOMMENDED: Check a trace of a without-arm run for where the path first appears before choosing; absent evidence, assume the harness note's wording prompts the Read. Basis: judgment
Sources read
plugins/planning/evals/wayfind-prototype-files-on-dossier-folder-and-safe-push/prompt.md:13: the harness note tells the agent to Read any skill's or plugin's files at their exact path; the prompt is a /planning:wayfind slash command; allowed_tools includes Read, Glob, Grep, Skill, Bash
plugins/planning/evals/wayfind-prototype-files-on-dossier-folder-and-safe-push: has prompt.md, graders and samples, and no case.yaml
plugins/planning/evals: 44 wayfind prompt.md files contain the same harness-note sentence
To answer, post a comment that starts with this line:
**Decision recorded.** <answer>
Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.
Activity
github-actions commentedon Nov 16, 2026
Size: Small
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: the planning eval case
wayfind-prototype-files-on-dossier-folder-and-safe-pushreturns INVALID because the without-plugin arm tries to Read the plugin copy'swayfind/SKILL.mdand the sandbox denies it. The denials void the comparison. The done-when is a VALID local run with no denied call in the without arm, plus a check of the other planning eval cases that share the harness note.What is missing: the trace doesn't record where the without arm learned the plugin copy's path. Reading the case's harness note and the run traces should settle that before choosing a fix, such as changing the harness note or the case setup.
No duplicate found among open issues. Related planning eval issues are #16304 and #14878, but they cover different failures. No merged pull request in the recent list appears to fix this.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 16, 2026
Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed
- [ ]criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 16, 2026
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: The without-plugin arm no longer makes a denied Read of the plugin copy's wayfind SKILL.md, so the case returns VALID, and the other wayfind cases that share the harness note are checked for the same verdict.
Related work
Open questions
RECOMMENDED: Reword the shared harness note, since the same Read-the-skill sentence sits in 44 wayfind prompts and the done-when asks to check the others. Do not tell the without arm to Read skill files. Basis: plugins/planning/evals/wayfind-prototype-files-on-dossier-folder-and-safe-push/prompt.md:13
claude plugin evalrun, which an unattended lane may not be able to perform. Is that run a person-run check, with the lane's criteria limited to file contents?RECOMMENDED: Yes. Treat the VALID local run and the check of the other 43 cases as checks a person runs, and keep the prompt edits as the lane's part. Basis: judgment
RECOMMENDED: Check a trace of a without-arm run for where the path first appears before choosing; absent evidence, assume the harness note's wording prompts the Read. Basis: judgment
Sources read
To answer, post a comment that starts with this line:
Automated refinement by the lanes App. Answer with a comment whose first line is
**Decision recorded.**.