Skip to content

planning eval case wayfind-prototype-files-on-dossier-folder-and-safe-push returns INVALID #16579

Open

Description

@kyle-sexton

The plugin eval case plugins/planning/evals/wayfind-prototype-files-on-dossier-folder-and-safe-push returned verdict: INVALID (2 permission denials in the traces) on two local claude plugin eval runs on 2026-11-16 (3 runs per arm each, branch of #16551 after it took main).

What happens

In the without-plugin arm, runs 1 and 3, the first tool call is a Read of the plugin copy's planning/skills/wayfind/SKILL.md, which the sandbox denies. The denials void the comparison, so the case has no valid result.

Likely cause

The case has only prompt.md. Its harness note tells the agent to "Read any skill's or plugin's files with the Read tool at their exact path", and the prompt is a /planning:wayfind work ... slash command the without arm cannot resolve, so that arm goes looking for the skill file. Where it learns the copy's path is not recorded in the trace.

Done when

  • The case returns VALID on a local run, with the without arm making no denied call.
  • Other cases in plugins/planning/evals that share the harness note are checked for the same verdict.

Found while verifying #16551; not caused by that change (the prompt text is the same on main).

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 16, 2026
  2. github-actions commented on Nov 16, 2026

    @github-actions
    Contributor

    Size: Small
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: the planning eval case wayfind-prototype-files-on-dossier-folder-and-safe-push returns INVALID because the without-plugin arm tries to Read the plugin copy's wayfind/SKILL.md and the sandbox denies it. The denials void the comparison. The done-when is a VALID local run with no denied call in the without arm, plus a check of the other planning eval cases that share the harness note.

    What is missing: the trace doesn't record where the without arm learned the plugin copy's path. Reading the case's harness note and the run traces should settle that before choosing a fix, such as changing the harness note or the case setup.

    No duplicate found among open issues. Related planning eval issues are #16304 and #14878, but they cover different failures. No merged pull request in the recent list appears to fix this.

    Automated triage. A maintainer reviews it.

  3. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 16, 2026
  4. melodic-automation-lanes commented on Nov 16, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed - [ ] criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  5. melodic-automation-lanes commented on Nov 16, 2026

    @melodic-automation-lanes
    Contributor

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: The without-plugin arm no longer makes a denied Read of the plugin copy's wayfind SKILL.md, so the case returns VALID, and the other wayfind cases that share the harness note are checked for the same verdict.

    Related work

    Open questions

    1. Which fix is wanted: reword the harness note in all 44 wayfind prompt.md files, or change only this case (for example by adding a case.yaml or setup that makes the skill path resolvable)?
      RECOMMENDED: Reword the shared harness note, since the same Read-the-skill sentence sits in 44 wayfind prompts and the done-when asks to check the others. Do not tell the without arm to Read skill files. Basis: plugins/planning/evals/wayfind-prototype-files-on-dossier-folder-and-safe-push/prompt.md:13
    2. The done-when needs a VALID local claude plugin eval run, which an unattended lane may not be able to perform. Is that run a person-run check, with the lane's criteria limited to file contents?
      RECOMMENDED: Yes. Treat the VALID local run and the check of the other 43 cases as checks a person runs, and keep the prompt edits as the lane's part. Basis: judgment
    3. Where does the without arm learn the plugin copy's path? The trace does not record it, and the fix depends on it: if it comes from the harness note's wording, a rewording fixes it, but if the runner leaks the path, the runner must change.
      RECOMMENDED: Check a trace of a without-arm run for where the path first appears before choosing; absent evidence, assume the harness note's wording prompts the Read. Basis: judgment

    Sources read

    • plugins/planning/evals/wayfind-prototype-files-on-dossier-folder-and-safe-push/prompt.md:13: the harness note tells the agent to Read any skill's or plugin's files at their exact path; the prompt is a /planning:wayfind slash command; allowed_tools includes Read, Glob, Grep, Skill, Bash
    • plugins/planning/evals/wayfind-prototype-files-on-dossier-folder-and-safe-push: has prompt.md, graders and samples, and no case.yaml
    • plugins/planning/evals: 44 wayfind prompt.md files contain the same harness-note sentence

    To answer, post a comment that starts with this line:

    **Decision recorded.** <answer>
    

    Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    needs-owner-decision

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions