Skip to content

fix(planning): two interview term eval cases cannot show a result (floor score, low skill fire rate) #16304

Closed

Description

@kyle-sexton

Problem

Two planning eval cases could not show whether a change to the interview skill's text helped or hurt, in the before and after eval recorded on #16290 (Sonnet, 6 runs per side, one measurement):

  • interview-context-unfamiliar-term-upstream scored 0.000 before and 0.083 after. answerable-without-follow-up failed 12 of 12 runs with every judge vote FAIL, and context-claims-referenced passed 0 of 6 before and 1 of 6 after. A case that sits at zero on both sides cannot register a gain or a loss.
  • interview-coded-terms-round-answerable was not decided: the skill loaded in 3 of 6 runs before and 4 of 6 after, under the 4-of-6 minimum on the before side.

A third case, interview-term-clash-qualified-name, was queued and not run (cost stop).

Likely cause (UNVERIFIED)

Not established. The record's reply excerpts end before the questions the graders judge, so they do not show whether the graders or the replies are at fault. Two readings to check:

  • interview-context-unfamiliar-term-upstream: either the graders ask for something the skill's text does not ask of a reply, or the skill's replies really miss it every time on main. context-claims-referenced is a regex over the lines labelled Context:; the file and line citations seen in the replies sit in the opening summary, outside those lines.
  • interview-coded-terms-round-answerable: the prompt does not reliably load the interview skill on Sonnet (fire rate 0.50 and 0.67).

Done when

The numeric bars in this list (runs per case, pass counts, means) are the filing session's judgment. They are not measured and not owner-approved. Whoever registers the eval sets them from the case's own measured pass rate and says so in the registration.

  • For interview-context-unfamiliar-term-upstream, the full last message of at least four failing runs on main is read against both graders, and the cause is stated with quotes: grader defect, sample defect, or a real miss in the skill's text.
  • That cause is fixed in its own pull request (the grader or sample, or the skill text), and the case's pass sample passes and its fail sample fails.
  • For interview-coded-terms-round-answerable, the reason the skill does not load in every run is stated from the traces, and the case either loads the skill in at least 5 of 6 Sonnet runs or carries a grader that makes an unloaded run fail visibly.
  • A recorded Sonnet run on main, 6 runs per case, shows interview-context-unfamiliar-term-upstream off the floor (mean above 0.5) or states why it cannot be, and gives the interview-coded-terms-round-answerable fire rate.
  • interview-term-clash-qualified-name is run 6 times on main in the same record, with its per-grader counts.

Out of scope

Re-running the before and after compare of #16290's change.

Related

Refs: #16252 (its paired-cases clause moved here by owner decision), #16290 (the record), #16092.

session: GitHub Support

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  2. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: two planning eval cases (interview-context-unfamiliar-term-upstream at the floor on both sides, and interview-coded-terms-round-answerable with a skill fire rate under the minimum) cannot show a result, and interview-term-clash-qualified-name has not been run. The cause is unverified. The work is to read the failing runs against the graders, fix whichever of grader, sample or skill text is at fault, make the second case load the skill reliably (or fail visibly when it doesn't), then record a 6-run Sonnet result on main for all three cases.

    Missing: nothing needed from the reporter. The full last messages of failing runs are the first thing to read.

    Related, not duplicates: #16290 (the record) and #16252, both named in the issue. I found no open duplicate and no merged pull request that already fixes this. The open issue about two other interview evals failing on main covers different cases.

    Automated triage. A maintainer reviews it.

  3. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  4. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    {"v":1,"actor_id":337595936,"title_sha256":"8a1b2eb3d72dbe413c35c5aa67d4d89e5dc4f4324078908b0189b03445e91d91","body_sha256":"99cc8b8f82b3c35592bba6518df98eb16aa5e10471371a381bf53a0cb2a188f5"}

  5. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor
    No description provided.
  6. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor
    No description provided.
  7. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
  8. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    The work-resolve-issue lane stopped on this issue (host-only): the run reports the work needs a host run the lane cannot make.
    It added host-run and set Work status to confirmed: an agent run on a host builds it, and it waits on no decision from a person.

    In the run's words:

    host-only: every done-when item of #16304 rests on live Sonnet eval runs of plugins/planning/evals and their transcripts, which this lane can neither make nor read. Item 1 needs the full last message of at least four failing runs of interview-context-unfamiliar-term-upstream on main, quoted, to say whether the grader, the sample or the skill's text is at fault. Item 3 needs the traces of interview-coded-terms-round-answerable to say why the interview skill loads in only some runs. Items 4 and 5 need a recorded Sonnet run on main, 6 runs per case, of all three cases. Bash here runs only the named check and verify wrappers: no eval harness, no interpreter and no model session. The repository holds no run record or transcript from #16290. Editing the regex grader or the skill text without those traces would put in a fix for a cause nobody has confirmed, and the issue marks that cause UNVERIFIED.

    What the run says it did not do:

    • Read the full last message of four or more failing runs of interview-context-unfamiliar-term-upstream and state the cause with quotes: this needs the eval harness and model runs, which a host run must make
    • Fix the confirmed cause in its own PR: this waits on the diagnosis above
    • Explain from the traces why the interview skill loads in only some runs of interview-coded-terms-round-answerable, and get it to 5 of 6 loads or add a grader that makes an unloaded run fail visibly: this needs run traces
    • Record a Sonnet run on main, 6 runs per case, of interview-context-unfamiliar-term-upstream, interview-coded-terms-round-answerable and interview-term-clash-qualified-name: this needs a host eval run
    • Tidy, review, verify and checks steps were skipped because nothing changed

    Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/43684541633

    To hand it back to the lane, remove host-run and set Work status to agent-ready again.

  9. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded. The host check registered for this issue ran on main after #16587 merged (302414dc). This is its record; the issue closes on it.

    Record: Sonnet on main, 6 runs per case, judge claude-opus-5-5

    Verdict as registered: VALID; R1 met; R2 met; the issue's 0.5 figure not reached.

    • Validity: each result file holds 6 launched runs, all 6 counted in each, no recorded stop.
    • R1 (deciding bar for item 4, set by the registration from the rate measured on main before the fix, 1 of 12 grader results): interview-context-unfamiliar-term-upstream mean at least 1/3. Measured 0.333 over 6 counted runs: met, at the bar exactly (4 of 12 grader results).
    • The issue's figure (reported, not deciding; the issue calls its numbers the filing session's judgment and asks the registrant to set the bar): mean above 0.5. Measured 0.333: not reached.
    • R2 (item 3): interview-coded-terms-round-answerable loads the skill in at least 5 of 6 launched runs. Measured 6 of 6: met.
    • R3 (item 5): interview-term-clash-qualified-name ran 6 times; counts below.
    Case Mean Skill loaded Grader Passed
    interview-context-unfamiliar-term-upstream 0.333 6 of 6 answerable-without-follow-up 0 of 6
    context-claims-referenced 4 of 6
    interview-coded-terms-round-answerable 0.750 6 of 6 answerable-without-session 3 of 6
    no-bare-ledger-code 6 of 6
    interview-term-clash-qualified-name 1.000 0 of 6 cites-existing-account 6 of 6
    proposes-qualified-name 6 of 6

    No run was excluded. Cost: 6.69 USD for the three record batches plus 0.71 USD for the 2-run smoke, 7.40 USD in all, judge included.

    Why the upstream case is not above 0.5 (item 4's "or states why it cannot be")

    • The regex grader moved off the floor: context-claims-referenced passed 4 of 6 against 1 of 12 before the fix. Its two failures:
      • Run 3: 13 Context lines the grader counts as uncited. Most cite README:<line>; the grader's pattern accepts a file:line only when the file name has an extension, so a correct citation of an extensionless file fails. Others carry no citation at all ("Whether that is happening to us is not visible in the repo.").
      • Run 5: one line, "The prototype is a throwaway (your request)."
    • The judge grader answerable-without-follow-up is still at zero: 6 of 6 runs fail, every judge vote FAIL (18 of 18). The result files keep the votes and not the judge's reasons, so this record does not establish which of the grader's three checks the replies miss. With that grader at zero the case mean cannot pass 0.5, whatever the regex grader does.

    What this record does not show

    • Why the judge grader fails every upstream run (above). The case registers a change through its regex grader only.
    • interview-term-clash-qualified-name passed 6 of 6 on both graders with the interview skill loaded in 0 of 6 runs, so it does not measure the skill: its replies pass without it.
    • answerable-without-session passed 3 of 6 on the coded-terms case, with the skill loaded in every run.

    Done-when

    1. Failing runs read against both graders, cause stated with quotes: fix(planning): define terms in every interview question and state the bar in the upstream-term eval #16587's summary.
    2. Cause fixed in its own pull request: fix(planning): define terms in every interview question and state the bar in the upstream-term eval #16587 (the case prompt and the skill's terms clause).
    3. Fire rate: 6 of 6, by the corrected load rule in fix(planning): define terms in every interview question and state the bar in the upstream-term eval #16587.
    4. Recorded Sonnet run on main: above, with the reason the 0.5 figure is not reached.
    5. Term-clash case run 6 times with per-grader counts: above.

    The eval harness is local and not committed, per this repository's rule for one-off eval tooling.

  10. removed
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 16, 2026
  11. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    Follow-up for what this record leaves weak (the upstream case's judge grader at zero with its reasons not kept, the term-clash case passing without the skill, and the regex grader rejecting a citation of an extensionless file): #16626.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions