Repository navigation
Description
Problem
Two planning eval cases could not show whether a change to the interview skill's text helped or hurt, in the before and after eval recorded on #16290 (Sonnet, 6 runs per side, one measurement):
interview-context-unfamiliar-term-upstreamscored 0.000 before and 0.083 after.answerable-without-follow-upfailed 12 of 12 runs with every judge vote FAIL, andcontext-claims-referencedpassed 0 of 6 before and 1 of 6 after. A case that sits at zero on both sides cannot register a gain or a loss.interview-coded-terms-round-answerablewas not decided: the skill loaded in 3 of 6 runs before and 4 of 6 after, under the 4-of-6 minimum on the before side.
A third case, interview-term-clash-qualified-name, was queued and not run (cost stop).
Likely cause (UNVERIFIED)
Not established. The record's reply excerpts end before the questions the graders judge, so they do not show whether the graders or the replies are at fault. Two readings to check:
interview-context-unfamiliar-term-upstream: either the graders ask for something the skill's text does not ask of a reply, or the skill's replies really miss it every time on main.context-claims-referencedis a regex over the lines labelledContext:; the file and line citations seen in the replies sit in the opening summary, outside those lines.interview-coded-terms-round-answerable: the prompt does not reliably load the interview skill on Sonnet (fire rate 0.50 and 0.67).
Done when
The numeric bars in this list (runs per case, pass counts, means) are the filing session's judgment. They are not measured and not owner-approved. Whoever registers the eval sets them from the case's own measured pass rate and says so in the registration.
- For
interview-context-unfamiliar-term-upstream, the full last message of at least four failing runs on main is read against both graders, and the cause is stated with quotes: grader defect, sample defect, or a real miss in the skill's text. - That cause is fixed in its own pull request (the grader or sample, or the skill text), and the case's pass sample passes and its fail sample fails.
- For
interview-coded-terms-round-answerable, the reason the skill does not load in every run is stated from the traces, and the case either loads the skill in at least 5 of 6 Sonnet runs or carries a grader that makes an unloaded run fail visibly. - A recorded Sonnet run on main, 6 runs per case, shows
interview-context-unfamiliar-term-upstreamoff the floor (mean above 0.5) or states why it cannot be, and gives theinterview-coded-terms-round-answerablefire rate. -
interview-term-clash-qualified-nameis run 6 times on main in the same record, with its per-grader counts.
Out of scope
Re-running the before and after compare of #16290's change.
Related
Refs: #16252 (its paired-cases clause moved here by owner decision), #16290 (the record), #16092.
session: GitHub Support
Metadata
Metadata
Assignees
Labels
Type
Fields
Work status
Priority
Activity
github-actions commentedon Nov 15, 2026
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: two planning eval cases (
interview-context-unfamiliar-term-upstreamat the floor on both sides, andinterview-coded-terms-round-answerablewith a skill fire rate under the minimum) cannot show a result, andinterview-term-clash-qualified-namehas not been run. The cause is unverified. The work is to read the failing runs against the graders, fix whichever of grader, sample or skill text is at fault, make the second case load the skill reliably (or fail visibly when it doesn't), then record a 6-run Sonnet result on main for all three cases.Missing: nothing needed from the reporter. The full last messages of failing runs are the first thing to read.
Related, not duplicates: #16290 (the record) and #16252, both named in the issue. I found no open duplicate and no merged pull request that already fixes this. The open issue about two other interview evals failing on main covers different cases.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 15, 2026
{"v":1,"actor_id":337595936,"title_sha256":"8a1b2eb3d72dbe413c35c5aa67d4d89e5dc4f4324078908b0189b03445e91d91","body_sha256":"99cc8b8f82b3c35592bba6518df98eb16aa5e10471371a381bf53a0cb2a188f5"}
github-actions commentedon Nov 15, 2026
github-actions commentedon Nov 15, 2026
melodic-automation-lanes commentedon Nov 15, 2026
The work-resolve-issue lane stopped on this issue (
host-only): the run reports the work needs a host run the lane cannot make.It added
host-runand set Work status to confirmed: an agent run on a host builds it, and it waits on no decision from a person.In the run's words:
What the run says it did not do:
Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/43684541633
To hand it back to the lane, remove
host-runand set Work status to agent-ready again.test(planning): re-pin the interview Stance section for the terminolo…
kyle-sexton commentedon Nov 16, 2026
Decision recorded. The host check registered for this issue ran on main after #16587 merged (
302414dc). This is its record; the issue closes on it.Record: Sonnet on main, 6 runs per case, judge
claude-opus-5-5Verdict as registered: VALID; R1 met; R2 met; the issue's 0.5 figure not reached.
interview-context-unfamiliar-term-upstreammean at least 1/3. Measured 0.333 over 6 counted runs: met, at the bar exactly (4 of 12 grader results).interview-coded-terms-round-answerableloads the skill in at least 5 of 6 launched runs. Measured 6 of 6: met.interview-term-clash-qualified-nameran 6 times; counts below.interview-context-unfamiliar-term-upstreamanswerable-without-follow-upcontext-claims-referencedinterview-coded-terms-round-answerableanswerable-without-sessionno-bare-ledger-codeinterview-term-clash-qualified-namecites-existing-accountproposes-qualified-nameNo run was excluded. Cost: 6.69 USD for the three record batches plus 0.71 USD for the 2-run smoke, 7.40 USD in all, judge included.
Why the upstream case is not above 0.5 (item 4's "or states why it cannot be")
context-claims-referencedpassed 4 of 6 against 1 of 12 before the fix. Its two failures:README:<line>; the grader's pattern accepts afile:lineonly when the file name has an extension, so a correct citation of an extensionless file fails. Others carry no citation at all ("Whether that is happening to us is not visible in the repo.").answerable-without-follow-upis still at zero: 6 of 6 runs fail, every judge vote FAIL (18 of 18). The result files keep the votes and not the judge's reasons, so this record does not establish which of the grader's three checks the replies miss. With that grader at zero the case mean cannot pass 0.5, whatever the regex grader does.What this record does not show
interview-term-clash-qualified-namepassed 6 of 6 on both graders with the interview skill loaded in 0 of 6 runs, so it does not measure the skill: its replies pass without it.answerable-without-sessionpassed 3 of 6 on the coded-terms case, with the skill loaded in every run.Done-when
The eval harness is local and not committed, per this repository's rule for one-off eval tooling.
kyle-sexton commentedon Nov 16, 2026
Follow-up for what this record leaves weak (the upstream case's judge grader at zero with its reasons not kept, the term-clash case passing without the skill, and the regex grader rejecting a citation of an extensionless file): #16626.