Repository navigation
feat: change questions, key-sourced grading and @hey that explain the MR - #24
Merged
Merged
Conversation
…1E102-138) GMS limits calls per key and production and development share the key, so each Responses request and each run summary is logged with an origin label. Direct local runs default to development and also append to .local/gms-usage.jsonl, since their stdout is not retained. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1E102-138) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…questions Round-2 real GMS outputs showed follow-ups naming an anchor whose concept they did not deepen, new interview questions solvable by one complexity formula, and an answer resting on an assumed OS facility. The shared exclusions now name the last two, and morecs requires the anchor to be the question the follow-up deepens and a new interview question to beat the remaining follow-ups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Independent scoring of round-3 real outputs found coreDepth 3.73 and falling grounding (3.62 -> 2.56 -> 2.31) across morecs rounds: later follow-ups added subsystems the change never has (replication, partitions, MAC encoding, version stores), repeated judgments under other anchors and filled distractors with formal variants. Follow-up assumptions may now change only one condition of a mechanism the change uses; otherwise a new core interview question is preferred. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prompt said a location only had to be a readable line, but the validator accepts only lines inside the diff hunks. Round-5 real runs cited workflow.ts:90/94 and controller.ts:113 (outside their hunks), failing a morecs job on all three attempts twice. The prompt now states the rule and points to the nearest hunk line that uses the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixed rubric, independent per-question scores, raw outputs for every round, run statistics and the iteration log. Final after+morecs scores: coreDepth 4.11, nonNiche 4.88, learningValue 4.98, 0 ambiguous and one bare-definition question; the new csinterview alone meets every target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The final independent scoring flagged s4 꼬리1 as bare-definition: its
goal sentence ('do not disclose beyond the target') was the correct
option. Both prompts now give this example, require the goal to state
only the outcome, and require every wrong option to claim it meets
that goal too; self-review checks both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The security, source-tool and hunk-line rules were copied into five prompts and had drifted: the change-question prompts never stated the validator's hunk-only rule that broke csinterview round 5. They now live in src/prompt-blocks.ts with the change-question explanation format and the bold rule. csinterview and morecs import text identical to their own, and a test pins both prompts' hashes so the CS bar stays unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Q1 now asks the effect a reviewer must grasp before approving and Q2 the principle the change's correctness most depends on, applied in this change's situation; both keep the basic level, the merge-gate role and the identifier-rename test. Answer keys explain behavior before vs after, why it matters and what goes wrong if misread, with wrong-option notes naming the misreading and what the code actually does. /hey whatelse moves to src/whatelse.ts and goes deeper into this change (trade-offs, failure boundaries, impact on callers/data/ops/compat, how to confirm the guarantee); CS depth stays with 면접/꼬리, as the prompt and help now say. The schema allowed only exactly five questions, so a short list could only be padded: it now returns up to five with a reason, shown under the batch. The ~80-character cap is gone in favour of every condition one answer needs, and both prompts state the validator's hunk-line rule and carry a good/bad example pair. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reasoned answer made the grader rewrite the explanation and every option note, so a grade and /hey showanswer could explain the same question differently. For a keyed question the grade now always shows key.explanation and key.options; the model writes only the verdict, feedback and evidence (expectedAnswer becomes nullable, and an older runner's full output still parses and is overridden). Questions saved before keys existed still get model-written explanations. The grading prompt is kind-aware: Q/심화 feedback credits what was right, names whether behavior, reason or impact was misread with code evidence and flags a right answer for a wrong reason; 면접/꼬리 keep the concept-and-terms rules. The payload now carries the diff. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every @Hey answer had to open with a CS concept's name and definition, so a lookup got a lecture; evidence was free text and the reply was flattened into one paragraph. Answers now correct a wrong premise first, answer, then explain why in terms of this change, list path:line evidence as code links and say what stayed unverified. A lookup needs no CS explanation; a principle appears only when the answer turns on it. The payload carries the diff and is bounded like learning inputs, evidence outside the pinned snapshot is dropped and counted, and long replies step down to stay within the comment budget. AnswerSchema changes shape (premise/answer/why/evidence/unverified); the next release canary must use the new fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing has called the review or verify phase since a087379 (2026-10-21): the n8n graph names controller steps, never model phases; stored jobs hold reports, not phase names; release canaries use answer, questions and grade. Removes both phases from ModelPhase, the provider schema registry and runner admission, plus VerificationSchema and the test-only finding validator/reconciler. AnalysisSchema, CandidateSchema, Finding and archivedFindings stay because stored reports still carry summary fields and archived findings. Runner and GMS adapter tests now exercise the answer phase. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README (ko/en), the plan's observable contract and the demo now say what each question family is for: Q1/Q2 as a merge gate on this change, 심화 going deeper into this change with fewer-than-five plus a reason, 면접/꼬리 owning CS depth, grades showing the stored key so they match showanswer, and @Hey answering premise-first with linked evidence. The demo replaces its CS-lecture keys with change-focused ones, shows a three-question 심화 batch with its shortfall notice, a right-answer-wrong-reason grade next to the same question's reveal, and adds .local/demo-hey.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…free Iteration 1 from blind scoring of the first real-model round (4 MRs): after-round 심화 averaged importance 3.32, nonNiche 3.53 and misreadings 3.95. Scorers found stems that already stated the deciding rule (s1 심화1, s2 심화1-2, s3 심화1-2), options contradicting the stem, scenarios resting on rare coinciding events (storage loss, hand-edited bot comments, filesystem stalls) and one question on pre-existing at-least-once behavior. Both change prompts now forbid stating the asked conclusion or deciding rule and premise-breaking distractors; 심화 picks topics that change real review/deploy/ops decisions, realistic failure conditions only, and only guarantees this change created or altered. Grade feedback that credits, names the misreading and cites code ran past three sentences: 3 of 17 grade attempts failed schema validation and retried whole commands. Feedback may now use four sentences; keys and option notes stay at three. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Iteration 2 from blind scoring of round 2: after-round Q1/Q2 met every target, but 심화 still had 3 padded questions and 1 lookup, importance 3.96 and nonNiche 4.08. The padded ones were mostly the fifth question (s1 심화5, s4 심화5, s2 심화4) re-asking an earlier theme or a second deploy-transition edge case; the lookup read a cache rule the change did not touch. Batches that stopped at four had no padding. Stems that listed every property to check left only one option standing. The 심화 prompt now says three or four independent topics are typical, drops the "aim for five" wording, admits mixed-version deploys only where the repository deploys components separately and at most one deploy-transition question per batch, bans re-asking an earlier question's guarantee and reading a pre-existing rule, and asks whether the batch loses anything without each question. The shared stem rule keeps stated goals to the outcome, as csinterview f15ff38 did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Iteration 3 from blind scoring of round 3: no padded, lookup or merge-gate failures and 2-3 심화 per batch with reasons, but understanding averaged 3.74 (Q1/Q2 3.63) and 심화 nonNiche 3.64. Q2 stems stated the change's decision and asked why, or carried a cue (prep files cleaned, copied by a writer) that eliminated the other options; 심화 still spent questions on deploy-window resumes of legacy jobs. The shared stem rule now ends with an explicit check (could someone who cannot read the code drop three options from the stem alone?) and a good/bad pair; Q2 gives the situation and asks the outcome instead of "why was this done"; deploy-transition topics appear only when the change's purpose is backward compatibility. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contract fix, not a prompt iteration. In the first real-model round, @Hey answers cited six to eight locations and two passing answers sat exactly at the new cap of eight; one wrong-premise answer failed schema validation and was retried by the application. The cause was not captured, but the earlier AnswerSchema allowed twelve entries, so that limit returns. The renderer keeps every entry while the reply fits and cuts to four only at its last comment-budget step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Hey docs/verification/change-questions-2026-10-27 holds the report (Korean), per-round rendered comments, the call log, blind A/B scores with their mapping and the aggregates for the before set and four after rounds. Final round: all seven rubric means are at least 4 (overall 4.63 vs 3.83 before), with 0 padded and 0 generic-CS questions, but one lookup / merge-gate-fail Q2 remains; grade vs showanswer disagreements went from 12/16 to 0/16 in both graded rounds. Examples are regenerated from final-prompt outputs: initial questions and a short 심화 batch with its reason on 252d5c8, a real grade reply, and new @Hey examples; the pre-89bbd89 whatelse-with-cs example is removed. The plan notes the four-sentence feedback limit, and progress.md logs the work unit, its unverified items and the Controller/Runner pairing needed at deploy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
codex/csinterview (7582a76) carries the csinterview session's final prompts (goal-restatement rule, enforced giveaway check), its verification records and a merge of main. Resolutions: - csinterview.ts takes that branch's prompt text and re-applies the shared prompt blocks; both prompts are byte-identical to 7582a76 and the hash pin moves to those hashes. - contracts.ts keeps QuestionBatchSchema's shortfall reason next to the new interview check schema; learning-command.ts keeps whatelse's own reason and the interview giveaway screening. - progress.md keeps CSI5-CSI7 from that branch, then CU1; operations.md and the csinterview verification records follow that branch. - The GMS usage-logging test arrived twice (the same commit exists on main under another SHA); the copy using the removed review phase is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs the unchanged final prompts (e7e76ea) on three MRs for change questions, grading, showanswer and @Hey, under a hard call counter that stays within the approved 140-call budget (135 used). All targets hold on this run: every rubric mean is at least 4 (overall 4.56 vs 3.96 before), 0 padded / lookup / merge-gate-fail questions, and 11/11 grades match showanswer; @Hey passed 9/9 with the twelve-location cap. The report keeps r4's lookup question and the sub-4 subset means (Q1/Q2 understanding 3.83, 심화 importance 3.86) as open items, and progress.md logs the run and the merge preparation for PR #24. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
emphasized()marker)./hey whatelse(심화) moved tosrc/whatelse.ts. It goes deeper into this change (trade-offs, failure boundaries, impact on callers/data/ops/compat, how to confirm the guarantee); CS depth stays with 면접/꼬리, as the prompt and help now say. The schema allows up to five with a reason instead of forcing exactly five, and the ~80-character cap is gone.key.explanationandkey.options, so it matches/hey showanswer; the model writes only verdict, feedback (≤4 sentences) and evidence. Older full-explanation outputs still parse and are overridden; unkeyed legacy questions still get model explanations. Feedback is kind-aware (Q/심화: behavior, reason or impact misread, right-for-wrong-reason; 면접/꼬리: concepts and terms).@heyanswers premise-first. Sections: premise correction, answer, 이유, 코드 근거 (validatedpath:linelinks), 확인하지 못한 것. Lookups get no CS lecture. The payload carries the diff; long replies step down to fit the comment budget.src/prompt-blocks.ts(csinterview/morecs prompts stay byte-identical, hash-pinned); one good/bad example pair per prompt; deadreview/verifymodel phases,VerificationSchemaand the test-only finding validator removed (no caller sincea087379).Verification
Real GMS
gpt-6.1-solon four commits of this repo, before vs after, blind A/B scoring (docs/verification/change-questions-2026-10-27/). Final round: all seven rubric means ≥ 4 (overall 4.63 vs 3.83 before), 0 padded and 0 generic-CS questions, but one lookup / merge-gate-fail Q2 remains (target missed). Grade vs showanswer disagreements 12/16 → 0/16 in both graded rounds.npm test193/193 andnode scripts/integration.mjs71/71 on this branch.Deploy notes
Controller and Runner must ship together (phase schemas changed; the Runner no longer admits
review/verify). The next release canary must use the new@heyfields (premise/answer/why/evidence/unverified).🤖 Generated with Claude Code