feat(core): context-aware comment adjudication with evidence-backed verdicts - #2
Merged
Merged
Conversation
added 11 commits
August 13, 2026 03:38
…traction
Each detected comment now carries the structural context of the code it
annotates, and the classifier's judgements are constrained by it.
## Detection (src/detect.rs)
`CommentContext` records the annotated code, whether that code is a
declaration, the scope (module/function/nested-block), the position role
(docstring-head/leading/trailing/inline), and whether the context is
reliable.
Python docstrings are found by walking the tree rather than by a
tree-sitter query. The bundled grammar makes a docstring a direct
`string` child of `module`/`block` with no `expression_statement`
wrapper, so the query matched nothing and Python docstrings were never
detected at all. The walk accepts both shapes.
The walk no longer descends into comment nodes. tree-sitter-rust nests a
marker-stripped content node inside `///`, which was emitted as a second,
phantom comment whose adjacent code was `"/"`.
Adjacent code is the code the comment annotates: the next non-comment
sibling for a leading or head comment, the previous one for a trailing or
inline comment. It was previously taken from the previous sibling in all
cases, so a Go doc comment reported `package main` instead of the
function it documents.
A comment is a docstring head only when nothing but trivia precedes it in
its body — a doc comment on a class's second member is a leading comment.
The file root counts as a head slot, so module docstrings are recognised.
`PositionRole::DocstringHead` is positional only: a `#` line comment
first in a file occupies the slot without being a docstring, and the
`comment_type == Docstring` gate in the public-contract rule is what
keeps that safe.
## Classification (src/classify.rs)
`restates_adjacent` scores how much of a comment's vocabulary already
appears in the code it annotates. Structural context can only *narrow* a
judgement, never widen it: a docstring is a public contract when it
carries contract markup AND is positioned as one (head slot, or leading a
declaration) AND does not merely echo that declaration. Position alone
must never justify — the corpus labels a bare summary line
(`"""Fetch the user."""`) a restatement, and an earlier draft of this
rule contradicted that.
Commented-out code now recognises assignment shape (`x = x + 1`), which
single `=` did not previously match. The check is deliberately narrow:
the left side must be one identifier path, so `set retries = 3 because …`
and the legend `1 = enabled, 2 = disabled` are not mistaken for code.
Edit/MultiEdit fragments are marked unreliable, and an unreliable
catch-all `RestatesCode` is downgraded to `Justified(NonObviousIntent)`.
Fragment edges lose adjacent context, so only a concrete rule may convict
there; the `Write` path is unaffected.
## Measurement (eval/corpus.json, tests/)
`eval/corpus.json` is the single source of truth, carrying a `kind` per
case; the binary label is derived from it. The gate reports per-kind and
per-language precision/recall and asserts a per-kind precision floor, so
a weak kind cannot hide inside the aggregate.
`every_case_is_detectable_end_to_end` and
`detected_path_reaches_f1_threshold` run all 50 cases through the real
parse → detect → classify path. The text-only gate cannot see detector
defects; this one fails when detection breaks. It immediately found a
corpus data bug: two cases carried `language: "python"` with `//` text,
which is not a Python comment. Corrected to rust and javascript.
`tests/context.rs` pins the derived context per grammar — every case
there is a defect that shipped once.
`.cargo/config.toml` is committed because it is load-bearing:
`tree-sitter-language-pack` is declared with `default-features = false`,
so every grammar comes from `TSLP_LANGUAGES`. Without it CI compiles zero
grammars and detection silently returns nothing.
## Verification
cargo fmt --check clean
cargo clippy --all-targets -D warns clean
cargo test --all-targets 58 passed
(37 classify, 9 context, 3 f1, 9 pipeline)
cargo mutants (src/classify.rs) 59 caught / 4 unviable / 0 missed
Detected-path per-kind precision: 48/50 cases kind-exact; the two
remaining are binary-label-equivalent (`// ref: https://…` scores
NonObviousIntent over Attribution).
The CI mutation step's path is corrected to the crate's real location.
Each corpus case now carries the adjacent code it annotates, its position and scope, so the gate exercises real parse->detect->classify against context-bearing code instead of comment text in isolation. - eval/corpus.json: 50 cases re-authored with authored context; labels re-verified under the kind-level rubric. Attribution vs intent marker precedence fixed: 'ref:'/'source:' provenance now classifies as Attribution, not NonObviousIntent (the old per-kind view hid a 2-case kind with no floor to trip). - tests/common: schema grows (code, position, scope); synthesize_source embeds the authored context; evaluate() reports per-kind actual / predicted / correct (recall now has a denominator) and per-language tp/fp/fn; per_kind_violations asserts precision and recall floors on every kind with at least 2 cases. - tests/f1: both the detected and the text-only path gate F1 >= 0.85 and the per-kind floors; malformed corpus fails loudly; zero-case kinds report gracefully; a crafted regression proves a kind-level failure trips the floor while the overall F1 stays above threshold.
- A comment inside a loop body reports Scope::NestedBlock with the loop statement as its adjacent code. - A real Edit payload that introduces a would-be restatement passes: the fragment-edge context is marked unreliable, the classifier falls back to the text-only floor, and the hook never convicts on context the fragment cannot vouch for.
…able (U3) RestatesCode is no longer a hand-waved catch-all: on reliable structural context the verdict carries the cited overlap (U3, KTD3). - comment.rs: UnnecessaryKind::RestatesCode now carries RestateEvidence (lexical tokens in comment order + (verb, operator) table matches); the empty-evidence form is the retained terminal text-only rule. - classify.rs: restate_evidence runs only on reliable context and fires on >= 50% lexical containment OR a verb->operator table match (increment <-> +=, decrement <-> -=, returns <-> return, assign, add/subtract/multiply/divide, double/halve). Word-like operators match as whole tokens so 'add' cannot fire on 'address'. The operator table requires the operator to actually appear in the adjacent code. INTENT markers gain 1-based/0-based (constraint conventions). - Attribution markers gain ref:/source: so provenance links classify as attribution, not generic intent (previously hidden by a 2-case kind with no display floor). - Corpus: +5 context-bearing cases pinning the new behaviour, including an inline paraphrase and the precision moat (throttle comment spared). - Report: the block reason cites the shared tokens and operator matches.
Two new branches in the restate detector needed dedicated pins after the first mutation run left them unprotected: - word-like operators match as whole tokens, not substrings (return_value cannot satisfy the 'return' operator); - evidence requires >= 50% containment, not any nonzero overlap. cargo mutants now reports 0 missed on classify.rs.
is_public_api_doc no longer requires the Docstring type: a line or block comment whose contract markup leads the comment (after its marker) at a contract position is interface documentation. Mid-sentence markup stays unjustified, trailing/inline tags document nothing, and the text-only / unreliable paths keep today's conservative docstring-only promotion. The U3 docstring echo guard is unchanged for docstrings; a contract tag on a line comment (the tag IS the contract) is not revoked for echoing the declaration. Corpus gains a Ruby '# Returns: the user' case.
…uct (U5)
Corpus mining found the shape today's rules only catch through the
blanket terminal rule: a comment that narrates a loop/iteration the code
already expresses, e.g. '# loop over each item' beside 'for item in ...'.
It now gets its own kind and a checkable reason naming the construct.
- comment.rs: UnnecessaryKind::NarratesControlFlow { construct }.
- classify.rs: flow_narration fires when a flow verb (loop/iterate/...)
in the comment matches a construct token (for/while/foreach/iter) in
the reliable adjacent code, matched against raw keyword tokens because
the stop-word list strips 'for'/'in'. Runs after the justification
tables, so a loop comment explaining why (backoff, rate limit) is
always spared; unreliable fragment context never convicts.
- Gate: NarratesControlFlow is asserted on the detected path and exempt
from the text-only per-kind floor, where it correctly degrades to
RestatesCode (no context, no construct).
- Corpus: 2 flow-narration cases + 1 precision case (retry-loop intent).
Three independent review passes (reuse, quality, efficiency) over the U3-U5 diff; all findings applied behavior-preservingly; the mutation gate stayed at 100% throughout (106/106 caught on the final run): - leads_with_contract_markup reuses any_starts; CONTRACT_LEAD_MARKUP is now derived from DOC_MARKUP (minus @see/@author, plus @note/@warning) so the two tables cannot drift. - One shared split_tokens + &str-based dedupe for the content-token and raw-keyword vocabularies (halves per-token allocations). - restate/flow share reliable_adjacent, containment_ratio and the lexical-overlap helper; the classify fallback tokenizes the comment once and shares it between the flow and restate paths; flow only tokenizes the adjacent code when a flow verb is present. - The lowercased adjacent-code copy is gone: symbolic operators are case-free and the only word-like operator matches via the token set. - NarratesControlFlow now cites the (verb, construct) pair, making the verb-match equality observable to tests; flow_construct picks the first verbatim-matching verb row (previously a masked-equivalent double scan that three equality/negation mutants survived). - KindMetrics::precision/recall unify matrix display and floor assertions; the corpus loader stores CommentType directly instead of a mirrored enum; the snippet builder special-cases only python function scope.
Twelve of fifteen review findings validated independent of the original
reviewers; all applied behavior-preservingly (the remaining three were
rejected: file-size preference, unreachable guard narrowing, and a public
API removal that would break library consumers).
- classify.rs: rewrite the stale module doc (the fallback is no longer a
pure fold); mask string literals before flow-construct keyword
extraction so print("for the win") cannot cite a phantom 'for'
construct; anchor ref:/source: attribution tags to comment starts.
- tests: pin the 0.4 containment band (two-of-five shared tokens must stay
empty), pin full-payload quote masking (two mutants survived until
both payload shapes were exercised), add a corpus row for the
string-literal precision case, exercise MultiEdit through the pipeline
(restatement supplies a pass, while rules still block).
- gate: the detected path now asserts at least one restatement verdict
cites evidence, so a detector that silently drops adjacent_code can no
longer alias through both F1 gates; per-kind floors now trip when a
single-case kind (GeneratedFile) goes wrong, closing the MIN_BUCKET
blind spot.
- corpus: the fn-scope python docstring case now embeds a valid statement
(return fetch_user(user_id)) instead of invalid Python.
- README: five kinds with the cited-evidence appendix, context-bearing
60-case corpus, line/block lead-markup sparing, and the Edit
restatement-disabled rule documented.
Verification: one-shot gate (fmt/clippy/90 tests) green; cargo mutants
117/117 caught (0 missed).
README now documents the five verdict kinds with their cited-evidence reasons, the context-bearing 60-case corpus behind the F1 gate, the line/block lead-markup public-API sparing rule, and the Edit/MultiEdit restatement-disabled behavior. AGENTS.md and CLAUDE.md carry the repo's distribution-layer governance updates.
Solves: how a deterministic classifier cites verified evidence, fails open on fragment-unreliable context, and gates the wiring (F1 evidence assertion, per-kind floors incl. the single-case-kind hole). Verified by 117/117 mutation, pipeline pins, and corpus floors; CONCEPTS.md seeded with the classifier domain vocabulary.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(core): context-aware comment adjudication with evidence-backed verdicts
Summary
The classifier no longer treats comments as bare text: it consumes the structural context (adjacent code, scope, position, language) supplied by the Claude Code hook, and adjudicates per kind with cited evidence. Every verdict reason is checkable against the code:
increment ↔ +=)loop,iterate,retry…) that narrates the raw keyword (for,while,map…) the code showsRestatement never convicts on unreliable context: an
Edit/MultiEditfragment cannot vouch for the code around a comment, so the hook falls back and passes instead of blocking the user's write. Fail-open is deliberate (PreferDontConvict floor) and now documented.Key decisions (settled, with plan)
docs/plans/2026-08-12-002-feat-sota-comment-adjudication-plan.md(implementation-ready, executed U1–U5).ref:/source:moved to the ATTRIBUTION pathway;1-based/0-basedintent markers recognized.# Returns:,@param,# panics) leading any comment kind at a contract position, not just docstrings.print("for the win")can never be cited as aforconstruct.Verification
cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test -- --test-threads=4— green.cargo mutants --file crates/comment-checker/src/classify.rs --timeout 90→ 117/117 caught (0 missed, 4 unviable) on the fix head; 106/106 on the simplify head.Review
Post-Deploy Monitoring & Validation
None required: a local CLI hook, no network/telemetry. The F1 gate and mutation gate are the regression surface; the corpus is the human-reviewable fixture for both precision and recall claims.
Related
Known residual risks
eval/corpus.jsonis a tracked evaluation surface; label edits are governed by plan process.Shipped by Compound Engineering — plan-driven, review-gated, verification-recorded. Feature branch
feat/sota-comment-adjudication.