Skip to content

fix(pr-review-cli): classify binary files by content, not the head's .gitattributes - #1110

Closed
kyle-sexton wants to merge 1 commit into
mainfrom
fix/1109-lockfile-review-coverage
Closed

kyle-sexton wants to merge 1 commit into
mainfrom
fix/1109-lockfile-review-coverage

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Refs: #1109

Summary

The full-diff step in pr-review-cli.yml lists a changed file as binary when git diff --numstat counts no lines for it. That numstat read the checked-out head's .gitattributes. So a text package-lock.json the consumer marks -diff was listed as binary, and the status line named it as "not reviewable as text".

The binary list alone does not make a review partial. isPartial in .github/actions/publish-review/publish.cjs checks only read < changed, over_budget and over_files; binary_files only adds text to the reason. In the melodic-software/claude-code-plugins#16458 run (log) the coverage was {"read":452,"changed":453,"binary_files":[".../package-lock.json"]}. The review's chunk 2 read the package files. The missing file was plugins/formal-verification/skills/chart/scripts/test/corpus/PickWorker/sany.out: chunk 11 reported files_read 8 of 9 because that file is one line of about 110k tokens, and the reads check warned no Read showed the whole change of 1 listed file(s): .../PickWorker/sany.out. The lockfile was only mislabelled in the message.

Two things are still wrong with letting the attribute decide, and this PR fixes both:

  • check-review-reads treats a file on the binary list as read once the lines before its first hunk are shown (README, "for a file on the job's binary-files list only the lines before its first hunk count"). A -diff attribute on any text file therefore lowered the read-evidence bar for that file under enforce mode.
  • A diff attribute on a real binary file removed it from the list.

The checkout is the pull request's head (ref: ...head.sha), so the pull request controlled both.

Fix

  • In all three legs, the numstat now runs as git --attr-source="$(git hash-object -t tree /dev/null)" diff --numstat .... Attributes come from the empty tree, so git's content check alone decides what is binary. No lockfile name list is needed. A consumer attribute cannot hide a binary file or get any text file listed as binary.
  • A failure of that pipe now emits a ::warning:: and names no file. Before, it named none silently.
  • The README "Binary files" paragraph states that classification is by content.

Direction chosen: the issue offered two options, reading -diff paths with --text or exempting lockfiles from coverage. Neither is needed. The review's diff already uses --text (git diff --no-color --no-ext-diff --no-textconv --text ..., the step's comment: "a head .gitattributes (-diff, binary) ... must not turn a changed file into 'Binary files differ'"). Exempting lockfiles would report a file as reviewed when nobody checked that it was read. With this fix the lockfile stays in the count and needs full read evidence like any text file, so the coverage claim stays true.

--attr-source arrived in git 2.41 (v2.41.0 release notes, "git --attr-source=<tree> cmd $args is a new way..."). That run's runner logs git version 2.56.1. $GIT_DIR/info/attributes and a global core.attributesFile still apply. Both belong to the runner, not the pull request.

Verification

  • Scratch repository, git 2.53.0, .gitattributes = package-lock.json -diff, *.lock -diff, forced.png diff:
    • git diff printed Binary files a/package-lock.json and b/package-lock.json differ for a JSON file.
    • numstat as the step ran it: 1 1 forced.png, - - logo.png, - - package-lock.json.
    • numstat with --attr-source=<empty tree>: - - forced.png, - - logo.png, 1 1 package-lock.json.
  • Test first. the full-diff step names the binary files in scope by content, not the head's .gitattributes, in every leg (.github/scripts/pr-review-cli.test.cjs) runs the step's shell in each leg against a head .gitattributes that marks a text pkg/package-lock.json and notes.txt -diff and a binary forced.png diff, beside a binary x/yarn.lock (a binary file named like a lockfile) and the existing .tar.gz archives.
    • Before the fix it failed with binary=5.
    • After: binary=4. The list is a/fixtures.tar.gz, b/fixtures.tar.gz, forced.png, x/yarn.lock. The text files are not listed.
  • a review that read every file is complete with a binary file in scope (.github/scripts/publish-review.test.cjs) pins the rule: a full read with binary_files posts reviewed, not partial.
  • node --test --test-concurrency=4 over pr-review-cli.test.cjs, publish-review.test.cjs and check-review-reads.test.cjs: 173 pass, 0 fail.

Remaining

  • Done-when item 3 (feat(formal-verification): add the chart and setup skills and the formal-area contract claude-code-plugins#16458's lanes pass on rerun) will not hold after this fix. That PR is partial because of PickWorker/sany.out, a single line longer than one Read can show, which no review can read whole. That needs its own decision: declare it generated data in claude-code-plugins, or teach the lane about files with over-long lines.
  • Done-when item 2 names publish-review.test.cjs. The classification lives in the workflow's full-diff step, so its test is in pr-review-cli.test.cjs. publish-review.test.cjs gets the coverage-rule test.
  • Release and pin: claude-code-plugins calls pr-review-cli.yml@main (pr-review-hosted.yml:338, pr-review-security-hosted.yml:342), so this fix reaches it at merge. A release cut with release-tag.yml (manual workflow_dispatch) is a changelog marker only, and no pin moves unless a caller is moved to a SHA.

Behavior change: under REVIEW_READS_CHECK_MODE=enforce, a text lockfile now needs read evidence for its whole section, where its header lines were enough before. A large lockfile can therefore make a review partial. The existing remedy is the consumer's generated-data list, read at the merge base, which leaves a file out of the review and names it "not reviewed". The same list fits sany.out.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c

….gitattributes

The full-diff step's numstat read the checked-out head's .gitattributes, so
a text lockfile the consumer marks -diff was listed as binary and a real
binary marked diff was not. Read attributes from the empty tree in all three
legs so content alone decides, warn when the list fails, and say so in the
README. The binary list also lowers the read-evidence bar to a file's header
lines, so the attribute could no longer weaken that check either.

Refs: #1109

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 17:41
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T17:44:48.910833Z 2c5e130 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown

claude-review reviewed 2c5e1303642453a29d7f782de0845a330a7d3584: read 4 of 4 changed file(s); 0 finding(s), 0 as inline comments.

No findings. I read the full diff for all 4 changed files and the PR context. This is a first review of this lane on the PR (no prior findings, not blocking-only). I ran nothing myself, so the test results and the scratch-repo output in the PR description are author-claimed and unverified.

What changed

  • All three legs of the full-diff step in pr-review-cli.yml now run the binary-file numstat as git --attr-source=<empty tree> diff --numstat .... Content alone decides what is binary, so the head's .gitattributes can no longer list a text -diff lockfile as binary or hide a binary file marked diff.
  • A failed pipe now emits a ::warning:: before naming no file. It did this silently before.
  • The README "Binary files" paragraph now says classification is by content.
  • Two tests were added or extended: a per-leg .gitattributes fixture test in pr-review-cli.test.cjs, and a rule test in publish-review.test.cjs.

What I checked

  • The three hunks in pr-review-cli.yml are identical, so no leg was missed.
  • The --attr-source and git hash-object -t tree /dev/null form is sound. Git treats the empty tree as built in, and hash-object without -w is enough to produce its id. On a git too old for --attr-source, the pipe fails and takes the new warning path, which names no file and fails safe.
  • The updated test expectations are consistent with the fixture, assuming the earlier head-commit writes sit in unchanged lines this diff does not show. files=9 is the old 4 plus the five new head files (.gitattributes, notes.txt, x/yarn.lock, forced.png, and the edited pkg/package-lock.json). binary=4 and the sorted list match the two .tar.gz archives plus forced.png and x/yarn.lock. odd name.bin is counted in scope but not named, as the test comment says.
  • The publish-review.test.cjs test pins the rule that a full read with binary_files is complete, not partial, which matches the PR description.
  • The PR description's behavior note stands. Under REVIEW_READS_CHECK_MODE=enforce, a large text lockfile now needs full read evidence, and the generated-data list is the remedy. This is a deliberate trade-off, not a defect.
  • There is no closing issue with acceptance criteria in the PR context. It only refs fix(review): a -diff lockfile makes both review lanes fail as partial #1109, which is not included as a section, so I could not check criteria against it.
  • The change touches no plugin manifest, so no manifest lines apply.
  • No cost hold: nothing in the change starts a billed resource or a plan or billing change.

Verifier dropped 0 of 0 candidates.

@kyle-sexton

Copy link
Copy Markdown
Member Author

Closing (hub, who filed #1109). The diagnosis behind #1109 was wrong: claude-code-plugins PR 16458's review went partial on a one-line sany.out of about 110k tokens, not on its lockfile (run 43909503487 log). publish.cjs lines 237-243 never fail a review on binary_files, which only adds words to the reason. This change fixes no observed failure, and it adds one: a large text lockfile could make a review partial in every caller that pins @main. The real fix is claude-code-plugins PR 16581. Branch kept; reopen if a lockfile ever does cause a failure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant