test_check_pr_size is red on main over four unclassified .agents/scripts paths, and it blocks every PR that must edit the checker

[localai-org-maint-bot]

Row: -

tests/scripts/test_check_pr_size.py fails on origin/main itself, and because check-pr-size.py runs that suite as the semantic evidence for any change to itself, the failure blocks every pull request that must touch the checker.

The failure

FAIL: test_every_tracked_and_current_change_path_is_classified
AssertionError: Lists differ: ['.agents/scripts/glm53-dsa-first-load.sh'...] != []

['.agents/scripts/glm53-dsa-first-load.sh',
 '.agents/scripts/glm53-dsa-streamed-load.sh',
 '.agents/scripts/glm53-rocm-memfit.hip',
 '.agents/scripts/glm53-rocm-memfit.sh']

Four tracked files under .agents/scripts/ are not classified by any rule in scripts/check-pr-size.py. They arrived in 5a948c249 ("spec(MODEL-MM-GLM53-FLASH-ROCM): make the memfit probe ASSERT its arch manifest, and run the committed script to get the third sample").

Reproduce

git checkout origin/main
python3 -m unittest -q tests.scripts.test_check_pr_size
# Ran 53 tests ... FAILED (failures=1)

I confirmed it reproduces on origin/main with no change applied, and on an unrelated external branch's parent, so it is the tree's and not any one pull request's.

Why this is worse than one red test

check-pr-size.py requires a change to itself to carry semantic mutation evidence, and it produces that evidence by running this suite. So the red is self-propagating: any pull request that must edit check-pr-size.py cannot get a green pr-size, whatever it does.

That is not hypothetical. #2363 creates a new checker (scripts/check-rocm-dp4a-intrinsic.py). A created-in-place checker is required to register a creation mutation in check-pr-size.py, otherwise the gate refuses to classify the change:

ERROR: PR size check could not classify the change: scripts/check-rocm-dp4a-intrinsic.py
is absent at BASE and has no closed creation mutation

So #2363 must edit the checker, and editing the checker runs this red suite. The contributor cannot win either way, and the failure they see names four files they never touched.

The shape of the fix

The classifier lists .agents/ paths explicitly and deliberately refuses a blanket directory exemption ("Never hide mutable files behind a general directory exemption"), so the repair is to classify these four, not to add a .agents/ catch-all. They are probe and evidence scripts belonging to a spec, so the existing SPEC_EVIDENCE shape is the closest precedent, but the extension set there is .patch|.json|.log and these are .sh and .hip — which is the actual decision to make, and why I am filing rather than guessing.

Red-before is free here: origin/main is red today.

Related

Blocks #2363. The pr-size gate is CI-only, which is why this was not caught locally: scripts/agent-preflight.sh reports check-pr-size.py under its standing SKIPs because it needs arguments preflight does not supply.

[localai-org-maint-bot]

Fixed on main by 6d1335568 ("fix(ENG-EXPERT-STREAM-DEVICE): classify .agents/scripts, whose absence left check-pr-size's own test red on main"), landed 2026-09-02 20:43 UTC. Closing on that evidence rather than re-specifying it.

Verified against the tree at 3d045ba1b:

$ python3 -m unittest -q tests.scripts.test_check_pr_size
Ran 53 tests in 0.469s
OK

The repair is the shape this issue argued for, and it did not take the blanket exemption:

AGENT_RUN_SCRIPT = re.compile(
    r"\.agents/scripts/[A-Za-z0-9](?:[A-Za-z0-9_.-]*[A-Za-z0-9])?"
    r"\.(?:sh|py|hip|cu)\Z"
)

Scoped to the directory and to four extensions, with the reasoning recorded beside it and verified rather than assumed: nothing under CMakeLists.txt, .github/, scripts/, src/ or include/ references .agents/scripts/, the only referents are spec files, nothing builds or installs them, and no entry point reaches them. That is the same class and the same argument as BENCH_EVIDENCE_RUN's .sh and .cu.

Its commit body also independently reached the diagnosis this issue recorded — that classify_path fails closed, so a directory with no class at all leaves already-tracked files red and errors any change touching one, and that it went unseen because check-pr-size.py is CI-only and agent-preflight.sh never runs it.

The author found this independently and did not cite this issue, so nothing auto-closed it. Closing by hand, and noting the fix arrived about four hours after the filing.

#2363 remains blocked, but no longer by this: its base predates 6d1335568, so it needs a rebase to pick the fix up, and it separately carries a commit missing the bare FOLLOWING_AGENTS_PROTOCOL paragraph.