Skip to content

Nothing refuses a comma in a doctest TEST_CASE name, and 1202 of them are already unselectable by -tc #2605

Description

@localai-org-maint-bot

Row: -

Owed by .agents/specs/unaligned-safetensors-consumers.md under ## Owed, added in #2602.

What is unguarded

doctest splits its -tc= filter on commas, so a TEST_CASE name containing one can never be selected by that filter. The binary reports test cases: 0, assertions: 0 and Status: SUCCESS! at rc 0 -- a skip wearing a pass for anyone running a focused gate by name.

#2601 is one instance of this, repaired by renaming a single case in #2602. Nothing stops the next one. Measured on tests/ at 872f7bfe4:

$ grep -rhcE 'TEST_CASE\("[^"]*,[^"]*"\)' tests/ | paste -sd+ | bc
1202

Four of those sit in the two files #2581 touched, so the shape is ordinary here and not a slip somebody made once.

Evidence that the mechanism is exactly as described

Reproduced against doctest 2.5.2 on a pre-existing case, so the finding does not depend on the case #2601 renamed:

$ ./test_exl3_gemm -tc="exl3 policy: the shape table is upstream's, value for value"
[doctest] test cases: 0 | 0 passed | 0 failed | 15 skipped
[doctest] assertions: 0 | 0 passed | 0 failed |
[doctest] Status: SUCCESS!                                        rc 0

The same case reached through a * glob selects 1 case and 25 assertions, and -tc="nomatchXYZ,exl3 policy: shape compatibility and the empty-block clamp" selects 1 case and 7 assertions. The filter therefore splits on the comma into independently full-matched patterns, and adding a space after the comma drops the selection back to 0.

Fix shape

A checker that refuses a comma inside a TEST_CASE (and SUBCASE) name, wired into scripts/agent-preflight.sh beside the other test-shape gates, with a red-before test. It must be diff-scoped or baselined: 1202 existing names would fail it on day one, and a gate that reds ordinary work is the defect rather than the discipline.

Renaming the 1202 is explicitly NOT what this issue asks for. The value is in stopping the 1203rd, which is written by somebody who does not know doctest does this.

Provenance

Raised as the durable follow-up to #2601 by the fresh review of #2602, which reported that the follow-up had no owner. It is filed rather than fixed here because a new checker changes gate semantics and wants its own spec, a red-before test, and a decision about the baseline -- none of which belong in a one-line rename.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions