[Hackathon] fix(test): test lint passes plans that test create / create-batch reject, and fails ones they accept #374

Verified against (commit SHA or released version)

commit 988c0355bb05631e9c6c1785e443937ff2a88c0e (main, v0.14.0)

Discord Username / User ID

1149447580477042769

What does this improvement do?

test lint promises (in --help) to "Validate plan/steps files offline with the same validators create runs". DOCUMENTATION.md places it in the authoring loop as the batch counterpart of test create --plan-from --dry-run. #98 accepted it on the condition that it reuse the create guards. Its directory reader and its type check, however, were written separately and have drifted from test create / test create-batch. Lint now gives the wrong answer in both directions, so a pre-commit hook or a CI step that runs test lint before create-batch can't be trusted.

False green: lint passes, create refuses.

1. A "type": "backend" plan.
   • Lint reports 1/1 valid and exits 0. Its error text even lists type: must be one of: "frontend", "backend".
   • test create --plan-from rejects the same file locally with exit 5: "Backend tests via the CLI require --code-file."
   • create-batch warns "server will return per-spec validation_error for these" and the spec fails.
   • schemas/plan.schema.json allows only "enum": ["frontend"].
2. A --plans JSONL with 51 specs. Lint reports 51/51 valid, exit 0. create-batch refuses it: "must contain at most 50 specs (got 51)", exit 5. The same 50-spec cap applies to --plan-from-dir.

False red / unchecked file: lint and create-batch read a directory differently.

3. Lint matches *.json case-sensitively, so B.JSON is never linted, yet create-batch ingests it.
4. Lint reports a non-plan JSON file in the directory (e.g. suite-index.json, which has no projectId/planSteps) as 4 errors and exits 5. create-batch deliberately skips it with [warn] Skipping suite-index.json: not a plan file and succeeds.

Reproduction

mkdir -p lintrepro/plandir && cd lintrepro
echo '{"projectId":"prj_1","type":"backend","name":"Health","planSteps":[{"type":"action","description":"GET /health"}]}' > be.plan.json
echo '{"projectId":"prj_1","type":"frontend","name":"Login","planSteps":[{"type":"action","description":"Open /login"}]}' > plandir/a.json
sed 's/Login/Login B/' plandir/a.json > plandir/B.JSON
echo '{"suite":"checkout","owner":"qa"}' > plandir/suite-index.json
for i in $(seq 1 51); do echo "{\"projectId\":\"prj_1\",\"type\":\"frontend\",\"name\":\"T$i\",\"planSteps\":[{\"type\":\"action\",\"description\":\"Open /\"}]}"; done > fiftyone.jsonl

testsprite test lint --plan-from be.plan.json;                       echo "exit=$?"   # 1/1 valid, exit=0
testsprite test create --plan-from be.plan.json --dry-run >/dev/null; echo "exit=$?"   # exit=5 (backend refused)
testsprite test lint --plans fiftyone.jsonl;                          echo "exit=$?"   # 51/51 valid, exit=0
testsprite test create-batch --plans fiftyone.jsonl --dry-run >/dev/null; echo "exit=$?" # exit=5 (cap)
testsprite test lint --plan-from-dir plandir;                         echo "exit=$?"   # 4 problems in suite-index.json, B.JSON not checked, exit=5
testsprite test create-batch --plan-from-dir plandir --dry-run;       echo "exit=$?"   # reads 3 files, skips suite-index.json, creates 2, exit=0

Expected. For any input, lint's verdict matches what the create path would do with it: anything create / create-batch refuses locally (or that is certain to fail as a backend spec) is a lint issue, and anything they accept or deliberately skip is not.

Details / implementation notes

Root cause (src/commands/test.ts):

• collectPlanIssues (lint) checks requireEnum(${prefix}type, obj.type, ['frontend', 'backend']). The backend refusal lives only in runCreateFromPlan (if (plan.type === 'backend') { … 'Backend tests via the CLI require --code-file.'), and the batch path only warns.
• runLint's --plan-from-dir branch filters with .filter(name => name.endsWith('.json')). readPlansFromDirGuarded uses .filter(f => extname(f).toLowerCase() === '.json') and applies the looksLikePlan skip heuristic, which lint lacks.
• The MAX_BATCH_SPECS (50) caps in readPlansJsonlGuarded / readPlansFromDirGuarded have no counterpart in runLint.

Proposed change (one concern: lint ↔ create parity; create behaviour unchanged):

1. collectPlanIssues reports type: "backend" as an issue, using the same reason test create --plan-from gives and pointing at test create --type backend --code-file.
2. Factor the directory filter and the looksLikePlan heuristic out of readPlansFromDirGuarded into two small helpers that both readers call, so they cannot drift again. Lint then checks *.JSON, and it skips non-plan files with the same [warn] Skipping … line on stderr, leaving the JSON report shape unchanged.
3. Lint reports a plans / plan-from-dir issue when the valid spec count exceeds MAX_BATCH_SPECS, as the create readers do. The 5 MB body cap can come along too if you want it.
4. Tests in the existing runLint block of src/commands/test.test.ts, one per drift above. Each runs lint and the create reader on the same fixture and asserts they agree, and each fails on main today.

I'd like to work on this once it's triaged.

Confirmations

• I have searched existing issues and this is not a duplicate.
• I have provided my Discord identity above for reward coordination.