Skip to content

fix(test): anchor Jest failure detection on the leading status marker - #4505

Open
houtinghang wants to merge 1 commit into
rtk-ai:developfrom
houtinghang:fix/test-jest-fail-substring-anchor
Open

houtinghang wants to merge 1 commit into
rtk-ai:developfrom
houtinghang:fix/test-jest-fail-substring-anchor

Conversation

@houtinghang

Copy link
Copy Markdown

Summary

rtk test npm test labels a fully green suite [FAIL] FAILURES: as soon as any test name contains the substring FAIL (#3741). The Jest branch of extract_test_summary did line.contains("FAIL"), so a passing ✔ contains uppercase word FAIL in the name line was collected as a failure, while the same block's OUTPUT tail still showed ℹ fail 0.

This is the follow-up KuSh asked for in #3753 after #988 moved the runner into src/core/runner.rs ("anchoring the Jest match … would be welcome"). It takes the smaller of the two suggested routes: no new TestEcosystem variant, just an anchored match.

  • Count a line as a failure only when its trimmed text starts with the runner's own status marker:
    • ✕ — Jest failed test
    • ✖ — node --test failed test (the reporter behind a bare npm test in the issue)
    • FAIL — Jest failed-suite header ( FAIL src/x.test.js after ANSI stripping, FAIL src/x.test.js without colour)
  • Tests: / Test Suites: summary handling is unchanged.
  • Regression tests next to the existing extract_test_summary tests: a green node --test run and a green Jest run whose test names contain FAIL stay green; a Jest FAIL header, a ✕ line and a node ✖ line are still reported, and the passing ✓ sibling is not quoted as a failure.

Closes #3741

Tests

  • cargo fmt --all --check
  • cargo clippy --all-targets
  • cargo test (full suite). One integration test, unwritable_hooks_dir_never_breaks_the_hook, fails in my WSL environment with and without this change because the tests run as root there and root ignores the chmod the test relies on; unrelated to this PR.
  • New tests fail on develop before the change and pass after it.
  • End-to-end: the issue's two-file reproducer (node --test via npm test) through the built binary: rtk test npm test no longer prints [FAIL] FAILURES:.

AI disclosure

Written with the help of Claude (Anthropic); I reviewed the change and the test results above before opening this PR.

🤖 Generated with Claude Code

`rtk test npm test` reported a fully passing suite as `[FAIL] FAILURES:`
whenever a test *name* contained the substring "FAIL", because the Jest
branch of extract_test_summary counted every line containing "FAIL" as a
failure. Tests about failure handling are commonly named that way, and
`npm test` often runs `node --test`, whose spec reporter prints the test
name right after the `✔` glyph.

Only the runner's own leading marker says a line is a failure: Jest opens
a failed suite with a `FAIL <path>` header and marks a failed test with
`✕`; node --test uses `✖`. Match on those at the start of the trimmed line
instead of anywhere in it, so a passing test is free to carry "FAIL" in its
own name.

Fixes rtk-ai#3741
@rtk-wshm-sync-bot rtk-wshm-sync-bot Bot added bug Something isn't working jest testing labels Oct 10, 2026
@rtk-wshm-sync-bot

Copy link
Copy Markdown

wshm · Automated triage by AI

📊 Automated PR Analysis

🐛 Type bug-fix
🟢 Risk low

Summary

Fixes a Jest/node --test failure-detection bug in src/core/runner.rs where any test name containing the substring "FAIL" was misclassified as a failure. The fix anchors failure detection to leading status markers (✕, ✖, or a "FAIL " suite header) instead of a bare substring match, and adds regression tests covering green and failing runs.

Review Checklist

  • Tests present
  • Breaking change
  • Docs updated

Linked issues: #3741


Analyzed automatically by wshm · This is an automated analysis, not a human review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working jest testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rtk test mislabels a fully-passing suite as [FAIL] when a test name contains the substring FAIL

1 participant