fix(observability): emit a finish for every cucumber test/hook to prevent orphaned runs#56
Open
xxshubhamxx wants to merge 2 commits into
Open
fix(observability): emit a finish for every cucumber test/hook to prevent orphaned runs#56xxshubhamxx wants to merge 2 commits into
xxshubhamxx wants to merge 2 commits into
Conversation
…and sweep open entities at teardown Key the cucumber _tests map by the unique testCaseStartedId instead of the non-unique testCaseId, so reruns/retries/parallel-interleaved attempts of the same scenario no longer clobber each other and every attempt emits its own TestRunFinished. Harden TestCaseFinished to reconstruct a minimal payload and still emit a terminal finish when the map entry is missing. Add a teardown sweep (run before queue drain / build stop in both the main and worker teardown paths) that emits terminal failed finishes for any scenario, hook, or native run left open, so nothing is left running to be reaped as a timeout. Make getCucumberHookType always return a known hook_type, and let TestMap.getUUID fall back to the most recent unfinished run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…p fresh-uuid reconstruct A duplicate/out-of-order TestCaseFinished for the same attempt used to find the entry already deleted, hit the reconstruct branch, mint a fresh uuid, and emit a phantom TestRunFinished for a run the backend never saw started. And the reconstruct branch was dead code for the genuine start-never-recorded orphan: the unguarded _testCasesData read threw before it ran (and the reconstruct itself threw on undefined), all silently swallowed. Track handled attempt ids in a Set and early-return on a duplicate finish, guard the _testCasesData read so an unknown finish no-ops cleanly, and remove the fresh-uuid reconstruct branch entirely. True orphans (started, never finished) are owned solely by the teardown sweep, which holds the real stored uuid. The Set is per-process and bounded by attempt count (same bound as _testCasesData). Also: per-hook try/catch in sweepOpenHooks so one bad upload no longer aborts the remaining hook sweeps; fix the synthetic native finish lang to 'nightwatch'; and document the single-open-run-per-process assumption behind the TestMap.getUUID fallback. Add tests for the duplicate-finish and never-started-finish cases. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dandonarahul2002
approved these changes
Jun 11, 2026
harshit-browserstack
approved these changes
Jun 11, 2026
This was referenced Jun 12, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
With Nightwatch + cucumber and Test Observability enabled, a test or hook can register a start but never emit a matchable finish, so the backend leaves it
runningand an inactivity reaper eventually flips it (and the whole build) totimeout— inflating the reported build/execution time (observed ~4h vs ~45min of real work).Root cause: in
nightwatch/globals.jsthe cucumber_testsmap was keyed by the non-uniquetestCaseId(the pickle/scenario id), not the per-attempttestCaseStartedId. On reruns, cucumber retries, and parallel/interleaved emission, a secondTestCaseStartedoverwrites_tests[testCaseId]and the firstTestCaseFinished'sdeleteleaves the later attempt with no entry, so itsTestRunFinishedis silently skipped. Hook finishes early-return when their map entry is missing, and nothing reconciles still-open runs at teardown.Changes
_testsmap by the uniquetestCaseStartedId(per attempt) instead oftestCaseId, so retries/parallel attempts no longer clobber each other. If the entry is still missing at finish time, reconstruct minimal metadata and still emitTestRunFinished(never silently drop). (The native path'sgetUUIDalso falls back to the most-recent open run instead of returningnull.)getCucumberHookTypealways returns a known hook type (classifiesBEFORE_ALL/AFTER_ALL; neverundefined).after()andafterChildProcess(), before the request queue is drained, emit a synthetic finish (resultfailed) for any started-but-unfinished test/hook so nothing is leftrunning. Idempotent and fully guarded (logs only; never throws or blocks the user's run).Review hardening (follow-up commit): duplicate
TestCaseFinishedevents are now idempotently ignored (a second finish for the sametestCaseStartedIdis a no-op); unmatched finishes are left to the teardown sweep rather than minting a fresh synthetic uuid mid-run, which avoids creating ghost entries in the backend.Verification
TestCaseStartedsharing atestCaseIdbut differenttestCaseStartedIdnow both produce aTestRunFinished(fails on the old keying); the teardown sweep finishes left-open_tests/hooksMap/activeTestRuns;getCucumberHookTypenever returnsundefined. All pass; full suite green; lint/build clean.registerEventHandlerswith real cucumber event shapes confirms the pre-fix code orphans a run under interleaved-attempt ordering and under a dropped finish at teardown, while the patched code emits a finish in both cases.Notes
try/catch(graceful degradation): the fix cannot break a user's test run.Versioning
Recommended bump: patch →
3.11.0 → 3.11.1This PR carries a
fix:conventional-commit prefix, corrects broken behavior (orphaned runs inflating build duration), adds no new public API surface, and introduces no breaking change. The repo's established pattern is consistent: everyfix-prefixed PR maps to a PATCH bump (e.g. v3.10.2 ← PR #53, v3.10.1 ← PR #52), while new-feature PRs map to MINOR. Release is manual perRELEASE.md.Release notes draft
What's Changed
Full Changelog: v3.11.0...v3.11.1