Skip to content

fix(observability): emit a finish for every cucumber test/hook to prevent orphaned runs#56

Open
xxshubhamxx wants to merge 2 commits into
nightwatchjs:mainfrom
xxshubhamxx:SDK-6280-orphan-fix
Open

fix(observability): emit a finish for every cucumber test/hook to prevent orphaned runs#56
xxshubhamxx wants to merge 2 commits into
nightwatchjs:mainfrom
xxshubhamxx:SDK-6280-orphan-fix

Conversation

@xxshubhamxx

@xxshubhamxx xxshubhamxx commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Problem

With Nightwatch + cucumber and Test Observability enabled, a test or hook can register a start but never emit a matchable finish, so the backend leaves it running and an inactivity reaper eventually flips it (and the whole build) to timeout — inflating the reported build/execution time (observed ~4h vs ~45min of real work).

Root cause: in nightwatch/globals.js the cucumber _tests map was keyed by the non-unique testCaseId (the pickle/scenario id), not the per-attempt testCaseStartedId. On reruns, cucumber retries, and parallel/interleaved emission, a second TestCaseStarted overwrites _tests[testCaseId] and the first TestCaseFinished's delete leaves the later attempt with no entry, so its TestRunFinished is silently skipped. Hook finishes early-return when their map entry is missing, and nothing reconciles still-open runs at teardown.

Changes

  1. Rerun-safe correlation — key the cucumber _tests map by the unique testCaseStartedId (per attempt) instead of testCaseId, so retries/parallel attempts no longer clobber each other. If the entry is still missing at finish time, reconstruct minimal metadata and still emit TestRunFinished (never silently drop). (The native path's getUUID also falls back to the most-recent open run instead of returning null.)
  2. Complete hook classificationgetCucumberHookType always returns a known hook type (classifies BEFORE_ALL/AFTER_ALL; never undefined).
  3. Teardown sweep — in both after() and afterChildProcess(), before the request queue is drained, emit a synthetic finish (result failed) for any started-but-unfinished test/hook so nothing is left running. Idempotent and fully guarded (logs only; never throws or blocks the user's run).

Review hardening (follow-up commit): duplicate TestCaseFinished events are now idempotently ignored (a second finish for the same testCaseStartedId is a no-op); unmatched finishes are left to the teardown sweep rather than minting a fresh synthetic uuid mid-run, which avoids creating ghost entries in the backend.

Verification

  • Unit tests added: two TestCaseStarted sharing a testCaseId but different testCaseStartedId now both produce a TestRunFinished (fails on the old keying); the teardown sweep finishes left-open _tests/hooksMap/activeTestRuns; getCucumberHookType never returns undefined. All pass; full suite green; lint/build clean.
  • An event-replay driving the actual registerEventHandlers with real cucumber event shapes confirms the pre-fix code orphans a run under interleaved-attempt ordering and under a dropped finish at teardown, while the patched code emits a finish in both cases.

Notes

  • All instrumentation paths remain inside try/catch (graceful degradation): the fix cannot break a user's test run.
  • No new dependencies.

Versioning

Recommended bump: patch3.11.0 → 3.11.1

This PR carries a fix: conventional-commit prefix, corrects broken behavior (orphaned runs inflating build duration), adds no new public API surface, and introduces no breaking change. The repo's established pattern is consistent: every fix-prefixed PR maps to a PATCH bump (e.g. v3.10.2 ← PR #53, v3.10.1 ← PR #52), while new-feature PRs map to MINOR. Release is manual per RELEASE.md.

Release notes draft

What's Changed

Full Changelog: v3.11.0...v3.11.1

xxshubhamxx and others added 2 commits June 10, 2026 01:29
…and sweep open entities at teardown

Key the cucumber _tests map by the unique testCaseStartedId instead of the
non-unique testCaseId, so reruns/retries/parallel-interleaved attempts of the
same scenario no longer clobber each other and every attempt emits its own
TestRunFinished. Harden TestCaseFinished to reconstruct a minimal payload and
still emit a terminal finish when the map entry is missing.

Add a teardown sweep (run before queue drain / build stop in both the main and
worker teardown paths) that emits terminal failed finishes for any scenario,
hook, or native run left open, so nothing is left running to be reaped as a
timeout. Make getCucumberHookType always return a known hook_type, and let
TestMap.getUUID fall back to the most recent unfinished run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…p fresh-uuid reconstruct

A duplicate/out-of-order TestCaseFinished for the same attempt used to find the
entry already deleted, hit the reconstruct branch, mint a fresh uuid, and emit a
phantom TestRunFinished for a run the backend never saw started. And the
reconstruct branch was dead code for the genuine start-never-recorded orphan: the
unguarded _testCasesData read threw before it ran (and the reconstruct itself threw
on undefined), all silently swallowed.

Track handled attempt ids in a Set and early-return on a duplicate finish, guard
the _testCasesData read so an unknown finish no-ops cleanly, and remove the
fresh-uuid reconstruct branch entirely. True orphans (started, never finished) are
owned solely by the teardown sweep, which holds the real stored uuid. The Set is
per-process and bounded by attempt count (same bound as _testCasesData).

Also: per-hook try/catch in sweepOpenHooks so one bad upload no longer aborts the
remaining hook sweeps; fix the synthetic native finish lang to 'nightwatch'; and
document the single-open-run-per-process assumption behind the TestMap.getUUID
fallback. Add tests for the duplicate-finish and never-started-finish cases.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants