Skip to content

fix(cli): don't lose a test's finish after browser.reloadSession() inside the test - #310

Open
rounak610 wants to merge 2 commits into
mainfrom
fix/wdio-reload-session-test-finish
Open

rounak610 wants to merge 2 commits into
mainfrom
fix/wdio-reload-session-test-finish

Conversation

@rounak610

Copy link
Copy Markdown
Collaborator

What is this about?

On the CLI flow, a Mocha test that calls browser.reloadSession() inside its body can lose its TestRunFinished. The test passes, but Test Reporting & Analytics keeps it in progress until the ~60-min reap, and the build is reported as timed out.

The observers for an event run one after another, and TestHubModule decided what to send from the test instance's current state instead of the state it was dispatched for. That state is shared by every trackEvent, so it can move on while an earlier observer is still awaiting:

  1. After a mid-test reload, the new session is not yet in AutomateModule's session map. Its TEST/POST observer (onAfterTest) registers the session and awaits the session-name PUT (since 9.35.1). Without a reload this is a de-duplicated no-op.
  2. WDIO runs the user's afterTest hook at the same time. If that hook takes a screenshot or prints to the console, the resulting LOG trackEvent sets the shared instance's current state to LOG while TEST/POST is still parked on the PUT.
  3. TestHubModule.onAllTestEvents then reads LOG and takes the log branch. The TEST/POST is never deferred or sent. Nothing is logged as an error.

This PR makes the observers act on the event they were called for:

  • TestFramework.runHooks puts the dispatched testFrameworkState / hookState on the observer args.
  • TestHubModule reads them in onAllTestEvents, sendTestFrameworkEvent, sendTestSessionEvent and sendLogCreatedEvent. It falls back to the instance's state for direct callers.
  • The LOG_REPORT/POST recovery path re-enters as TEST/POST through a copy of the args instead of relying only on the instance state.

The same race could also drop a TEST/PRE or a hook finish if a log landed while an earlier observer was awaiting. The fix covers those too.

The other getCurrentTestState() readers (custom tags, attachments, log loading, instance resolution) run from user commands or from trackEvent itself. They want the live state, so they are unchanged. AutomateModule.onAfterTest still awaits the session-name PUT. Making it fire-and-forget would race the de-dupe against the end-of-worker sweep, and once the observers act on their own state the await is harmless.

The v8 line is not affected: its onAfterTest does not await a session-name PUT.

Verification

End to end, on real BrowserStack sessions (wdio 9.32 mocha, Chrome on Windows 11, Test Observability on). The sample spec has 4 tests that each call browser.reloadSession() mid-test, plus 1 test without a reload. A config-level afterTest hook either takes a screenshot or prints a console line 50 ms in, which always lands inside the session-name PUT.

@wdio/browserstack-service afterTest Builds Reload tests where a LOG landed during the PUT TEST/POST dropped Tests with no TestRunFinished Observability
9.41.0 (published) screenshot 3 1 of 9 1 1 that test stays in progress
9.41.0 (published) console line 1 4 of 4 4 4 passed 1, in progress 4
9.41.0 + this PR console line 1 4 of 4 0 0 passed 5
9.41.0 + this PR screenshot 1 0 of 4 0 0 passed 5

With the published package, a screenshot only hits when the PUT is slower than the screenshot command (here 1.3 s vs ~0.6 s). The test without a reload was never affected. With this PR the LOG still lands inside the PUT, but every TEST/POST is deferred and sent, and log delivery is unchanged (5 log events in each run).

Unit tests: new tests/cli/modules/testHubModule.dispatchedState.test.ts (5). It uses the real TestFramework.runHooks, event dispatcher and TestFrameworkInstance:

  • an earlier TEST/POST observer is parked on a promise while a LOG is tracked, and the TEST/POST is still sent;
  • TEST/PRE and AFTER_EACH/POST are sent with their own state after the instance has moved on to LOG;
  • the test session event reports TEST/PRE in the same situation;
  • the LOG_REPORT/POST recovery still yields exactly one deferred TEST/POST.

The first 4 fail without the fix. The related suites (TestHub, skip reporter, mocha/cucumber frameworks, Automate module, event dispatcher, custom tags, attachments) pass 194/194 on Node 22. The full suite has no new failures; its 78 pre-existing failures come from the local network proxy and are the same set with and without this change. npm run lint and npm run build (including tsc) are clean.

Related Jira task/s

  • SDK-8004 (TRA SDK ingestion stability)

Release (mandatory for every PR — required for the ready-for-review label)

Version bump: (required — tick exactly one)

  • minor (backwards-compatible feature)
  • patch (bug fix or other small change)

Release notes type: (optional)

  • New Feature
  • Bug Fix
  • Other Improvement

Release notes (customer-facing): (optional but encouraged)

  • Fixed tests that call browser.reloadSession() staying "in progress" on Test Reporting & Analytics, and their builds being reported as timed out, when an afterTest hook takes a screenshot or logs to the console.

Release notes (internal): (required — engineer-facing; what actually changed / why)

  • TestFramework.runHooks now passes the dispatched testFrameworkState / hookState to observers, and TestHubModule acts on it instead of the shared instance's current state.
  • Previously, a LOG tracked while AutomateModule.onAfterTest awaited the session-name PUT for a session reloaded mid-test sent TEST/POST down the log branch, so its TestRunFinished was never sent.

Checklist

  • Ready to review
  • Has it been tested locally?

PR Validations

Run Tests: Comment RUN_TESTS to trigger sanity tests.

🤖 Generated with Claude Code

…rver awaits

TestHubModule branched on the shared test instance's current state instead of
the state its observer was dispatched for. Observers of an event run one after
another. When the test body calls browser.reloadSession(), AutomateModule's
TEST/POST observer awaits the session-name PUT for the new session before
TestHub's observer runs. A screenshot or console log from the user's own
afterTest, tracked meanwhile, moves the instance's current state to LOG, so
TEST/POST took the LOG branch and its TestRunFinished was neither deferred nor
sent. The test stayed in progress until the ~60-min reap and the build was
reported as timed out.

TestFramework.runHooks now puts the dispatched testFrameworkState/hookState on
the observer args. TestHubModule (onAllTestEvents, sendTestFrameworkEvent,
sendTestSessionEvent, sendLogCreatedEvent) acts on it, falling back to the
instance's state for direct callers.

Refs: SDK-8004

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rounak610
rounak610 requested a review from a team as a code owner October 26, 2026 18:15
@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Central YAML (base), Organization UI (inherited), Workspace UI (inherited); Enforced overrides: Organization
  • Review profile: ASSERTIVE
  • Plan: Enterprise
  • Run ID: 2d0220e7-f8ca-4f9d-a0b5-b04a4b7b2cfc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@rounak610

Copy link
Copy Markdown
Collaborator Author

✅ Good to go

File Confidence
.changeset/pr-310.md ✅ All Clear
packages/browserstack-service/src/cli/frameworks/testFramework.ts ✅ All Clear
packages/browserstack-service/src/cli/modules/testHubModule.ts ✅ All Clear
packages/browserstack-service/tests/cli/modules/testHubModule.dispatchedState.test.ts ✅ All Clear

Change map (generated deterministically from the diff)

graph LR
  subgraph nwdio_service["wdio-service"]
    npackages_browserstack_service_tests_cli_modules_testHubModule_dispatchedState_test_ts["⚠ testHubModule.dispatchedState.test.ts<br/>~124 lines"]
    npackages_browserstack_service_src_cli_modules_testHubModule_ts["testHubModule.ts<br/>~33 lines"]
    n_changeset_pr_310_md["pr-310.md<br/>~5 lines"]
    npackages_browserstack_service_src_cli_frameworks_testFramework_ts["testFramework.ts<br/>~5 lines"]
  end
Loading

↻ This verdict comment is the review anchor — it's updated in place on each run (the gate posts its status separately).

— SDK PR Review Agent

@github-actions

Copy link
Copy Markdown
Contributor

🟢 SDK PR Review gate is green — the SDK PR Review Agent has run on the current head commit (verdict: success).

This gate confirms a review ran on the latest commit. The verdict itself is advisory — read the findings and use your judgement; it does not block merge. A native GitHub reviewer approval is still separately required by branch protection before this PR can merge.

1 similar comment
@github-actions

Copy link
Copy Markdown
Contributor

🟢 SDK PR Review gate is green — the SDK PR Review Agent has run on the current head commit (verdict: success).

This gate confirms a review ran on the latest commit. The verdict itself is advisory — read the findings and use your judgement; it does not block merge. A native GitHub reviewer approval is still separately required by branch protection before this PR can merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant