[Aaryan430]

Why

startAccessibilityScan could set the WCAG version, best practices, needs review and advanced rules, but not experimental rules. The public create-scan API had no field for them. browserstack/accessibility#5649 added scanSettings.experimentalRules, and it is now deployed.

What

• New experimentalRules argument (true / false) on startAccessibilityScan, sent as scanSettings.experimentalRules. When it's left out, the API's default (on) applies. false is sent as a real setting, not dropped as missing.
• The settings line includes it. For example: Scan settings, as the scan ran: WCAG 2.2 AAA, best practices on, needs review on, advanced rules on, experimental rules off. The run summary that the other settings come from has no experimental field. So this one is read from the scan's own settings (/scans/{id}/overview, scanConfigInfo.experimental), which a scan started here has for this run alone. Both reads go out in parallel. If the overview can't be read, the rest of the line still comes from the run summary.
• docs/TOOLS.md lists the scan settings, and says what accessibility: true does on setupBrowserStackAutomateTests.

The argument description is within the 120-character budget, and the tool's description is unchanged.

Verified

• npm run build passes on Node 22: lint, prettier, 1564 tests, tsc. New tests cover:

  • false is sent
  • the setting is read back from the overview
  • the rest is still read when the overview fails
  • the setting is passed through and reported
  • only booleans are accepted

• Live, against the deployed API:

  • experimentalRules: "yes" is refused with a 422 (scanSettings.experimentalRules field must be a valid Boolean), and no scan is created.
  • A scan created with experimentalRules: false shows experimental: false in its overview.
  • startAccessibilityScan through this branch's build on bstackdemo.com, with experimentalRules: false, wcagVersion: "wcag22aaa" and bestPractices: true, completed in about a minute. Its answer read: Scan settings, as the scan ran: WCAG 2.2 AAA, best practices on, needs review on, advanced rules on, experimental rules off.

• Every scan setting, both ways. Two more scans of bstackdemo.com went through this branch's build, each with every setting set explicitly. Each answer read back exactly what was asked:

  • WCAG 2.0 A, best practices off, needs review off, advanced rules off, experimental rules on.
  • WCAG 2.2 AAA, best practices on, needs review on, advanced rules on, experimental rules off.

  In the reports:

  • Best-practice issues appear only with bestPractices on (8 against 0).
  • WCAG 2.1/2.2 AA criteria (2.5.8 target size, 1.3.5 autocomplete, non-text contrast) appear only at wcag22aaa.
  • bypass-blocks-skip-links appears only with advanced rules on.
  • The page has no needs-review or experimental-rule issues, so those two show only in the read-back.
  • The wcag2a run still reports color-contrast (SC 1.4.3, a level AA criterion). The scan ran at wcag2a by its own summary, so this is the scan engine's rule selection, not something this change sends.

Also in this PR (QA eval bugs)

ATA-1461: the Automate setup now writes accessibility: true. setupBrowserStackAutomateTests had no accessibility option. Asked for accessibility testing, it returned a browserstack.yml without it and said to replace the existing file.

• It now takes accessibility (off by default).
• When it's on, the YAML has an accessibility: true section, and a last step says how to read the issues: fetchAccessibilityIssues with the Automate build link or the SDK's build id.
• The YAML step now says to add these keys to an existing browserstack.yml and keep its other settings, instead of "update it with the following content".

ATA-1465: a page that didn't load in a re-run no longer reads as fixed. A page that fails to load is still scanned, as an error page filed under its address. Its real issues then read as resolved and the error page's as new. The scan's own logs say "success" for that page, and the dashboard counts it the same way.

• When comparing scan runs, an issue whose HTML snippet shows an error page marks that page as not loaded. The markers are BrowserStack's Local connection page (its links carry ref=dashboard-privoxy-page) and Chrome's network error page (id="main-frame-error", class="neterror", chrome-error://).
• The page's earlier issues are listed as not checked instead of resolved, and the error page's issues aren't counted as new. A warning names the page, quoted as untrusted data.
• The mirror case is handled too: for a page that didn't load in the earlier run, the error page's issues aren't counted as resolved, and the page's issues now are listed as not checked against the earlier run instead of new. A page that failed in both runs isn't compared.
• A page whose earlier issues all gave way to new ones gets a softer warning, without being reclassified.

Verified live on QA's case, scan 4263536, where page B didn't load in run 2382506. The dashboard says 2 new, 6 resolved, 2 still open. The comparison now gives 0 new, 0 resolved, 2 still open, with page B's 6 issues listed as not checked and the warning. With the two runs swapped (the earlier-run case), it gives 0 new, 0 resolved, 2 still open, with the 6 issues listed as not checked against the earlier run. setupBrowserStackAutomateTests with accessibility: true was checked through the built server.

Changes after review

• The run comparison is decided page by page. The "changed page" check used to re-filter the whole earlier run for every resolved issue, parsing a URL each time, and it blocked the process while it ran (3.3 s for 3,000 issues per run). Both runs are now grouped by page once: about 10 ms. A test holds it under 500 ms.
• A page that didn't load in the earlier run is set apart (above). Issues on a page that failed in either run are no longer counted as still open.
• The error-page markers are stricter. A link to the Local docs, or a class that merely starts with neterror, no longer marks a page as not loaded, since a real page can have either. QA's privoxy page is still caught by its ref=dashboard-privoxy-page links.
• The warnings agree in number ("1 page … its place … the 1 issue … is").
• Cypress and WebdriverIO get accessibility too (ATA-1461). They return before the browserstack.yml step, so accessibility: true was dropped silently. Cypress now gets "accessibility": true under run_settings in browserstack.json, WebdriverIO gets accessibility: true in the BrowserStack service's options, and both get the step on reading the issues. Both flags were checked in browserstack-cypress-cli 1.37.1 and @wdio/browserstack-service 9.41.0.

Build: lint, prettier, 1581 tests, tsc.

Note for review

#517 also changes scanner.ts: it routes runSettings's request through a timed get. Whichever lands second needs a small merge in runSettings, where both reads should go through that get.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Central YAML (base), Organization UI (inherited), Workspace UI (inherited); Enforced overrides: Organization
• Review profile: ASSERTIVE
• Plan: Enterprise
• Run ID: 7db1a882-b349-4311-9bc6-9bddb147a94a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}