test(regression): 100% E2E coverage for CLI configs (PER-8250)#2312
Draft
AkashBrowserStack wants to merge 5 commits into
Draft
test(regression): 100% E2E coverage for CLI configs (PER-8250)#2312AkashBrowserStack wants to merge 5 commits into
AkashBrowserStack wants to merge 5 commits into
Conversation
Covers every non-excluded CLI config option (percy/snapshot/discovery/ project + cli-snapshot static/sitemap + per-snapshot capture options) via `percy snapshot --dry-run`, asserting the CLI loads & validates each with no "Invalid config:" output. Runs token-free and discovery-free, so it can gate every PR in CI. Includes a negative self-test fixture proving the validator detector fires. PER-8250
…s (Track V) New pages + snapshots.yml entries exercise scope/scopeOptions, per-snapshot percyCSS override, execute + additionalSnapshots, domTransformation, discovery.scrollToBottom, disableShadowDOM, forceShadowAsLightDOM, and responsiveSnapshotCapture. Visual diffs are reviewed via Percy's dashboard on token-gated runs; the suite skips cleanly without PERCY_TOKEN. Bumped Track C expected snapshot count 16 -> 25 to match. PER-8250
Adds gated server routes that record the headers/auth/cookies/user-agent Percy sends during discovery plus a cross-origin disallowed-hostname probe. functional.test.js runs one real snapshot and asserts on those server observations (robust to log-format changes) for discovery.requestHeaders, authorization, cookies, userAgent, captureSrcset and disallowedHostnames. Token-gated; skips cleanly without PERCY_TOKEN. Server-route behavior is verified token-free; the end-to-end Percy assertions run when a token is set. PER-8250
…cks in CI Adds COVERAGE.md (per-option E2E coverage matrix with explicit onlyAutomate exclusions), rewrites the regression README around the three tracks, and adds config-validation (token-free) + functional (token-gated) steps to the regression job in test.yml. PER-8250
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+10
to
+15
| const child = spawn('npx', ['percy', ...args], { | ||
| env: { ...process.env, ...env }, | ||
| cwd, | ||
| stdio: ['ignore', 'pipe', 'pipe'], | ||
| shell: true | ||
| }); |
- lib/percy-cli.js: justify spawn shell:true with a nosemgrep comment (static, test-controlled args — no injection surface; mirrors regression.test.js). - functional-discovery.html: load the cross-origin disallowed-host probe via @import instead of a <link> so it needs no Subresource Integrity hash (meaningless for a dynamic localhost test resource); discovery still attempts the request. PER-8250
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds 100% E2E coverage for the
percy snapshotCLI config surface(PER-8250). The existing
test/regression/harness covered only a subset of asset-discovery options;this extends it to every non-excluded option across the
percy,snapshot,discovery, andprojectnamespaces plus the cli-snapshotstatic/sitemapand per-snapshot capture options.
Three complementary tracks (full per-option matrix in
test/regression/COVERAGE.md):config-validation.test.js) — token-freepercy snapshot --dry-runthat loads fixtures setting every option andasserts the CLI accepts them all (no
Invalid config:). This is theliteral-100% backbone and gates every PR (including forks) without a token.
Includes a negative self-test proving the validator actually fires.
regression.test.js, newpages/config-*.html) — render-affectingoptions reviewed as visual diffs: scope/scopeOptions, per-snapshot percyCSS
override, execute + additionalSnapshots, domTransformation,
discovery.scrollToBottom, disableShadowDOM, forceShadowAsLightDOM,
responsiveSnapshotCapture.
functional.test.js, gatedserver.jsroutes) — discoveryoptions asserted on what the test servers observed (robust to log-format
changes): requestHeaders, authorization, cookies, userAgent, captureSrcset,
disallowedHostnames.
CI: the
regressionjob intest.ymlnow runs config validation token-freealways, plus visual + functional with
PERCY_REGRESSION_TOKEN.Scope
onlyAutomatesnapshot options (fullPage,freezeAnimation,freezeAnimatedImage,freezeAnimatedImageOptions,ignoreRegions,considerRegions) are out of scope — unreachable via thepercy snapshotweb flow; they belong to the SDK/Automate path. Test-only changes; no
production code touched.
Test plan
yarn test:regression:config— passes locally, token-free (12 assertionsincl. negative self-test; 25 + 8 snapshots enumerated, no
Invalid config:)recorded; auth route 401s without creds)
eslint test/regression/cleanPERCY_REGRESSION_TOKEN)🤖 Generated with Claude Code