Fix e2e drift, pin wp-env versions, and add a WP 6.9–7.1 + nightly matrix - #37
Conversation
Two e2e tests in exclude-with-post-in.spec.js began failing on main with no code change, timing out in editor setup waiting for the "Posts to Include" combobox. Cause: Advanced Query Loop 5.0.0 (released 2026-08-31), which .wp-env.json pulled in via the unversioned advanced-query-loop.zip. That release moved the post parameter controls into a ToolsPanel, and "Include posts" is not one of the items shown by default, so the field is not rendered at all until it is enabled from the panel's options menu. WordPress core was ruled out as the cause: FormTokenField and its token-input (which renders role="combobox") are byte-identical between the 6.9 and 6.9.7 builds of wp-includes/js/dist/components.js, and still emit role="combobox". Changes: - Reveal the field via the "AQL: Post options" menu before using it, matching the pattern AQL uses in its own e2e suite, and locate it by label rather than by FormTokenField's implicit ARIA role, which is an upstream implementation detail. - Pin core to the exact tag 6.9.7 rather than the floating #6.9 branch, and pin Advanced Query Loop to 5.0.0. Floating refs let an upstream release break CI with no change here, and made re-running an old green commit depend on the day it ran. - Add a scheduled "E2E (latest upstream)" workflow that reuses the Playwright job with those pins overridden to latest, giving early warning of the next such break. It never runs on pull requests, so it cannot block a merge; on failure it opens or comments on a single rolling issue. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uc1T67ac4VvePFBhENPbki
Playwright test results
Details
|
WordPress is now on 7.1, so the suite only exercising a single core version leaves both the current release and the next one untested. Replace the single Playwright job with a matrix. Core is still pinned per lane to an exact tag rather than a branch, so re-running an old commit resolves to the same WordPress: 6.9 WordPress/WordPress#6.9.7 blocking 7.0 WordPress/WordPress#7.0.4 blocking 7.1 WordPress/WordPress#7.1 blocking nightly WordPress/WordPress#master non-blocking The nightly lane deliberately tracks trunk for early warning. It never fails its job, so upstream breakage cannot block a merge; a failure surfaces as a warning annotation and a job summary instead. Lanes are resolved as JSON in a small `lanes` job so a caller can narrow the matrix without duplicating the test job. Core is selected per lane with WP_ENV_CORE, which takes precedence over .wp-env.json for both the dev and tests environments, so .wp-env.json now only carries the local development default (bumped to 7.1). An aggregate job keeps the id `test` so the existing required status check still resolves — the per-version lanes publish names branch protection does not know about. Artifact names and Playwright report tags are per lane, since upload-artifact v4 rejects duplicate names and the report comment would otherwise overwrite itself. The scheduled workflow becomes the Advanced Query Loop canary only, since the nightly lane now covers core trunk on every push. It runs one pinned-core lane with AQL un-pinned. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uc1T67ac4VvePFBhENPbki
roborourke
changed the title
Playwright test results — WP 7.1
Details
Failed testschromium › elasticpress-toggle.spec.js › ElasticPress Toggle › should show the Use ElasticSearch toggle in the panel |
Playwright test results — WP 7.0
Details
Failed testschromium › elasticpress-toggle.spec.js › ElasticPress Toggle › should show the Use ElasticSearch toggle in the panel |
Playwright test results — WP 6.9
Details
|
The new 7.0, 7.1 and nightly matrix lanes all failed before running a single test, timing out in global-setup.js waiting for login to reach wp-admin. tests/e2e/database.sql is a dump taken on WordPress 6.9, so it pins `db_version` to 60717. WordPress 7.0.4, 7.1 and trunk all expect 61833, so core treated the imported database as out of date and redirected every admin request to wp-admin/upgrade.php. That matches neither `**/wp-admin/` nor `#wpadminbar`, so login timed out and the suite reported 0 tests. The 6.9 lane was unaffected because its db_version matches the dump. Run `wp core update-db` after importing, which keeps the fixture version-agnostic and is a no-op when the versions already match. Also report the landing URL when login fails. The timeout alone gave no indication that the browser was sitting on upgrade.php. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uc1T67ac4VvePFBhENPbki
With login fixed, the 7.0 lane ran all 22 tests and 8 failed. The failures correlate exactly with one call: every spec that opens the settings sidebar in the site editor failed, and every spec that does not passed. editor.openDocumentSettingsSidebar() from @wordpress/e2e-test-utils-playwright requires a button named exactly "Settings" inside the "Editor top bar" region. That button is not reachable in the site editor on WordPress 7.x, so the locator never resolved and the helper timed out before any assertion ran. The same specs pass in the post editor on the same WordPress. Ruled out along the way: - Not a stale test-utils package. openDocumentSettingsSidebar is byte-identical in 1.33.2 (wp-6.9), 1.40.1 (wp-7.0), 1.50.0 (pinned here) and 1.54.0, so bumping it changes nothing. - Not a renamed landmark. The "Editor top bar" and "Editor settings" region labels are unchanged between the 6.9.7 and 7.1 builds of editor.js. - Not a renamed panel class. components-panel__body-title still exists in the 7.1 build of components.js. - Not a broken site editor. The test asserting the site editor layout loads passed on 7.0. Return early when the sidebar is already open, and fall back to any Settings toggle when the core helper cannot find its own. No assertion is weakened and no test is skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uc1T67ac4VvePFBhENPbki
The previous commit got the sidebar open on WordPress 7.x — the
openDocumentSettingsSidebar timeout is gone from the 7.0 log. The same 8 tests
still fail, but now on a different locator: the plugin's own controls
("Query Preset", and the "Extra Query Loop Settings" panel behind the posts
per page and ElasticPress tests) are not found.
Those panels are block inspector controls on core/query. They render correctly
in the post editor on WordPress 7.0 — multiple-post-templates exercises the
plugin's own inspector controls there and passes — so the panels themselves are
fine and the site editor sidebar is simply not showing the Block tab.
Select the Block tab when the editor offers one. It is a no-op where that tab
is already active, so the 6.9 lane is unaffected.
This does not cover everything on 7.1, which additionally fails
multiple-post-templates and a frontend preset assertion. Those are separate and
still unexplained.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uc1T67ac4VvePFBhENPbki
The nightly lane failed wp-env start with ERR_SOCKET_CLOSED_BEFORE_CONNECTION — a Docker/network blip, not a test failure — and that failed the whole run. The non-blocking mechanism only exempted the Playwright step, so any earlier step failing still failed the lane's job. Because the aggregate gate keyed off the matrix result, and the matrix contained trunk, an infrastructure hiccup on trunk blocked the PR. That is the opposite of what marking it experimental was meant to achieve. Move the experimental lanes into their own job and leave it out of the aggregate's `needs` entirely, so "non-blocking" no longer depends on which step happened to fail. To keep the blocking and experimental paths from drifting, the shared steps move into a composite action. `continue-on-error` is not available to composite steps, so the suite records its own `outcome` output and each caller decides whether that is fatal. Also stop `wp-env stop` failing the job when the environment never came up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uc1T67ac4VvePFBhENPbki
Status: WP 7.x lanes are red, and I'm stopping rather than guessing furtherWP 6.9 is green (22/22) and has stayed green through every commit here, including the CI restructure. What is failing
On 7.0, all 8 failures are site editor tests that need this plugin's Fixed along the way
Ruled out (so nobody repeats this)
What blocks finishing itI cannot reproduce WordPress 7.x to inspect the editor: Docker image blobs, Handing the scope decision to @roborourke: land the matrix with 7.0/7.1 non-blocking plus a tracking issue, or expand this PR into real WP 7.x compatibility work. Generated by Claude Code |
The matrix works and WordPress 6.9 is green, but 7.0 (8 failures) and 7.1 (11) fail on genuine compatibility gaps that predate this work: the plugin's core/query inspector panels are not found in the 7.x site editor, and 7.1 also fails multiple-post-templates and a frontend preset assertion. Gating merges on that would block every PR on pre-existing breakage, so move both lanes into the experimental job. They still run and report on every PR — no test is skipped or weakened — and move back to DEFAULT_BLOCKING once the gaps are closed. Lanes now carry a `comment` flag rather than the job deciding, so 7.0 and 7.1 keep reporting into the PR thread while nightly stays out of it. Also repoint the scheduled AQL canary from 7.1 to 6.9. Running it against a known-red lane would have failed every week on the WordPress 7.x gaps and buried the signal it exists to give. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uc1T67ac4VvePFBhENPbki
roborourke
changed the title
3 participants
Two e2e tests in
tests/e2e/exclude-with-post-in.spec.jsstarted failing onmainwith no code change. Fixing that, then fixing the class of problem behind it, then widening coverage to current WordPress.1. The original failure
Advanced Query Loop 5.0.0, released 2026-08-31 — after the last green run on 2026-08-17.
.wp-env.jsonpulled it in via the unversionedadvanced-query-loop.zip, so it landed with no change in this repo.5.0.0 moved the post parameter controls into a
ToolsPanel(newsrc/groups/post-parameters.js). The "Include posts"ToolsPanelItemhas noisShownByDefault— unlike the "Post types" item above it — so theFormTokenFieldis not rendered until enabled from the panel's options menu. Hence a timeout in editor setup, before either test asserted anything.WordPress core was ruled out, not assumed:
FormTokenFieldand itstoken-inputmodule are byte-identical between the6.9and6.9.7builds ofcomponents.js, and still renderrole="combobox". The old locator was valid; the element wasn't there.Fix: reveal the field via the "AQL: Post options" menu (matching AQL's own e2e suite), and locate it with
getByLabelrather than an implicit ARIA role that upstream owns.2. Pinning
Pins are now exact tags/releases, never branches — floating refs let an upstream release break CI with no change here, and made re-running an old green commit depend on the day it ran.
3. Version matrix
6.9WordPress/WordPress#6.9.77.0WordPress/WordPress#7.0.47.1WordPress/WordPress#7.1nightlyWordPress/WordPress#masterCore is selected per lane via
WP_ENV_CORE;.wp-env.jsonnow only carries the local development default. Non-blocking lanes still run and report on every PR — nothing is skipped or weakened.What the matrix found
A latent bug that would have broken the suite on any core bump.
tests/e2e/database.sqlpinsdb_versionto 60717 (6.9.7); 7.0/7.1/trunk expect 61833, so core redirected every admin request towp-admin/upgrade.phpand the suite ran 0 tests. Fixed withwp core update-dbafter the fixture import.Real WordPress 7.x compatibility gaps. 7.0 fails 8 tests, 7.1 fails 11. The plugin's
core/queryinspector panels are not found in the 7.x site editor, and 7.1 additionally failsmultiple-post-templatesand a frontend preset assertion that cannot be a selector problem. These predate this PR and are tracked in #38, with everything already ruled out written down so nobody re-treads it. 7.0/7.1 are non-blocking until that is closed; they move back toDEFAULT_BLOCKINGthen.4. CI structure — three points for reviewers
test. That is the name branch protection resolves; the per-version lanes publish names (WP 7.1) it does not know about. Renaming or removing it silently strands any required status check.needs. They were originally in the blocking matrix with only the test step exempted — then a nightlywp-env startblip (ERR_SOCKET_CLOSED_BEFORE_CONNECTION) failed the run and blocked the PR. "Non-blocking" has to mean the lane is out of the gate entirely, not that one step is tolerated..github/actions/e2e-suite) shared by both jobs so they cannot drift.continue-on-erroris unavailable to composite steps, so the suite records its ownoutcomeand each caller decides whether it is fatal. Artifact names and report tags are per lane, sinceupload-artifact@v4rejects duplicate names.The scheduled canary is now AQL-only and pinned to the 6.9 lane — pointed at a known-red lane it would fail weekly on the 7.x gaps and bury its own signal.
Verification
I could not run WordPress in this sandbox: Docker image blobs,
downloads.wordpress.organd CI artifact storage are all blocked by the egress proxy. CI is the real check, and every WordPress 7.x conclusion here is inferred from logs rather than observed — stated plainly because it bounds how much to trust the 7.x diagnosis in #38.Verified locally instead:
WP_ENV_COREgenuinely overrides.wp-env.jsonfor both the dev andtestsenvironments, checked per lane against wp-env's ownparseConfig— if it did not, every lane would silently test the same core and look green for nothing..wp-env.jsonand the scheduled job's override parse throughparseConfig; the versioned AQL zip resolves to basenameadvanced-query-loop.actionlintclean on all workflows; everyrun:body parses underbash -n;npm run buildandwp-scripts lint-jsclean.Confirmed by CI: 6.9 green at 22/22 across every commit here, the aggregate
testcheck reporting, per-lane artifacts not colliding, and the nightly lane reporting success while failing internally.🤖 Generated with Claude Code
https://claude.ai/code/session_01Uc1T67ac4VvePFBhENPbki