SOLR-8474: Add Selenium-based JUnit tests for the Admin UI - #4738
SOLR-8474: Add Selenium-based JUnit tests for the Admin UI#4738janhoy wants to merge 47 commits into
Conversation
- New opt-in JettyConfig.enableAdminUi flag makes JettySolrRunner serve the Admin UI static files and LoadAdminUiServlet like production web.xml does - New test sourceSet in solr/webapp with AdminUiTestBase: starts a 2-node cloud cluster with the UI enabled and drives it with headless Chrome via Selenium WebDriver; tests skip cleanly when no Chrome binary is found - First test: AdminUiDashboardTest asserts the dashboard displays versions, JVM and system stats matching the /admin/info/system API - Selenium 4.47.0 test-only dependency with license bookkeeping
Exclude opentelemetry (provided by solr-core), auto-service and jspecify annotation jars, and drop selenium-support in favor of a small poll-based wait helper. Only the 8 core selenium jars and a byte-buddy version bump remain as new dependencies.
Navigates every node-level, cloud, collection and core screen, waiting for each screen's main content element and asserting no severe browser console errors. Runs the test cluster with metricsEnabled=true so metrics-backed screens (Plugins) work.
Selenium requires 1.18.11 on the webapp test classpath; align the shared version so only a single byte-buddy version remains in solr/licenses.
Java properties, thread dump, logging tree, cloud nodes/tree views, collections detail, core admin, security and login screens, verified against the corresponding admin APIs. Serves a minimal stub of the generated js-client bundle so the Collections screen's CollectionsV2 service instantiates in tests.
Collection screens: query execution, analysis, schema browser, files, segments, plugins, documents form, paramsets, overview. Write actions through the UI: create and delete a collection, index a document, change and revert a log level - all verified via the corresponding APIs.
github-actions
Bot
added
documentation
|
Well, this was impressive. In under one hour, Fable 5 did all this without any hand-holding beyond the initial promt (except me asking it to slim down dependencies). It figured out what tests to write, how to write them, how to wire in adminUI in test framework etc. The doc We'll then be able to catch regressions from any V2 work or any other commits to the UI. I ran all current 27 tests in 33 seconds, so this thing is fast too. Edit: I kicked off the next prompt, going warp speed here.. |
…ole log capture Adds createFixtureCollection/coreNameOnNode0/waitUntil helpers, an optional security.json for the test cluster, browser console logs in failure artifacts, a standard test log4j2 config so the log watcher sees events, and filters for two benign UI console errors.
…tions Groups the collections display and write tests in one feature class: create/delete collection, create/delete alias, add/delete replica and reload, each verified through the corresponding API.
Feature-grouped classes: query execution with rows/fl parameters, the documents indexing form, and paramset create/delete via the UI.
…gner Schema browser display, field flags and term info, add/delete field via the dialogs. Nightly Schema Designer happy path: new schema from a sample document.
Logging level editor and events viewer, core reload, streaming expression execution, and replication screen rendering in cloud mode.
… depth Removes AdminUiWriteActionsTest (its tests moved to the feature classes), trims the node/collection display classes accordingly, and deepens the cloud coverage: graph SVG replica circles and ZK status ensemble info.
I think the big reason we are pushing to have this now is that the odds of breaking the UI are actually GOING UP, becasue we are migrating to the V2 api and we've labeled them Experimental which allows us to modify them more casually. I don't love the nightly because I don't think anyone sees them... Definitly no casual contributor is subscribed to that mailing list, and it new casual contributors who are doing much of the V2 migration work... Lastly, one of Jason's big concerns was flakyness in this new tooling. The best way to iron out any flakyness, or identify and decide TOO flaky is to mkae these tests part of our regular testing. |
Stubbing the js-client only grows as more v2 API calls land in the Admin UI. The tests now serve the generated bundle unconditionally and skip themselves when it is unavailable (js-client build disabled). See apache#4738 (comment)
…um=true Addresses maturity/stability concerns raised in review: the Selenium suites no longer run by default. A new @SeleniumTest test group annotation (disabled by default, aligned with @Nightly/@monster mechanics but its own group) gates them; enable with -Ptests.selenium=true. Since a run is now an explicit choice, a missing Chrome or non-starting ChromeDriver fails the tests instead of skipping them.
With the js-client bundle excluded from the build the Admin UI tests cannot meaningfully run, and a stubbed client would only grow stale as the UI adopts more of the v2 API. The build now disables :solr:webapp tests with a warning when -PdisableJsClient=true is set, and fails with an error when -Ptests.selenium=true explicitly requests them at the same time. See apache#4738 (comment)
Runs on pull requests that touch the webapp (UI code, tests, build), the v2 API contract in solr/api, or the v2 API implementations in solr/core/.../handler/admin/api — the places a UI regression can originate. Uses the Chrome preinstalled on ubuntu-latest and archives screenshots, page sources and browser logs on failure.
Pick up the bundle from the js-client build output when the Gradle-set sysprop is absent, so the suites also run from an IDE after one Gradle build. Point the two UI-bug bullets in the doc at SOLR-18347.
|
We could both make these tests nightly, and add a GitHub Actions workflow that runs specific things like this as well. |
Aligns the browser-binary override with the tests.selenium opt-in flag. A blank value is now treated as unset rather than overriding discovery.
Only wire the js-client bundle (and thereby its node/npm toolchain) into :solr:webapp test tasks when -Ptests.selenium=true enables the Admin UI tests; a default test run no longer touches the js-client build at all. See apache#4738 (comment)
|
I've now made the whole suite opt-in, which I think addresses the stability/maturity concerns raised here:
On nightly (@dsmiley): I deliberately didn't group them into Docs in |
Navigating to a URL that only differs in the #-fragment does not reload the page, so the Angular app instance — including in-flight refresh callbacks of the previous test's screen — survived across tests. On a slow CI runner such a stale callback chain interleaved with the new test's navigation and fired the collections controller's "No collection or alias" alert spuriously, failing testReloadCollectionViaUi. openPage now forces a reload when already on index.html, giving each test a fresh app instance.
dsmiley
left a comment
dsmiley
left a comment
There was a problem hiding this comment.
Thanks for addressing my concerns Jan! Big effort here, thank you.
epugh
left a comment
epugh
left a comment
There was a problem hiding this comment.
while I don't love YAGW: Yet Another Github Workflow, I'll take the progress to unblock things. This looks great, I can't wait to have it. And I look forward to saying "thank you to the Admin UI for all your hard work" and "Goodbye to the entire Admin UI" in the not too distant future ;-).
|
Btw, @gerlowskija is reviewing the improvments I made to schema designer v2 api, and i'm looking forward to getting that merged, getting the new v2 use in admin ui merged, and then being able to remove the |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 54 out of 54 changed files in this pull request and generated 2 comments.
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
solr/webapp/src/test/org/apache/solr/webapp/AdminUiNodeScreensTest.java:57
- This test does not drain or assert the browser console, unlike the other methods in this class. Because the same ChromeDriver is reused and test order is randomized, any severe error from this page is instead consumed by whichever test runs next and is reported against the wrong screen.
solr/webapp/src/test-files/log4j2.xml:30 - Correct the spelling of “synchonous” to “synchronous.”
Fetching the browser log from ChromeDriver empties its buffer, so a failed console assertion left an empty console.log artifact. The assertion now accumulates what it drains and the failure watcher writes the accumulated entries plus any remainder, cleared at test start.
Changes to the embedded-Jetty harness, the SeleniumTest annotation or the test-group wiring can break the browser tests; include those paths in the workflow filter.
A Chrome update surfaced timing races the harness previously got away with: - waitFor(...).click() could hit a StaleElementReferenceException when the page re-rendered between lookup and click (reproducible on the logging screen's jstree). All click sites now go through a click(By) helper that retries the whole find-and-click, like the existing setText helper. - Booting the app directly on a deep link races its initial data loading, intermittently leaving the target screen unresolved (empty collection in the per-collection menu, hidden action buttons). openPage now boots a fresh app instance on the default route and then navigates in-app via a hash change, like a user would. - WAIT_TIMEOUT raised from 15s to 30s for loaded CI machines; successful waits still return as soon as their condition holds.
|
Heads-up on the latest commit (5df3e5d): a Chrome auto-update (151.0.7922.137 → .170) surfaced three timing races locally — a stale-element click on the logging screen's jstree, a data-loading race when the app boots directly on a deep link, and tight 15s waits under load. All clicks now retry via a |
https://issues.apache.org/jira/browse/SOLR-8474
The old (AngularJS) Admin UI has had no automated test coverage. This PR adds JUnit tests that drive the real UI in a headless Chrome browser via Selenium WebDriver — pure JVM dependencies, no Node.js toolchain.
What's included
JettyConfig.Builder#enableAdminUi(boolean)makesJettySolrRunner(and thusMiniSolrCloudCluster) serve the Admin UI static files andLoadAdminUiServletexactly as the productionweb.xmldoes — addressing the long-standingTODOinJettySolrRunner.solr/webapp/src/test):AdminUiTestBasestarts a 2-node cloud cluster with the UI enabled and a headless Chrome;AdminUiStandaloneTestBasecovers standalone (user-managed, no ZooKeeper) nodes — including a leader/follower pair — whose UI differs from cloud mode. Tests skip cleanly (JUnitAssume) on machines without Chrome — e.g. ASF Jenkins — and run wherever Chrome is installed (-Dtests.ui.chrome.binary=to override discovery). On failure a screenshot, the page source and the browser console log are saved. An optionalsecurity.jsonhook supports authenticated clusters, and a stub of the generated js-client bundle keeps the Collections screen alive from a source checkout.AdminUiSmokeTest— navigates every screen, asserting no severe console errorsAdminUiDashboardTest— versions/JVM/memory vs/admin/info/systemAdminUiNodeScreensTest— java properties, threads, cloud nodes/tree/graph/zkstatus, security, loginAdminUiCollectionsScreenTest— detail display; create/delete collection, create/delete alias, add/delete replica, reload — all verified via APIsAdminUiQueryScreenTest— query execution, rows/fl, paramsets dropdown, dismax/edismax toggles, raw query parametersAdminUiDocumentsScreenTest— indexing form; document indexed via UI becomes searchableAdminUiSchemaScreenTest— field browsing, flags and term info vs schema API; add/delete field via the dialogsAdminUiParamsetsScreenTest— create/delete paramset via the formAdminUiLoggingScreenTest— level editor set/unset; events viewerAdminUiCoreAdminScreenTest— core listing, reload core (cloud)AdminUiCoreAdminStandaloneTest— add/rename/swap/unload core on a standalone node, plus standalone menu differencesAdminUiCollectionScreensTest— analysis, files, segments, plugins, overviewsAdminUiStreamScreenTest— streaming expression executionAdminUiSqlScreenTest— SQL query via the form (sql module as a test-only dependency; its jars were already licensed)AdminUiReplicationScreenTest— cloud-mode renderingAdminUiReplicationStandaloneTest— leader/follower pair: disable polling, replicate-now, enable pollingAdminUiSecurityAuthTest— BasicAuth login flow, security screen, add user/role/permissionAdminUiSchemaDesignerTest(@AwaitsFix) — designer happy path; disabled until the designer's backend flakiness is addressedselenium-*jars (Apache-2.0) plus a byte-buddy version alignment; opentelemetry comes from solr-core,selenium-supportwas avoided by a small poll-based wait helper.gradlew -p solr/webapp beast -Ptests.dups=N --tests <Class>works (the generated test tasks needed explicit sourceSet wiring in this war project); the full suite is stable across order-shuffled beast runs.dev-docs/admin-ui-tests.mdThis PR was developed almost fully AI-assisted (Claude Fable).
Coverage checklist
@AwaitsFixdue to designer backend flakiness)Remaining TODOs for full coverage
ubuntu-latest)dev-docs/admin-ui-tests.md, and re-enableAdminUiSchemaDesignerTestonce the designer races are fixed