Skip to content

CAMEL-24485: windows-validator - stop killing slow-but-passing PowerShell suite runs - #26568

Merged
davsclaus merged 1 commit into
apache:mainfrom
ammachado:CAMEL-24485
Sep 18, 2026
Merged

davsclaus merged 1 commit into
apache:mainfrom
ammachado:CAMEL-24485

Conversation

@ammachado

Copy link
Copy Markdown
Contributor

Description

Fixes CAMEL-24485: the windows-validator job of the Package-native validation workflow is very unstable.

Root cause

Across the 62 runs of the workflow since it was added (2026-07-15 to 2026-09-17), windows-validator failed in 33 of 54 completed runs. By failure signature:

Failure Count
Surefire fork timeout in "Run Windows installer unit tests" 19
camel-buildtools:*-SNAPSHOT not found in "Install camel-repackager-maven-plugin" (also hits the POSIX legs; out of scope here) 7
Real test failures 3
Early July setup breakage, since fixed 4

The java.net.SocketException: Connection reset quoted in the ticket is not a network problem. It is the Develocity capture thread losing its socket when surefire kills the forked JVM.

The suite is not hanging, it is slow. WebsiteInstallTest$WindowsPowerShell (38 tests) took 626s to 1198s in the runs that passed, and camel-launcher sets camel.surefire.forkTimeout=1200. 8 of the 21 passing runs finished less than 10 seconds before they would have been killed. There is no trend over time and the image (windows-2025) did not change, so the failures are just the slow end of the same distribution.

Changes

  1. Fork timeout (stops the red): the Windows test step passes -Dcamel.surefire.forkTimeout=2700, and the job gets timeout-minutes: 60 so a genuine hang still fails well below GitHub's 6h default. Passing jobs took up to about 25 minutes end to end. I checked with help:effective-pom that the CLI property overrides the pom value (1200 → 2700).
  2. Measurement:
    • WebsiteInstallerFixture prints one [WebsiteInstallerFixture] <ms>: <command> line per spawned process. Surefire streams test stdout to the build log live, so the timings survive a fork kill, where no XML report is written.
    • The job uploads target/surefire-reports (if: always(), 7-day retention).
  3. Speed-up attempts (need measuring on this PR's run):
    • Every test points LOCALAPPDATA at a new, empty home, so each powershell.exe launch probably rebuilds its module analysis cache when auto-loading Expand-Archive / Get-FileHash. The fixture now sets PSModuleAnalysisCachePath to one shared file under target/.
    • The fixture passes TEMP/TMP through, so the C# compile that install.ps1's Add-Type runs on every launch no longer writes into the per-test home. install.ps1 itself never uses the temp directory.
  4. Bounded helper processes: several helpers called readAllBytes() before waitFor(30, SECONDS). The read blocks until every holder of the pipe closes it, so the timeout never applied. The POSIX assertVersionInstalled used waitFor() with no timeout at all. The bounded runner that fixture.run() already had is now WebsiteInstallerFixture.execute(ProcessBuilder), and all 9 helper call sites use it. A helper that times out now fails the test instead of carrying on.

Testing

  • macOS, locally: WebsiteInstallTest has 0 failures and 0 errors; all 37 POSIX tests pass, and the 11 reported skips are Windows-only tests. That exercises the refactored run() / execute() path, and the timing lines appear in the console. WebsiteInstallerFixtureTest passes 5/5.
  • Not verified locally: the Windows code paths, and whether Windows PowerShell 5.1 honors PSModuleAnalysisCachePath (the Microsoft docs I found cover 7.x). If 5.1 ignores it, nothing breaks; there is just no speed-up. This PR's own windows-validator run is the first measurement.
  • SelfUpdateIntegrationTest fails 6 of 8 tests locally ("unable to determine how the Camel CLI was installed"). The same 6 fail on unmodified main, and the test does not call any code touched here.

Opened as a draft until the Windows run on this PR produces timings.

🤖 Generated with Claude Code on behalf of Adriano Machado (@ammachado)

This was generated by an AI agent and may contain inaccuracies. Please verify before relying on it.

…hell suite runs

The windows-validator job failed in 33 of 54 completed runs. 19 of those
were surefire fork timeouts: WebsiteInstallTest$WindowsPowerShell takes
10-20 minutes on windows-latest (626s-1198s across passing runs) against
camel-launcher's 1200s forkTimeout, so slow runs were killed. The
"Connection reset" in the log is the Develocity event socket dying with
the killed fork, not a network problem.

- Raise the fork timeout to 2700s for this job's test step only, and cap
  the job at 60 minutes so a real hang still fails fast.
- Upload surefire reports, and print one timing line per spawned process
  from the fixture; surefire streams that live, so it survives a fork
  kill where no XML report is written.
- Pass TEMP/TMP through and point PSModuleAnalysisCachePath at one shared
  file: every test gives PowerShell a fresh LOCALAPPDATA, so each launch
  would rebuild its module analysis cache.
- Route every helper process through a bounded
  WebsiteInstallerFixture.execute(): readAllBytes() before waitFor(30s)
  blocks until the pipe closes, so the timeout never applied (the POSIX
  assertVersionInstalled had no timeout at all).

_Claude Code on behalf of Adriano Machado (@ammachado)_

_This was generated by an AI agent and may contain inaccuracies.
Please verify before relying on it._
@github-actions github-actions Bot added the dsl label Sep 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🌟 Thank you for your contribution to the Apache Camel project! 🌟
🤖 CI automation will test this PR automatically.

🐫 Apache Camel Committers, please review the following items:

  • First-time contributors require MANUAL approval for the GitHub Actions to run
  • You can use the command /component-test (camel-)component-name1 (camel-)component-name2.. to request a test from the test bot although they are normally detected and executed by CI.
  • You can label PRs using skip-tests and test-dependents to fine-tune the checks executed by this PR.
  • Build and test logs are available in the summary page. Only Apache Camel committers have access to the summary.

⚠️ Be careful when sharing logs. Review their contents before sharing them publicly.

@github-actions

Copy link
Copy Markdown
Contributor

🧪 CI tested the following changed modules:

  • dsl/camel-jbang/camel-launcher

🔬 Scalpel shadow comparison — Scalpel: 1 tested, 0 compile-only — current: 1 all tested

Maveniverse Scalpel detected 1 affected modules (current approach: 1).

⚠️ Modules only in Scalpel (1)
  • camel-launcher
Modules only in current approach (1)
  • camel-launcher-container

Skip-tests mode would test 1 modules (1 direct + 0 downstream), skip tests for 0 (generated code, meta-modules)

Modules Scalpel would test (1)
  • camel-launcher

ℹ️ Shadow mode — Scalpel observes but does not affect test execution. Learn more

💡 Manual integration tests recommended:

You modified dsl/camel-jbang/camel-launcher. The related integration tests in dsl/camel-jbang/camel-jbang-it are excluded from CI. Consider running them manually:

mvn verify -f dsl/camel-jbang/camel-jbang-it -Djbang-it-test
All tested modules (2 modules, 44.4s total)

Total reactor time: 44.4s

Module Duration Status
Camel :: Launcher 43.5s SUCCESS
Camel :: Launcher :: Container 0.9s SUCCESS

Top 20 slowest modules:

  • Camel :: Launcher (43.5s)
  • Camel :: Launcher :: Container (0.9s)

⚙️ View full build and test results

@ammachado
ammachado marked this pull request as ready for review September 18, 2026 12:50
@davsclaus davsclaus added this to the 4.23.0 milestone Sep 18, 2026
@davsclaus davsclaus added the test label Sep 18, 2026
@davsclaus
davsclaus merged commit 7ab6c29 into apache:main Sep 18, 2026
12 checks passed
@ammachado
ammachado deleted the CAMEL-24485 branch September 18, 2026 18:05
davsclaus pushed a commit that referenced this pull request Sep 19, 2026
…ll.ps1 on Windows

Follow-up to CAMEL-24485 (#26568). The WindowsPowerShell suite in
windows-validator still took ~1200s, with 99% of it in the 46 install.ps1
launches (median 25.4s each).

WebsiteInstallerFixture clears the child environment, and with PSModulePath
unset (or pointing at Windows PowerShell's built-in module directory) the
first cmdlet call in install.ps1 spends ~18s on windows-latest scanning
Program Files\WindowsPowerShell\Modules (AWSPowerShell, Microsoft.Graph,
SqlServer). Pointing PSModulePath at the isolated per-test home's own empty
user module directory brings that call down to 0.2s while Windows
PowerShell still finds its built-in cmdlets, and keeps the suite
independent of whatever modules the runner has installed.

Also drops the PSModuleAnalysisCachePath setting from #26568, which the
bisect showed has no effect.

Result: the WindowsPowerShell suite runs in 90s (was 901-1206s), install.ps1
launches take a median of 1.96s (was 19.5-25.4s), and the windows-validator
job finishes in ~5 minutes. Production is unaffected: SelfUpdateCommand only
clears the installer environment under its test-only override.

Closes #26609

Co-authored-by: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants