ci(windows): add Windows MSVC CI jobs by pimlock · Pull Request #2738 · NVIDIA/OpenShell

pimlock · 2026-08-13T23:29:29Z

Summary

Run the supported Windows MSVC lint and test workload natively on x64 and ARM64 for pull requests and merge queues. Keep each architecture's Cargo and compiler caches warm on main, then build both release binaries without publishing them.

Related Issue

No issue required: localized CI infrastructure work following #2496.

Changes

expose architecture-specific Windows lint and nextest entry points from tasks/windows.toml
run them on windows-2025 (x64) and windows-11-arm (ARM64) for pull-request mirrors and merge queues
run the same per-architecture workload on main and manual dispatches to seed the matching Cargo target and sccache namespaces
build openshell-gateway.exe and openshell.exe after the cache-seed job succeeds on main or a manual dispatch
use the official prebuilt Z3 release instead of compiling Z3 from source on Windows
run Windows Rust tests with the same nextest CI profile and server test-support feature set used by main
cache Cargo target artifacts and use the GitHub Actions sccache backend
validate and build ARM64 natively on the hosted Windows ARM runner
update the Windows build architecture notes and maintenance skill
use enterprise-approved pinned revisions of jdx/mise-action and dtolnay/rust-toolchain

Why prebuilt Z3 on Windows

The previous bundled mode built Z3's C++ sources locally through CMake and MSBuild. That made clean Windows jobs substantially heavier and added another native build configuration, especially for x64-to-ARM64 cross-builds, without changing how OpenShell uses Z3. The z3-sys gh-release mode instead downloads the official architecture-matched Z3 4.16.0 static library and embeds it in the resulting OpenShell binary. This keeps the same pinned Z3 version and static-linking behavior while removing the Z3 source compilation from the critical CI path.

CI supplies its read-only GitHub workflow token to avoid unauthenticated API throttling, and the extracted library/build output is retained by the Cargo target cache. The existing Z3_LIBRARY_PATH_OVERRIDE path remains available for developers who need to use a locally supplied Z3 build.

Testing

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture docs updated

copy-pr-bot · 2026-08-13T23:29:33Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

pimlock · 2026-08-13T23:39:34Z

Windows MSVC cache experiments

I ran the hosted windows-2025 workflow repeatedly while changing one major cache/build variable at a time. Times below are end-to-end workflow/job times and can vary slightly with runner setup and GitHub cache transfer latency.

Configuration	Cache state	Runtime	Notes
Original workflow / bundled Z3	effectively cold	~75m48s–78m22s	Baseline; Z3 compiled from source and the Rust target tree was not providing a useful warm reuse path. run, repeat
Cargo target cache / bundled Z3	cold	~75m16s	Populated the target archive. run
Cargo target cache / bundled Z3	warm	~18m39s	Large improvement; target artifacts were the most effective cache layer. run
sccache only / bundled Z3	cold	~80m47s	Target cache disabled to isolate sccache. Remote writes made the cold run slower. run
sccache only / bundled Z3	warm	~67m55s	1,157 hits / 122 misses (90.46% hit rate), but linking, non-cacheable crate types, Z3 work, and remote reads left total time high. run
Cargo target cache + prebuilt Z3	cold	~37m28s	`z3-sys` downloaded the pinned official prebuilt Z3 archive instead of compiling Z3 via CMake/MSBuild. run
Cargo target cache + prebuilt Z3	warm	~16m24s	Check 1m14s, release build 6m36s, tests 4m46s, focused test 2m12s; target restore ~58s. run
Target cache + prebuilt Z3 + sccache	target warm, sccache cold	~17m09s	62 misses, 62 successful writes, 0 errors; cache writes took 32.6s. run
Target cache + prebuilt Z3 + sccache	both warm	~15m45s	62/62 sccache hits, 0 misses/errors; check 1m07s, release build 5m19s, tests 5m06s, focused test 2m25s. run
Warm hybrid cache, redundant focused test removed	both warm	~12m33s	Check 1m01s, release build 5m07s, full tests 4m58s. The full suite already includes the Windows unsupported-driver contract test. run

Conclusions

Cargo target caching was the largest win: roughly 75 minutes cold to roughly 19 minutes warm with bundled Z3.
Prebuilt Z3 materially improved cold builds: the cold target-cache run dropped from roughly 75 minutes to roughly 37 minutes.
sccache alone was not competitive, despite a high hit rate. It does not cache final linking or several Rust crate types, and GitHub remote reads still cost time.
sccache is useful as a secondary layer on top of the target cache: it reduced the warm release build from roughly 6m36s to 5m19s and produced a clean 100% hit rate for the 62 cacheable compiler requests in the measured run.
The best measured stack is Cargo target cache + prebuilt Z3 + sccache, with the redundant focused test removed.
The PR/main split is intended to keep PR validation focused on Windows Clippy + Rust tests, run that same workload on main to warm the exact PR cache namespaces, and build (but not upload/publish) release binaries after the seed succeeds.

Current validation note

The earlier single-job hybrid workflow is proven by the successful runs above. The new multi-job split currently receives an opaque GitHub startup_failure before any job is created, on both manual dispatch and the PR mirror. actionlint reports the workflow as valid, and a temporary ref at the last known-good commit starts normally, so I am continuing to isolate the workflow-graph difference before treating the split as ready.

pimlock · 2026-08-13T23:42:00Z

Validation update: the opaque pre-job startup_failure was isolated to jdx/mise-action v4.2.4 on windows-2025. Restoring the previously measured v4.2.0 pin fixes workflow startup. The PR mirror is now running PR lint and test (x64), and a manual dispatch is independently running Seed cache (x64) followed by Build binaries (x64). The newer Swatinem/rust-cache v2.9.2 pin remains in place.

pimlock · 2026-08-19T16:34:29Z

Pausing this PR until the compute-driver refactor stack lands:

refactor(compute): unify gateway restart reconciliation #2743 — unify gateway restart reconciliation
refactor(compute): support external driver parity #2744 — negotiate external driver behavior
refactor(compute): register compiled drivers #2786 — register compiled drivers and move first-party driver composition into the new openshell-gateway crate

#2786 overlaps 8 of the 18 files changed here and establishes a cleaner Windows boundary by keeping first-party driver dependencies out of openshell-core and openshell-server. The current strict Windows Clippy fixes add 65 cfg attributes, many of which are likely to become unnecessary after that refactor. Continuing now would create avoidable code and rebase churn.

After the stack merges, the plan is to:

Rebase this PR again.
Keep the Windows PR/main/cache-seed job split, target cache, sccache, and prebuilt Z3 work.
Update lint/build coverage for the new openshell-gateway composition crate.
Remove obsolete platform-scoping changes and handle any remaining Kubernetes or UDS boundaries at module level where possible.
Rerun the hosted Windows timing and cache validation.

The branch is currently rebased onto main; local mise run pre-commit passes.

pimlock · 2026-08-20T22:42:21Z

Update to the earlier waiting note: #2786 has merged, but the stack was split before merge. The merged #2786 is only the compute-driver registry pre-factor and intentionally keeps first-party driver composition in openshell-server.

The remaining relevant stack is now:

feat(compute): add standalone first-party drivers #2822 — add standalone first-party driver binaries
refactor(compute): decouple gateway driver composition #2823 — move first-party composition into openshell-gateway and decouple openshell-server

#2823 is the architectural change we were originally waiting for. It overlaps 8 of this PR’s 18 files, including the core/server configuration and Kubernetes-auth files where most of the Windows cfg changes currently live. This PR is also now conflicting with main, so rebasing and simplifying it before #2823 would create another round of avoidable churn.

Plan: continue holding this PR until #2822 and #2823 land, then rebase, retain the Windows workflow/cache/prebuilt-Z3 work, update lint coverage for openshell-gateway, and recreate only the platform fixes still required by strict hosted Windows Clippy.