Skip to content

Fix CI: Tests run on windows-latest, which no longer provides VS 2022 - #71

Draft
mcgroarty with Copilot wants to merge 5 commits into
mainfrom
copilot/test-windows-2022-and-windows-latest
Draft

Fix CI: Tests run on windows-latest, which no longer provides VS 2022#71
mcgroarty with Copilot wants to merge 5 commits into
mainfrom
copilot/test-windows-2022-and-windows-latest

Conversation

Copilot AI commented Aug 2, 2026

Copy link
Copy Markdown

Problem

GitHub Actions windows-latest is a rolling label that GitHub updates over time. The CI matrix used only windows-latest but the test test_vstoolset_set hardcoded Visual Studio 2022 (AUTOBUILD_VSVER=170). When windows-latest moved to a newer Visual Studio, the test started failing.

Changes

.github/workflows/ci.yaml

  • Added windows-2022 to the OS matrix so that the VS 2022 toolchain stays tested even as windows-latest moves forward.
  • Added a comment pointing to MAINTENANCE.md explaining when and how to update this list.

tests/test_source_environment.py

  • NEWEST_KNOWN_VS_MAJOR — a constant near the top of the file set to "18" (Visual Studio 2026). This is the single value a developer bumps when a new Visual Studio ships.
  • test_vstoolset_set — updated to dynamically detect the installed VS version via _available_vsvers() and look up the expected toolset from _VSTOOLSETS, so it works correctly on any supported VS.
  • test_newest_vs_known (new) — intentionally fails (not skips) when the detected VS major version exceeds NEWEST_KNOWN_VS_MAJOR. This is the early-warning signal that autobuild needs updating. The failure message points directly to MAINTENANCE.md.
  • Both the constant and the new test have inline comments linking to MAINTENANCE.md.

MAINTENANCE.md (new)

Documents the maintenance strategy and provides step-by-step instructions for a developer who hits a test_newest_vs_known failure:

  1. Add the new VS major version to _VSTOOLSETS (and the CMake generator map) in autobuild_tool_source_environment.py.
  2. Bump NEWEST_KNOWN_VS_MAJOR in tests/test_source_environment.py.
  3. Add the outgoing windows-latest image (e.g. windows-2025) to the os matrix in ci.yaml.
  4. Optionally remove end-of-life runner images from the matrix.

A summary table lists every file that needs updating.

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

CLA Assistant Lite bot:
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


0 out of 2 committers have signed the CLA.
❌ @Copilot
@mcgroarty
You can retrigger this bot by commenting recheck in this Pull Request

@mcgroarty

Copy link
Copy Markdown
Contributor

recheck

Copilot AI changed the title [WIP] Implement CI tests for both windows-2022 and windows-latest images CI: test windows-2022 + windows-latest; auto-detect VS version in test Aug 2, 2026
Copilot AI requested a review from mcgroarty August 2, 2026 05:42
@mcgroarty
mcgroarty requested a review from brad-linden August 2, 2026 05:49
@mcgroarty mcgroarty changed the title CI: test windows-2022 + windows-latest; auto-detect VS version in test Fix CI: Tests run on windows-latest, which no longer provides VS 2022 Aug 2, 2026
@mcgroarty

Copy link
Copy Markdown
Contributor

@copilot Add an additional test in tests/test_source_environment.py

If we encounter a toolchain with a major version number newer than the Visual Studio 2026 series, the test should fail. Make this hard-coded version a configurable variable near the top of the file.

The failure message informs the user to refer to MAINTENANCE.md to see what needs updating. That document explains the steps and rationale:

Strategy:

  • We code the newest-known toolchain in the test with the assumption it's supported by windows-latest.
  • We maintain earlier still-supported versions in ci.yml to ensure tests run on all supported toolchains.
  • When windows-latest is updated with a newer, unseen major Visual Studio release, we fail but point the developer at exactly what maintenance is needed.

Maintenance steps for a developer encountering a failure:

  • Update test_source_environment.py to recognize the next Windows compiler chain
  • Add the Windows runner for the just-removed toolchain to ci.yml
  • Determine whether to remove no-longer-supported runners from ci.yml in the same edit

The spots that will require updates should be clearly identified in MAINTENANCE.md, and the corresponding test and ci.yml should have comments pointing back to MAINTENANCE.md

Use language that would be clear to a mid-level engineer with only basic Linden AutoBuild exposure. Explain why, not just what.

Copilot AI commented Aug 2, 2026

Copy link
Copy Markdown
Author

@copilot Add an additional test in tests/test_source_environment.py

If we encounter a toolchain with a major version number newer than the Visual Studio 2026 series, the test should fail. Make this...

Done in the latest commit. Here's what was added:

tests/test_source_environment.py

  • NEWEST_KNOWN_VS_MAJOR = "18" # Visual Studio 2026 near the top of the file — this is the only value a developer needs to change when a new VS ships.
  • New test_newest_vs_known test (Windows/Cygwin only) that fails (not skips) with an explicit message pointing to MAINTENANCE.md if the detected VS major version exceeds the sentinel.
  • Both the sentinel constant and the new test have comments linking to MAINTENANCE.md.

.github/workflows/ci.yaml

  • The os matrix now has a comment explaining when and how to update it, linking back to MAINTENANCE.md.

MAINTENANCE.md (new)

  • Explains the strategy (sentinel test, pinned runners, retiring old runners).
  • Gives step-by-step instructions for a developer who hits the failure: update _VSTOOLSETS and the AUTOBUILD_WIN_CMAKE_GEN mapping in the autobuild source, bump NEWEST_KNOWN_VS_MAJOR, add the outgoing runner image to ci.yml, and optionally retire old images.
  • Includes a summary table of every file that needs touching.

Copilot AI requested a review from mcgroarty August 2, 2026 18:18
Updated comments in CI workflow for clarity.
@mcgroarty

Copy link
Copy Markdown
Contributor

@copilot The PR description is stale. Update it to reflect the actual implementation. Keep it concise, and targeted at a med-level developer with a basic Linden AutoBuild understanding. Provide adequate context such that they do not need to read the full thread.

Copilot AI commented Aug 2, 2026

Copy link
Copy Markdown
Author

@copilot The PR description is stale. Update it to reflect the actual implementation. Keep it concise, and targeted at a med-level developer with a basic Linden AutoBuild understanding. Provide adequa...

PR description updated. It now covers the full implementation: the windows-2022 matrix addition, the NEWEST_KNOWN_VS_MAJOR sentinel, the updated test_vstoolset_set, the new test_newest_vs_known failure test, and the new MAINTENANCE.md — with enough context that a developer doesn't need to read the thread.

Copilot AI requested a review from mcgroarty August 2, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

autobuild CI depends on "windows-latest" image, but fails because it expects Visual Studio 2022

2 participants