mutations-core intermittently hangs until the 60-minute job timeout

[iancooper]

Summary

The mutations-core job in mutation-tests.yml intermittently stalls and runs until the job's timeout-minutes: 60, so the run ends up cancelled. A normal run takes 7 to 8 minutes. gh run rerun <id> --failed has cleared it every time, with no code change.

It's frequent: 13 hangs in 77 workflow runs since 2026-08-26, about one run in six. It has only ever hit core. The other four targets (extensions, legacy, ratelimiting, testing) have never hung.

The hang is in the unmutated test suite, not a mutant

In every hung job, the last Stryker log line is one of the two steps that run Paramore.Fences.Core.Tests before any mutant is tested:

Stalls at	Jobs
Number of tests found: 698/700 … Initial test run started.	8
Capture mutant coverage using 'CoverageBasedTest' mode.	5

So this isn't a mutant causing an infinite loop, which Stryker would report as a Timeout mutant and move on from. Something in the Core test suite itself occasionally never finishes under Stryker's test runner. A plain local dotnet test test/Paramore.Fences.Core.Tests -f net10.0 also hung once in testhost on 2026-09-26, and a re-run with --blame-hang-timeout 90s then completed, so the hang may not be Stryker-specific.

Hung runs

Run	Branch	Date	Stalled at
36324157998	#45	2026-09-27	initial test run
36324334125	dependabot Test.Sdk	2026-09-27	coverage capture
36324051803	dependabot cake.tool	2026-09-27	initial test run
36109378060	#35	2026-09-25	coverage capture
35842152348	main	2026-09-23	initial test run
35522888773	main	2026-09-20	initial test run
35369065842	dependabot	2026-09-18	initial test run
34621245255	dependabot	2026-09-11	coverage capture
34164667916	main	2026-09-07	initial test run
33893497372	dependabot	2026-09-04	initial test run
33261573459	main	2026-08-29	initial test run
33154102952	dependabot	2026-08-28	coverage capture
32994115652	dependabot	2026-08-26	coverage capture

Related: the mutation score is also non-deterministic

Separately from the hang, main failed mutations-core on the #44 merge (36323895894) with a score of 99.72% against the 100% break threshold. #44 only changed two build properties. #45's run on the same main code scored 100% and passed.

One mutant survived: src/Paramore.Fences.Core/Utils/Pipeline/ExecutionTrackingComponent.cs:54, a Negate expression on if (_timeProvider.GetElapsedTime(start) > Timeout) in the DisposeAsync wait loop. Whether it's killed depends on timing. The line's // stryker disable once equality comment only covers equality mutations, so the negation isn't excluded. The run also reported 22 Timeout mutants, which suggests the suite is timing-sensitive under mutation generally.

Cost

• Each hang costs an hour of runner time plus a manual re-run, and it blocks the PR's checks until then.
• A spurious 99.x% score looks exactly like a real regression. That's the same trap Coverage collection is unreliable, and blocks the coverlet 10 upgrade #14 describes for coverage.

Possible directions (not decided)

• Find the hanging test: run the Core suite in CI with --blame-hang-timeout (outside Stryker) enough times to catch a hang dump. Or check whether Stryker's own test-runner settings can surface it.
• Fail fast: a much lower timeout-minutes for core (for example 20) would turn a one-hour wait into a quick, obvious failure. It doesn't fix the cause.
• Timing-dependent mutants: decide whether ExecutionTrackingComponent's wait-loop condition should be excluded explicitly or tested deterministically.
• Test-runner changes that could affect this: Port upstream xunit v3 / Microsoft.Testing.Platform v2 migration (App-vNext/Polly@1a80392b) #34 (xunit v3 / Microsoft.Testing.Platform v2) and Bump dotnet-stryker from 4.16.0 to 5.0.0 #39 (Stryker 4.16 → 5.0).

🤖 Generated with Claude Code