Repository navigation
Fork-per-mutant inherits the parent's dead state; async tests are recorded as timeouts instead of their verdicts #578
Description
Disclaimer: AI-drafted, verified by me before filing. I answer follow-ups.
mutmut run executes the test suite in its own process (the stats, clean-tests and forced-fail passes) and then forks a child per mutant. Only the forking thread survives a fork, so anything process-global those passes started -- a thread pool, a connection pool -- is inherited dead. An async test that awaits it blocks in run_until_complete until the wall-clock deadline.
Expected: each mutant is killed or survives.
Actual: four of eight are recorded as timeouts, and two of those four are survivors the run never reports. The verdicts are lost, not merely slowed.
| mutant | mutation | mutmut | fresh interpreter |
|---|---|---|---|
| 3 | run_in_executor(_pool, None) |
timeout | failed (killed), 0.29 s |
| 6 | lambda: None |
timeout | failed (killed), 0.29 s |
| 7 | lambda: x / 2 |
timeout | passed (survived), 0.29 s |
| 8 | lambda: x * 3 |
timeout | passed (survived), 0.29 s |
On a FastAPI/asyncpg service this made 15 of 15 mutants of one function time out; a fresh pytest process per mutant gave 15 verdicts in a tenth of the time.
Reproduce -- three files, mutmut 3.7.0, pytest 9.1.1, pytest-asyncio 1.0.0 and 1.4.0 (identical), CPython 3.12.12
# mylib/core.py
import asyncio
from concurrent.futures import ThreadPoolExecutor
_pool = ThreadPoolExecutor(max_workers=1)
async def double(x: int) -> int:
loop = asyncio.get_running_loop()
return await loop.run_in_executor(_pool, lambda: x * 2)# tests/test_core.py
from mylib.core import double
async def test_double():
assert await double(0) == 0# pyproject.toml
[project]
name = "mylib"
version = "0"
requires-python = ">=3.12"
[tool.pytest.ini_options]
asyncio_mode = "auto"
[tool.mutmut]
source_paths = ["mylib/"]Plus an empty mylib/__init__.py.
$ mutmut run
8/8 🎉 3 🫥 0 ⏰ 4 🤔 0 🙁 1 🔇 0 🧙 0
$ mutmut results
mylib.core.x_double__mutmut_2: survived
mylib.core.x_double__mutmut_3: timeout
mylib.core.x_double__mutmut_6: timeout
mylib.core.x_double__mutmut_7: timeout
mylib.core.x_double__mutmut_8: timeoutThe table above is those four run by hand from the generated mutants/ tree, one fresh interpreter each:
$ MUTANT_UNDER_TEST=mylib.core.x_double__mutmut_3 PYTHONPATH=. pytest -q
FAILED tests/test_core.py::test_double - TypeError: 'NoneType' object is not callable
1 failed in 0.29sCause -- the child's executor counts a worker that did not survive the fork
import os
from concurrent.futures import ThreadPoolExecutor, TimeoutError
pool = ThreadPoolExecutor(max_workers=1)
print("parent:", pool.submit(lambda: 1 * 2).result(timeout=5))
if os.fork() == 0:
fut = pool.submit(lambda: 1 * 2)
print("child threads:", len(pool._threads), "alive:", [t.is_alive() for t in pool._threads], flush=True)
try:
print("child:", fut.result(timeout=3), flush=True)
except TimeoutError:
print("child: still queued after 3s", flush=True)
os._exit(0)
os.wait()parent: 2
child threads: 1 alive: [False]
child: still queued after 3s_threads holds the dead thread and the idle-semaphore token was inherited from the parent, so _adjust_thread_count starts no replacement and submit queues work nothing will run. The awaiting test then sits in the event loop until mutmut's deadline; a hung child's stack, via faulthandler:
File ".../selectors.py", line 468 in select
File ".../asyncio/base_events.py", line 1961 in _run_once
File ".../asyncio/base_events.py", line 645 in run_forever
File ".../asyncio/base_events.py", line 678 in run_until_complete
File ".../pytest_asyncio/plugin.py", line 773 in inner
File ".../_pytest/python.py", line 157 in pytest_pyfunc_call
The runner shape alone reproduces it, without mutmut -- 25 lines
Run pytest in-process, fork, run pytest in-process again in the child. The parent run passes and the child never finishes.
# forkprobe.py -- run with PYTHONPATH=. python forkprobe.py, against the files above
import faulthandler, os, signal, sys, time
import pytest
args = ["-q", "-p", "no:cacheprovider", "tests"]
print("=== parent run ===", flush=True)
print("parent exit:", pytest.main(args), flush=True)
pid = os.fork()
if pid == 0:
dump = open("child-trace.txt", "w")
faulthandler.register(signal.SIGUSR1, file=dump)
os._exit(int(pytest.main(args)))
deadline = time.time() + 20
while time.time() < deadline:
done, status = os.waitpid(pid, os.WNOHANG)
if done:
print("=== child exited:", os.waitstatus_to_exitcode(status), flush=True)
sys.exit(0)
time.sleep(0.2)
print("=== child still running after 20s; SIGUSR1 traceback follows ===", flush=True)
os.kill(pid, signal.SIGUSR1)
time.sleep(1)
os.kill(pid, signal.SIGKILL)
os.waitpid(pid, 0)
sys.exit(1)A thread pool is the smallest thing that shows this; the same holds for anything else the pre-passes leave behind.
Related: #566 (open PR, fork-safe runners) targets this shape. #454 was the sync-trampoline defect for async functions, which fails fast rather than hanging.
Activity
nicklafleur commentedon Sep 12, 2026
fixed in #566