Skip to content

Fork-per-mutant inherits the parent's dead state; async tests are recorded as timeouts instead of their verdicts #578

Closed

Description

@MaxWolf-01

Disclaimer: AI-drafted, verified by me before filing. I answer follow-ups.

mutmut run executes the test suite in its own process (the stats, clean-tests and forced-fail passes) and then forks a child per mutant. Only the forking thread survives a fork, so anything process-global those passes started -- a thread pool, a connection pool -- is inherited dead. An async test that awaits it blocks in run_until_complete until the wall-clock deadline.

Expected: each mutant is killed or survives.
Actual: four of eight are recorded as timeouts, and two of those four are survivors the run never reports. The verdicts are lost, not merely slowed.

mutant mutation mutmut fresh interpreter
3 run_in_executor(_pool, None) timeout failed (killed), 0.29 s
6 lambda: None timeout failed (killed), 0.29 s
7 lambda: x / 2 timeout passed (survived), 0.29 s
8 lambda: x * 3 timeout passed (survived), 0.29 s

On a FastAPI/asyncpg service this made 15 of 15 mutants of one function time out; a fresh pytest process per mutant gave 15 verdicts in a tenth of the time.

Reproduce -- three files, mutmut 3.7.0, pytest 9.1.1, pytest-asyncio 1.0.0 and 1.4.0 (identical), CPython 3.12.12
# mylib/core.py
import asyncio
from concurrent.futures import ThreadPoolExecutor

_pool = ThreadPoolExecutor(max_workers=1)


async def double(x: int) -> int:
    loop = asyncio.get_running_loop()
    return await loop.run_in_executor(_pool, lambda: x * 2)
# tests/test_core.py
from mylib.core import double


async def test_double():
    assert await double(0) == 0
# pyproject.toml
[project]
name = "mylib"
version = "0"
requires-python = ">=3.12"

[tool.pytest.ini_options]
asyncio_mode = "auto"

[tool.mutmut]
source_paths = ["mylib/"]

Plus an empty mylib/__init__.py.

$ mutmut run
8/8  🎉 3 🫥 0  ⏰ 4  🤔 0  🙁 1  🔇 0  🧙 0

$ mutmut results
    mylib.core.x_double__mutmut_2: survived
    mylib.core.x_double__mutmut_3: timeout
    mylib.core.x_double__mutmut_6: timeout
    mylib.core.x_double__mutmut_7: timeout
    mylib.core.x_double__mutmut_8: timeout

The table above is those four run by hand from the generated mutants/ tree, one fresh interpreter each:

$ MUTANT_UNDER_TEST=mylib.core.x_double__mutmut_3 PYTHONPATH=. pytest -q
FAILED tests/test_core.py::test_double - TypeError: 'NoneType' object is not callable
1 failed in 0.29s
Cause -- the child's executor counts a worker that did not survive the fork
import os
from concurrent.futures import ThreadPoolExecutor, TimeoutError

pool = ThreadPoolExecutor(max_workers=1)
print("parent:", pool.submit(lambda: 1 * 2).result(timeout=5))

if os.fork() == 0:
    fut = pool.submit(lambda: 1 * 2)
    print("child threads:", len(pool._threads), "alive:", [t.is_alive() for t in pool._threads], flush=True)
    try:
        print("child:", fut.result(timeout=3), flush=True)
    except TimeoutError:
        print("child: still queued after 3s", flush=True)
    os._exit(0)
os.wait()
parent: 2
child threads: 1 alive: [False]
child: still queued after 3s

_threads holds the dead thread and the idle-semaphore token was inherited from the parent, so _adjust_thread_count starts no replacement and submit queues work nothing will run. The awaiting test then sits in the event loop until mutmut's deadline; a hung child's stack, via faulthandler:

  File ".../selectors.py", line 468 in select
  File ".../asyncio/base_events.py", line 1961 in _run_once
  File ".../asyncio/base_events.py", line 645 in run_forever
  File ".../asyncio/base_events.py", line 678 in run_until_complete
  File ".../pytest_asyncio/plugin.py", line 773 in inner
  File ".../_pytest/python.py", line 157 in pytest_pyfunc_call
The runner shape alone reproduces it, without mutmut -- 25 lines

Run pytest in-process, fork, run pytest in-process again in the child. The parent run passes and the child never finishes.

# forkprobe.py -- run with PYTHONPATH=. python forkprobe.py, against the files above
import faulthandler, os, signal, sys, time
import pytest

args = ["-q", "-p", "no:cacheprovider", "tests"]
print("=== parent run ===", flush=True)
print("parent exit:", pytest.main(args), flush=True)

pid = os.fork()
if pid == 0:
    dump = open("child-trace.txt", "w")
    faulthandler.register(signal.SIGUSR1, file=dump)
    os._exit(int(pytest.main(args)))

deadline = time.time() + 20
while time.time() < deadline:
    done, status = os.waitpid(pid, os.WNOHANG)
    if done:
        print("=== child exited:", os.waitstatus_to_exitcode(status), flush=True)
        sys.exit(0)
    time.sleep(0.2)
print("=== child still running after 20s; SIGUSR1 traceback follows ===", flush=True)
os.kill(pid, signal.SIGUSR1)
time.sleep(1)
os.kill(pid, signal.SIGKILL)
os.waitpid(pid, 0)
sys.exit(1)

A thread pool is the smallest thing that shows this; the same holds for anything else the pre-passes leave behind.

Related: #566 (open PR, fork-safe runners) targets this shape. #454 was the sync-trampoline defect for async functions, which fails fast rather than hanging.

Activity

  1. nicklafleur commented on Sep 12, 2026

    @nicklafleur
    Collaborator

    fixed in #566

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions