feat: process isolation and fork-safe test runners

ARCHITECTURE.rst

@@ -15,6 +15,13 @@ We start by copying ``source_paths`` to ``mutants/`` and then mutate the ``*.py`
 
 The mutated files contains the original code and the mutants. The active mutant is stored in process-local state (and mirrored to the ``MUTANT_UNDER_TEST`` environment variable). If a mutant is not enabled, it will run the original code.
 
+When ``mutate_only_covered_lines`` is set, we also run the test suite under coverage.py
+first, to find out which lines are worth mutating. That run happens in a forked child
+(``store_lines_covered_by_tests``) which sends back only the line numbers it measured.
+Running it in the main process would import the test suite and all of its dependencies
+there, and the later phases import the same modules again -- which some native extensions
+do not survive.
+
 
 Collecting tests and stats
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -48,3 +55,17 @@ We finally check, which mutations are caught by the test suite.
 For each mutant, we execute the test suite. If any of the tests fails, we successfully killed the mutant. To optimize performance, we only execute the tests that could cover the mutant and sort them by mutation time. We also skip mutants, which already have a result from a previous run.
 
 The results are stored in the ``.meta`` files.
+
+Every mutant is tested in its own process. Which process that one is forked from is
+owned by a ``MutantRunner`` (``src/mutmut/workers/isolation.py``), selected by the
+``process_isolation`` config. ``ForkRunner`` forks each worker straight from mutmut's
+main process, which by this point has imported pytest and run the suite once to collect
+stats. ``ForkServerRunner`` instead keeps the main process free of pytest: it forks a
+dedicated fork server child, and that child imports pytest once and forks a worker per
+mutant, streaming results back over a length-prefixed pipe. That extra generation exists
+for test setups that are not fork-safe, where inheriting the main process would hang or
+crash the workers.
+
+The runner also owns the surrounding test operations (stats collection, clean tests,
+forced fail, test listing) so that, under ``forkserver``, those too can run in short-lived
+forks and leave the main process clean.

HISTORY.rst

@@ -20,8 +20,12 @@ Unreleased
 
 * Fix mutants stopped by the CPU-time limit being reported as killed instead of timed out (`#565`)
 
+* Fix ``mutate_only_covered_lines`` breaking when the test suite imports a dependency that cannot be initialized twice in one process, which raised for numpy and libyaml, produced misleading type errors for cryptography, and segfaulted for asyncpg. Coverage is now gathered in a separate process, so the modules the coverage run imports are no longer unloaded and re-imported in the main process (`#528`, `#566`)
+
 * Mutate the condition of ternary expressions (`#196`, `#546`)
 
+* Add a ``process_isolation`` config for choosing which process mutant workers are forked from. The new ``forkserver`` strategy keeps mutmut's main process free of pytest and your ``conftest.py``, routing every fork through a dedicated fork server process, for test setups that are not fork-safe (``gevent``, ``grpc``, ``torch``). Tune what it preloads with ``forkserver_warmup`` (`#578`, `#566`)
+
 * Add a `badge` command that writes a shields.io endpoint file, for publishing the mutation score (`#549`)
 
 * Support python3.15 (`#551`)

README.rst

@@ -236,6 +236,108 @@ macOS at your own risk, or to disable it on other platforms), set ``use_setproct
     use_setproctitle = false
 
 
+Process isolation (``fork`` vs ``forkserver``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: toml
+
+    # pyproject.toml
+    [tool.mutmut]
+    process_isolation = "fork"  # or "forkserver"
+
+Both strategies require ``os.fork``, so they are POSIX-only.
+
+``fork`` (default)
+    Mutmut's main process imports pytest and runs your test suite to collect
+    stats, then forks one child per mutant. This is fast and needs no
+    configuration, but every mutant worker inherits whatever your test setup
+    left behind in that process: monkey-patched sockets, running event loops,
+    background threads, open database connections, CUDA contexts. If any of
+    that is not fork-safe, the workers can hang, segfault, or report nonsense.
+
+``forkserver``
+    Mutmut's main process never imports pytest or your ``conftest.py``. It forks
+    a dedicated *fork server* child, and that child does the importing once and
+    forks a worker per mutant::
+
+        main process (clean) -> fork server (imports pytest) -> N mutant workers
+
+    Because the fork server only imports and forks, and never runs a test
+    itself, every worker starts from the same predictable state. This is the
+    same design as the ``forkserver`` start method in Python's
+    ``multiprocessing``, and it is still only one pytest import for the whole
+    run, so it is not slower than ``fork`` in the normal case.
+
+**Which one should you use?** Start with ``fork``. Switch to ``forkserver`` if
+your run hangs, crashes, or gives results you cannot reproduce by running the
+tests by hand, and your test setup touches something known to be fork-unsafe,
+such as ``gevent.monkey.patch_all()``, ``grpc``, or ``torch``. It is also the
+better choice if you simply do not want your session-scoped setup living in the
+process that forks mutants.
+
+
+Tuning what the fork server preloads
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``forkserver_warmup`` controls how much the fork server imports before it starts
+forking workers. It only applies when ``process_isolation = "forkserver"``.
+Anything imported in the fork server is inherited by every  worker, and anything
+left out is imported in each worker.
+
+.. code-block:: toml
+
+    # pyproject.toml
+    [tool.mutmut]
+    process_isolation = "forkserver"
+    forkserver_warmup = "collect"  # "collect" (default), "import" or "none"
+
+``collect`` (default)
+    Run ``pytest --collect-only``, which imports your ``conftest.py``, your
+    plugins and your test modules. This is the right default for most projects
+    and usually the fastest.
+
+``import``
+    Import only the modules listed in ``preload_modules_file`` (one dotted
+    module name per line; blank lines and ``#`` comments are ignored, and
+    modules that fail to import are skipped). Use this when test *collection*
+    itself has side effects you do not want every worker to share, but you still
+    want your slow imports warmed up.
+
+``none``
+    Import nothing beyond what running a single test requires. This is the
+    slowest option, and the most isolated. It is mostly useful for working out
+    whether a warmup side effect is what is breaking your run.
+
+.. code-block:: toml
+
+    # pyproject.toml
+    [tool.mutmut]
+    process_isolation = "forkserver"
+    forkserver_warmup = "import"
+    preload_modules_file = "mutmut_preload.txt"
+
+If the fork server itself crashes, mutmut starts a new one and re-submits the
+mutants that were in flight, up to ``max_forkserver_restarts`` times (default
+``3``) before giving up with a ``ForkServerCrashError``:
+
+.. code-block:: toml
+
+    # pyproject.toml
+    [tool.mutmut]
+    max_forkserver_restarts = 3
+
+The fork server and its workers cannot write to the console without mangling
+mutmut's progress output, so they log to a file instead. Logging is off unless
+you ask for it with ``log_to_file`` (or ``debug``):
+
+.. code-block:: toml
+
+    # pyproject.toml
+    [tool.mutmut]
+    log_to_file = true
+    log_file_path = "mutants/mutmut-debug.log"
+
+
 Disabling mutation on specific code
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

e2e_projects/benchmark_1k/README.md

@@ -1,10 +1,10 @@
 # Benchmark 1K
 
-A synthetic benchmark project with 1000 mutants for validating mutmut's process isolation and hot-fork warmup strategy performance.
+A synthetic benchmark project with 1000 mutants for validating mutmut's process isolation and forkserver warmup strategy performance.
 
 **TL;DR:**
 - `fork` is fastest and nearly immune to import delays (requires fork-safe libraries)
-- `collect` (hot-fork default) is 2-9x faster than `import`/`none` depending on import cost
+- `collect` (forkserver default) is 2-9x faster than `import`/`none` depending on import cost
 - Higher import delays dramatically penalize `import` and `none` strategies
 
 

e2e_projects/benchmark_1k/mutmut_preload.txt

@@ -1,5 +1,5 @@
 # Modules to preload for the 'import' warmup strategy
-# These are imported in the orchestrator and inherited by forked workers
+# These are imported in the fork server and inherited by forked workers
 pytest
 
 #normally you would avoid importing the module itself in real projects

e2e_projects/benchmark_1k/pyproject.toml

@@ -14,6 +14,6 @@ packages = ["src/benchmark"]
 [tool.mutmut]
 log_to_file = true
 source_paths = ["src/"]
-process_isolation = "hot-fork"
+process_isolation = "forkserver"
 debug = false
-hot_fork_warmup = "none"
+forkserver_warmup = "none"

e2e_projects/benchmark_1k/run_benchmark.py

@@ -69,10 +69,10 @@ def run_mutmut(
     show_output: bool = False,
 ) -> dict:
     """Run mutmut with specified strategy."""
-    config = get_pyproject_content(debug=verbose, process_isolation=strategy if strategy == "fork" else "hot-fork")
+    config = get_pyproject_content(debug=verbose, process_isolation=strategy if strategy == "fork" else "forkserver")
 
     if strategy != "fork":
-        config += f'hot_fork_warmup = "{strategy}"\n'
+        config += f'forkserver_warmup = "{strategy}"\n'
         if strategy == "import":
             config += 'preload_modules_file = "mutmut_preload.txt"\n'
 

e2e_projects/forkserver_basic/pyproject.toml

@@ -0,0 +1,14 @@
+[project]
+name = "fs-calc"
+version = "0.1.0"
+requires-python = ">=3.10"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.mutmut]
+process_isolation = "forkserver"
+
+[tool.pytest.ini_options]
+asyncio_default_fixture_loop_scope = "function"

e2e_projects/forkserver_basic/src/fs_calc/__init__.py

@@ -0,0 +1,14 @@
+def add(a, b):
+    return a + b
+
+
+def sub(a, b):
+    return a - b
+
+
+def mul(a, b):
+    return a * b
+
+
+def untested(x):
+    return x + 1

e2e_projects/forkserver_basic/tests/test_fs_calc.py

@@ -0,0 +1,15 @@
+from fs_calc import add
+from fs_calc import mul
+from fs_calc import sub
+
+
+def test_add():
+    assert add(1, 2) == 3
+
+
+def test_sub():
+    assert sub(5, 2) == 3
+
+
+def test_mul():
+    assert mul(3, 4) == 12

e2e_projects/mutate_only_covered_lines/src/mutate_only_covered_lines/ignore_single_init_dep.py

@@ -0,0 +1,21 @@
+"""Stands in for a dependency that cannot be initialized twice in one process.
+
+Regression guard for #528, where numpy and libyaml raise, cryptography fails a type
+check and asyncpg segfaults. Named to match ``do_not_mutate = ["*ignore*"]``, so it is
+copied and imported like any dependency but stays out of the result snapshot.
+"""
+
+import os
+
+# Outside the module, or it would be evicted with it. Keyed by pid so forks, which
+# inherit a copy of the environment, each get their own initialization.
+_INITIALIZED_IN_PID = "MUTMUT_E2E_SINGLE_INIT_DEP_PID"
+
+if os.environ.get(_INITIALIZED_IN_PID) == str(os.getpid()):
+    raise ImportError("cannot load module more than once per process")
+
+os.environ[_INITIALIZED_IN_PID] = str(os.getpid())
+
+
+def single_init_value():
+    return 42

e2e_projects/mutate_only_covered_lines/tests/main/test_mutate_only_covered_lines.py

@@ -1,4 +1,6 @@
 from mutate_only_covered_lines.ignore_me import this_function_shall_NOT_be_mutated
+# Raises on a second import in the same process; see the module docstring (#528).
+from mutate_only_covered_lines.ignore_single_init_dep import single_init_value
 from mutate_only_covered_lines import hello_mutate_only_covered_lines, mutate_only_covered_lines_multiline, function_with_pragma, do_not_mutate_external_ommited_function
 from mutate_only_covered_lines.exclude_me import excluded_function, excluded_branch, excluded_by_coverage_config, excluded_case
 
@@ -33,3 +35,6 @@ def test_excluded_by_coverage_config():
 def test_excluded_case():
     assert excluded_case("a") == 1
     assert excluded_case("b") == 2
+
+def test_single_init_dep():
+    assert single_init_value() == 42