Mutating a typing.Protocol stub adds generated members to the Protocol and breaks type_check_command #590

Version: mutmut 3.8.0, mypy 2.4.0, Python 3.12, Linux.

What happens

With type_check_command set to mypy, mutmut run stops with:

Exception: Could not find mutant for type error .../mutants/pkg/store.py:85752 (Argument 1 to "project_current_state" has incompatible type "PgVectorStore"; expected "StateStore").

The unmutated project type checks clean.

Cause, as far as I can tell

The Protocol has a method stub with a default argument:

class StateStore(Protocol):
    def iter_chunks(self, batch_size: int = 1000) -> Iterable[Chunk]: ...

mutmut mutates that default (1000 to 1001) and rewrites the class with the trampoline decorator plus xǁStateStoreǁiter_chunks__mutmut_orig and xǁStateStoreǁiter_chunks__mutmut_1. Those two methods are now members of the Protocol. Any concrete class that satisfied it structurally no longer does, and mypy says so in its hint:

"PgVectorStore" is missing following "StateStore" protocol members:
    xǁStateStoreǁiter_chunks__mutmut_1, xǁStateStoreǁiter_chunks__mutmut_orig

Every call that passes the concrete class where the Protocol is expected then becomes a type error. One of them sat in the original method body behind the trampoline decorator, which no mutant owns, so filter_mutants_with_type_checker raised. The other 139 sat inside mutant functions, so without the crash those mutants would have been discarded as caught by the type checker, although none of them caused the error.

Minimal reproduction

mini/__init__.py is empty.

mini/proto.py:

from typing import Protocol


class Source(Protocol):
    def read(self, size: int = 10) -> str: ...


def consume(source: Source) -> str:
    return source.read(10)

mini/impl.py:

from mini.proto import consume


class File:
    def read(self, size: int = 10) -> str:
        return "x" * size

    def text(self) -> str:
        return consume(self)

tests/test_mini.py:

from mini.impl import File


def test_text() -> None:
    assert File().text() == "x" * 10

setup.cfg:

[mutmut]
source_paths=
    mini/
also_copy=
    tests/
type_check_command=
    mypy
    --output
    json
    mini/proto.py
    mini/impl.py

mypy mini/proto.py mini/impl.py reports no issues. mutmut run then fails:

Exception: Could not find mutant for type error .../mutants/mini/impl.py:26 (Argument 1 to "consume" has incompatible type "File"; expected "Source").

Line 26 of the mutated impl.py is the original body of File.text, under @_mutmut_mutated(mutants_xǁFileǁtext__mutmut). The mutated Protocol reads:

class Source(Protocol):
    @_mutmut_mutated(mutants_xǁSourceǁread__mutmut)
    def read(self, size: int = 10) -> str: ...
    def xǁSourceǁread__mutmut_orig(self, size: int = 10) -> str: ...
    def xǁSourceǁread__mutmut_1(self, size: int = 11) -> str: ...

Control: the same project with the stub's default removed (def read(self, size: int) -> str: ...) leaves Source unmutated, and mutmut run completes all 7 mutants (3 killed, 1 survived, 3 caught by the type checker).

Related

This is the same exception as #541, with a different trigger. #522 covered errors in files without mutants and is fixed; here the file does have mutants. It is also a concrete case of the concern raised in #541, that type errors may not stay inside the mutated function: the rewritten Protocol puts an error at every call site that passes a conforming class. In my project, 139 of the 140 resulting errors sat inside unrelated mutant functions, so those mutants would have been discarded as caught by the type checker, and a plain continue on the one unowned error would not have prevented that.

Suggestions

1. Do not mutate methods of a class that inherits from typing.Protocol. A stub's default is never executed, so such a mutant cannot be killed, and the rewrite changes the Protocol's set of members. This removes the cause rather than the symptom.
2. For the strict or flexible option discussed in Exception: Could not find mutant for type error #541: an error whose message or hint names a generated identifier (xǁ...__mutmut_...) is caused by the rewrite rather than by any mutant, so it could be ignored for attribution in either mode.

I worked around it with a wrapper around mypy that drops errors naming generated identifiers and errors outside every mutant, and I am happy to share it.