Skip to content

Use runtime member set for protocol checks - #5644

Open
amas127 wants to merge 10 commits into
pyrefly-org:mainfrom
amas127:fix/protocol-runtime-members
Open

amas127 wants to merge 10 commits into
pyrefly-org:mainfrom
amas127:fix/protocol-runtime-members

Conversation

@amas127

@amas127 amas127 commented Nov 13, 2026

Copy link
Copy Markdown

Summary

Fixes #5614

issubclass() rejected @runtime_checkable protocols that declare __slots__ in their body, reporting invalid-argument even though CPython accepts the call. Protocols from typeshed are affected, so the false positive reaches ordinary library code:

from typing import Protocol, SupportsInt, runtime_checkable

@runtime_checkable
class Buffer(Protocol):
    __slots__ = ()
    def __buffer__(self, flags: int, /) -> memoryview: ...

issubclass(int, SupportsInt)        # before: error, after: no error
issubclass(bytes, Buffer)           # before: error, after: no error

One list of names was serving two different questions:

  • which names a conforming implementation must provide
  • which names CPython inspects when it checks a protocol at runtime

The structural check skips __slots__ because it is class machinery, not an instance member. The runtime check asked the same question of the same list with no such exception, so __slots__, a non-callable class attribute, was enough to make the protocol a data protocol, and correct code was rejected.

The runtime check now reads its own member list, built by the same rules CPython uses to decide what a protocol declares. It also decides whether a member is a method the way CPython does, from the value the name resolves to on the class rather than from the declared type. That fixes the reverse case too: a subclass method, or a decorated method,
that shadows an inherited slot is now accepted.

A few corners still diverge from CPython: an unresolvable __slots__ is treated as non-data, an annotation-only callable member is accepted even though CPython finds no class attribute, and private slot names are not mangled for the overlap check. Each one has a bug test marking the divergence so a follow-up can close it.

  • I am a human, and I fully understand the code I am submitting.

Test Plan

  • cargo test -p pyrefly --lib test::protocol::: 126 passed, including 27 new tests for slot declarations in each form, the two typeshed protocols, version boundaries, unsafe overlap, and shadowing through the MRO in both directions.
  • cargo test -p pyrefly --lib protocol: 240 passed, which adds the solver-side protocol tests. The neighbouring suites pass too: test::slots (49), test::attributes (310), test::class_subtyping (52).
  • Formatting and linting are clean. Clippy reports only the pre-existing operators.rs warning.
  • The classification was checked against CPython 3.15.0 for a method shadowing an inherited slot, a property, and a slot and a method on separate bases in both orders.

mypy_primer

Not run yet.

A runtime-checkable protocol declaring `__slots__ = ()` was classified as
a data protocol, so `issubclass(bytes, BufferProtocol)` and
`issubclass(int, SupportsInt)` reported `invalid-argument` even though
CPython accepts both. The same happened for typeshed protocols such as
`typing.SupportsInt` and `collections.abc.Buffer`, reported in pyrefly-org#5614.

`ProtocolMetadata.members` answered two questions at once: what structural
subtyping requires, and what the runtime protocol check inspects.
`__slots__` is an ordinary class-body field, so it entered `members`; the
protocol subsetting code already skipped it, which is why the consumers
disagreed.

Add `runtime_members`, built from the class-body fields minus the names
CPython's `typing._get_protocol_attrs` excludes, plus the slot names that
survive the same filter. `has_data_slots` records whether the declaration
proves a data attribute, because a slot name can resolve to an inherited
callable and the member type alone cannot decide it.

`is_data_protocol` and the unsafe-overlap presence check now read
`runtime_members`. The overlap check compares types only for names in
`members`: a slot-only name is part of the runtime contract but has no
protocol-side type to compare. `_abc_*` is skipped by prefix, and
`__class_getitem__` (3.9+), `__match_args__` (3.13+), and the `__annotate__`
family (3.14+) are gated to match the CPython versions that exclude them.
Inheritance merges both sets and the flag from protocol bases.

An unresolvable `__slots__` value is treated as non-data. The runtime may
reject the `issubclass()` call, but the checker cannot prove it; this is a
deliberate false negative.

Tests cover empty, non-empty, excluded, and inherited slots; string and
annotation-only declarations; typeshed protocols; the version boundaries;
and the unsafe-overlap presence rule.
A subclass method that shadows an inherited `__slots__` name was reported as
a data member, so `issubclass()` rejected the protocol even though CPython
accepts the call. The `has_data_slots` flag was inherited by every protocol
subclass, so it could not tell which class actually binds the name.

CPython decides whether a member is a method with `callable(getattr(cls,
name))`, so a data member is a name whose runtime value is non-callable.
`is_data_protocol` now walks the MRO per runtime member: a class-body binding
lets the declared read type decide, and a slot declaration that no nearer
class body initializes is a data member. A body that only annotates the name
creates no class attribute, so it does not shadow a slot. Members with no
runtime binding are classified by their declared type, which keeps the check
aligned with the type an implementation must provide; CPython would reject
those calls. The flag is no longer needed and is removed.

The test for a child method shadowing an inherited slot now expects no error.
New tests cover a slot and a method on separate bases in both orders, an
annotation-only member that still exposes the inherited slot, and the
declared-type rule for a callable annotation-only member and a property whose
getter returns a callable.
CPython decides whether a protocol member is a method with
`callable(getattr(cls, name))`. A `property` object is not callable, so a
runtime-checkable protocol whose member is a property always rejects
`issubclass()`.

`is_data_protocol` classified a class-body binding by its declared read type,
which for a property is the getter's return type. That accepted the call and,
for a property shadowing an inherited slot, removed an error the previous rule
reported. A property binding is now a data member. Other bindings, and members
with no runtime binding, still follow the declared read type.

Tests cover a property shadowing an inherited slot, a protocol whose only
member is a property, and decorated methods shadowing an inherited slot. Two
shapes where the check trusts the declaration over the runtime value keep their
divergence from CPython and now carry `bug` markers.
@meta-cla

meta-cla Bot commented Nov 13, 2026

Copy link
Copy Markdown

Hi @amas127!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

@meta-cla meta-cla Bot added the cla signed label Nov 13, 2026
@github-actions github-actions Bot added size/l and removed size/l labels Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

✅ This PR meets all requirements from our pull request checklist.

Optional recommendations:

  • This PR is on the larger side. If parts of it could be merged on their own, consider splitting it into separate PRs for faster review. If you've already split it as far as is practical, no action is needed.

Note: this comment updates automatically after changes to the PR. Updates may take up to half an hour.

@meta-codesync

meta-codesync Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

This pull request has been imported. If you are a Meta employee, you can view this in D131788114. (Because this pull request was imported automatically, there will not be any future comments.)

@stroxler stroxler self-assigned this Nov 13, 2026

@stroxler stroxler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I think this overall approach looks good.

A round of AI-assisted review found two issues: one regression that I think is worth digging into, and an easy fix where there's another dunder name we might as well include in our exclusions.

A. unsafe-overlap regression

Head location: special_calls.rs:700–716.

from typing import Protocol, runtime_checkable

@runtime_checkable
class P(Protocol):
    _abc_marker: int
    def f(self) -> int: ...

class C:
    _abc_marker: str = "wrong"
    def f(self) -> int:
        return 0

isinstance(C(), P)
p: P = C()

In this example:

  • The protocol requires _abc_marker: int.
  • The implementation provides _abc_marker: str.
  • Python ignores _abc_marker during isinstance(), so the check returns True.
  • Pyrefly still rejects assigning that implementation to the protocol, because str is incompatible with int.

That combination is what the unsafe-overlap warning catches: the runtime check succeeds even though the object doesn't satisfy the protocol's types.

The PR correctly removes _abc_marker from the runtime check. But it uses that same shortened list for checking types, so it loses the warning.

The suggested fix: keep the shortened list for checking which attributes Python looks for; use Pyrefly's ordinary protocol-assignment logic to check whether their types satisfy the protocol. Don't make _abc_marker a runtime requirement again.

incomplete runtime exclusions (minor but easy to fix)

Head location: class_metadata.rs:99–135.

from typing import Protocol, runtime_checkable

@runtime_checkable
class P(Protocol):
    __type_params__: tuple[object, ...]
    def f(self) -> int: ...

class C:
    def f(self) -> int:
        return 0

issubclass(C, P)

Targeting Python 3.14, both base and head emit the non-method-members diagnostic; CPython accepts the call. Thus this is an uncompleted case of the stated excluded-attribute problem, not a newly introduced regression. The same exact-head probe confirms the combined __protocol_attrs__ / __non_callable_proto_members__ annotation case is rejected.

The tested CPython 3.14 exclusion set also contains __static_attributes__ and __firstlineno__, absent from the new table. These entries are corroborated by pinned CPython v3.14.0 source. Use version-aware tests for the omitted names; do not infer a generated-name regression where Pyrefly never gathers the injected field.

@github-actions

Copy link
Copy Markdown

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

main moved the explicit_slots call after metaclass computation, while this branch needs the slots before final protocol metadata. Compute both after the metaclass is known and before parsed_results is consumed.
The unsafe-overlap check only iterated runtime members, so it skipped members that CPython excludes from its runtime check. Keep the runtime presence check over runtime_members, then compare the declared types of the protocol members the implementation provides. A missing runtime-excluded member stays optional, while an incompatible annotation or assignment is reported.
The unsafe-overlap check used runtime_members for both the presence gate and
the type comparison, so a runtime-excluded member was skipped even when the
implementation omitted it. Keep the presence gate on runtime_members, then
check every static protocol member. A missing runtime-excluded member now
reports an unsafe overlap, while a missing ordinary member still fails the
presence gate first.

Remove exclusions with no observed manual declaration: `_MutableMapping__marker`
and the internal PEP 749 companions `__annotate_func__` and
`__annotations_cache__`.

Add tests for missing members on both sides of the gate and for the
`__match_args__` 3.12/3.13 boundary.
Rename tests so the API in the name matches the call in the body: several
`test_issubclass_*` tests actually call `isinstance`.

Correct comments that gave the wrong reason for unsafe-overlap warnings or
described runtime-excluded names as static members. Fix typos in the
version-gated exclusion notes.

Make the private-name test observable by adding a type mismatch: pyrefly
now reports the false positive on the unmangled implementation and misses
the warning on the mangled one, so the bug marker covers both directions.
Only `pyrefly/lib/alt/special_calls.rs` conflicted. Main's display and
`TypeEnv` refactor changed `SubsetError::to_error_msg` to take the solver,
while this branch rewrote the runtime protocol overlap check. Keep the new
overlap logic and pass `self` to the error-message helper.
@amas127

amas127 commented Nov 16, 2026

Copy link
Copy Markdown
Author

@stroxler Thanks for your review and helpful comments! I believe issue A is real, while issue B is a deliberate FP misalignment with CPython runtime behavior after careful consideration.

A. unsafe-overlap regression: FIXED

The unsafe-overlap check used runtime_members for two different questions:

  • which names must be present for CPython's runtime check to pass;
  • which names to compare by type.

When the CPython-excluded names were removed from the runtime set, they also disappeared
from the type comparison.

The two questions now use separate sets:

  • The presence gate iterates runtime_members only. A missing ordinary runtime member fails the gate, so no unsafe overlap is reported; this matches CPython returning False and avoids warning about an overlap that does not exist.
  • Once the gate passes, the type check iterates the static members set. A missing runtime-excluded member and a present member with an incompatible type both produce unsafe overlap. The check skips only the four protocol-machinery dunders (__init__, __new__, __slots__, __class_getitem__), the same set subset.rs skips.

A testcase has been added to pyrefly/lib/test/protocol.rs to fix this behavior.

testcase!(
    test_isinstance_raise_unsafe_overlap_by_wrong_typing_with_abc_marker,
    r#"
from typing import Protocol, runtime_checkable

@runtime_checkable
class P(Protocol):
    _abc_marker: int
    def m(self) -> None: ...

class AnnotatedWrong:
    _abc_marker: str
    def m(self) -> None: ...

class AssignedWrong:
    _abc_marker = "wrong"
    def m(self) -> None: ...

class Missing:
    def m(self) -> None: ...

isinstance(AnnotatedWrong(), P)  # E: Runtime checkable protocol `P` has an unsafe overlap with type `AnnotatedWrong`
isinstance(AssignedWrong(), P)   # E: Runtime checkable protocol `P` has an unsafe overlap with type `AssignedWrong`
isinstance(Missing(), P)  # E: Runtime checkable protocol `P` has an unsafe overlap with type `Missing`
"#,
);

B. incomplete runtime exclusions: DEFERRED

This is deliberate. The exclusion table keeps a version gate only for names with a realistic manual-declaration surface: __class_getitem__ (3.9+), __match_args__ (3.13+), and __annotate__ (3.14+). The omitted names have no observed protocol-body declarations; adding them would create version branches for names ordinary code never writes.

Pyrefly used to treat every explicitly declared name as a protocol member. It now excludes the names real code writes while reports on the compiler- and runtime-injected ones, keeping its strict styles for implementation details.

Omitted attribute names are most likely to be compiler- or runtime-injected, and there is very few, if any, practical scenarios for them to be declared in a Protocol. In order to show that, I ran a "Debian Code Search" automatic audit with AI assistance:

Name CPython runtime exclusion Class-body declarations Protocol-body declarations Source
Observed in protocol bodies (retained)
__slots__ all versions (baseline) not counted separately 15 (11 typeshed + 4 asyncio) typeshed Buffer/SupportsInt; asyncio protocols
__orig_bases__ all versions (baseline) not counted separately ≥2 openai/anthropic InheritsGeneric
__subclasshook__ all versions (baseline) not counted separately 8 (4 runtime-checkable) CPython/typing_extensions tests
__module__ all versions (baseline) not counted separately 1 numpy _utils._HasModule
__init__ all versions (baseline) not counted separately 12 (9 typeshed + 3 asyncio) typeshed protocol classes
__new__ all versions (baseline) not counted separately 1 typeshed boltons _RSub
__class_getitem__ 3.9+ 908 1 beartype's Protocol
__match_args__ 3.13+ 879 8 CPython/typing_extensions tests
Class-body declarations, no protocol body (retained)
__annotate__ 3.14+ 45 0 typeshed on type/function
_is_protocol all versions (baseline) 28 0 typing internals/tests
_is_runtime_protocol all versions (baseline) 19 0 typing internals/tests
__abstractmethods__ all versions (baseline) 7 0 ABCMeta stubs/metaclasses
__annotations__ all versions (baseline) 75 (100-hit sample) 0 wrapt/toolz descriptors
__dict__ all versions (baseline) 74 (100-hit sample) 0 wrapt/charset-normalizer
__doc__ all versions (baseline) 56 (100-hit sample) 0 pyasn1
__orig_class__ all versions (baseline) 45 0 universal-pathlib/aiohttp
__parameters__ all versions (baseline) 53 0 genx/typing alias methods
__weakref__ all versions (baseline) 81 (100-hit sample) 0 dask/anyio
No protocol-body declarations (not added)
__type_params__ 3.12+ 0 in corpus; typeshed on type/function 0 PEP 695 runtime-injected
__protocol_attrs__ 3.12+ 0 0 runtime-injected
__callable_proto_members_only__ 3.12.0/3.12.1 0 0 runtime-injected
__non_callable_proto_members__ 3.12.2+ 0 0 runtime-injected
__static_attributes__ 3.13+ 0 0 compiler-injected
__firstlineno__ 3.13+ 0 (5 hits, not declarations) 0 compiler-injected
__typing_is_deprecated_inherited_runtime_protocol__ 3.15+ 0 0 runtime-injected
__final__ 3.11.5 only 0 0 decorator-injected
__annotate_func__ 3.14+ 0 0 internal PEP 749 companion
__annotations_cache__ 3.14+ 0 0 internal PEP 749 companion
_MutableMapping__marker all versions (baseline) 0 0 stale typing internal; removed from Pyrefly's baseline

Additional Changes

  • Remove unnecessary comments.
  • Split tests and remove redundant testcases.

@amas127
amas127 requested a review from stroxler November 16, 2026 20:38

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

False positive: Protocol with EXCLUDED_ATTRIBUTES cannot be used with issubclass()

2 participants