Repository navigation
fix: Let an owning null handle be copied, and build as C++20 - #2
Conversation
`Handle::incr_ref` incremented the reference count unconditionally, while `decr_ref` already tolerated a null pointer. An owning handle can hold a null pointer: `Iter_handle::set_next` stores the result of `PyIter_Next` with `allow_null`, so an exhausted iterator carries an owning null value. Copying such an iterator, which `std::vector<Handle>(it.begin(), it.end())` does, dereferenced the null pointer and crashed. This is what brings down gristmill on a product with no indices at all (gristmill#45): `std::vector<Handle> dims(dims_inp.begin(), dims_inp.end())` over an empty list. Use `Py_XINCREF`, matching `decr_ref`, and cover the exhausted-iterator copy, the container built from an empty iterable, and the plain null copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…old on modern CPython The library header already compiles as C++20, since gristmill builds it that way, but the project's own build was pinned to C++14 with a Catch2 2.0.1 single header that no longer compiles under C++20 (std::uncaught_exception is gone) nor on arm64 (its breakpoint trap is x86 assembly). Set C++20, fetch Catch2 v3.11.0 the way fbitset and libparenth do, and find Python with FindPython3 instead of the deprecated FindPythonLibs. The reference-count tests assumed that the count of a small integer or of a static type moves when a handle is taken. From Python 3.12 on those objects are immortal and their count is frozen, so 48 assertions failed on any current interpreter. The handle tests now use large integers, which are fresh objects, and the two tests that need the cached small integer for identity check its count only when it can move. CI runs on ubuntu-latest with the system CMake and Python, Debug and Release. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Fixes a null-dereference crash in Handle reference counting (triggered by copying exhausted iterators that hold owning-null handles) and modernizes the project build/test setup to align with C++20, Catch2 v3, and modern Python discovery.
Changes:
- Fix
Handle::incr_refto tolerate owning-null handles by usingPy_XINCREF. - Update tests for iterator exhaustion/null-handle copying and for Python ≥3.12 “immortal” objects where refcounts don’t change.
- Modernize build/CI: C++20 (no extensions), Catch2 v3 via FetchContent,
FindPython3, and a simplified GitHub Actions matrix.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
include/cpypp.hpp |
Fixes owning-null handle copying by using Py_XINCREF and updates related documentation. |
CMakeLists.txt |
Moves project build to C++20 and switches Python discovery to FindPython3. |
test/CMakeLists.txt |
Replaces Catch2 v2 single-header download with Catch2 v3 FetchContent + target linking. |
.github/workflows/ci.yml |
Updates CI to ubuntu-latest with Debug/Release matrix and system dependencies. |
test/testmain.cpp |
Switches to Catch2 v3 session runner and initializes/finalizes Python around tests. |
test/handle.cpp |
Updates refcount-sensitive tests to use large integers (but currently with a portability issue). |
test/iteratorprotocol.cpp |
Adds regression coverage for copying exhausted iterators and owning-null handles. |
test/parsebuild.cpp |
Adjusts refcount assertions to account for Python ≥3.12 immortal small integers. |
test/numericobjects.cpp |
Adjusts refcount assertions to account for Python ≥3.12 immortal small integers. |
test/fundamentalobjects.cpp |
Adjusts static-type refcount assertions for Python ≥3.12 immutability behavior. |
test/exc.cpp |
Catch2 v3 include update. |
test/objectprotocol.cpp |
Catch2 v3 include update. |
test/numberprotocol.cpp |
Catch2 v3 include update. |
test/sequenceobjects.cpp |
Catch2 v3 include update. |
test/otherobjects.cpp |
Catch2 v3 include update. |
Suppressed comments (2)
test/handle.cpp:35
- Same portability issue here:
1L << 41overflows/UB whenlongis 32-bit. Use a 64-bit constructor (orPy_BuildValue("L", ...)) instead of shifting along.
PyObject* two = Py_BuildValue("l", 1L << 41);
test/handle.cpp:356
- These two
Py_BuildValue("l", 1L << ...)calls have the same LLP64 portability problem as above (32-bitlongon Windows). UsePyLong_FromLongLong(or"L"format with1LL << ...) to avoid UB and varargs type mismatch.
PyObject* one = Py_BuildValue("l", 1L << 40);
PyObject* two = Py_BuildValue("l", 1L << 41);
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // Large integers are used, since small integers are cached by CPython | ||
| // and immortal from Python 3.12 on, where their reference count never | ||
| // moves. These two are fresh objects whose count we fully control. | ||
| PyObject* one = Py_BuildValue("l", 1L << 40); |
There was a problem hiding this comment.
Done in a921674: the three integers are now built with Py_BuildValue("L", static_cast<long long>(1) << 40) and << 41, so the shift and the format agree on every platform.
| /** Increments the reference count only for owning reference to a non-null. | ||
| * | ||
| * An owning handle can hold a null pointer, for instance the value of an | ||
| * exhausted `Iter_handle`, so this has to tolerate null in the same way as | ||
| * `decr_ref` does. | ||
| */ |
There was a problem hiding this comment.
Reworded in a921674: "Increments the reference count for an owning reference, if not null", with the note that null is tolerated here just as in decr_ref.
Copilot pointed out that 1L << 40 is not valid where long is 32 bits. Use long long, and reword the incr_ref comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes the crash behind DrudgeCAS/gristmill#45, and brings the project's own build in line with fbitset and libparenth.
The crash
Handle::incr_refcalledPy_INCREFon whatever pointer it held, whiledecr_refalready usedPy_XDECREF. An owning handle can hold a null pointer:Iter_handle::set_nextstores the result ofPyIter_Nextwithallow_null, so an exhausted iterator carries an owning null value. Copying that iterator dereferences the null pointer.That copy happens whenever an iterator is used with a standard container or algorithm that copies its arguments. In gristmill it is
std::vector<Handle> dims(dims_inp.begin(), dims_inp.end());over an empty list, which is what a product carrying no indices at all hands in. The interpreter dies with no traceback.
incr_refnow usesPy_XINCREF, so a null owning handle copies the way it already destructs. Three cases are added to the iterator tests: copying an exhausted iterator, building a vector from an empty iterable, and copying a plain owning null handle.Verified with the gristmill wrapper under ASan and UBSan:
parenth([], 0, [[]], 1, True)crashed before and now returns the trivial memoir cleanly.C++20
The header itself already compiles as C++20; gristmill builds it that way. What did not was the project's own test build: C++14 in CMake, and a Catch2 2.0.1 single header that neither compiles under C++20 (
std::uncaught_exceptionis gone) nor on arm64 (its breakpoint trap is x86 assembly). The build now sets C++20 with extensions off, fetches Catch2 v3.11.0 the way fbitset and libparenth do, and finds Python withFindPython3in place of the deprecatedFindPythonLibs.Compiled locally as C++20 with
-Wall -Wextraunder ASan and UBSan; the header produces no warnings.Tests on modern CPython
48 assertions in five test cases failed on any Python 3.12 or later, before this change. They assumed that the reference count of a small integer or of a static type moves when a handle is taken. Those objects are immortal now and their count is frozen. The handle tests use large integers, which are fresh objects, and the two tests that need the cached small integer for identity check its count only when it can move.
CI moves to ubuntu-latest with the system CMake and Python, Debug and Release. Locally on Python 3.13: all 374 assertions in 15 test cases pass.
🤖 Generated with Claude Code