Skip to content

fix: Let an owning null handle be copied, and build as C++20 - #2

Merged
chenpeizhi merged 3 commits into
masterfrom
fix/null-handle-copy
Aug 16, 2026
Merged

chenpeizhi merged 3 commits into
masterfrom
fix/null-handle-copy

Conversation

@chenpeizhi

Copy link
Copy Markdown
Contributor

Fixes the crash behind DrudgeCAS/gristmill#45, and brings the project's own build in line with fbitset and libparenth.

The crash

Handle::incr_ref called Py_INCREF on whatever pointer it held, while decr_ref already used Py_XDECREF. An owning handle can hold a null pointer: Iter_handle::set_next stores the result of PyIter_Next with allow_null, so an exhausted iterator carries an owning null value. Copying that iterator dereferences the null pointer.

That copy happens whenever an iterator is used with a standard container or algorithm that copies its arguments. In gristmill it is

std::vector<Handle> dims(dims_inp.begin(), dims_inp.end());

over an empty list, which is what a product carrying no indices at all hands in. The interpreter dies with no traceback.

incr_ref now uses Py_XINCREF, so a null owning handle copies the way it already destructs. Three cases are added to the iterator tests: copying an exhausted iterator, building a vector from an empty iterable, and copying a plain owning null handle.

Verified with the gristmill wrapper under ASan and UBSan: parenth([], 0, [[]], 1, True) crashed before and now returns the trivial memoir cleanly.

C++20

The header itself already compiles as C++20; gristmill builds it that way. What did not was the project's own test build: C++14 in CMake, and a Catch2 2.0.1 single header that neither compiles under C++20 (std::uncaught_exception is gone) nor on arm64 (its breakpoint trap is x86 assembly). The build now sets C++20 with extensions off, fetches Catch2 v3.11.0 the way fbitset and libparenth do, and finds Python with FindPython3 in place of the deprecated FindPythonLibs.

Compiled locally as C++20 with -Wall -Wextra under ASan and UBSan; the header produces no warnings.

Tests on modern CPython

48 assertions in five test cases failed on any Python 3.12 or later, before this change. They assumed that the reference count of a small integer or of a static type moves when a handle is taken. Those objects are immortal now and their count is frozen. The handle tests use large integers, which are fresh objects, and the two tests that need the cached small integer for identity check its count only when it can move.

CI moves to ubuntu-latest with the system CMake and Python, Debug and Release. Locally on Python 3.13: all 374 assertions in 15 test cases pass.

🤖 Generated with Claude Code

chenpeizhi and others added 2 commits August 16, 2026 11:40
`Handle::incr_ref` incremented the reference count unconditionally, while
`decr_ref` already tolerated a null pointer.  An owning handle can hold a null
pointer: `Iter_handle::set_next` stores the result of `PyIter_Next` with
`allow_null`, so an exhausted iterator carries an owning null value.  Copying
such an iterator, which `std::vector<Handle>(it.begin(), it.end())` does,
dereferenced the null pointer and crashed.

This is what brings down gristmill on a product with no indices at all
(gristmill#45): `std::vector<Handle> dims(dims_inp.begin(), dims_inp.end())`
over an empty list.

Use `Py_XINCREF`, matching `decr_ref`, and cover the exhausted-iterator copy,
the container built from an empty iterable, and the plain null copy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…old on modern CPython

The library header already compiles as C++20, since gristmill builds it that
way, but the project's own build was pinned to C++14 with a Catch2 2.0.1
single header that no longer compiles under C++20 (std::uncaught_exception is
gone) nor on arm64 (its breakpoint trap is x86 assembly).  Set C++20, fetch
Catch2 v3.11.0 the way fbitset and libparenth do, and find Python with
FindPython3 instead of the deprecated FindPythonLibs.

The reference-count tests assumed that the count of a small integer or of a
static type moves when a handle is taken.  From Python 3.12 on those objects
are immortal and their count is frozen, so 48 assertions failed on any current
interpreter.  The handle tests now use large integers, which are fresh objects,
and the two tests that need the cached small integer for identity check its
count only when it can move.

CI runs on ubuntu-latest with the system CMake and Python, Debug and Release.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 16, 2026 04:26

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a null-dereference crash in Handle reference counting (triggered by copying exhausted iterators that hold owning-null handles) and modernizes the project build/test setup to align with C++20, Catch2 v3, and modern Python discovery.

Changes:

  • Fix Handle::incr_ref to tolerate owning-null handles by using Py_XINCREF.
  • Update tests for iterator exhaustion/null-handle copying and for Python ≥3.12 “immortal” objects where refcounts don’t change.
  • Modernize build/CI: C++20 (no extensions), Catch2 v3 via FetchContent, FindPython3, and a simplified GitHub Actions matrix.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
include/cpypp.hpp Fixes owning-null handle copying by using Py_XINCREF and updates related documentation.
CMakeLists.txt Moves project build to C++20 and switches Python discovery to FindPython3.
test/CMakeLists.txt Replaces Catch2 v2 single-header download with Catch2 v3 FetchContent + target linking.
.github/workflows/ci.yml Updates CI to ubuntu-latest with Debug/Release matrix and system dependencies.
test/testmain.cpp Switches to Catch2 v3 session runner and initializes/finalizes Python around tests.
test/handle.cpp Updates refcount-sensitive tests to use large integers (but currently with a portability issue).
test/iteratorprotocol.cpp Adds regression coverage for copying exhausted iterators and owning-null handles.
test/parsebuild.cpp Adjusts refcount assertions to account for Python ≥3.12 immortal small integers.
test/numericobjects.cpp Adjusts refcount assertions to account for Python ≥3.12 immortal small integers.
test/fundamentalobjects.cpp Adjusts static-type refcount assertions for Python ≥3.12 immutability behavior.
test/exc.cpp Catch2 v3 include update.
test/objectprotocol.cpp Catch2 v3 include update.
test/numberprotocol.cpp Catch2 v3 include update.
test/sequenceobjects.cpp Catch2 v3 include update.
test/otherobjects.cpp Catch2 v3 include update.
Suppressed comments (2)

test/handle.cpp:35

  • Same portability issue here: 1L << 41 overflows/UB when long is 32-bit. Use a 64-bit constructor (or Py_BuildValue("L", ...)) instead of shifting a long.
    PyObject* two = Py_BuildValue("l", 1L << 41);

test/handle.cpp:356

  • These two Py_BuildValue("l", 1L << ...) calls have the same LLP64 portability problem as above (32-bit long on Windows). Use PyLong_FromLongLong (or "L" format with 1LL << ...) to avoid UB and varargs type mismatch.
    PyObject* one = Py_BuildValue("l", 1L << 40);
    PyObject* two = Py_BuildValue("l", 1L << 41);

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread test/handle.cpp Outdated
// Large integers are used, since small integers are cached by CPython
// and immortal from Python 3.12 on, where their reference count never
// moves. These two are fresh objects whose count we fully control.
PyObject* one = Py_BuildValue("l", 1L << 40);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in a921674: the three integers are now built with Py_BuildValue("L", static_cast<long long>(1) << 40) and << 41, so the shift and the format agree on every platform.

Comment thread include/cpypp.hpp Outdated
Comment on lines 639 to 644
/** Increments the reference count only for owning reference to a non-null.
*
* An owning handle can hold a null pointer, for instance the value of an
* exhausted `Iter_handle`, so this has to tolerate null in the same way as
* `decr_ref` does.
*/

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworded in a921674: "Increments the reference count for an owning reference, if not null", with the note that null is tolerated here just as in decr_ref.

Copilot pointed out that 1L << 40 is not valid where long is 32 bits.  Use
long long, and reword the incr_ref comment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@chenpeizhi
chenpeizhi merged commit 2247a05 into master Aug 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants