Repository navigation
Avoid creating incomplete/invalid objects #36
Description
From capi-workgroup/problems#56
Related to #20 (Disallow mutating immutable objects)
New API should not allow creation of incomplete/invalid objects.
In particular, a traverse function must be safe to call right after an object is tracked with the GC.
Reacted by Antoine Pitrou
Activity
scoder commentedon Oct 24, 2023
These are actually two independent requirements. It's easy to construct an API for creating container objects incrementally that keeps the object out of GC tracking while it's being built. This is independent of the question whether the object has been created in one go or incrementally before it is added to the GC tracking.
encukou commentedon Oct 26, 2023
Yes. This guideline should disallow both:
traverse(a particularly tricky class of invalid objects)That could be solved with one-shot creation API, and/or a "builder" approach that makes the user do all the steps, in the proper order. (And if those approaches can't be fast enough, it's time for an exception -- see #4.)
vstinner commentedon Oct 30, 2023
In Python 3.13 alpha1, I modified PyList_SET_ITEM() to restrict index to
[0; Py_SIZE(list)[range. It caused issues in Cython, and so it was changed to allow index in[0; list->allocated[instead: python/cpython#111480I dislike PyList_New() + PyList_SET_ITEM() API since the list is immediately tracked by the GC and so calling gc.get_objects() can expose an invalid list object (ex: calling
repr(list)can crash).With python/cpython#111480 change,
PyList_SetItem()andPyList_SET_ITEM()become inconsistent:PyList_SetItem()useslist->ob_size, whereasPyList_SET_ITEM()useslist->allocated.PyList_SET_ITEM()is a strange beast: it doesn't DECREF the old value, it must only be used to initialize a list newly allocated by PyList_New(). ButPyList_New(size)setslist->ob_sizetosize.PyList_SetItem()is more regular: it calls DECREF on the old item.Well, python/cpython#111480 was not about designing a correct API, but to migrate Python 3.12 code to Python 3.13. Previously,
PyList_SET_ITEM()could write safely in[0; allocated[range, and in Python 3.13 it's limited to[0; ob_size[.vstinner commentedon Oct 30, 2023
For list, we need an API to transfer objects from a C array to a list without the expensive INCREF/DECREF dance: make
PyObject * _PyList_FromArraySteal(PyObject *const *src, Py_ssize_t n)public, but rename it to avoid "Steal" misleading name. In fact, ownership is "moved" or "transfered", not "stolen". cc @erlend-aaslandvstinner commentedon Oct 30, 2023
I created an issue for that: python/cpython#111489
vstinner commentedon Jul 4, 2024
I added
PyUnicodeWriterAPI to solve this problem for Unicode strings: https://docs.python.org/dev/c-api/unicode.html#pyunicodewriter It provides a convenient API to build a string. The object is only exposed to Python once the caller is done and calls thePyUnicodeWriter_Finish()method.I'm working on a similar API to create a Python int object from an array of digits,
PyLongWriter: PR python/cpython#121339.scoder commentedon Jul 5, 2024
vstinner commentedon Jul 5, 2024
PyUnicodeWriter allocates a structure on the heap memory. It's not a Python object.
gvanrossum commentedon Jul 5, 2024
This is unfortunate (creating a string this way now requires two heap allocations) but if we were to use a stack-allocated writer object instead, it's hard to imagine how this API could ever be promoted to the stable ABI, and it would still require a separate API call (to reveal the created object in its final form). So I've made peace with this design.
vstinner commentedon Jul 13, 2024
I created python/cpython#121710: [C API] Add PyBytesWriter API.