Repository navigation
Conversation
…ws it A std::string_view loaded from a bytearray points into the bytearray's storage. Keeping only the object alive does not stop Python code that runs during the call from resizing it, which frees the buffer under the view. On CPython, keep a memoryview alive instead so that resizing raises BufferError until the call returns.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When a bound function takes
std::string_viewand is passed abytearray,string_caster::load_rawgives the view a pointer straight into the bytearray's storage and only registers the bytearray object itself withloader_life_support. That keeps the object alive but not its buffer: a bytearray is resizable, so any Python code that runs while the C++ function is still using the view (a callback it invokes, or another thread when the function usespy::call_guard<py::gil_scoped_release>()or runs on a free-threaded build) can reallocate it. In the new test,string_view_call_then_copy(buf, lambda: buf.extend(b"y" * 1_000_000))reads the view after the callback returns, and an ASan build withPYTHONMALLOC=mallocreports a heap-use-after-free in the bound function with the free inPyByteArray_Resize; without ASan the returned bytes start with a stale allocator pointer.bytesandstrare immutable, so only the bytearray path is affected. On CPython the caster now keeps a memoryview of the bytearray alive for the call instead, which is the same export mechanismmemoryview/numpy rely on, so a resize during the call raisesBufferErrorand the bytearray is resizable again as soon as the call returns. PyPy and GraalPy keep the old behaviour: there the export is only released when the GC collects it, which would leave the bytearray locked well after the call, and I could not reproduce the corruption on PyPy 3.11. Tested with the full suite on CPython 3.12 (Debug,PYBIND11_WERROR=ON),test_builtin_casterson 3.13t, PyPy 3.11 and C++11, and the new test fails without thecast.hchange.Suggested changelog entry:
bytearraypassed as astd::string_viewargument was resized while the bound function was running; on CPython thebytearraynow cannot be resized for the duration of the call.