Skip to content

fix: hold a buffer export on bytearray while a std::string_view borrows it - #6219

Open
kitkatt7 wants to merge 1 commit into
pybind:masterfrom
kitkatt7:string-view-bytearray-export
Open

kitkatt7 wants to merge 1 commit into
pybind:masterfrom
kitkatt7:string-view-bytearray-export

Conversation

@kitkatt7

Copy link
Copy Markdown

Description

When a bound function takes std::string_view and is passed a bytearray, string_caster::load_raw gives the view a pointer straight into the bytearray's storage and only registers the bytearray object itself with loader_life_support. That keeps the object alive but not its buffer: a bytearray is resizable, so any Python code that runs while the C++ function is still using the view (a callback it invokes, or another thread when the function uses py::call_guard<py::gil_scoped_release>() or runs on a free-threaded build) can reallocate it. In the new test, string_view_call_then_copy(buf, lambda: buf.extend(b"y" * 1_000_000)) reads the view after the callback returns, and an ASan build with PYTHONMALLOC=malloc reports a heap-use-after-free in the bound function with the free in PyByteArray_Resize; without ASan the returned bytes start with a stale allocator pointer. bytes and str are immutable, so only the bytearray path is affected. On CPython the caster now keeps a memoryview of the bytearray alive for the call instead, which is the same export mechanism memoryview/numpy rely on, so a resize during the call raises BufferError and the bytearray is resizable again as soon as the call returns. PyPy and GraalPy keep the old behaviour: there the export is only released when the GC collects it, which would leave the bytearray locked well after the call, and I could not reproduce the corruption on PyPy 3.11. Tested with the full suite on CPython 3.12 (Debug, PYBIND11_WERROR=ON), test_builtin_casters on 3.13t, PyPy 3.11 and C++11, and the new test fails without the cast.h change.

Suggested changelog entry:

  • Fixed a use-after-free when a bytearray passed as a std::string_view argument was resized while the bound function was running; on CPython the bytearray now cannot be resized for the duration of the call.

…ws it

A std::string_view loaded from a bytearray points into the bytearray's
storage. Keeping only the object alive does not stop Python code that
runs during the call from resizing it, which frees the buffer under
the view. On CPython, keep a memoryview alive instead so that resizing
raises BufferError until the call returns.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant