Repository navigation
Fix data race in registering instances - #6215
Conversation
rwgk
left a comment
There was a problem hiding this comment.
codex GPT-6.1-Sol ultra review:
Found one P2 regression at pybind11.h:1735: custom-holder construction loses Python virtual dispatch.
An intrusive holder can invoke a virtual callback while acquiring a reference. Moving init_holder() before registration prevents get_override() from finding the Python instance, so it silently calls the C++ fallback. I reproduced this on ordinary CPython: the Python callback executes at the base revision but is skipped at the PR head. Preserve reentrant instance lookup during holder initialization and add coverage for this case.
Reproducer:
|
Thanks for catching the regression @rwgk! You are right, the reordering broke the lookups. I updated the change; the reordering only applies to holders that can't run user code during the construction. I also added the regression repro test. This PR keeps the data race with the holder that could run user code as a result. Fixing it for the type seems to require more thinking so I'd prefer to handle that in a separate PR if that's ok. Please let me know what you think! Note the 5 failing CI jobs appear to be failing due to gitlab related issues. |
rwgk
left a comment
There was a problem hiding this comment.
LGTM and codex GPT-6.1-Sol ultra.
For a moment I was worried that test_holder_ctor_calls_python_override might be flaky in free-threaded builds. Codex explained that the callback completes synchronously before the assertion, and the test doesn't expose the instance to another thread during initialization. It also passed 10,000 repetitions with the GIL disabled.
Description
class_::init_instance()added the new instance toregistered_instancesbefore setting its status flags and constructing its holder. On free-threaded builds another thread can find the instance in the map and read those flags (e.g.simple_layoutinvalue_and_holder) while they are still being written. TSan reports this as a data race when several threads cast the same C++ object.init_instance()overloads (classic and smart_holder). The instance map lock then orders every write to the instance before any reader that finds it there. There are no new locks and no layout or ABI changes.test_concurrent_cast_of_shared_cpp_objectintest_thread.pycovers both overloads. Under TSan on 3.14t it fails without this change and passes with it. It won't fail on a regular build, since the race only shows up under TSan.smart_holder_from_unique_ptr/smart_holder_from_shared_ptr) callkeep_alive_impl()afterinit_instance(), and that writeshas_patientsafter the instance is already visible.Suggested changelog entry:
📚 Documentation preview 📚: https://pybind11--6215.org.readthedocs.build/