Add `py::potentially_slicing_shared_ptr(handle)` function #5624

nsoblath · 2025-04-17T22:19:10Z

Description

Closes #5623.

Please see the added documentation and source code comments for more information.

For quick access, this is the new section in the documentation:

https://pybind11--5624.org.readthedocs.build/en/5624/advanced/classes.html#avoiding-inheritance-slicing-and-std-weak-ptr-surprises

rwgk · 2025-04-18T06:04:33Z

tests/test_class_sh_trampoline_weak_ptr.cpp

+    virtual ~WpBase() = default;
+};
+
+struct PyWpBase : WpBase {


struct PyWpBase : WpBase, py::trampoline_self_life_support {

Does that help?

(I just fixed an oversight in my previous comment.)

Suggested change

struct PyWpBase : WpBase {

struct PyWpBase : WpBase, py::trampoline_self_life_support {

rwgk · 2025-04-19T07:41:31Z

I simplified the reproducer and changed the variable names to make them more intuitive (to me).

I'm guessing we're building an intermediate shared_ptr<VirtBase> that has a different control block than the shared_ptr held by the smart_holder.

I wrote that code four years ago. Quite likely this will take me several hours to re-learn what exactly is happening. Not sure when I'll get to it.

@nsoblath How did you arrive at that test? Did you have code that worked with std::shared_ptr as holder?

…s holder). Tweak test_class_sh_trampoline_weak_ptr.py to pass, with `# THIS NEEDS FIXING` comment.

rwgk · 2025-04-19T19:40:00Z

@nsoblath How did you arrive at that test? Did you have code that worked with std::shared_ptr as holder?

On that suspicion, I added test_class_sp_trampoline_weak_ptr.cpp,py, and it seems to work: it certainly works locally, but I'm waiting for GitHub Actions to finish.

Assuming the new test works reliably on all platforms, I'd guess that it's feasible to fix the smart_holder equivalent.

``` /__w/pybind11/pybind11/tests/test_class_sh_trampoline_weak_ptr.cpp:23:43: error: the parameter 'sp' is copied for each invocation but only used as a const reference; consider making it a const reference [performance-unnecessary-value-param,-warnings-as-errors] 23 | void set_wp(std::shared_ptr<VirtBase> sp) { wp = sp; } | ^ | const & 4443 warnings generated. ``` ``` /__w/pybind11/pybind11/tests/test_class_sp_trampoline_weak_ptr.cpp:23:43: error: the parameter 'sp' is copied for each invocation but only used as a const reference; consider making it a const reference [performance-unnecessary-value-param,-warnings-as-errors] 23 | void set_wp(std::shared_ptr<VirtBase> sp) { wp = sp; } | ^ | const & 4430 warnings generated. ```

rwgk · 2025-04-19T21:23:37Z

Before I wrote:

I'm guessing we're building an intermediate shared_ptr<VirtBase> that has a different control block than the shared_ptr held by the smart_holder.

That's here (conclusive):

pybind11/include/pybind11/detail/type_caster_base.h

Lines 779 to 782 in bc4a66d

    
           std::shared_ptr<T> to_be_released( 
        
               type_raw_ptr, shared_ptr_trampoline_self_life_support(loaded_v_h.inst)); 
        
           vptr_gd_ptr->released_ptr = to_be_released; 
        
           return to_be_released;

Note that the smart_holder instance is readily available in the same context:

pybind11/include/pybind11/detail/type_caster_base.h

Line 762 in bc4a66d

smart_holder &hld = holder();

hld.vptr is the shared_ptr instance with the control block that we need in order to fix #5623.

So in line 782 we need to return a shared_ptr that

has the same control block as hld.vptr
but also owns a Python reference count for loaded_v_h.inst

Any ideas, please let me know. (I haven't thought about it much yet; the penny didn't want to drop immediately.)

@virtuald for visibility

nsoblath · 2025-04-19T23:59:10Z

@nsoblath How did you arrive at that test? Did you have code that worked with std::shared_ptr as holder?

I copied the test_class_sh_trampoline_shared_ptr_cpp_arg test, modified it to store a std::weak_ptr and stripped out everything that wasn't needed. It replicates the essential parts of the application I have that's not working. I didn't try using std::shared_ptr because the application I have also needs the benefits of py::smart_holder.

rwgk · 2025-04-20T05:28:05Z

I've now turned this over in my mind for a couple hours, and consulted ChatGPT, which I believe got very confused/confusing, but please see for yourself here.

Stepping out of the box:

The code I pointed out before is pulling a trick to avoid inheritance slicing ([BUG] Problem when creating derived Python objects from C++ (inheritance slicing) #1333), by keeping the derived Python object alive as long as the returned shared_ptr exists.
But you don't actually want a shared_ptr, you actually want a weak_ptr, which isn't meant to keep anything alive.
So in your particular case, the trick to avoid inheritance slicing backfires. It fixes one problem, but creates another one.

Solving this is a real brain teaser. I'm not sure I can help. I'm not sure if there is a viable solution. I have a couple ideas, but each of them might take hours of work only to find out it leads to a dead end.

Could you try to work around this limitation? For example, instead of owning the C++ weak_ptr, own a Python weakref instead (in WpOwner). Dereference the Python weakref to see if the Python object is still alive, then you can safely extract the C++ pointer if it is (in WpOwner::get_code).

…s does not inherit from this class, preserve established Inheritance Slicing behavior. rwgk reached this point with the help of ChatGPT: * https://chatgpt.com/share/68056498-7d94-8008-8ff0-232e2aba451c The only production code change in this commit is: ``` diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h index d4f9a41..f3d4530 100644 --- a/include/pybind11/detail/type_caster_base.h +++ b/include/pybind11/detail/type_caster_base.h @@ -776,6 +776,14 @@ struct load_helper : value_and_holder_helper { if (released_ptr) { return std::shared_ptr<T>(released_ptr, type_raw_ptr); } + auto *self_life_support + = dynamic_raw_ptr_cast_if_possible<trampoline_self_life_support>(type_raw_ptr); + if (self_life_support == nullptr) { + std::shared_ptr<void> void_shd_ptr = hld.template as_shared_ptr<void>(); + std::shared_ptr<T> to_be_released(void_shd_ptr, type_raw_ptr); + vptr_gd_ptr->released_ptr = to_be_released; + return to_be_released; + } std::shared_ptr<T> to_be_released( type_raw_ptr, shared_ptr_trampoline_self_life_support(loaded_v_h.inst)); vptr_gd_ptr->released_ptr = to_be_released; ```

rwgk · 2025-04-20T21:29:50Z

I'm testing commit 4638e01 to see if that works on all platforms.

See https://chatgpt.com/share/68056498-7d94-8008-8ff0-232e2aba451c for how I got there / rationale.

I'm not sure if 4638e01 is a wise twist to add.

Fundamentally, it'd be better to support std::weak_ptr conversions more directly, but that's certainly a lot more work than I'm able to invest into pybind11.

``` 11>D:\a\pybind11\pybind11\tests\test_class_sp_trampoline_weak_ptr.cpp(44,50): error C2220: the following warning is treated as an error [D:\a\pybind11\pybind11\build\tests\pybind11_tests.vcxproj] 11>D:\a\pybind11\pybind11\tests\test_class_sp_trampoline_weak_ptr.cpp(44,50): warning C4458: declaration of 'sp' hides class member [D:\a\pybind11\pybind11\build\tests\pybind11_tests.vcxproj] D:\a\pybind11\pybind11\tests\test_class_sp_trampoline_weak_ptr.cpp(54,31): see declaration of 'pybind11_tests::class_sp_trampoline_weak_ptr::SpOwner::sp' ```

rwgk · 2025-04-21T04:30:02Z

@nsoblath I feel I'm pretty clear now about the situation. I think it's best not to change the smart_holder code, but to go with the approach outlined before:

instead of owning the C++ weak_ptr, own a Python weakref instead (in WpOwner). Dereference the Python weakref to see if the Python object is still alive, then you can safely extract the C++ pointer if it is (in WpOwner::get_code).

I'll explain more when I get to it, probably next weekend.

(The only thing I want to change, probably, is to add a static_assert, to enforce that py::trampoline_self_life_support is used if the holder is smart_holder, and not used otherwise. This doesn't change anything at runtime, but is to avoid leaving users unclear.)

nsoblath · 2025-04-21T22:32:40Z

@rwgk Thank you very much for your time and attention looking into this. I really appreciate it.

Though my understanding of the internals of pybind11 is quite incomplete, my thoughts about what might be going wrong were similar to what you suggested: that the control block for the shared_ptr<VirtBase> that's extracted is different from the one held by the smart_holder. Do I understand correctly that the change you added in 4638e01 fixes this problem? That's what it looks like since the test_class_sh_trampoline_weak_ptr tests now pass.

In your later message you said

I think it's best not to change the smart_holder code, . . .

Does that mean you're preferring to not use the change in 4638e01? Is there a downside to making sure that the control block of the returned shared_ptr matches the one held by the smart_holder in this particular situation?

Regarding the alternative solution you proposed:

instead of owning the C++ weak_ptr, own a Python weakref instead (in WpOwner). Dereference the Python weakref to see if the Python object is still alive, then you can safely extract the C++ pointer if it is (in WpOwner::get_code).

I believe I can implement something to this effect. It's not ideal because the library where I'm using this is sometimes used as a pure-C++ library, sometimes used with its Python bindings. WpOwner (which in reality holds a collection of weak_ptrs) can simultaneously hold pointers to pure C++ classes and Python classes. When the library is used without Python bindings, I don't want to require the user to also have Python and Pybind11 installed. However, I can probably make a struct that combines weak_ptr and weakref and handles the source differences between when the Python bindings are used and when they're not.

rwgk · 2025-04-21T22:55:15Z

Does that mean you're preferring to not use the change in 4638e01?

Yes.

Is there a downside to making sure that the control block of the returned shared_ptr matches the one held by the smart_holder in this particular situation?

Yes, it'll be very confusing to many (I think), while only being useful in a few niche cases.

It is not obvious at all that the shared_ptr that works for you is subject to inheritance slicing.

I feel it'll be better to keep the smart_holder feature simpler and the behavior consistent. It will be easier to reason about the bindings.

We could probably add an interface that makes it relatively easy for you to get the inheritance-slicing-shared_ptr from the Python object, which you could use from a lambda function in the pybind11 bindings.

What I didn't explain: I convinced myself that it is impossible to get a shared_ptr that does both:

Keeps the PyDrvd alive (in other words, isn't subject to inheritance slicing).
Has the same control block as the shared_ptr inside the holder.

Reason: This would introduce a hard reference cycle, because the Python object (PyDrvd) owns the holder.

rwgk · 2025-04-24T06:36:03Z

@nsoblath If you can wait a week or two, I think (although I'm not sure) that I can give you something like this:

    py::classh<WpOwner>(m, "WpOwner")
        ...
        .def("set_wp", [](WpOwner& self, py::handle obj) {
            self.set_wp(py::slicing_shared_ptr_from_smart_holder<VirtBase>(obj));
        })
        ...

nsoblath · 2025-04-24T23:00:05Z

@nsoblath If you can wait a week or two, I think (although I'm not sure) that I can give you something like this:
    py::classh<WpOwner>(m, "WpOwner")
        ...
        .def("set_wp", [](WpOwner& self, py::handle obj) {
            self.set_wp(py::slicing_shared_ptr_from_smart_holder<VirtBase>(obj));
        })
        ...

@rwgk I can certainly wait, and if that works, it'd be great. Thank you!

Incidentally I have been working on the workaround we talked about, but this would be a cleaner solution (at least on the side of my application).

rwgk · 2025-04-24T23:41:08Z

Incidentally I have been working on the workaround we talked about, but this would be a cleaner solution (at least on the side of my application).

I wouldn't call the proposed "slicing" option the cleaner solution. It's only easier. — You might still get a shared_ptr from the weak_ptr, but the virtual overrides in Python might be gone (sliced). I convinced myself that's not UB, but depending on the exact circumstances, it can be very surprising.

Also undo the corresponding test change in test_class_sh_trampoline_weak_ptr.py But keep all extra debugging code for now.

…cast to `std::shared_ptr<VirtBase>` for now.

…ssage and add test.

…h smart_holder and std::shared_ptr as holder.

… it got lost accidentally in commit 56d23dc

``` /__w/pybind11/pybind11/tests/test_potentially_slicing_shared_ptr.cpp:30:9: error: 'magic_token' should be initialized in a member initializer of the constructor [cppcoreguidelines-prefer-member-initializer,-warnings-as-errors] 29 | trampoline_is_alive_simple(const trampoline_is_alive_simple &other) { | : magic_token(other.magic_token) 30 | magic_token = other.magic_token; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /__w/pybind11/pybind11/tests/test_potentially_slicing_shared_ptr.cpp:33:9: error: 'magic_token' should be initialized in a member initializer of the constructor [cppcoreguidelines-prefer-member-initializer,-warnings-as-errors] 32 | trampoline_is_alive_simple(trampoline_is_alive_simple &&other) noexcept { | : magic_token(other.magic_token) 33 | magic_token = other.magic_token; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ```

rwgk · 2025-04-28T07:43:52Z

@nsoblath I'm mostly done, if you want to take a look. The only part missing is an addition to the trampoline documentation.

… surprises" section in advanced/classes.rst

…_ptr.py

rwgk · 2025-04-28T17:59:50Z

@nsoblath All done now. Could you please review the added documentation?

@virtuald Is there a chance that you could help with a review? — The production code changes are small and nearly trivial.

Most of my effort for this PR went into the new stress-test level unit test, documentation, and source code comments.

nsoblath added 3 commits April 15, 2025 17:25

Added weak_ptr test (currently failing)

50c3c1e

Merge branch 'master' into weak_ptr_test_only

528d1cc

Cleanup

b160f6c

nsoblath requested a review from henryiii as a code owner April 17, 2025 22:19

nsoblath mentioned this pull request Apr 17, 2025

[BUG]: std::weak_ptr expires for derived Python objects using py::smart_holder #5623

Open

3 tasks

rwgk reviewed Apr 18, 2025

View reviewed changes

[skip ci] Simplify test case.

2fe82b2

Add test_class_sp_trampoline_weak_ptr.cpp,py (using std::shared_ptr a…

ff70657

…s holder). Tweak test_class_sh_trampoline_weak_ptr.py to pass, with `# THIS NEEDS FIXING` comment.

rwgk added 3 commits April 19, 2025 13:39

PYPY, GRAALPY: skip last part of test_weak_ptr_base

32858da

Rename test_weak_ptr_base → test_weak_ptr_owner

e978d93

rwgk added 3 commits April 20, 2025 10:54

Add SpOwner, test_with_sp_owner, test_with_sp_and_wp_owners

1c0b700

Remove debug printf in include/pybind11/pybind11.h

6cc6368

rwgk mentioned this pull request Apr 24, 2025

[WIP] Preparation for v3.0.0 release: Upgrade guide & Changelog #5589

Draft

rwgk added 2 commits April 26, 2025 12:08

[skip ci] Undo the production code change under 4638e01

2f672eb

Also undo the corresponding test change in test_class_sh_trampoline_weak_ptr.py But keep all extra debugging code for now.

[skip ci] Introduce lambda in WpOwner::set_wp bindings, but simply …

888a295

…cast to `std::shared_ptr<VirtBase>` for now.

rwgk added 6 commits April 26, 2025 13:50

Add py::potentially_slicing_shared_ptr()

b20f557

Add type_id<T>() to py::potentially_slicing_shared_ptr() error me…

3b4b28c

…ssage and add test.

test_potentially_slicing_shared_ptr.cpp,py (for smart_holder only)

56d23dc

Generalize test_potentially_slicing_shared_ptr.cpp,py for testing wit…

ff0792e

…h smart_holder and std::shared_ptr as holder.

Add back test_potentially_slicing_shared_ptr_not_convertible_error(),…

eb1787c

… it got lost accidentally in commit 56d23dc

Add simple trampoline state assertions.

10f6e78

rwgk changed the title ~~Failing test for std::weak_ptr used with derived Python class and py::smart_holder~~ Add py::potentially_slicing_shared_ptr(handle) function Apr 28, 2025

rwgk added 2 commits April 27, 2025 23:22

Add a (long) C++ comment for py::potentially_slicing_shared_ptr<>()

b42bd11

rwgk added 2 commits April 28, 2025 09:43

[skip ci] Add new "Avoiding Inheritance Slicing and std::weak_ptr…

f160450

… surprises" section in advanced/classes.rst

[skip ci] Add introductory comment to test_potentially_slicing_shared…

78178d7

…_ptr.py

[skip ci] Merge branch 'master' into weak_ptr_test_only

62d2242

rwgk mentioned this pull request Apr 28, 2025

Add static_asserts to enforce that py::smart_holder is combined with py::trampoline_self_life_support #5633

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `py::potentially_slicing_shared_ptr(handle)` function #5624

Add `py::potentially_slicing_shared_ptr(handle)` function #5624

nsoblath commented Apr 17, 2025 •

edited by rwgk

Loading

rwgk Apr 18, 2025 •

edited

Loading

rwgk Apr 18, 2025

henryiii Apr 18, 2025

rwgk commented Apr 19, 2025

rwgk commented Apr 19, 2025

rwgk commented Apr 19, 2025

nsoblath commented Apr 19, 2025

rwgk commented Apr 20, 2025 •

edited

Loading

rwgk commented Apr 20, 2025

rwgk commented Apr 21, 2025

nsoblath commented Apr 21, 2025

rwgk commented Apr 21, 2025

rwgk commented Apr 24, 2025

nsoblath commented Apr 24, 2025

rwgk commented Apr 24, 2025

rwgk commented Apr 28, 2025

rwgk commented Apr 28, 2025

	struct PyWpBase : WpBase {
	struct PyWpBase : WpBase, py::trampoline_self_life_support {

Add py::potentially_slicing_shared_ptr(handle) function #5624

Are you sure you want to change the base?

Add py::potentially_slicing_shared_ptr(handle) function #5624

Conversation

nsoblath commented Apr 17, 2025 • edited by rwgk Loading

Description

rwgk Apr 18, 2025 • edited Loading

Choose a reason for hiding this comment

rwgk Apr 18, 2025

Choose a reason for hiding this comment

henryiii Apr 18, 2025

Choose a reason for hiding this comment

rwgk commented Apr 19, 2025

rwgk commented Apr 19, 2025

rwgk commented Apr 19, 2025

nsoblath commented Apr 19, 2025

rwgk commented Apr 20, 2025 • edited Loading

rwgk commented Apr 20, 2025

rwgk commented Apr 21, 2025

nsoblath commented Apr 21, 2025

rwgk commented Apr 21, 2025

rwgk commented Apr 24, 2025

nsoblath commented Apr 24, 2025

rwgk commented Apr 24, 2025

rwgk commented Apr 28, 2025

rwgk commented Apr 28, 2025

Add `py::potentially_slicing_shared_ptr(handle)` function #5624

Add `py::potentially_slicing_shared_ptr(handle)` function #5624

nsoblath commented Apr 17, 2025 •

edited by rwgk

Loading

rwgk Apr 18, 2025 •

edited

Loading

rwgk commented Apr 20, 2025 •

edited

Loading