_pickle.PicklingError on 3.14.0a1+ where it didn't before due to the start method change from fork to forkserver

[progval]

Bug report

Bug description:

Code dating back to Python 2 had to do super(OwnClassName, self) instead of super(). Modules designed to be reloadable could not use the super(OwnClassName, self) syntax in their non-constructor method, as OwnClassName was re-binded to the class of the new module, rather than the class that had the name when the object was constructed (which differs when the object was constructed before a reload). Therefore, such modules defined classes like this:

class Subclass(Superclass):
    def __init__(self):
        self.__parent = super(Subclass, self)

    def method1(self, arg):
        self.__parent.method1(arg + 2)

Python 3.14.0a1+ breaks existing code using this trick on subclasses of multiprocessing.Process.

For example:

import multiprocessing

class MyProcess(multiprocessing.Process):
    def __init__(self, target=None, args=(), kwargs={}):
        self.__parent = super(MyProcess, self)
        self.__parent.__init__(target=target, args=args, kwargs=kwargs)

    def run(self):
        self.__parent.run()
    
if __name__ == "__main__":
    p = MyProcess()
    p.start()

worked fine on previous versions, but now errors with:

Traceback (most recent call last):
  File "/home/dev-irc/Limnoria/repro.py", line 12, in <module>
    p.start()
    ~~~~~~~^^
  File "/home/dev-irc/.local-py-git/lib/python3.14/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ~~~~~~~~~~~^^^^^^
  File "/home/dev-irc/.local-py-git/lib/python3.14/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/dev-irc/.local-py-git/lib/python3.14/multiprocessing/context.py", line 300, in _Popen
    return Popen(process_obj)
  File "/home/dev-irc/.local-py-git/lib/python3.14/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/dev-irc/.local-py-git/lib/python3.14/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
    ~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/dev-irc/.local-py-git/lib/python3.14/multiprocessing/popen_forkserver.py", line 47, in _launch
    reduction.dump(process_obj, buf)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/home/dev-irc/.local-py-git/lib/python3.14/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
_pickle.PicklingError: first argument to __newobj__() must be <class 'super'>, not <class '__main__.MyProcess'>
when serializing super object
when serializing dict item '_MyProcess__parent'
when serializing MyProcess state
when serializing MyProcess object

Full version string: Python 3.14.0a1+ (heads/main:c8fd4b12e3d, Oct 18 2024, 22:51:39) [GCC 12.2.0] on linux

CPython versions tested on:

3.14, CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-125714: Doc: Explain that multiprocessing only supports unpicklable functions through the fork method #125750

[JelleZijlstra]

Your repro case also fails for me on 3.11 through 3.13. Simpler code that fails similarly:

import pickle
class X: pass
s = super(X, X())
pickle.dumps(s)

[progval]

Strange, my repro case fails passes on 3.11.2; and it's based on a codebase that passed on 3.12 and 3.13 so far.

Yours does fail on 3.11.2 too, though.

[progval]

Can you try this?

import multiprocessing

def f():
    def g():
        print("hello")

    multiprocessing.Process(target=g).start()
    
if __name__ == "__main__":
    f()

For me it passes on 3.11.2, but fails on 3.14.0a1+, even though function g is unpicklable no matter the version.

[JelleZijlstra]

Oh, this is because I am on MacOS. The relevant change is listed in https://docs.python.org/3.14/whatsnew/3.14.html#deprecated: the default start method moved from fork to forkserver on Linux. With fork, I guess we didn't need things to be picklable.

cc @gpshead as the author of this change, but I think this will be a won't fix: We made that change after careful consideration, and this sort of issue was a known consequence. If you want the old behavior, you can either explicitly use the fork start method or refactor your code so the relevant objects can be pickled.

There might be room for improvements in how super objects are pickled, though.

[progval]

Ah, I understand. Thanks!

I feel there is some room for improvement regarding documentation here, as this is a breaking change and not a deprecation as far as I can tell. And neither the changelog nor "Contexts and start methods" mentions only fork supports unpicklable Process attributes or targets

[gpshead]

Agreed, this is mostly a documentation issue. But I don't have a good feel for exactly what to state. Care to draft up some text in a PR @progval? I think we'd want this to be clarified in both what's new 3.14 and multiprocessing docs.

[progval]

Done in #125750

[furkanonder]

CC: @serhiy-storchaka

[JelleZijlstra]

@progval If you're interested, feel free to open an issue specifically about pickling super objects. I couldn't find any existing issue about that, and the code for super doesn't have anything geared towards pickling support. It's possible that we could fix this to make super objects properly picklable.

>>> import pickle
>>> class X: pass
... 
>>> pickle.dumps(super(X, X()))
Traceback (most recent call last):
  File "<python-input-6>", line 1, in <module>
    pickle.dumps(super(X, X()))
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^
_pickle.PicklingError: first argument to __newobj__() must be <class 'super'>, not <class '__main__.X'>
when serializing super object

[serhiy-storchaka]

Created a separate issue for pickling super: #125767.

[gpshead]

Given the user reports (now plural) on this one, we should also catch and improve the error message within multiprocessing when pickle fails to at least point people in the right direction.