Should we ship the code for optional extension modules?

[brettcannon]

E.g. do we ship OpenSSL, zlib, SQLite, etc.?

[emmatyping]

I think this is basically a requirement to make this distribution palatable/of interest to users if we expect it to be used by consumers and not just a build to be re-packaged. A lot of things break if you don't ship at least OpenSSL and zlib. And a lot of software is written with common optional modules expected to be present (whether they should or not is a different story).

I think another related question is: What do we do if there is a CVE in a dependency? Are we going to need to release more patch releases to handle CVEs in dependencies? The process there seems complicated and would be good to flesh out.

[indygreg]

Security or convenience: pick one.

We need an artifact with 3rd party library dependencies.

But there is a world of variations to possibly play with.

e.g. we could provide distro compatible artifacts for e.g. just Ubuntu that fall back to system libraries.

e.g. we could provide an executable that is able to download 3rd party library updates. Or modify ELF to use libraries from local system if possible.

But there's no getting around the need to own the transitive supply chain and stay on top of CVEs. CPython on Windows already has this problem. So it's scope bloat, not a completely new domain of responsibility.

[brettcannon]

This is where the Manylinux question comes up and whether that's the baseline as some of these dependencies can be assumed to already exist.

But I've assumed that it would be like Windows and as Gregory said, "scope bloat, not a completely new domain of responsibility."

[zooba]

I think there's a strong argument for "both" (i.e. two different packages). There's plenty of useful application for a Python runtime that doesn't have that functionality at all, and that's typically a case where it needs to be more relocatable.

It seems okay to assume that users who choose this "minimal" runtime (the "embeddable distro", in terms of existing releases¹) may have other ways to get modules. So they can get the ones they need - sys.path still works, of course.

Footnotes

1. Though for Windows, all the modules are there, they're just easily removable. Statically linking them all was an unreasonable looking pain, so we never even tried. But if the plan is to statically link them by default, then we'd need two separate packages to omit them. ↩

[freakboy3742]

+1 the idea that modules like OpenSSL and SQLite need to be present or many users are going to be very surprised.

However, I wonder if revisiting the "minimal viable Python" idea might be worthwhile in this context. If the binary artefact is a "minimal" Python, but it's possible to pip install ssl or pip install sqlite3, then we allow for an embeddable distribution to be minimal; we are able to distribute security updates to binary packages independent of the main build. Much in the same way that ensurepip is able to guarantee that pip is available, it would be possible to ship a minimal embedded distribution, and then let the rest of a full standard library be installed.

I appreciate that MVPy efforts haven't been very successful in the past (for a variety of reason); but I figured that it's worth mentioning given the overlap of concerns that exists with optional (and especially optional binary) extension modules.

[emmatyping]

I agree with @AA-Turner there ought to be at least a distribution with the full stdlib eventually. I think we could also distribute a minimal version which downstream projects could experiment with pluggable, opt-in extensions. Users will also likely want to make custom builds with their own selection of what to include/exclude.

[freakboy3742]

Is this something people would be interested in?

Yes please, with fudge sauce and sprinkles on top :-)

If it's helpful, I'd be happy to draft a PEP to this effect

Count me in on this - I'd be happy to be involved in drafting, co-authoring, sponsoring - whatever helps move this forward.

On a side note, pip install ssl might be hard because pip itself is likely going to need HTTPS to talk to PyPI.

Agreed this is a complicated bootstrapping problem. However, if the distribution includes ssl and zlib (or whatever the minimum pip-required set is) pre-installed, then we'd be in a position to run pip, but with the binary dependencies not bound to the core interpreter.

I think for upstream to be distributing 'Python', it should come with the full stdlib

I agree with @AA-Turner there ought to be at least a distribution with the full stdlib eventually.

Not disagreeing with this as a goal for the thing that is actually distributed. However, if these binary dependencies are externally packaged - even if they're "installed by default" - the interpreter doesn't need to be updated if there's a CVE against OpenSSL.

[zooba]

The Linux distros who split Python up are already very unpopular for splitting it up, so we probably want to be careful going down that path.

That said, I do think that tkinter and everything downstream of it would make more sense as a separate project (that is still integrated for our core distributions, though it's been optional on Windows for years with virtually no problems).

Part of the reason for starting the TLS PEP was to let us have something smaller than "all of OpenSSL" in the stdlib, so that people who want all of OpenSSL can go get it. But without some kind of curl-level functionality, we can't leave it out of a standard distro (and need some way to bootstrap it as well).

But overall, this has been discussed and debated pretty heavily in the past, and normally ends up being picked apart until it's not worth the effort any more. I'd suggest we probably shouldn't try and drive it from here, but if there's real energy then take the proposal upstream (again) and get those modules moved to PyPI and bundled, rather than being part of the stdlib. Let's treat this forum as "distributing CPython as it is today", or we'll never get anywhere for all the great ideas.

[cmaloney]

Agree strongly with we need to ship all of CPython as it exists today.

As far as paths to "install profiles", "minimal" variant, or optional features/dependencies https://discuss.python.org/t/pep-771-default-extras-for-python-software-packages-round-2/94905 but for the core interpreter would be nice to standardize this eventually. In particular with PEP-723 how would a single-file script with inline metadata (https://packaging.python.org/en/latest/specifications/inline-script-metadata/) say "I need a Python interpreter with this feature set"

[brettcannon]

To really harp on what Steve said:

I do think that tkinter and everything downstream of it would make more sense as a separate project (that is still integrated for our core distributions, though it's been optional on Windows for years with virtually no problems).

I.e. people can feel free to give it a try, but I don't think we should do it here as part of this project as it is not a blocker for success here, it would just make things simpler.