pep517 requiring PEP-517 support to build creates a circular dependency for build frontends depending on pep517

[jameshilliard]

As pointed out by @layday pep517 currently requires a PEP-517 compatible build frontend to build, however this means no PEP-517 build frontends like build may depend on pep517 without introducing a circular dependency chain.

One potential way of solving this issue is using setuptools along the lines of build which supports a simple setup.py shim for bootstrapping to break the dependency cycle.

[takluyver]

My suggestion for this was to add a python -m flit_core.wheel CLI entry point that can roughly be used in place of build for bootstrapping, to make wheels of installer, tomli, pep517 and suchlike low-level packages. So bootstrapping would go something like this:

• Install flit_core using its bootstrap install script (Add bootstrap install script for flit_core flit#481) - i.e. it makes a wheel of itself, then that is unpacked to site-packages.
• Build a wheel of installer using python -m flit_core.wheel, then use installer from source to install that.
• For each low-level package using Flit (tomli, pep517, maybe packaging in the future, ...)
  • Build a wheel with python -m flit_core.wheel
  • Install using python -m installer
• Use build from source to build a wheel of itself, install with installer

[jameshilliard]

My suggestion for this was to add a python -m flit_core.wheel CLI entry point that can roughly be used in place of build for bootstrapping

So all distros wanting to use build would then have to maintain both the python -m flit_core.wheel frontend and build?

So bootstrapping would go something like this

That seems to be excessively complex vs just using setuptools for bootstrapping this sort of thing(since it has a simple setup.py shim designed for this sort of thing already it seems).

[takluyver]

All distros wanting to use anything packaged by Flit will need to package flit_core. There's nothing extra they would need to do for the python -m flit_core.wheel interface to work if we add that.

setuptools already deprecated setup.py install, and without that you will face similar issues bootstrapping from setuptools as you do with flit_core. We're going to need to figure out how to do bootstrapping in a post-PEP-517 world. I'm getting bored of repeating this, but we want to move forwards and work these problems out, not go back to a world where everything is based on setuptools.

[layday]

There's a parallel discussion happening at pypa/build#430. I suggest closing either one of these issues.

[jameshilliard]

All distros wanting to use anything packaged by Flit will need to package flit_core.

But flit_core itself requires PEP-517, so we can't package it without a PEP-517 frontend. Can we try to avoid having distros needing to maintain multiple PEP-517 frontends for bootstrapping?

setuptools [already deprecated](already deprecated) setup.py install, and without that you will face similar issues bootstrapping from setuptools as you do with flit_core

Argh, even the docs in the deprecation commit it don't list any replacement useful for bootstrapping...I really get the feeling things are being made needlessly painful for no good reason. At least it's just a warning from the looks of it so at least at the moment...

I'm getting bored of repeating this, but we want to move forwards and work these problems out, not go back to a world where everything is based on setuptools.

Normally one moves forwards by providing a migration period where you support both the old and new way of doing things. That's usually the simplest way of dealing with these problems, all the other solutions seem to be excessively complex(well maybe other than just vendoring the entire build+install toolchain in a single package or something).

[pfmoore]

But flit_core itself requires PEP-517, so we can't package it without a PEP-517 frontend.

I'm confused. The bootstrap script linked by @takluyver here doesn't appear to require pep517. Conceded, that PR's not merged yet - but where's the urgency that we have to have a solution right now? We're working on it.

[takluyver]

flit_core itself requires PEP-517, so we can't package it without a PEP-517 frontend

I've already suggested (both in this thread and various others) how to bootstrap flit_core. Please do read my messages before replying, there's not much point in me writing them if you ignore what I'm saying.

I really get the feeling things are being made needlessly painful for no good reason

The benefits of these mechanisms apply to other people in the ecosystem - we haven't spent years creating new tools and standards just to irritate you. I'm sorry that it makes your life harder. But you have multiple Python packaging people engaging with you to come up with improvements. We're just not interested in the 'solution' of abandoning the last few years of work and going back to relying on setuptools as a special case.

At @layday's suggestion, I'll close this, and we can keep the discussion going at pypa/build#430 .

[jameshilliard]

I've already suggested (both in this thread and various others) how to bootstrap flit_core. Please do read my messages before replying, there's not much point in me writing them if you ignore what I'm saying.

I am reading them, I'm just trying to make sure that there isn't some simpler solution being missed here.

The benefits of these mechanisms apply to other people in the ecosystem - we haven't spent years creating new tools and standards just to irritate you. I'm sorry that it makes your life harder. But you have multiple Python packaging people engaging with you to come up with improvements. We're just not interested in the 'solution' of abandoning the last few years of work and going back to relying on setuptools as a special case.

Ok, so I guess the policy here is that setuptools has to be removed as a bootstrapping dependency(I still don't see a great reason for that but at least that's clarifying the situation here), so I guess if that's the case then flit_core would be the replacement bootstrapping frontend, it just seems weird to me that we have this multiple frontends needed for bootstrapping a PEP-517 toolchain situation.

[takluyver]

Ok, so I guess the policy here is that setuptools has to be removed as a bootstrapping dependency

There's no policy that setuptools itself has to be removed. We are trying to get away from direct installations (setup.py install) in favour of the pattern of build wheel, install wheel. There are reasons we want to do this, which I can try to explain if you're interested, but it's kind of a diversion at this point. Once you get rid of direct installations, I think we're closer to a sensible bootstrapping story based on flit_core than we are for setuptools.

I guess if that's the case then flit_core would be the replacement bootstrapping frontend, it just seems weird to me that we have this multiple frontends needed for bootstrapping

flit_core wouldn't really be acting as a PEP 517 frontend, because it would only build things which were packaged with Flit. The point of the frontend/backend separation is that a frontend like build can be used regardless of which backend a package chooses.

[pfmoore]

Ok, so I guess the policy here is that setuptools has to be removed as a bootstrapping dependency

No, there's no "policy" as such here. Individual tools make their own decision as to which backend to use. Bootstrapping should be a process of starting from "somewhere", and building a working stack from there. There's a process being developed for flit_core that starts from nothing, which would be sufficient to get the flit stack in place. I don't know what setuptools' bootstrapping process is, but that's something you'd need to take up with them.

Once you have the backends bootstrapped, you need a front end, which can be pip or build, which will allow you to build wheels for other projects. Pip builds using setuptools, as does build. Pip can be bootstrapped using get-pip.py which has no dependencies, but I suspect you'll object because it uses a pre-build wheel internally. I don't know what build's bootstrapping process is.

That gives you a full installation stack for Python projects. Which is all you need for anything else you do. Yes, it's complex, but that's a consequence of the (deliberate) decision not to mandate a single build backend, but allow projects to choose among many.

[jameshilliard]

flit_core wouldn't really be acting as a PEP 517 frontend, because it would only build things which were packaged with Flit.

I mean if flit_core acts as a flit specific PEP 517 build+install frontend and all the build/install dependencies use flit_core then that gets us all the way to a working PEP-517 frontend without circular dependencies.

There's no policy that setuptools itself has to be removed.

No, there's no "policy" as such here. Individual tools make their own decision as to which backend to use.

IMO a more official policy doc for bootstrapping dep requirements would be kinda useful here IMO. I'm having to interpret the effective policy/direction of these things otherwise.

Bootstrapping should be a process of starting from "somewhere", and building a working stack from there. There's a process being developed for flit_core that starts from nothing, which would be sufficient to get the flit stack in place.

Yeah, the tricky part is fitting it all in with our downstream tooling cleanly which makes certain approaches difficult, I'm trying to avoid a super hacky integration that's hard to maintain.

I don't know what setuptools' bootstrapping process is, but that's something you'd need to take up with them.

They have a self-bootstrap setup.py it seems, I know we have it working reasonably well already with our distribution tooling.

I don't know what build's bootstrapping process is.

Well it has a setuptools shim at the moment so it's pretty much just setuptools->build dependency stages for building/installing, the dependencies are the problem since they require a PEP-517 frontend(at least the ones that have removed the distutils setup.py shim flit used to generate).

Yes, it's complex, but that's a consequence of the (deliberate) decision not to mandate a single build backend, but allow projects to choose among many.

That seems to mostly be a separate issue, I mean the bootstrapping backends should be as limited as possible(IMO it seems best to standardize on flit_core for that I guess since that seems to be the direction maintainers are going), once bootstrapped though adding additional PEP-517 backends is probably not hard as long as they are PEP-517 compliant(they need to have no dependency cycles as per the standard, setuptools does not actually appear to be PEP-517 compliant I guess but we currently only do setup.py style builds with it at the moment so it hasn't yet been an issue).