Twine 6.2.0 breaking changes: Remove hacks that support --skip-existing for indexes other than PyPI and TestPyPI.

[ppearcy]

Publishes to AWS CodeArtifact are failing with twine 6.2.0. It appears publib pulls latest twine version and then sets --skip-existing flag which results in this error now on CodeArtifact:

ERROR    UnsupportedConfiguration: The configured repository
--
'https://******.d.codeartifact.us-west-2.amaz
onaws.com/pypi/***/' does not have support for the following
features: --skip-existing and is an unsupported configuration

There is no workaround AFAICT as the twine version nor the --skip-existing flag is configurable.

Details on the twine change are here:
https://twine.readthedocs.io/en/stable/changelog.html

Remove hacks that support --skip-existing for indexes other than PyPI and TestPyPI.

To date, these hacks continue to accrue and there have been numerous issues with them, not the least of which being that every time we update them, the paid index providers change things to break the compatibility we implement for them. Beyond that, these hacks do not work when text is internationalized in the response from the index provider.

[bhrutledge]

Is the fix to remove --skip-existing here?

publib/bin/publib-pypi

Lines 39 to 40 in aa23062

	# Basic upload command options
	upload_opts="--verbose --skip-existing"

I tried to work around this by running pip install 'twine<6.2.0' before publib-pypi, but Twine was upgraded anyway because of the --upgrade option:

publib/bin/publib-pypi

Line 54 in aa23062

python3 -m pip install --user --upgrade $packages

[mrgrain]

Happy to take a PR to make this option optional and/or an option to use a specific version of twine.

[mrgrain]

We can also try and rebuild the skip existing support in pulib. Maven is doing something similar.

[bhrutledge]

We can also try and rebuild the skip existing support in publib.

Something like this would be my vote. It seems like Twine and --skip-existing is an implementation detail that would be better to keep encapsulated (rather than exposing it as an option). This part of publib-ca feels related:

publib/src/codeartifact/staging/pip.ts

Lines 28 to 33 in aa23062

	// Even though twine supports uploading all packages in one go, we have to upload them
	// individually since CodeArtifact does not support Twine's `--skip-existing`. Fun beans.
	await parallelShell(packages, async (pkg, output) => {
	console.log(`⏳ ${pkg}`);
	await shell(['twine', 'upload', '--verbose', pkg], {

Aside: I used to be a maintainer of Twine, and I was the last person to touch some of the code that was removed.

[toxygene]

To elaborate, the Twine devs have stated:

Twine was and always will be PyPI first.
pypa/twine#1265 (comment)

AWS CodeArtifact implements only a subset of the PyPI repository API (https://docs.aws.amazon.com/codeartifact/latest/ug/python-compatibility.html), so Twine isn't guaranteed to work with it.

[mrgrain]

AWS CodeArtifact implements only a subset of the PyPI repository API (https://docs.aws.amazon.com/codeartifact/latest/ug/python-compatibility.html), so Twine isn't guaranteed to work with it.

Yeah. Coincidently AWS CodeArtifact is returning the correct HTTP status code (409) which Twine does check for. Except that the option was just deactivated for all non PyPI repos.

[antonu17]

We could also switch from twine to uv publish https://docs.astral.sh/uv/guides/package/#publishing-your-package

[gradybarrett]

We could also switch from twine to uv publish https://docs.astral.sh/uv/guides/package/#publishing-your-package

Seems like switching tools might be the right answer given twine's stance (pypi first). I respect it, but it means it's not the right tool for flexible publishing. Won't weigh in on the tool wars, other than to say that uv is compelling, even if it is VC-only backed atm.

[sigmavirus24]

pypa/twine#1270 might also be of interest to folks here

[mrgrain]

Seems like switching tools might be the right answer given twine's stance (pypi first). I respect it, but it means it's not the right tool for flexible publishing. Won't weigh in on the tool wars, other than to say that uv is compelling, even if it is VC-only backed atm.

Not sure what alternatives we have. uv doesn't necessarily make it easier, e.g. it requires a config file for alternative repository urls. I also cannot tell if it supports attestations yet.

[gradybarrett]

Not sure what alternatives we have. uv doesn't necessarily make it easier, e.g. it requires a config file for alternative repository urls. I also cannot tell if it supports attestations yet.

Yeah, just sorta moves the problem around a bit. Poetry has similar config requirements. But the main thing is does it support diverse registries better than twine long-term and can the configs be reasonably abstracted.

I kinda think the tool shouldn't be exposed - like instead of twine env vars, there's env vars for like "PYTHON_REGISTRY_USERNAME", that can then be wrangled into whatever config or env var setup the underlying tool uses. Then you can swap tools at will. I'm less worried about npm, maven, and others. But the Python package tool ecosystem is just rough in my experience, so that isolation might be useful.