revert HTML page change, broke artifactory

[morotti]

Hello,

The new release of pypi warehouse changed the HTML output and broke the HTML API.
It's causing our artifactory to fail to fetch packages from pypi, as it can no longer parse the page.

Can you please revert this change?
this file https://github.com/pypi/warehouse/blob/main/warehouse/templates/api/simple/detail.html
PR #18266

Explanation, the PR automatically reformatted all the HTML templates with a formatting tool and it changed the output.
The changed output is breaking tools that are no longer able to parse it and it's worse for performance (more whitespaces).

Output now:

Output before: I can't take a screenshot since it gone, but it should be exactly one package per line and no whitespace.
PR:

Explanation: The HTML page is the pypi API, it is downloaded and processed by numerous python packaging tools to identify available packages.
Historically pypi did not a have a json API, the HTML page is the API.
We've noticed that our Artifactory started failing to mirror/download packages from pypi.org on Friday, we tracked down the issue to this recent PR. (obviously artifactory wrote their parser poorly!).

Second issue: the reformatting added whitespaces and new lines, that makes every page a few percent larger, which makes both warehouse and packaging tools slower (a lot of packages have thousands of files available and the page is very large!).
It's bad for performance, the whitespace needs to be removed.
The parsing of the /simple HTML page is a noticeable amount of the time taken to install packages (for old pip and tools that haven't been updated to use the json API instead). I bet it's a fair amount of the time taken for warehouse to generate the page too.

Can you revert the change please? Don't need to revert all the PR, just the changes to this file.
(or if you prefer to adjust the template instead, the jinja template needs to have a minus sign {%- and -%} to strip whitespaces before/after the template.)

[morotti]

@miketheman @ewdurbin ping

[claylong]

Agree, we are also suddenly seeing issues with Artifactory's ability to read the index document. I can't speak to JFrog's implementation b/c I can't see the code, but it does seem to rely on parsing the "listing" HTML page to get the list of packages.

I can't confirm that it's this change yet, or if the problem i just the whitespace or any other structural changes in the HTML response.

Our Artifactory instance can no longer work with pypi.org

[sythel]

Hitting this issue as well

Started around friday (6/13/25) ~12:17 PM EST

[ERROR] (o.a.a.p.r.r.PypiRemoteIndexProvider:132) - Could not retrieve remote index from https://pypi.org/simple/build: String index out of range: -1

While waiting for fix we have setup a middleman proxy server to modify pypi responses to format before PR #18266

[woodruffw]

(Non-maintainer opinion.)

Reverting this makes sense to me personally, but it would be really good to have some confirmation that Artifactory themselves are aware of this upstream (ideally, in the form of some kind of communication from multiple paying customers of theirs) -- it's baffling (and concerning IMO) that they're performing structured HTML parsing here.

To whit: have any of the folks who've raised here already raised this with Artifactory?

[claylong]

(Non-maintainer opinion.)

Reverting this makes sense to me personally, but it would be really good to have some confirmation that Artifactory themselves are aware of this upstream (ideally, in the form of some kind of communication from multiple paying customers of theirs) -- it's baffling (and concerning IMO) that they're performing structured HTML parsing here.

To whit: have any of the folks who've raised here already raised this with Artifactory?

It looks like the newest version of Artifactory does not have this issue, so presumably they have already fixed up their parsing log.

[claylong]

We created a man-in-the-middle proxy to modify the pypi.org/simple response body, removing the extra line breaks inside and after the tags. Testing revealed Artifactory started working, confirming this is issue is likely the problem.

[demaziej]

To whit: have any of the folks who've raised here already raised this with Artifactory?

Yes they are aware, I have an open ticket with JFrog with a link to this github issue.

[woodruffw]

Thanks both!

Yes they are aware, I have an open ticket with JFrog with a link to this github issue.

Out of curiosity, did they respond to you? Per above it sounds like they've done a fix already, but I'm curious if they've acknowledged this.

[morotti]

it's baffling (and concerning IMO) that they're performing structured HTML parsing here.

Agreed, though it's not entirely surprising for very old code.
Think about it, it's the same reason why the bug exists in the first place, pypi had no API, this HTML page is the official API :D
and it was not supposed to change and it needed to be processed as fast as possible. (I don't think people realize but generating and parsing this page is one of the slowest part of pip install).

I can confirm the issue has been resolved in recent versions of artifactory. (I have an instance running on a version from 12 months ago, works fine).
I think it's been fixed for quite some time but it's hard to identify the cutoff.

Unrelated: artifactory has finally added support for the json API, just now with the latest release.
https://jfrog.com/help/r/jfrog-release-information/artifactory-7.111.4-self-hosted

[claylong]

... this HTML page is the official API :D and it was not supposed to change and it needed to be processed as fast as possible. (I don't think people realize but generating and parsing this page is one of the slowest part of pip install).

This. In a nutshell, whether clients are implemented well or not, this Index HTML page is an interface that can suffer from breaking changes.

Hopefully someday soon this won't be an issue and another interface will fully take over.

[woodruffw]

This. In a nutshell, whether clients are implemented well or not, this Index HTML page is an interface that can suffer from breaking changes.

I want to emphasize: this isn't a breaking change. The interface in question is defined by PEP 503, which stipulates:

This URL must respond with a valid HTML5 page with a single anchor element per file for the project.

Beyond that, the index is permitted to include any other tags (or make any whitespace formatting decisions) it wants. For example from the PEP:

There may be any other HTML elements on the API pages as long as the required anchor elements exist.

PEP 503 has been in place for over a decade, so I don't think the pre-standard state of PyPI's index APIs is a good excuse for consumers not properly parsing the index responses here.

What's happening here is not a breaking change, but an example of Hyrum's Law -- instead of following the specification, Artifactory was relying on the observed behavior of the system. That's always going to be brittle, because the observed behavior is necessarily a subset of the specified behavior.

Hopefully someday soon this won't be an issue and another interface will fully take over.

That would be PEP 691, which specifies the JSON API and has been standardized for about 3 years 🙂

[claylong]

This. In a nutshell, whether clients are implemented well or not, this Index HTML page is an interface that can suffer from breaking changes.

I want to emphasize: this isn't a breaking change. The interface in question is defined by PEP 503, which stipulates:

This URL must respond with a valid HTML5 page with a single anchor element per file for the project.

Beyond that, the index is permitted to include any other tags (or make any whitespace formatting decisions) it wants. For example from the PEP:

There may be any other HTML elements on the API pages as long as the required anchor elements exist.

PEP 503 has been in place for over a decade, so I don't think the pre-standard state of PyPI's index APIs is a good excuse for consumers not properly parsing the index responses here.

What's happening here is not a breaking change, but an example of Hyrum's Law -- instead of following the specification, Artifactory was relying on the observed behavior of the system. That's always going to be brittle, because the observed behavior is necessarily a subset of the specified behavior.

Hopefully someday soon this won't be an issue and another interface will fully take over.

That would be PEP 691, which specifies the JSON API and has been standardized for about 3 years 🙂

Sure, in theory I agree completely.