Support GraalPy #1538

timfel · 2023-06-28T09:10:28Z

This PR adds support for GraalPy.

A couple of questions:

GraalPy is not included in manylinux images, so I added a facility to download Pythons there as well. Does that seem ok?
I couldn't find a way to exclude GraalPy by default - I'm not sure, but maybe it would be good to be conservative here and let people opt-in to building wheels on GraalPy via the CIBW_BUILD variable explicitly.

henryiii · 2023-07-08T13:40:55Z

GraalPy is not included in manylinux images

Would it make sense to include it?

I couldn't find a way to exclude GraalPy by default

Maybe it's time to think about adding this. It comes up occasionally. I'd like to also remove pypy37 and pypy38 from the default, as they are no longer supported by upstream.

joerick · 2023-07-09T13:46:27Z

GraalPy is not included in manylinux images

Would it make sense to include it?

It would certainly be neater if it was included. Also, I've never heard of GraalPy, so I can't assess wether it's inclusion and maintenance here would be worth it for users. If Manylinux were to adopt it, that would make the decision much easier!

I couldn't find a way to exclude GraalPy by default

Maybe it's time to think about adding this. It comes up occasionally. I'd like to also remove pypy37 and pypy38 from the default, as they are no longer supported by upstream.

Hm. Yes, agreed. It would also allow us to soft-drop EoL CPythons, while keeping the functionality. The simplest way to do this would be to set a default for CIBW_BUILD. E.g. the default could be CIBW_BUILD=cp* pp{39,310}-*. One downside of this would be some projects (example) set CIBW_BUILD to *, for example, in a matrix where some other configurations are set. The fix in such cases, I suppose, would be to set to empty string instead "". On the upside, it does change the default as we want, and doesn't add additional complexity.

timfel · 2023-07-12T15:48:02Z

GraalPy is not included in manylinux images

I was under the assumption that manylinux (like e.g. conda) wants to build everything into their images from source, but now I see that they also just download PyPy binaries and put them in there, so I can open a PR there to do the same for GraalPy. This will simplify this PR.

The simplest way to do this would be to set a default for CIBW_BUILD [...] the default could be CIBW_BUILD=cp* pp{39,310}-*

That's what I also thought. I did this, which is just skipping any Python specifier starting with g. Rather than a negative match which may exclude too much (with fnmatch we cannot do curly brace expansion iirc, so cannot exclude gp), a positive default may be a good idea, but I'm the wrong person to decide that :)

joerick · 2023-08-30T08:14:21Z

Converting to draft until manylinux support lands.

mayeut · 2024-06-22T12:37:27Z

manylinux support has landed. It will be available in cibuildwheel once the update workflow runs & gets merge.

timfel · 2024-07-17T09:01:32Z

I've updated the PR to use the manylinux support, but in the meantime uv has become included in manylinux and used by default, and that doesn't support GraalPy, yet. See also astral-sh/uv#5141

timfel · 2024-07-19T20:01:49Z

uv now supports GraalPy on main, once the next release is out we need to include it in manylinux images and then I'll revisit this PR

mayeut · 2024-07-20T08:41:23Z

Unfortunately, the new version of uv still does not work on manylinux (I commented in the uv PR with more details).
@timfel, you can opt-out from using uv with graalpy for now, c.f. #1941 for a sample opting out PyPy 3.8 on Windows.

joerick · 2025-01-20T09:29:51Z

@timfel, I don't know if you're still actively working on this, but we have a v3.0 release coming up, which would be a perfect time to land a feature like this!

timfel · 2025-01-20T09:40:48Z

@timfel, I don't know if you're still actively working on this, but we have a v3.0 release coming up, which would be a perfect time to land a feature like this!

What is the timeframe for 3.0? We are still working on this, but needed a few fixes in tox, uv, and virtualenv to go in first. The virtualenv fix is still not merged because we can only pass it with the new GraalPy release that is due out in March

joerick · 2025-01-22T09:13:25Z

Ah, I didn't realise there were blockers.

What is the timeframe for 3.0?

No definitive timeline, (when it's ready ;) ) but probably some time over the next few weeks.

joerick · 2025-01-22T09:15:00Z

March is probably too far out for us to hold the 3.0 release for specifically. Not a problem, it can arrive in a point update, I just thought it would be neat :)

joerick · 2025-04-11T10:05:38Z

Something we need to consider: how this impacts CI times. It's considerable-

	main	this PR
ubuntu-latest (3.13)	37min	65min
macOS-13 (3.13)	32min	43min
windows-latest (3.13)	49min	77min

Perhaps there's a way we can flag what's enabled for an integration test run - e.g. don't run the test suite on all the EnableGroups by default? The test duration is less important on main, but it does affect how fast we can move in PRs.

joerick · 2025-04-13T17:41:29Z

Perhaps this is a good model to copy - pyodide-build has this code in their CI config to add flags to their tests based on PR labels pyodide/pyodide-build@794ed7a

joerick · 2025-04-13T20:49:45Z

I've drawn up that approach in #2357

timfel · 2025-04-15T13:46:49Z

I've drawn up that approach in #2357

This looks very reasonable to me :-) Will you let me know if/when I should rebase this PR on top of that one?

henryiii · 2025-04-15T13:56:27Z

Will ci: add Windows ARM #2362 affect this?

timfel · 2025-04-15T16:15:33Z

Will ci: add Windows ARM #2362 affect this?

We don't release Windows arm binaries of GraalPy so far, so as long as it is not enabled for that platform it's fine.

henryiii · 2025-04-17T02:54:20Z

Doesn't seem to be causing a problem after rebasing, so I think it's fine.

mayeut · 2025-04-19T06:06:58Z

test/test_0_basic.py

-        assert "WARNING: Running pip as the 'root' user can result" not in stream
+        # manylinux-interpreters ensure will run pip on GraalPy as the root
+        # user once, so we will get the warning below. We make sure it is only
+        # because of manylinux and nothing else. We do this by checking that
+        # the error message only occurs directly after a call to graalpy's
+        # ensurepip in the manylinux `/opt/` directory
+        start = 0
+        bad_msg = "WARNING: Running pip as the 'root' user can result"
+        assert_msg = f"{bad_msg} is only tolerated from manylinux's ensure for graalpy"
+        for m in re.finditer(bad_msg, stream):
+            idx = stream.rfind("/opt/", start, m.end())
+            start = m.end()
+            m2 = re.match(
+                "/opt/.*/graalpy.*/bin/python -m ensurepip --default-pip", stream[idx : m.start()]
+            )
+            assert m2 is not None, assert_msg
+            assert idx + m2.end() + 1 == m.start(), assert_msg


In light of the much longer CI times, I'm checking how not excluding graalpy from image warm-up impacts CI time on Linux, which would most likely make this workaround no-op:

diff --git a/test/conftest.py b/test/conftest.py index 8c0c9504..f2b575df 100644 --- a/test/conftest.py +++ b/test/conftest.py @@ -56,9 +56,8 @@ def docker_warmup(request: pytest.FixtureRequest) -> None: images = [build_options.manylinux_images[arch] for arch in archs] + [ build_options.musllinux_images[arch] for arch in archs ] - # exclude GraalPy as it's not a target for cibuildwheel command = ( - "manylinux-interpreters ensure $(manylinux-interpreters list 2>/dev/null | grep -v graalpy) &&" + "manylinux-interpreters ensure-all &&" "cpython3.13 -m pip download -d /tmp setuptools wheel pytest" ) for image in images:

Applied in a47e354 and I removed this workaround in the same commit.

mayeut · 2025-04-19T06:30:16Z

Something we need to consider: how this impacts CI times. It's considerable-

Updating warm-up on Linux images allows to reduce CI times on Linux.

	main	this PR	Linux warm-up
ubuntu-latest (3.13)	39min	66min	50min
ubuntu-24.04-arm (3.13)	24min	42min	34min
macOS-13 (3.13)	29min	48min	N/A
macOS-15 (3.13)	21min	29min	N/A
windows-latest (3.13)	47min	65min	N/A

joerick · 2025-04-28T17:49:13Z

Let's apply @mayeut's patch here then, and if we can get the conflicts sorted I think this is ready. @timfel would you be able to do that? We'll merge #2357 afterwards.

timfel · 2025-04-28T18:12:23Z

Let's apply @mayeut's patch here then, and if we can get the conflicts sorted I think this is ready. @timfel would you be able to do that? We'll merge #2357 afterwards.

~~I'll look at the conflicts~~

Sorry, I think I misunderstood your message, you meant me to apply @mayeut's patch, right @joerick? That's what I did now plus rebase.

for more information, see https://pre-commit.ci

Signed-off-by: Henry Schreiner <[email protected]>

… it late

henryiii · 2025-04-28T22:25:23Z

Thanks!

timfel force-pushed the graalpy branch 5 times, most recently from ad26df1 to ab43cbb Compare July 2, 2023 11:43

This was referenced Jul 18, 2023

Add GraalPy to manylinux images pypa/manylinux#1509

Closed

Basic GraalPy Support PyO3/pyo3#3247

Merged

mayeut mentioned this pull request Aug 13, 2023

Rework PyPy installation pypa/manylinux#1518

Merged

joerick marked this pull request as draft August 30, 2023 08:13

timfel force-pushed the graalpy branch from ab43cbb to 80668a0 Compare July 16, 2024 11:59

timfel force-pushed the graalpy branch 3 times, most recently from 5dfb310 to 69760d9 Compare July 21, 2024 04:05

timfel force-pushed the graalpy branch 3 times, most recently from 60b55f6 to c012a0f Compare September 17, 2024 14:58

henryiii mentioned this pull request Oct 17, 2024

Non-default identifiers? #2045

Closed

henryiii mentioned this pull request Feb 12, 2025

Release plans: 2.22 and 3.0 #2047

Closed

joerick mentioned this pull request Apr 13, 2025

Test enable groups as specified by PR labels #2357

Merged

henryiii force-pushed the graalpy branch 2 times, most recently from 7bba507 to 7446d63 Compare April 15, 2025 19:07

mayeut reviewed Apr 19, 2025

View reviewed changes

timfel and others added 11 commits April 28, 2025 20:18

Properly close files used for testing

e64cfe6

Add support for GraalPy

f90aa4e

Help GraalPy discover build tools on Windows

60358e9

Expect manylinux-interpreters ensure graalpy* warning in pip

fb9c855

Workaround GraalPy bugs on Windows

cb2d4e9

Workaround oracle/graalpython#491 also when uv is not available

d808a34

[pre-commit.ci] auto fixes from pre-commit.com hooks

4a425f9

for more information, see https://pre-commit.ci

Update azure-pipelines.yml

b06c429

Update azure-pipelines.yml

a59a1e1

refacotor: use pathlib.write_text

a4410f5

Signed-off-by: Henry Schreiner <[email protected]>

Include GraalPy in docker_warmup and remove workaround for installing…

a47e354

… it late

timfel force-pushed the graalpy branch from 7446d63 to a47e354 Compare April 28, 2025 18:21

henryiii merged commit 020a91b into pypa:main Apr 28, 2025
27 checks passed

timfel deleted the graalpy branch April 29, 2025 06:20

mayeut mentioned this pull request May 25, 2025

Using requires_python skips GraalPy identifiers #2410

Closed

Support GraalPy #1538

Support GraalPy #1538

Uh oh!

Conversation

timfel commented Jun 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

henryiii commented Jul 8, 2023

Uh oh!

joerick commented Jul 9, 2023

Uh oh!

timfel commented Jul 12, 2023

Uh oh!

joerick commented Aug 30, 2023

Uh oh!

mayeut commented Jun 22, 2024

Uh oh!

timfel commented Jul 17, 2024

Uh oh!

timfel commented Jul 19, 2024

Uh oh!

mayeut commented Jul 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joerick commented Jan 20, 2025

Uh oh!

timfel commented Jan 20, 2025

Uh oh!

joerick commented Jan 22, 2025

Uh oh!

joerick commented Jan 22, 2025

Uh oh!

joerick commented Apr 11, 2025

Uh oh!

joerick commented Apr 13, 2025

Uh oh!

joerick commented Apr 13, 2025

Uh oh!

timfel commented Apr 15, 2025

Uh oh!

henryiii commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

timfel commented Apr 15, 2025

Uh oh!

henryiii commented Apr 17, 2025

Uh oh!

mayeut Apr 19, 2025

Choose a reason for hiding this comment

Uh oh!

timfel Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

mayeut commented Apr 19, 2025

Uh oh!

joerick commented Apr 28, 2025

Uh oh!

timfel commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

henryiii commented Apr 28, 2025

Uh oh!

Uh oh!

timfel commented Jun 28, 2023 •

edited

Loading

mayeut commented Jul 20, 2024 •

edited

Loading

henryiii commented Apr 15, 2025 •

edited

Loading

timfel commented Apr 28, 2025 •

edited

Loading