Support GraalPy #1538

timfel · 2023-06-28T09:10:28Z

This PR adds support for GraalPy.

A couple of questions:

GraalPy is not included in manylinux images, so I added a facility to download Pythons there as well. Does that seem ok?
I couldn't find a way to exclude GraalPy by default - I'm not sure, but maybe it would be good to be conservative here and let people opt-in to building wheels on GraalPy via the CIBW_BUILD variable explicitly.

henryiii · 2023-07-08T13:40:55Z

GraalPy is not included in manylinux images

Would it make sense to include it?

I couldn't find a way to exclude GraalPy by default

Maybe it's time to think about adding this. It comes up occasionally. I'd like to also remove pypy37 and pypy38 from the default, as they are no longer supported by upstream.

joerick · 2023-07-09T13:46:27Z

GraalPy is not included in manylinux images

Would it make sense to include it?

It would certainly be neater if it was included. Also, I've never heard of GraalPy, so I can't assess wether it's inclusion and maintenance here would be worth it for users. If Manylinux were to adopt it, that would make the decision much easier!

I couldn't find a way to exclude GraalPy by default

Maybe it's time to think about adding this. It comes up occasionally. I'd like to also remove pypy37 and pypy38 from the default, as they are no longer supported by upstream.

Hm. Yes, agreed. It would also allow us to soft-drop EoL CPythons, while keeping the functionality. The simplest way to do this would be to set a default for CIBW_BUILD. E.g. the default could be CIBW_BUILD=cp* pp{39,310}-*. One downside of this would be some projects (example) set CIBW_BUILD to *, for example, in a matrix where some other configurations are set. The fix in such cases, I suppose, would be to set to empty string instead "". On the upside, it does change the default as we want, and doesn't add additional complexity.

timfel · 2023-07-12T15:48:02Z

GraalPy is not included in manylinux images

I was under the assumption that manylinux (like e.g. conda) wants to build everything into their images from source, but now I see that they also just download PyPy binaries and put them in there, so I can open a PR there to do the same for GraalPy. This will simplify this PR.

The simplest way to do this would be to set a default for CIBW_BUILD [...] the default could be CIBW_BUILD=cp* pp{39,310}-*

That's what I also thought. I did this, which is just skipping any Python specifier starting with g. Rather than a negative match which may exclude too much (with fnmatch we cannot do curly brace expansion iirc, so cannot exclude gp), a positive default may be a good idea, but I'm the wrong person to decide that :)

joerick · 2023-08-30T08:14:21Z

Converting to draft until manylinux support lands.

mayeut · 2024-06-22T12:37:27Z

manylinux support has landed. It will be available in cibuildwheel once the update workflow runs & gets merge.

timfel · 2024-07-17T09:01:32Z

I've updated the PR to use the manylinux support, but in the meantime uv has become included in manylinux and used by default, and that doesn't support GraalPy, yet. See also astral-sh/uv#5141

timfel · 2024-07-19T20:01:49Z

uv now supports GraalPy on main, once the next release is out we need to include it in manylinux images and then I'll revisit this PR

mayeut · 2024-07-20T08:41:23Z

Unfortunately, the new version of uv still does not work on manylinux (I commented in the uv PR with more details).
@timfel, you can opt-out from using uv with graalpy for now, c.f. #1941 for a sample opting out PyPy 3.8 on Windows.

joerick · 2025-01-20T09:29:51Z

@timfel, I don't know if you're still actively working on this, but we have a v3.0 release coming up, which would be a perfect time to land a feature like this!

timfel · 2025-01-20T09:40:48Z

@timfel, I don't know if you're still actively working on this, but we have a v3.0 release coming up, which would be a perfect time to land a feature like this!

What is the timeframe for 3.0? We are still working on this, but needed a few fixes in tox, uv, and virtualenv to go in first. The virtualenv fix is still not merged because we can only pass it with the new GraalPy release that is due out in March

joerick · 2025-01-22T09:13:25Z

Ah, I didn't realise there were blockers.

What is the timeframe for 3.0?

No definitive timeline, (when it's ready ;) ) but probably some time over the next few weeks.

joerick · 2025-01-22T09:15:00Z

March is probably too far out for us to hold the 3.0 release for specifically. Not a problem, it can arrive in a point update, I just thought it would be neat :)

henryiii · 2025-04-01T21:13:13Z

Wild guess: is it trying to build for both 32-bit and 64-bit on Windows, but actually producing the 64-bit wheels both times? I see that's a cp wheel, maybe it's not picking up the GraalPy tag?

henryiii · 2025-04-02T03:53:59Z

Does GraalPy support the stable ABI? I see one test failing for that, I'd assume it's not supported. I also see <frozen graalpy.pip_hook>:48: RuntimeWarning: You are using an untested version of pip. GraalPy provides patches and workarounds for a number of packages when used with compatible pip versions. We recommend to stick with the pip version that ships with this version of GraalPy. but maybe that's on one of the pinning tests?

henryiii · 2025-04-03T15:01:34Z

It looks to me that gp* is not producing a graalpy wheel, but instead the host Python's wheel.

+ python --version
GraalPy 3.11.7 (Oracle GraalVM Native 24.2.0)
...
+ python -m build 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-2\\popen-gw2\\test0\\project' --wheel '--outdir=C:\\Users\\runneradmin\\AppData\\Local\\Temp\\cibw-run-sxknk4dt\\gp242-win_amd64\\built_wheel' --no-isolation
* Getting build dependencies for wheel...
stored_version 3.13.2 (tags/v3.13.2:4f8bb39, Feb  4 2025, 15:23:48) [MSC v.1942 64 bit (AMD64)]
sys.version 3.13.2 (tags/v3.13.2:4f8bb39, Feb  4 2025, 15:23:48) [MSC v.1942 64 bit (AMD64)]
...
Successfully built spam-0.1.0-cp313-cp313-win_amd64.whl

Could this be an issue with build? Since this is a Windows issue, I guess I can't just use setup-python to test it in pypa/build's test suite.

timfel · 2025-04-03T17:39:26Z

Could this be an issue with build? Since this is a Windows issue, I guess I can't just use setup-python to test it in pypa/build's test suite.

Yes, GraalPy has a few bugs on Windows when using build. I got past most of them now I think, failed at 98% of the tests just earlier so I hope I'll have this ready by tomorrow.

timfel · 2025-04-03T18:28:36Z

Ok, everything green now in Github actions. Let's see if circleci and azure agree :)

henryiii · 2025-04-03T22:03:51Z

Looks like it's hitting a "couldn't find uv" error. Our tests run with or without uv. Is uv required? We could skip if it's not present.

timfel · 2025-04-04T07:00:15Z

Looks like it's hitting a "couldn't find uv" error. Our tests run with or without uv. Is uv required? We could skip if it's not present.

Ah, yeah, I linked oracle/graalpython#491 - we have a bug when build uses virtualenv and isolation for building only on Windows, so my workaround was to force using uv with GraalPy on Windows. ~~I guess we'll have to skip then.~~

Only when building the GraalPy wheels on windows with build[virtualenv] as build I just disable build isolation, this passes the tests, but for actual builds this might fail, so I added a warning that to build for GraalPy on Windows we should use uv until the next GraalPy release.

timfel · 2025-04-04T13:32:04Z

Ok, Windows is now green on Azure, not sure why the Linux build got cancelled considering it passed before 🤔

timfel · 2025-04-08T08:32:29Z

Thank you @henryiii for fixing the pipeline :) Just to make sure no one is waiting here for us now, is there anything else I should do about this PR right now, or is it fine to just wait for review comments?

henryiii · 2025-04-08T15:15:46Z

cibuildwheel/resources/build-platforms.toml

@@ -18,6 +18,7 @@ python_configurations = [
  { identifier = "pp39-manylinux_x86_64", version = "3.9", path_str = "/opt/python/pp39-pypy39_pp73" },
  { identifier = "pp310-manylinux_x86_64", version = "3.10", path_str = "/opt/python/pp310-pypy310_pp73" },
  { identifier = "pp311-manylinux_x86_64", version = "3.11", path_str = "/opt/python/pp311-pypy311_pp73" },
+  { identifier = "gp242-manylinux_x86_64", version = "3.11", path_str = "/opt/python/graalpy311-graalpy242_311_native" },


Wheels have to be made for each minor version too? I guess this means we can't drop all the workarounds until we drop gp242 (unless there's a patch release planned)?

We are doing patch releases, I missed the merge window for a patch release that goes out next week, but I'm confident I can get the required fixes in for our patch release in June, at which time we can drop this.

Versioning is a bit confusing right now, we used to do calendar versioning with two releases per year, but we are switching to align with the OpenJDK release schedule and will then have 2 releases per year. At that point every 4th release will be a long-term support release, releases in between will only receive support for 6 months. The upcoming release 25 (due out in Sep) will be an LTS, so we could then stick to that for 2 years and also have 26/27/28 each for 6 months until we go to 29 at which point 25 is out of support.

So in terms of ABI stability, a sample wheel I just produced was given the filename
spam-0.1.0-graalpy311-graalpy242_311_native-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_28_x86_64.whl. So it does look like there's only ABI stability within each minor release of GraalPy (i.e. across patch releases), is that right?

(I just want to check the logic behind this build identifier, as most have the python version at the start, not the interpreter version. But if the ABI stability is tied to the GraalPy minor version, this makes sense to me.)

henryiii

This can be batched with other changes if someone else requests changes.

henryiii · 2025-04-08T18:55:48Z

azure-pipelines.yml

@@ -16,18 +16,19 @@ jobs:
        docker run --rm --privileged docker.io/tonistiigi/binfmt:latest --install all
        python -m pip install dependency-groups
        python -m dependency_groups test | xargs python -m pip install -e.
-        python ./bin/run_tests.py
+        python ./bin/run_tests.py --num-processes 2


Suggested change

python ./bin/run_tests.py --num-processes 2

python ./bin/run_tests.py

henryiii · 2025-04-08T18:55:56Z

azure-pipelines.yml

  steps:
    - task: UsePythonVersion@0
      inputs:
        versionSpec: '3.11'
    - bash: |
        python -m pip install dependency-groups
        python -m dependency_groups test | xargs python -m pip install -e.
-        python ./bin/run_tests.py --num-processes 2
+        python ./bin/run_tests.py --num-processes 3


Suggested change

python ./bin/run_tests.py --num-processes 3

python ./bin/run_tests.py

henryiii · 2025-04-08T18:56:04Z

azure-pipelines.yml

@@ -39,4 +40,4 @@ jobs:
    - bash: |
        python -m pip install dependency-groups
        python -m dependency_groups test | xargs python -m pip install -e.
-        python ./bin/run_tests.py
+        python ./bin/run_tests.py --num-processes 2


Suggested change

python ./bin/run_tests.py --num-processes 2

python ./bin/run_tests.py

for more information, see https://pre-commit.ci

joerick · 2025-04-09T14:40:11Z

cibuildwheel/resources/build-platforms.toml

@@ -18,6 +18,7 @@ python_configurations = [
  { identifier = "pp39-manylinux_x86_64", version = "3.9", path_str = "/opt/python/pp39-pypy39_pp73" },
  { identifier = "pp310-manylinux_x86_64", version = "3.10", path_str = "/opt/python/pp310-pypy310_pp73" },
  { identifier = "pp311-manylinux_x86_64", version = "3.11", path_str = "/opt/python/pp311-pypy311_pp73" },
+  { identifier = "gp242-manylinux_x86_64", version = "3.11", path_str = "/opt/python/graalpy311-graalpy242_311_native" },


So in terms of ABI stability, a sample wheel I just produced was given the filename
spam-0.1.0-graalpy311-graalpy242_311_native-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_28_x86_64.whl. So it does look like there's only ABI stability within each minor release of GraalPy (i.e. across patch releases), is that right?

(I just want to check the logic behind this build identifier, as most have the python version at the start, not the interpreter version. But if the ABI stability is tied to the GraalPy minor version, this makes sense to me.)

joerick · 2025-04-09T14:46:12Z

test/test_before_build.py

-        """python -c "import sys; open('{project}/pythonversion_bb.txt', 'w').write(sys.version)" && """
-        f'''python -c "import sys; open('{{project}}/pythonprefix_bb.txt', 'w').write({SYS_PREFIX})"'''
+        """python -c "import sys; f = open('{project}/pythonversion_bb.txt', 'w'); f.write(sys.version); f.close()" && """
+        f'''python -c "import sys; f = open('{{project}}/pythonprefix_bb.txt', 'w'); f.write({SYS_PREFIX}); f.close()"'''


What's the reason for this change? I wouldn't expect a problem here given that the Python process is exiting immediately - is the behaviour on GraalPy different?

timfel force-pushed the graalpy branch 5 times, most recently from ad26df1 to ab43cbb Compare July 2, 2023 11:43

This was referenced Jul 18, 2023

Add GraalPy to manylinux images pypa/manylinux#1509

Closed

Basic GraalPy Support PyO3/pyo3#3247

Merged

mayeut mentioned this pull request Aug 13, 2023

Rework PyPy installation pypa/manylinux#1518

Merged

joerick marked this pull request as draft August 30, 2023 08:13

timfel force-pushed the graalpy branch from ab43cbb to 80668a0 Compare July 16, 2024 11:59

timfel force-pushed the graalpy branch 3 times, most recently from 5dfb310 to 69760d9 Compare July 21, 2024 04:05

timfel force-pushed the graalpy branch 3 times, most recently from 60b55f6 to c012a0f Compare September 17, 2024 14:58

henryiii mentioned this pull request Oct 17, 2024

Non-default identifiers? #2045

Closed

henryiii mentioned this pull request Feb 12, 2025

Release plans: 2.22 and 3.0 #2047

Open

henryiii mentioned this pull request Apr 2, 2025

[Bot] Update dependencies #2346

Merged

This was referenced Apr 2, 2025

io has no attribute _WindowsConsoleIO oracle/graalpython#490

Open

venv created from virtualenv seeded graalpy does not work on Windows oracle/graalpython#491

Open

Build tools fail to discover MSVC on GraalPy oracle/graalpython#492

Open

timfel force-pushed the graalpy branch from f45e89d to 07a0347 Compare April 3, 2025 18:28

timfel force-pushed the graalpy branch from e4f5351 to 2561489 Compare April 4, 2025 11:11

henryiii reviewed Apr 8, 2025

View reviewed changes

henryiii approved these changes Apr 8, 2025

View reviewed changes

henryiii mentioned this pull request Apr 8, 2025

refactor: pull out dependency_constraint #2347

Merged

henryiii reviewed Apr 8, 2025

View reviewed changes

timfel and others added 9 commits April 9, 2025 10:00

Properly close files used for testing

dfe5831

Add support for GraalPy

595c650

Help GraalPy discover build tools on Windows

1e7fadf

Expect manylinux-interpreters ensure graalpy* warning in pip

97f084f

Workaround GraalPy bugs on Windows

96996e7

Workaround oracle/graalpython#491 also when uv is not available

8042bc9

[pre-commit.ci] auto fixes from pre-commit.com hooks

904d628

for more information, see https://pre-commit.ci

Update azure-pipelines.yml

8976a75

Update azure-pipelines.yml

a809218

henryiii force-pushed the graalpy branch from 6c064b3 to a809218 Compare April 9, 2025 14:02

joerick reviewed Apr 9, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support GraalPy #1538

Support GraalPy #1538

timfel commented Jun 28, 2023 •

edited

Loading

henryiii commented Jul 8, 2023

joerick commented Jul 9, 2023

timfel commented Jul 12, 2023

joerick commented Aug 30, 2023

mayeut commented Jun 22, 2024

timfel commented Jul 17, 2024

timfel commented Jul 19, 2024

mayeut commented Jul 20, 2024 •

edited

Loading

joerick commented Jan 20, 2025

timfel commented Jan 20, 2025

joerick commented Jan 22, 2025

joerick commented Jan 22, 2025

henryiii commented Apr 1, 2025 •

edited

Loading

henryiii commented Apr 2, 2025

henryiii commented Apr 3, 2025 •

edited

Loading

timfel commented Apr 3, 2025

timfel commented Apr 3, 2025 •

edited

Loading

henryiii commented Apr 3, 2025 •

edited

Loading

timfel commented Apr 4, 2025 •

edited

Loading

timfel commented Apr 4, 2025 •

edited

Loading

timfel commented Apr 8, 2025

henryiii Apr 8, 2025 •

edited

Loading

timfel Apr 9, 2025

joerick Apr 9, 2025

henryiii left a comment

henryiii Apr 8, 2025

henryiii Apr 8, 2025

henryiii Apr 8, 2025

joerick Apr 9, 2025

joerick Apr 9, 2025

	python ./bin/run_tests.py --num-processes 2
	python ./bin/run_tests.py

Support GraalPy #1538

Are you sure you want to change the base?

Support GraalPy #1538

Conversation

timfel commented Jun 28, 2023 • edited Loading

henryiii commented Jul 8, 2023

joerick commented Jul 9, 2023

timfel commented Jul 12, 2023

joerick commented Aug 30, 2023

mayeut commented Jun 22, 2024

timfel commented Jul 17, 2024

timfel commented Jul 19, 2024

mayeut commented Jul 20, 2024 • edited Loading

joerick commented Jan 20, 2025

timfel commented Jan 20, 2025

joerick commented Jan 22, 2025

joerick commented Jan 22, 2025

henryiii commented Apr 1, 2025 • edited Loading

henryiii commented Apr 2, 2025

henryiii commented Apr 3, 2025 • edited Loading

timfel commented Apr 3, 2025

timfel commented Apr 3, 2025 • edited Loading

henryiii commented Apr 3, 2025 • edited Loading

timfel commented Apr 4, 2025 • edited Loading

timfel commented Apr 4, 2025 • edited Loading

timfel commented Apr 8, 2025

henryiii Apr 8, 2025 • edited Loading

Choose a reason for hiding this comment

timfel Apr 9, 2025

Choose a reason for hiding this comment

joerick Apr 9, 2025

Choose a reason for hiding this comment

henryiii left a comment

Choose a reason for hiding this comment

henryiii Apr 8, 2025

Choose a reason for hiding this comment

henryiii Apr 8, 2025

Choose a reason for hiding this comment

henryiii Apr 8, 2025

Choose a reason for hiding this comment

joerick Apr 9, 2025

Choose a reason for hiding this comment

joerick Apr 9, 2025

Choose a reason for hiding this comment

timfel commented Jun 28, 2023 •

edited

Loading

mayeut commented Jul 20, 2024 •

edited

Loading

henryiii commented Apr 1, 2025 •

edited

Loading

henryiii commented Apr 3, 2025 •

edited

Loading

timfel commented Apr 3, 2025 •

edited

Loading

henryiii commented Apr 3, 2025 •

edited

Loading

timfel commented Apr 4, 2025 •

edited

Loading

timfel commented Apr 4, 2025 •

edited

Loading

henryiii Apr 8, 2025 •

edited

Loading