Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support the free-threaded build of CPython 3.13 #1456

Open
wants to merge 37 commits into
base: master
Choose a base branch
from

Conversation

lysnikolaou
Copy link
Contributor

@lysnikolaou lysnikolaou commented Dec 20, 2024

What do these changes do?

  • Add a matrix config for free-threaded Python to run tests with it.
  • Change configuration to correctly build wheels under the free-threaded build.
  • Fix thread-safety issues in the Cython extension module.

Are there changes in behavior for the user?

No.

Related issue number

Resolves #1455

Checklist

  • I think the code is well written
  • Unit tests for the changes exist
  • Documentation reflects the changes

@psf-chronographer psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Dec 20, 2024
Copy link

codspeed-hq bot commented Jan 23, 2025

CodSpeed Performance Report

Merging #1456 will not alter performance

Comparing lysnikolaou:free-threading-support (ace56a4) with master (3209565)

Summary

✅ 99 untouched benchmarks

Copy link

codecov bot commented Jan 23, 2025

Codecov Report

Attention: Patch coverage is 95.55556% with 6 lines in your changes missing coverage. Please review.

Project coverage is 95.22%. Comparing base (3209565) to head (ace56a4).

Files with missing lines Patch % Lines
yarl/_quoting_c.pyx 76.19% 5 Missing ⚠️
packaging/pep517_backend/_backend.py 83.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1456      +/-   ##
==========================================
- Coverage   95.57%   95.22%   -0.35%     
==========================================
  Files          33       33              
  Lines        6010     6032      +22     
  Branches      364      364              
==========================================
  Hits         5744     5744              
- Misses        240      262      +22     
  Partials       26       26              
Flag Coverage Δ
CI-GHA 95.22% <95.55%> (-0.35%) ⬇️
MyPy 49.78% <99.10%> (+0.06%) ⬆️
OS-Linux 89.90% <96.12%> (-8.94%) ⬇️
OS-Windows 89.44% <100.00%> (-9.42%) ⬇️
OS-macOS 89.71% <96.12%> (-8.88%) ⬇️
Py-3.10.11 89.57% <96.12%> (-9.00%) ⬇️
Py-3.10.16 89.79% <96.12%> (-9.00%) ⬇️
Py-3.11.11 89.79% <96.12%> (-9.00%) ⬇️
Py-3.11.9 89.57% <96.12%> (-9.00%) ⬇️
Py-3.12.8 89.57% <96.12%> (-9.00%) ⬇️
Py-3.12.9 89.79% <96.12%> (-9.00%) ⬇️
Py-3.13.1 89.57% <96.12%> (-9.00%) ⬇️
Py-3.13.1t 89.72% <100.00%> (?)
Py-3.13.2 89.79% <96.12%> (-9.00%) ⬇️
Py-3.9.13 89.53% <96.12%> (-9.00%) ⬇️
Py-3.9.21 89.75% <96.12%> (-9.00%) ⬇️
Py-pypy7.3.16 89.33% <100.00%> (-9.42%) ⬇️
Py-pypy7.3.18 ?
Py-pypy7.3.19 89.35% <100.00%> (-9.42%) ⬇️
VM-macos-latest 89.71% <96.12%> (-8.88%) ⬇️
VM-ubuntu-latest 89.90% <96.12%> (-8.94%) ⬇️
VM-windows-latest 89.44% <100.00%> (-9.42%) ⬇️
pytest 89.90% <96.12%> (-8.94%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@lysnikolaou lysnikolaou force-pushed the free-threading-support branch 3 times, most recently from 78f84fd to 1c857ca Compare February 17, 2025 13:04
@lysnikolaou lysnikolaou force-pushed the free-threading-support branch from 9bfa0d3 to 44ee894 Compare February 17, 2025 13:06
lysnikolaou and others added 3 commits February 17, 2025 15:30
Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <[email protected]>
@lysnikolaou lysnikolaou force-pushed the free-threading-support branch from 4c8fe2f to 3527edc Compare February 17, 2025 18:00
@lysnikolaou
Copy link
Contributor Author

FTR the CI is failing due to actions/setup-python#1034 and the following fix will need to be implemented here: aio-libs/frozenlist#622 + aio-libs/frozenlist#633.

One alternative to this would be to use setup-uv instead of setup-python, which could also speed up dependency installation etc. If you feel okay about this, I can try it out in a separate PR. Otherwise, I can also move the custom actions to this repo as well.

@webknjaz
Copy link
Member

FTR the CI is failing due to actions/setup-python#1034 and the following fix will need to be implemented here: aio-libs/frozenlist#622 + aio-libs/frozenlist#633.

One alternative to this would be to use setup-uv instead of setup-python, which could also speed up dependency installation etc. If you feel okay about this, I can try it out in a separate PR. Otherwise, I can also move the custom actions to this repo as well.

I'm still looking into the uv integration, I'm not sure it's ready for prime time just yet. I'll be looking into that separately. But feel free to send a PR with the caching fix separately.

@bdraco
Copy link
Member

bdraco commented Feb 20, 2025

The performance regression in the writer looks real. I can't come up with a better solution though that is thread-safe.

@lysnikolaou
Copy link
Contributor Author

The performance regression in the writer looks real. I can't come up with a better solution though that is thread-safe.

We could try to make it so that the global buffer is used in the non-free-threaded build, but not in the free-threaded one. Cython, however, makes that particularly hard. There's two possible solutions:

  • Rewrite the buffer handling in C, where we can use #ifdef Py_GIL_DISABLED.
  • Use the Cython compile-time DEF and IF features, that have been deprecated and have no good alternatives.

Both of the solutions are sub-optimal. Thoughts?

@bdraco
Copy link
Member

bdraco commented Feb 21, 2025

The C solution sounds better because it will work long term vs the DEF and IF are guaranteed to break sooner

@Dreamsorcerer
Copy link
Member

Are there any plans to expose Py_GIL_DISABLED in Cython? If so, might make more sense to come back and fix it when that happens.

If we do want to write it without Cython, should we consider Rust instead of C?

Comment on lines +98 to +102
buf = <char *> PyMem_Malloc(BUF_SIZE)
if buf == NULL:
PyErr_NoMemory()
return
writer.buf = buf
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lysnikolaou this might be because the test job installs older cython: https://github.com/aio-libs/yarl/actions/runs/13526871108/job/37800294520?pr=1456#step:8:3. Could you try updating it there to see if it helps?

Copy link
Contributor Author

@lysnikolaou lysnikolaou Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably happening because the quansight-labs/setup-python action returns the version number without the trailing t so one set of coverage results might be overriding the other. I've fixed that.

Also opend #1480 to get rid of the hypothesis warnings.

@lysnikolaou lysnikolaou force-pushed the free-threading-support branch from 41c79ea to ace56a4 Compare March 11, 2025 12:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bot:chronographer:provided There is a change note present in this PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support the free-threaded build of CPython 3.13
4 participants