Add details about JIT build infrastructure and updating dependencies to Tools/jit
#141167
Conversation
savannahostrowski
requested review from
brandtbucher and
diegorusso
as code owners
savannahostrowski
added
skip issue
skip news
topic-JIT
infra
|
|
||
| ### Other notes | ||
| - You must make sure that the name of the artifact matches exactly what is expected in `Tools/jit/_llvm.py` and `PCbuild/get_externals.py`. | ||
| - We don't need multiple release artifacts for each architecture because LLVM can cross-compile for different architectures on Windows; x86_64 is sufficient. |
There was a problem hiding this comment.
The reason why it works is because on Windows on Arm the binary emulation (https://learn.microsoft.com/en-us/windows/arm/apps-on-arm-x86-emulation) executes the x86_64 binary on the platform and, being a compiler, it can cross compile to Arm64.
Ideally, on Windows on Arm, it should be run natively by installing the Arm64 version (clang+llvm-21.1.4-aarch64-pc-windows-msvc.tar.xz) and then compile natively.
There was a problem hiding this comment.
Yep, it's definitely possible to do this. I have it on my list to explore 1) automating some of this process (i.e. make tarball extraction smarter) and 2) making it so that we can handle platform-specific tarballs. Just haven't gotten around it yet.
| ```bash | ||
| tar -xf clang+llvm-21.1.4-x86_64-pc-windows-msvc.tar.xz | ||
| mv clang+llvm-21.1.4-x86_64-pc-windows-msvc llvm-21.1.4.0 | ||
| tar -cf - llvm-21.1.4.0 | pv | xz > llvm-21.1.4.0.tar.xz |
There was a problem hiding this comment.
is pv really needed here?
There was a problem hiding this comment.
Re. this and the comment below, I've included pv here so that you can see progress as the binary is being rezipped. It's fairly large so the process is not instantaneous. However, this is just an example, folks can choose to do this as they see fit 😄
| To update the LLVM release artifact for Windows builds, follow these steps: | ||
| 1. Go to the [LLVM releases page](https://github.com/llvm/llvm-project/releases). | ||
| 1. Download x86_64 Windows artifact for the desired LLVM version (e.g. `clang+llvm-21.1.4-x86_64-pc-windows-msvc.tar.xz`). | ||
| 1. Extract and repackage the tarball with the correct directory structure. For example: |
There was a problem hiding this comment.
More generic question: can this be avoided at all?
IIUC, we are doing all of this just to set the right directory within the archive. Can we actually deal with the original directory structure in CPython?
There was a problem hiding this comment.
Some thoughts here: https://github.com/python/cpython/pull/141167/files#r2504767448
...but TL;DR: yes, we could potentially automate the creation of LLVM release artifacts in cpython-bin-deps and make tarball extraction smarter. I'd tackle this in pieces and as a separate issue though!
2 participants
I'm not going anywhere 😄 but figured that as our resident LLVM upgrade specialist ™️ 🫡 , I'd jot down the process of bumping the dep in
cpython-bin-depsso others have visibility.