Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-43951: [CI][Python] Use GitHub Packages for vcpkg cache #44644

Open
wants to merge 33 commits into
base: main
Choose a base branch
from
Open

GH-43951: [CI][Python] Use GitHub Packages for vcpkg cache #44644

wants to merge 33 commits into from
+117 −47

Conversation

kou
Copy link
Member

@kou kou commented Nov 5, 2024 •
edited
Loading

Rationale for this change

We're using only Docker level cache for vcpkg used for wheels. If we have any vcpkg related changes, all vcpkg ports are rebuilt. It's time consuming.

What changes are included in this PR?

Enable NuGet + GitHub Packages based cache. It's port level cache. So we don't need to rebuild all ports when we have any vcpkg related changes.

See also: https://learn.microsoft.com/en-us/vcpkg/consume/binary-caching-github-packages

NuGet + GitHub Packages based cache isn't enabled with manylinux2014 + aarch64. Because EPEL for CentOS 7 + aarch64 provides old Mono. (FYI: EPEL for CentOS 7 + x86_64 provides newer Mono.) We can't use old Mono to run NuGet on Linux.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 5, 2024

⚠️ GitHub issue #43951 has been automatically assigned in GitHub to PR creator.

@github-actions github-actions bot added the awaiting committer review Awaiting committer review label Nov 5, 2024
@kou
Copy link
Member Author

kou commented Nov 5, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 5, 2024 

Failed to render template `python-wheels/github.linux.yml` with UndefinedError: 'secrets' is undefined
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/11678507794

@kou kou force-pushed the ci-wheel-manylinux-vcpkg-cache branch from c931193 to f401684 Compare November 5, 2024 06:08
@kou
Copy link
Member Author

kou commented Nov 5, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

@github-actions GitHub Actions

This comment was marked as outdated.

Sign in to view
@kou
Copy link
Member Author

kou commented Nov 5, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

@github-actions GitHub Actions

This comment was marked as outdated.

Sign in to view
@kou
Copy link
Member Author

kou commented Nov 5, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

@github-actions GitHub Actions

This comment was marked as outdated.

Sign in to view
@kou
Copy link
Member Author

kou commented Nov 5, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

@github-actions GitHub Actions

This comment was marked as outdated.

Sign in to view
@kou
Copy link
Member Author

kou commented Nov 5, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

@github-actions GitHub Actions

This comment was marked as outdated.

Sign in to view
@kou
Copy link
Member Author

kou commented Nov 6, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

1 similar comment
@kou
Copy link
Member Author

kou commented Nov 6, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

@kou kou changed the title GH-43951: [CI][Python] Use GitHub Packages for vcpkg cache GH-43951: [CI][Python] Use GitHub Actions cache for vcpkg cache Nov 6, 2024
@kou kou force-pushed the ci-wheel-manylinux-vcpkg-cache branch from 78ee981 to 8b4f8ab Compare November 6, 2024 08:00
@kou kou mentioned this pull request Nov 6, 2024
Closed
@kou
Copy link
Member Author

kou commented Nov 6, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

2 similar comments
@kou
Copy link
Member Author

kou commented Nov 6, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

@kou
Copy link
Member Author

kou commented Nov 6, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

@kou kou changed the title GH-43951: [CI][Python] Use GitHub Actions cache for vcpkg cache GH-43951: [CI][Python] Use GitHub Packages for vcpkg cache Nov 6, 2024
@kou
Copy link
Member Author

kou commented Nov 6, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

1 similar comment
@kou
Copy link
Member Author

kou commented Nov 6, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

@github-actions GitHub Actions

This comment was marked as outdated.

Sign in to view
@kou kou force-pushed the ci-wheel-manylinux-vcpkg-cache branch from 63d255c to 6730902 Compare November 13, 2024 07:33
@github-actions github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Nov 13, 2024
@kou
Copy link
Member Author

kou commented Nov 13, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

@github-actions GitHub Actions

This comment was marked as outdated.

Sign in to view
@kou
Copy link
Member Author

kou commented Nov 13, 2024

@github-actions crossbow submit -g wheel java-jars

@github-actions GitHub Actions
Copy link

Revision: 52b6ee9

Submitted crossbow builds: ursacomputing/crossbow @ actions-f4d0e5be13

Task Status
java-jars GitHub Actions
python-sdist GitHub Actions
wheel-macos-monterey-cp310-cp310-amd64 GitHub Actions
wheel-macos-monterey-cp310-cp310-arm64 GitHub Actions
wheel-macos-monterey-cp311-cp311-amd64 GitHub Actions
wheel-macos-monterey-cp311-cp311-arm64 GitHub Actions
wheel-macos-monterey-cp312-cp312-amd64 GitHub Actions
wheel-macos-monterey-cp312-cp312-arm64 GitHub Actions
wheel-macos-monterey-cp313-cp313-amd64 GitHub Actions
wheel-macos-monterey-cp313-cp313-arm64 GitHub Actions
wheel-macos-monterey-cp313-cp313t-amd64 GitHub Actions
wheel-macos-monterey-cp313-cp313t-arm64 GitHub Actions
wheel-macos-monterey-cp39-cp39-amd64 GitHub Actions
wheel-macos-monterey-cp39-cp39-arm64 GitHub Actions
wheel-manylinux-2-28-cp310-cp310-amd64 GitHub Actions
wheel-manylinux-2-28-cp310-cp310-arm64 GitHub Actions
wheel-manylinux-2-28-cp311-cp311-amd64 GitHub Actions
wheel-manylinux-2-28-cp311-cp311-arm64 GitHub Actions
wheel-manylinux-2-28-cp312-cp312-amd64 GitHub Actions
wheel-manylinux-2-28-cp312-cp312-arm64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313-amd64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313-arm64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313t-amd64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313t-arm64 GitHub Actions
wheel-manylinux-2-28-cp39-cp39-amd64 GitHub Actions
wheel-manylinux-2-28-cp39-cp39-arm64 GitHub Actions
wheel-manylinux-2014-cp310-cp310-amd64 GitHub Actions
wheel-manylinux-2014-cp310-cp310-arm64 GitHub Actions
wheel-manylinux-2014-cp311-cp311-amd64 GitHub Actions
wheel-manylinux-2014-cp311-cp311-arm64 GitHub Actions
wheel-manylinux-2014-cp312-cp312-amd64 GitHub Actions
wheel-manylinux-2014-cp312-cp312-arm64 GitHub Actions
wheel-manylinux-2014-cp313-cp313-amd64 GitHub Actions
wheel-manylinux-2014-cp313-cp313-arm64 GitHub Actions
wheel-manylinux-2014-cp313-cp313t-amd64 GitHub Actions
wheel-manylinux-2014-cp313-cp313t-arm64 GitHub Actions
wheel-manylinux-2014-cp39-cp39-amd64 GitHub Actions
wheel-manylinux-2014-cp39-cp39-arm64 GitHub Actions
wheel-windows-cp310-amd64 GitHub Actions
wheel-windows-cp311-amd64 GitHub Actions
wheel-windows-cp312-amd64 GitHub Actions
wheel-windows-cp313-amd64 GitHub Actions
wheel-windows-cp39-amd64 GitHub Actions

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Nov 13, 2024
raulcd
raulcd reviewed Nov 13, 2024
View reviewed changes
Copy link
Member

@raulcd raulcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a minor nit and a question

ci/docker/python-wheel-manylinux.dockerfile Outdated Show resolved Hide resolved
ci/scripts/install_vcpkg.sh
@@ -53,4 +53,33 @@ if [ -f "${vcpkg_ports_patch}" ]; then
echo "Patch successfully applied to the VCPKG port files!"
fi

if [ -n "${GITHUB_TOKEN:-}" ] && \
[ -n "${GITHUB_REPOSITORY_OWNER:-}" ] && \
[ "${VCPKG_BINARY_SOURCES:-}" = "clear;nuget,GitHub,readwrite" ] ; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in which case would VCPKG_BINARY_SOURCES be different to clear;nuget,GitHub,readwrite? on manylinux2014_aarch64 this wouldn't be present but not different, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If VCPKG_BINARY_SOURCES is not clear;nuget,GitHub,readwrite (including not present case on manylinux2014_aarch64), we don't configure NuGet automatically.

Ignoring not clear;nuget,GitHub,readwrite VCPKG_BINARY_SOURCES is for using other cache. For example, we may want to use local file cache for local archery docker run. But it's out-of-scope of this PR and it's not tested. If we need it, we can work on it as a separated task.

@kou @raulcd
Fix English
b2238ef 
Co-authored-by: Raúl Cumplido <[email protected]>
@github-actions github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Nov 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pitrou pitrou pitrou left review comments

@raulcd raulcd raulcd left review comments

@assignUser assignUser assignUser requested changes

@jonkeane jonkeane Awaiting requested review from jonkeane jonkeane is a code owner

Assignees
No one assigned
Labels
awaiting changes Awaiting changes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kou @raulcd @pitrou @assignUser