Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI]: git clone revamp, part II #4031

Open
apostasie opened this issue Mar 21, 2025 · 0 comments
Open

[CI]: git clone revamp, part II #4031

apostasie opened this issue Mar 21, 2025 · 0 comments

Comments

@apostasie
Copy link
Contributor

apostasie commented Mar 21, 2025

What is the problem you're trying to solve

This is a second breakout from #4021.

  1. git clones are currently done for every target platform. There is no reason for that, and clone could (should) be done only once per run, at maximum.

  2. The same goes for go modules.
    They are currently retrieved at build time - which means they are retrieved as many times as we build (per architecture, per version).
    They could (should) instead be retrieved once, at clone time, using go mod vendor.

  3. Also, go mod could (should) use a caching mount for the download cache location, to avoid redownloading modules that have been fetched already (eg: by another project), and also avoid putting every independent stage go mod cache into buildkit cache

  4. Finally, it is convenient to be able to override the git repository location of a given project (be it because of maintenance or to test a fork). I recently faced exactly that problem (I am building libslirp, but they went offline for maintenance for a week).

The proposal here is to address these issues.

Describe the solution you'd like

The updated form of the Dockerfile would look something like (pseudo code, should be very close to working):

FROM        --platform=$BUILDPLATFORM builder AS download-bypass4netns
ARG         BYPASS4NETNS_VERSION
ARG         BYPASS4NETNS_REPO
RUN         --mount=target=/root/go/pkg/mod,type=cache \
            git clone --quiet --depth 1 --branch "${BYPASS4NETNS_VERSION%@*}" https://"$BYPASS4NETNS_REPO".git . && \
            git-checkout-tag-with-hash.sh "$VERSION" && \
            go mod vendor

FROM        --platform=$BUILDPLATFORM builder AS build-bypass4netnsd
ARG         TARGETARCH
RUN         --mount=from=download-bypass4netns,type=bind,target=/src,source=/src,rw \
            exec 42>.lock; flock -x 42; \
            GOPROXY=off GOFLAGS=--mod=vendor make blablbabla; \
            flock -u 42

Let me know your thoughts on this to infzasx (cat is typing :-)) to inform upcoming PR.

Additional context

No response

@apostasie apostasie changed the title CI: cloning revamp, part II [CI]: git clone revamp, part II Apr 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant