Skip to content

NO-JIRA: Switch layered build to treefile-apply, drain get-ocp-repo.sh #1780

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Apr 4, 2025
Merged

NO-JIRA: Switch layered build to treefile-apply, drain get-ocp-repo.sh #1780

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
2294939
Nuke `okd-c9s` variant
jlebon Mar 28, 2025
5551bac
c9s.repo: use canonical /etc gpgkey paths
jlebon Mar 30, 2025
69a0b96
Containerfiles: use heredoc to clean up gnarly `RUN`s
jlebon Mar 30, 2025
e2d149d
extensions-okd-c9s: add all mirror repos
jlebon Mar 30, 2025
a2c632d
extensions/Dockerfile: drop `VARIANT` arg
jlebon Mar 30, 2025
6596f19
Containerfile: move to treefile-apply, drop apply-manifest
jlebon Mar 30, 2025
d0fbf44
packages-openshift: define repos list there
jlebon Mar 30, 2025
b4ce075
get-ocp-repo.sh: drastically simplify
jlebon Mar 30, 2025
bee6596
okd: add filtered repo for 4.19 plashet
jlebon Apr 2, 2025
c04f311
Containerfiles: move logic to separate shell script
jlebon Apr 3, 2025
7997d1a
extensions-ocp-rhel-9.6.yaml: add rhel-9.6-fast-datapath repo
jlebon Apr 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 1 addition & 9 deletions Containerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,7 @@

FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev:c9s-coreos as build
ARG OPENSHIFT_CI=0
# Avoid shipping modified .pyc files. Due to https://github.com/ostreedev/ostree/issues/1469,
# any Python apps that run (e.g. dnf) will cause pyc creation.
RUN --mount=type=bind,target=/run/src --mount=type=secret,id=yumrepos,target=/etc/yum.repos.d/secret.repo \
find /usr -name '*.pyc' -exec mv {} {}.bak \; && \
if [ "${OPENSHIFT_CI}" != 0 ]; then /run/src/ci/get-ocp-repo.sh --ocp-layer /run/src/packages-openshift.yaml --output-dir /etc/yum.repos.d; fi && \
/run/src/scripts/apply-manifest /run/src/packages-openshift.yaml && \
if [ "${OPENSHIFT_CI}" != 0 ]; then /run/src/ci/get-ocp-repo.sh --output-dir /etc/yum.repos.d --cleanup; fi && \
find /usr -name '*.pyc.bak' -exec sh -c 'mv $1 ${1%.bak}' _ {} \; && \
ostree container commit
RUN --mount=type=bind,target=/run/src --mount=type=secret,id=yumrepos,target=/etc/yum.repos.d/secret.repo /run/src/build-node-image.sh

FROM build as metadata
RUN --mount=type=bind,target=/run/src /run/src/scripts/generate-metadata
Expand Down
3 changes: 0 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,6 @@ supported:
- `rhel-9.6`: RHEL 9.6-based CoreOS; without OpenShift components.
- `ocp-rhel-9.6`: RHEL 9.6-based CoreOS; including OpenShift components.
- `c9s`: CentOS Stream-based CoreOS, without OKD components.
- `okd-c9s`: CentOS Stream-based CoreOS, including OpenShift components. This
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still see the okd-c9s variant used in the okd/scos build pipeline [1] run in MOC and more specifically in the latest commit okd-project/okd-coreos-pipeline@d4be53e for 4.19.
But according to openshift/release#62296 the scos imagestream (to be used as node image) is now populated by the OpenShift CI itself instead of the MOC pipeline.

So maybe we should decommission the MOC pipeline [2] before merging this patch ? What do you think @Prashanth684 ? It's not a blocker though, the MOC builds would just fail and can be deal as a follow-up.

[1] https://github.com/search?q=repo%3Aokd-project%2Fokd-coreos-pipeline%20okd-c9s&type=code
[2] https://github.com/okd-project/okd-coreos-pipeline

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The okd-c9s "variant" used in that pipeline is for the extensions build, not the base OS AFAIK.

That said, there is indeed a small cleanup possible there which is that it no longer needs to provide a VARIANT argument to the extensions build since that's auto-detected now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So maybe we should decommission the MOC pipeline [2] before merging this patch ? What do you think @Prashanth684 ? It's not a blocker though, the MOC builds would just fail and can be deal as a follow-up.

Correct. MOC is only used for 4.18. Once we release 4.19 as stable, we will stop those also. We are working to migrate off MOC (we still do OKD release promotions from there) to an internal cluster.

currently includes some packages from RHEL because not all packages required
by OpenShift are provided in CentOS Stream.

In the future, the `ocp-*` variants will be removed. Instead, OpenShift
components will be layered by deriving from the `rhel-9.X`/`c9s` images.
Expand Down
43 changes: 43 additions & 0 deletions build-node-image.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
#!/bin/bash
set -euo pipefail

# This script builds the OpenShift node image. It's called from `Containerfile`. set -xeuo pipefail

# Avoid shipping modified .pyc files. Due to
# https://github.com/ostreedev/ostree/issues/1469, any Python apps that
# run (e.g. dnf) will cause pyc creation. We do this by backing them up and
# restoring them at the end.
find /usr -name '*.pyc' -exec mv {} {}.bak \;

# fetch repos from in-cluster mirrors if we're running in OpenShift CI
if [ "${OPENSHIFT_CI}" != 0 ]; then
/run/src/ci/get-ocp-repo.sh /etc/yum.repos.d/ocp.repo
fi

source /etc/os-release

# XXX: For SCOS, only allow certain packages to come from ART; everything else
# should come from CentOS. We should eventually sever this.
if [ $ID = centos ]; then
# this says: "if the line starts with [.*], turn off printing. if the line starts with [our-repo], turn it on."
awk "/\[.*\]/{p=0} /\[rhel-9.6-server-ose-4.19\]/{p=1} p" /etc/yum.repos.d/*.repo > /etc/yum.repos.d/okd.repo.tmp
sed -i -e 's,rhel-9.6-server-ose-4.19,rhel-9.6-server-ose-4.19-okd,' /etc/yum.repos.d/okd.repo.tmp
echo 'includepkgs=openshift-*,ose-aws-ecr-*,ose-azure-acr-*,ose-gcp-gcr-*' >> /etc/yum.repos.d/okd.repo.tmp
mv /etc/yum.repos.d/okd.repo{.tmp,}
fi

# XXX: patch cri-o spec to use tmpfiles
# https://github.com/CentOS/centos-bootc/issues/393
mkdir -p /var/opt

# this is where all the real work happens
rpm-ostree experimental compose treefile-apply \
--var id=$ID /run/src/packages-openshift.yaml

# cleanup the repo file we injected
if [ "${OPENSHIFT_CI}" != 0 ]; then
rm /etc/yum.repos.d/ocp.repo
fi

find /usr -name '*.pyc.bak' -exec sh -c 'mv $1 ${1%.bak}' _ {} \;
ostree container commit
8 changes: 4 additions & 4 deletions c9s-mirror.repo
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,28 +9,28 @@ baseurl=https://mirror.stream.centos.org/9-stream/BaseOS/$basearch/os
gpgcheck=1
repo_gpgcheck=0
enabled=1
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[c9s-appstream-mirror]
name=CentOS Stream 9 - AppStream
baseurl=https://mirror.stream.centos.org/9-stream/AppStream/$basearch/os
gpgcheck=1
repo_gpgcheck=0
enabled=1
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[c9s-nfv-mirror]
name=CentOS Stream 9 - NFV
baseurl=https://mirror.stream.centos.org/9-stream/NFV/$basearch/os
gpgcheck=1
repo_gpgcheck=0
enabled=1
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[c9s-rt-mirror]
name=CentOS Stream 9 - RT
baseurl=https://mirror.stream.centos.org/9-stream/RT/$basearch/os
gpgcheck=1
repo_gpgcheck=0
enabled=1
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
16 changes: 8 additions & 8 deletions c9s.repo
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,15 @@ baseurl=https://composes.stream.centos.org/production/latest-CentOS-Stream/compo
gpgcheck=1
repo_gpgcheck=0
enabled=1
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[c9s-appstream]
name=CentOS Stream 9 - AppStream
baseurl=https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/AppStream/$basearch/os
gpgcheck=1
repo_gpgcheck=0
enabled=1
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[c9s-extras-common]
name=CentOS Stream 9 - Extras packages
Expand All @@ -29,44 +29,44 @@ baseurl=https://mirror.stream.centos.org/SIGs/9-stream/extras/x86_64/extras-comm
gpgcheck=1
repo_gpgcheck=0
enabled=1
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-SIG-Extras-SHA512
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Extras-SHA512

[c9s-nfv]
name=CentOS Stream 9 - NFV
baseurl=https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/NFV/$basearch/os
gpgcheck=1
repo_gpgcheck=0
enabled=1
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[c9s-rt]
name=CentOS Stream 9 - RT
baseurl=https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/RT/$basearch/os
gpgcheck=1
repo_gpgcheck=0
enabled=1
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[c9s-sig-nfv]
name=CentOS Stream 9 - SIG NFV
baseurl=https://mirror.stream.centos.org/SIGs/9-stream/nfv/$basearch/openvswitch-2/
gpgcheck=1
repo_gpgcheck=0
enabled=1
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-SIG-NFV
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-NFV

[c9s-sig-virtualization]
name=CentOS Stream 9 - SIG Virtualization
baseurl=https://mirror.stream.centos.org/SIGs/9-stream/virt/$basearch/kata-containers/
gpgcheck=1
repo_gpgcheck=0
enabled=1
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-SIG-Virtualization
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Virtualization

[c9s-sig-cloud-okd]
name=CentOS Stream 9 - SIG Cloud OKD 4.19
baseurl=https://mirror.stream.centos.org/SIGs/9-stream/cloud/$basearch/okd-4.19/
gpgcheck=1
repo_gpgcheck=0
enabled=1
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-SIG-Cloud
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud
206 changes: 12 additions & 194 deletions ci/get-ocp-repo.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,203 +3,21 @@ set -euo pipefail

# This script is used when running within the OpenShift CI clusters to fetch
# the RHEL and OCP yum repo files from an in-cluster service that mirrors the
# content. It's called from three places:
# - prow-entrypoint.sh: CI tests that build & and test different variants
# - extensions/Dockerfile: when building the extensions container in OpenShift CI
# - Containerfile: when building the node image in CI
# content.

print_usage_and_exit() {
cat 1>&2 <<'EOF'
Usage: $0 <MODE> [OPTIONS]

Fetch mirrored RHEL/OCP yum repo files from OpenShift CI's in-cluster service.
The following modes are supported:

--cosa-workdir PATH Get RHEL and OCP versions from manifests in cosa workdir
--ocp-layer MANIFEST Get RHEL version from /usr/lib/os-release and OCP version from manifest

The following options are supported

--output-dir PATH Directory to which to output ocp.repo file
EOF
exit 1
}

info() {
echo "INFO:" "$@" >&2
}

cleanup_repos() {
# if we had installed the packages and created symlinks, remove it
if rpm -q centos-release-cloud; then
dnf remove -y centos-release-{cloud,nfv,virt}-common
find "/usr/share/distribution-gpg-keys/centos" -type l -exec rm -f {} \;
echo "Removed all symbolic links and packages installed for scos"
fi
# remove ocp.repo file
if [ -n "$ocp_manifest" ]; then
if [ -z "$output_dir" ]; then
output_dir=$(dirname "$ocp_manifest")
fi
else
if [ -z "$output_dir" ]; then
output_dir="$cosa_workdir/src/config"
fi
fi
rm "$output_dir/ocp.repo"
echo "Removed repo file $output_dir/ocp.repo"
}

create_gpg_keys() {
# Check if centos-stream-release is installed and centos-release-cloud is not
# enablerepo added in case the repo is disabled (when building extensions)
if rpm -q centos-stream-release && ! rpm -q centos-release-cloud; then
dnf install -y centos-release-{cloud,nfv,virt}-common --enablerepo extras-common
fi

# Create directory for CentOS distribution GPG keys
mkdir -p /usr/share/distribution-gpg-keys/centos
# Create symbolic links for GPG keys
if [ ! -e "/usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official" ]; then
ln -s /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial /usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Cloud
ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Extras-SHA512
ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-NFV
ln -s {/etc/pki/rpm-gpg,/usr/share/distribution-gpg-keys/centos}/RPM-GPG-KEY-CentOS-SIG-Virtualization
fi
}

cosa_workdir=
ocp_manifest=
output_dir=
rc=0
options=$(getopt --options h --longoptions help,cosa-workdir:,ocp-layer:,output-dir:,cleanup -- "$@") || rc=$?
[ $rc -eq 0 ] || print_usage_and_exit
eval set -- "$options"
while [ $# -ne 0 ]; do
case "$1" in
-h | --help) print_usage_and_exit;;
--cosa-workdir) cosa_workdir=$2; shift;;
--ocp-layer) ocp_manifest=$2; shift;;
--output-dir) output_dir=$2; shift;;
--cleanup) cleanup_repos; exit 0;;
--) break;;
*) echo "$0: invalid argument: $1" >&2; exit 1;;
esac
shift
done

if [ -n "$ocp_manifest" ]; then
# --ocp-layer path
ocp_version=$(rpm-ostree compose tree --print-only "$ocp_manifest" | jq -r '.metadata.ocp_version')
ocp_version=${ocp_version//./-}
info "Got OpenShift version $ocp_version from $ocp_manifest"
# osname is used lower down, so set it
osname=$(source /usr/lib/os-release; if [ $ID == centos ]; then echo scos; fi)

if [ -z "$output_dir" ]; then
output_dir=$(dirname "$ocp_manifest")
fi

# get rhel version corresponding to the release so we can get the
# correct OpenShift rpms from those for scos. These packages are not
# available in CentOS Stream
if [ "$osname" = scos ]; then
workdir=$(dirname "$ocp_manifest")
manifest="$workdir/manifest.yaml"
json=$(rpm-ostree compose tree --print-only "$manifest")
version=$(jq -r '.["automatic-version-prefix"]' <<< "$json")
rhel_version=$(cut -f2 -d. <<< "$version")
info "Got RHEL version $rhel_version from rhel manifest for scos"
else
rhel_version=$(source /usr/lib/os-release; echo ${VERSION_ID//./})
info "Got RHEL version $rhel_version from /usr/lib/os-release"
fi
else
[ -n "$cosa_workdir" ]
# --cosa-workdir path

# the OCP version always comes from packages-openshift.yaml
ocp_version=$(rpm-ostree compose tree --print-only "$cosa_workdir/src/config/packages-openshift.yaml" | jq -r '.metadata.ocp_version')
ocp_version=${ocp_version//./-}
info "Got OpenShift version $ocp_version from packages-openshift.yaml"

# the RHEL version comes from the target manifest

# first, make sure we're looking at the right manifest
manifest="$cosa_workdir/src/config/manifest.yaml"
if [ -f "$cosa_workdir/src/config.json" ]; then
variant="$(jq --raw-output '."coreos-assembler.config-variant"' 'src/config.json')"
manifest="$cosa_workdir/src/config/manifest-${variant}.yaml"
fi

# flatten manifest and query a couple of fields
json=$(rpm-ostree compose tree --print-only "$manifest")
osname=$(jq -r '.metadata.name' <<< "$json")
is_ocp_variant=$(jq '.packages | contains(["cri-o"])' <<< "$json")

if [ "$osname" = scos ] && [ "$is_ocp_variant" = false ]; then
# this is the pure SCOS case; we don't need any additional repos at all
info "Building pure SCOS variant. Exiting..."
exit 0
elif [ "$osname" = scos ]; then
# We still need the OCP repos for now unfortunately because not
# everything is in the Stream repo. For the RHEL version, just use the
# default variant's one.
json=$(rpm-ostree compose tree --print-only "$cosa_workdir/src/config/manifest.yaml")
fi
version=$(jq -r '.["automatic-version-prefix"]' <<< "$json")
if [ "$is_ocp_variant" = true ]; then
# RHEL version is second field
info "Building OCP variant"
rhel_version=$(cut -f2 -d. <<< "$version")
else
# RHEL version is first and second field
info "Building pure variant"
rhel_version=$(cut -f1-2 -d. <<< "$version")
rhel_version=${rhel_version//./}
fi
info "Got RHEL version $rhel_version from automatic-version-prefix value $version"

if [ -z "$output_dir" ]; then
output_dir="$cosa_workdir/src/config"
fi
fi

mkdir -p "$output_dir"
repo_path="$output_dir/ocp.repo"

set -x
curl --fail -L "http://base-${ocp_version}-rhel${rhel_version}.ocp.svc.cluster.local" -o "$repo_path"
set +x

if [ "${rhel_version}" = 96 ]; then
# XXX: also currently also add 9.4 repos for crun-wasm when building extensions
urls=(
# theoretically that's the only one we need
"http://base-4-19-rhel96.ocp.svc.cluster.local"
# XXX: but also currently add 9.4 repos for crun-wasm when building extensions
# https://github.com/openshift/os/issues/1680
# https://github.com/openshift/os/pull/1682
# https://issues.redhat.com/browse/COS-3075
curl --fail -L http://base-4-19-rhel94.ocp.svc.cluster.local >> "$repo_path"
fi
"http://base-4-19-rhel94.ocp.svc.cluster.local"
)

# If we're building the SCOS OKD variant, then strip away all the RHEL repos and just keep the plashet.
# Temporary workaround until we have all packages for SCOS in CentOS Stream.
if [ "$osname" = scos ]; then
info "Neutering RHEL repos for SCOS"
awk '/server-ose/,/^$/' "$repo_path" > "$repo_path.tmp"
# only pull in certain Openshift packages as the rest come from the c9s repo
sed -i '/^baseurl = /a includepkgs=openshift-* ose-aws-ecr-* ose-azure-acr-* ose-gcp-gcr-*' "$repo_path.tmp"
# add the contents of the CentOS Stream repo
workdir="$cosa_workdir/src/config"
if [ -n "$ocp_manifest" ]; then
workdir=$(dirname "$ocp_manifest")
fi
# pull in the mirror repo as well in case there are newer versions in the composes
# and we require older versions - this happens because we build the node images async
# and the composes move fast.
cat "$workdir/c9s.repo" >> "$repo_path.tmp"
cat "$workdir/c9s-mirror.repo" >> "$repo_path.tmp"
mv "$repo_path.tmp" "$repo_path"
create_gpg_keys
fi
dest=$1; shift

cat "$repo_path"
rm -f "$dest"
for url in "${urls[@]}"; do
curl --fail -L "$url" >> "$dest"
done
2 changes: 1 addition & 1 deletion ci/prow-entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ cosa_init() {

# Initialize the .repo files
prepare_repos() {
src/config/ci/get-ocp-repo.sh --cosa-workdir .
src/config/ci/get-ocp-repo.sh src/config/ocp.repo
}

# Do a cosa build & cosa build-extensions only.
Expand Down
Loading