NO-JIRA: Switch layered build to `treefile-apply`, drain `get-ocp-repo.sh` #1780

jlebon · 2025-03-28T16:02:51Z

See individual commit messages.

openshift-ci-robot · 2025-03-28T16:03:02Z

@jlebon: This pull request explicitly references no jira issue.

In response to this:

This is not built anywhere by anyone. OKD has moved to the new layered node image model and uses the output from the c9s variant we currently build internally.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

jlebon · 2025-03-28T16:50:47Z

[1/3] STEP 9/9: RUN --mount=type=secret,id=yumrepos,target=/os/secret.repo if [[ -n "${VARIANT}" ]]; then MANIFEST="manifest-${VARIANT}.yaml"; EXTENSIONS="extensions-${VARIANT}.yaml"; else MANIFEST="manifest.yaml"; EXTENSIONS="extensions.yaml"; fi && rpm-ostree compose extensions --rootfs=/ --output-dir=/usr/share/rpm-ostree/extensions/ ./"${MANIFEST}" ./"${EXTENSIONS}"
error: Can't open file "./manifest-okd-c9s.yaml" for reading: No such file or directory (os error 2)

Man, this get-ocp-repo.sh script has become quite the beast. I'm thinking of reworking how that works entirely to make it saner.

jlebon · 2025-03-31T14:43:55Z

This requires coreos/rpm-ostree#5351 and coreos/coreos-assembler#4054.

jlebon · 2025-03-31T17:25:36Z

cc @Prashanth684 since this also touches OKD

jbtrystram

Awesome !
Just a couple of questions :)

common.yaml

extensions-okd-c9s.yaml

Containerfile

jcapiitao · 2025-04-01T10:34:01Z

README.md

@@ -16,9 +16,6 @@ supported:
 - `rhel-9.6`: RHEL 9.6-based CoreOS; without OpenShift components.
 - `ocp-rhel-9.6`: RHEL 9.6-based CoreOS; including OpenShift components.
 - `c9s`: CentOS Stream-based CoreOS, without OKD components.
- `okd-c9s`: CentOS Stream-based CoreOS, including OpenShift components. This


I still see the okd-c9s variant used in the okd/scos build pipeline [1] run in MOC and more specifically in the latest commit okd-project/okd-coreos-pipeline@d4be53e for 4.19.
But according to openshift/release#62296 the scos imagestream (to be used as node image) is now populated by the OpenShift CI itself instead of the MOC pipeline.

So maybe we should decommission the MOC pipeline [2] before merging this patch ? What do you think @Prashanth684 ? It's not a blocker though, the MOC builds would just fail and can be deal as a follow-up.

[1] https://github.com/search?q=repo%3Aokd-project%2Fokd-coreos-pipeline%20okd-c9s&type=code
[2] https://github.com/okd-project/okd-coreos-pipeline

The okd-c9s "variant" used in that pipeline is for the extensions build, not the base OS AFAIK.

That said, there is indeed a small cleanup possible there which is that it no longer needs to provide a VARIANT argument to the extensions build since that's auto-detected now.

So maybe we should decommission the MOC pipeline [2] before merging this patch ? What do you think @Prashanth684 ? It's not a blocker though, the MOC builds would just fail and can be deal as a follow-up.

Correct. MOC is only used for 4.18. Once we release 4.19 as stable, we will stop those also. We are working to migrate off MOC (we still do OKD release promotions from there) to an internal cluster.

jcapiitao · 2025-04-01T16:18:24Z

It looks like the CI base image does not contain the /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial file containing the CentOS Stream GPG keys.

jlebon · 2025-04-01T16:27:34Z

It looks like the CI base image does not contain the /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial file containing the CentOS Stream GPG keys.

Yeah, it needed coreos/coreos-assembler#4054. That's merged now but there's no point retriggering the tests until coreos/rpm-ostree#5351 percolates down too. We're working on that.

ci/get-ocp-repo.sh

jlebon · 2025-04-02T18:59:43Z

/retest

jlebon · 2025-04-02T19:10:37Z

OK, we have new enough rpm-ostree and cosa now. Let's try this out!

jlebon · 2025-04-02T19:11:31Z

@Prashanth684 Can you take of this bit once this lands:

That said, there is indeed a small cleanup possible there which is that it no longer needs to provide a VARIANT argument to the extensions build since that's auto-detected now.

dustymabe

mostly LGTM - some comments

Containerfile

dustymabe · 2025-04-02T20:10:08Z

extensions/Dockerfile

+# buildah doesn't seem to support heredoc output
+# redirection like buildkit so do it manually here


is there a bug for this we can link to?

I didn't find one, but I could file one I guess.

Ahhh OK, just saw your other comment below related to this. The feature I'm talking about here isn't just bash redirection, but a RUN feature so you can write e.g.

RUN <<EOF > /out echo foobar

and it'll go to /out. Which would've been perfect for our use case here.

dustymabe · 2025-04-02T20:24:09Z

extensions/Dockerfile

+  MANIFEST="manifest-rhel-9.6.yaml"
+  EXTENSIONS="extensions-ocp-rhel-9.6.yaml"


Does this imply these paths will need to be updated each time we say bump to the next version of RHEL? (i.e. RHEL 9.7 or 9.8 are here?)

I would kind of prefer that we didn't have to remember to update these variables when that happens.

Not 9.7, but 9.8 yeah. It would be part of all the tree updates we do after branching for 4.22 I guess? When bumping the base variant from rhel-9.6 to rhel-9.8. Though by then we should be on top of rhel-bootc which likely means we can rework this too since it's then driven by the base RHEL version of the rhel-bootc image.

ci/get-ocp-repo.sh

Containerfile

jlebon · 2025-04-03T03:40:05Z

Hmm, the builder is choking on the heredocs. I think possibly the Dockerfile parser there (which happens before it's handed off to buildah) is getting tripped up on something. :( I'll dig into this a bit, but if it can't be easily worked around, I think I'll just add a commit for now that moves the heredocs to shell scripts for now.

c9s builds are failing on

 error: Installing packages: Updating rpm-md repo 'c9s-baseos-mirror': Failed to download gpg key for repo 'c9s-baseos-mirror': Curl error (37): Could not read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial]

which I have no idea why. It's using new enough cosa with coreos/coreos-assembler#4054. And using that new cosa locally I can't reproduce this.

jcapiitao

See my inline comment, sorry I missed it in my previous review.
Otherwise LGTM 👍

c9s.repo

jcapiitao · 2025-04-03T12:25:00Z

Hmm, the builder is choking on the heredocs. I think possibly the Dockerfile parser there (which happens before it's handed off to buildah) is getting tripped up on something. :( I'll dig into this a bit, but if it can't be easily worked around, I think I'll just add a commit for now that moves the heredocs to shell scripts for now.

c9s builds are failing on
 error: Installing packages: Updating rpm-md repo 'c9s-baseos-mirror': Failed to download gpg key for repo 'c9s-baseos-mirror': Curl error (37): Could not read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial] 
which I have no idea why. It's using new enough cosa with coreos/coreos-assembler#4054. And using that new cosa locally I can't reproduce this.

It's because the latest run CI job pull cosa commit b45a4066b16a2332517f659111d4f474372f77d5 [1]

+ [[ -d /cosa ]]
+ jq .
{
  "date": "2025-04-02T17:18:40Z",
  "git": {
    "commit": "b45a4066b16a2332517f659111d4f474372f77d5",
    "origin": "https://github.com/coreos/coreos-assembler.git",
    "branch": "HEAD",
    "dirty": "false"
  },

and not the latest one with the changes we want coreos/coreos-assembler@ae0e86a

Maybe some cache issue or race condition issue ? I'm not yet familiar with the CI workflow to be sure, maybe a simple /retest should work

[1] https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_os/1780/pull-ci-openshift-os-master-scos-9-build-test-qemu/1907634163477909504/build-log.txt

jcapiitao · 2025-04-03T12:28:14Z

Hmm, the builder is choking on the heredocs. I think possibly the Dockerfile parser there (which happens before it's handed off to buildah) is getting tripped up on something. :( I'll dig into this a bit, but if it can't be easily worked around, I think I'll just add a commit for now that moves the heredocs to shell scripts for now.
c9s builds are failing on
 error: Installing packages: Updating rpm-md repo 'c9s-baseos-mirror': Failed to download gpg key for repo 'c9s-baseos-mirror': Curl error (37): Could not read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial] 
which I have no idea why. It's using new enough cosa with coreos/coreos-assembler#4054. And using that new cosa locally I can't reproduce this.
It's because the latest run CI job pull cosa commit b45a4066b16a2332517f659111d4f474372f77d5 [1]
+ [[ -d /cosa ]]
+ jq .
{
  "date": "2025-04-02T17:18:40Z",
  "git": {
    "commit": "b45a4066b16a2332517f659111d4f474372f77d5",
    "origin": "https://github.com/coreos/coreos-assembler.git",
    "branch": "HEAD",
    "dirty": "false"
  },
and not the latest one with the changes we want coreos/coreos-assembler@ae0e86a

Maybe some cache issue or race condition issue ? I'm not yet familiar with the CI workflow to be sure, maybe a simple /retest should work

[1] https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_os/1780/pull-ci-openshift-os-master-scos-9-build-test-qemu/1907634163477909504/build-log.txt

Hmm forget about it, wrong assumption, the codebase up to b45a4 contains the changes we want from https://github.com/coreos/coreos-assembler/pull/4054/commits
That's odd it does not work on CI

jlebon · 2025-04-03T12:42:25Z

OK, coreos/coreos-assembler#4059 should fix the GPG issue!

jbtrystram · 2025-04-03T14:00:49Z

/retest

Instead of having to explicitly pass in the `VARIANT`, we can autodetect it based on the node image we're building `FROM`.

`apply-manifest` was essentially folded back into rpm-ostree in: coreos/rpm-ostree#5274 The only thing we need to keep is the workaround for cri-o's `/var/opt`, which... we should just try to get fixed.

Previously, when building the layered node image, we were relying on the default repo enablement settings. This though is at the source of a lot of complexity because then we need to make sure that we only inject just the repos that we need with the right enablement. See e.g. the complex logic in `get-ocp-repo.sh`. Let's instead match the semantics already in use by the base compose and extensions builds, both of which explicitly list the repos to enable. This means that we can be a lot less careful in what repo definitions we inject into the build environment, knowing only the necessary ones will be enabled. This is pretty easy to do now that (1) rpm-ostree suppports inlined treefiles, and (2) `treefile-apply` supports a `--var` option to define variables at invocation time.

Now that (1) we've reworked the layered node image build to only enable the repos it needs, and (2) we've simplified the CentOS Stream GPG keys, we can delete all of the complex logic in this repo. It basically just boils down to curl'ing down all the repo files we may need to build the various artifacts that use this script.

We only want certain packages to come from the 4.19 plashet. And we can't just rely on NVRs because the plashet may sometimes win. Long-term we should sever that dependence on ART packages, but for now, let's add a hack to essentially generate a repo on the fly from the 4.19 repo with the filters we need. The advantage of doing it this way instead of e.g. in the `get-ocp-repo.sh` script is that this applies both in CI and locally.

The OCP builder API path isn't parsing the heredoc correctly for some reason: error: build error: EOF: unterminated heredoc This will be fixed by openshift/builder#469. Anyway, just work around this for now by moving all the logic to scripts. It does make the Containerfiles cleaner at least now that it has gotten so larger and we get syntax highlighting, ShellCheck, etc... so probably for the best.

Before we inherited this from the ocp-rhel-9.6 manifest. But now that we're inheriting from the rhel-9.6 manifest, that repo isn't enabled by default there since it's not strictly needed (because we don't ship openvswitch in the base). So we need to enable it here ourselves.

jlebon · 2025-04-04T17:22:31Z

OK, the OKD node image build is failing on

Error: Unknown repo: 'c9s-baseos'

which I know why. Got a fix for that, but let's see if CI for the other tests pass. If they do, then let's get this in, and I'll add my fix to #1498 instead when I rebase it.

dustymabe · 2025-04-04T17:25:51Z

/lgtm

openshift-ci · 2025-04-04T17:26:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dustymabe, jbtrystram, jlebon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dustymabe,jbtrystram,jlebon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2025-04-04T17:28:31Z

/retest-required

Remaining retests: 0 against base HEAD 48a1891 and 2 for PR HEAD 7997d1a in total

jlebon · 2025-04-04T18:49:11Z

This will known fail. It'll be fixed in #1498.

/override ci/prow/okd-scos-images

OK, as for the RHCOS failure, it seems to have just... timed out?

 --- FAIL: ext.config.shared.rpm-ostree.kernel-replace (1206.73s)
        harness.go:106: TIMEOUT[20m0s]: ssh: sudo /usr/local/bin/kolet run-test-unit kola-runext.service
        harness.go:106: TIMEOUT[20m0s]: ssh: journalctl -t kola-runext-kernel-replace

Nothing in particular in the journal logs. Last operation was

Apr  4 18:20:36.204321 rpm-ostreed.service[1391]: Fetching ostree-unverified-image:oci-archive:/var/tmp/coreos-derived.ociarchive

So possibly a genuine timeout because of e.g. slow I/O.

Anyway, don't think it needs to block this. It'll rerun in #1498.

openshift-ci · 2025-04-04T18:50:03Z

@jlebon: Overrode contexts on behalf of jlebon: ci/prow/okd-scos-images

In response to this:

This will known fail. It'll be fixed in #1498.

/override ci/prow/okd-scos-images

OK, as for the RHCOS failure, it seems to have just... timed out?
--- FAIL: ext.config.shared.rpm-ostree.kernel-replace (1206.73s)
       harness.go:106: TIMEOUT[20m0s]: ssh: sudo /usr/local/bin/kolet run-test-unit kola-runext.service
       harness.go:106: TIMEOUT[20m0s]: ssh: journalctl -t kola-runext-kernel-replace 
Nothing in particular in the journal logs. Last operation was
Apr  4 18:20:36.204321 rpm-ostreed.service[1391]: Fetching ostree-unverified-image:oci-archive:/var/tmp/coreos-derived.ociarchive
So possibly a genuine timeout because of e.g. slow I/O.

Anyway, don't think it needs to block this. It'll rerun in #1498.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci · 2025-04-04T18:50:05Z

@jlebon: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/okd-scos-e2e-aws-ovn	`7997d1a`	link	false	`/test okd-scos-e2e-aws-ovn`
ci/prow/rhcos-9-build-test-qemu	`7997d1a`	link	true	`/test rhcos-9-build-test-qemu`
ci/prow/e2e-aws	`7997d1a`	link	false	`/test e2e-aws`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

We now need to support both EL9 and EL10. Using the conditionnal includes for treefiles added in [1], update `osversion` to contain the variant (centos/rhel) and the major version. This allows the layered build to source `/etc/release` and include the correct repos. Update denylist entries to matcht that. [1] openshift#1780

jlebon changed the title ~~Nuke okd-c9s variant~~ NO-JIRA: Nuke okd-c9s variant Mar 28, 2025

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 28, 2025

openshift-ci bot requested review from c4rt0 and marmijo March 28, 2025 16:03

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 28, 2025

jlebon mentioned this pull request Mar 28, 2025

COS-2902: Add C10s variant #1498

Merged

jlebon changed the title ~~NO-JIRA: Nuke okd-c9s variant~~ NO-JIRA: Switch layered build to treefile-apply, drain get-ocp-repo.sh Mar 31, 2025

jlebon mentioned this pull request Mar 31, 2025

Add CentOS Stream keys, add cosa podman-build coreos/coreos-assembler#4054

Merged

jbtrystram approved these changes Apr 1, 2025

View reviewed changes

common.yaml Outdated Show resolved Hide resolved

extensions-okd-c9s.yaml Show resolved Hide resolved

Containerfile Outdated Show resolved Hide resolved

jcapiitao reviewed Apr 1, 2025

View reviewed changes

Prashanth684 reviewed Apr 1, 2025

View reviewed changes

ci/get-ocp-repo.sh Show resolved Hide resolved

jlebon force-pushed the pr/nuke-okd-c9s branch from b8fbcd9 to ba35f21 Compare April 2, 2025 19:10

dustymabe reviewed Apr 2, 2025

View reviewed changes

jlebon force-pushed the pr/nuke-okd-c9s branch from ba35f21 to 7bbe649 Compare April 3, 2025 03:19

jcapiitao reviewed Apr 3, 2025

View reviewed changes

c9s.repo Outdated Show resolved Hide resolved

jlebon mentioned this pull request Apr 3, 2025

cmdlib.sh: copy all GPG keys into supermin VM coreos/coreos-assembler#4059

Merged

jlebon added 3 commits April 4, 2025 09:18

extensions/Dockerfile: drop VARIANT arg

a2c632d

Instead of having to explicitly pass in the `VARIANT`, we can autodetect it based on the node image we're building `FROM`.

Containerfile: move to treefile-apply, drop apply-manifest

6596f19

`apply-manifest` was essentially folded back into rpm-ostree in: coreos/rpm-ostree#5274 The only thing we need to keep is the workaround for cri-o's `/var/opt`, which... we should just try to get fixed.

jlebon force-pushed the pr/nuke-okd-c9s branch 3 times, most recently from 100d9f8 to ebd3ff7 Compare April 4, 2025 14:18

jlebon added 2 commits April 4, 2025 10:50

jlebon force-pushed the pr/nuke-okd-c9s branch from ebd3ff7 to ac98e93 Compare April 4, 2025 14:51

jlebon added 2 commits April 4, 2025 11:26

jlebon force-pushed the pr/nuke-okd-c9s branch from ac98e93 to 7997d1a Compare April 4, 2025 15:35

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 4, 2025

jlebon merged commit 902882d into openshift:master Apr 4, 2025
10 of 14 checks passed

jlebon deleted the pr/nuke-okd-c9s branch April 4, 2025 18:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NO-JIRA: Switch layered build to `treefile-apply`, drain `get-ocp-repo.sh` #1780

NO-JIRA: Switch layered build to `treefile-apply`, drain `get-ocp-repo.sh` #1780

jlebon commented Mar 28, 2025 •

edited

Loading

openshift-ci-robot commented Mar 28, 2025

jlebon commented Mar 28, 2025

jlebon commented Mar 31, 2025 •

edited

Loading

jlebon commented Mar 31, 2025

jbtrystram left a comment

jcapiitao Apr 1, 2025

jlebon Apr 1, 2025

Prashanth684 Apr 1, 2025

jcapiitao commented Apr 1, 2025

jlebon commented Apr 1, 2025

jlebon commented Apr 2, 2025

jlebon commented Apr 2, 2025

jlebon commented Apr 2, 2025

dustymabe left a comment

dustymabe Apr 2, 2025

jlebon Apr 2, 2025

dustymabe Apr 2, 2025

jlebon Apr 2, 2025

jlebon commented Apr 3, 2025

jcapiitao left a comment

jcapiitao commented Apr 3, 2025

jcapiitao commented Apr 3, 2025

jlebon commented Apr 3, 2025

jbtrystram commented Apr 3, 2025

jlebon commented Apr 4, 2025

dustymabe commented Apr 4, 2025

openshift-ci bot commented Apr 4, 2025

openshift-ci-robot commented Apr 4, 2025

jlebon commented Apr 4, 2025

openshift-ci bot commented Apr 4, 2025

openshift-ci bot commented Apr 4, 2025

		# buildah doesn't seem to support heredoc output
		# redirection like buildkit so do it manually here

		MANIFEST="manifest-rhel-9.6.yaml"
		EXTENSIONS="extensions-ocp-rhel-9.6.yaml"

NO-JIRA: Switch layered build to treefile-apply, drain get-ocp-repo.sh #1780

NO-JIRA: Switch layered build to treefile-apply, drain get-ocp-repo.sh #1780

Conversation

jlebon commented Mar 28, 2025 • edited Loading

openshift-ci-robot commented Mar 28, 2025

jlebon commented Mar 28, 2025

jlebon commented Mar 31, 2025 • edited Loading

jlebon commented Mar 31, 2025

jbtrystram left a comment

Choose a reason for hiding this comment

jcapiitao Apr 1, 2025

Choose a reason for hiding this comment

jlebon Apr 1, 2025

Choose a reason for hiding this comment

Prashanth684 Apr 1, 2025

Choose a reason for hiding this comment

jcapiitao commented Apr 1, 2025

jlebon commented Apr 1, 2025

jlebon commented Apr 2, 2025

jlebon commented Apr 2, 2025

jlebon commented Apr 2, 2025

dustymabe left a comment

Choose a reason for hiding this comment

dustymabe Apr 2, 2025

Choose a reason for hiding this comment

jlebon Apr 2, 2025

Choose a reason for hiding this comment

dustymabe Apr 2, 2025

Choose a reason for hiding this comment

jlebon Apr 2, 2025

Choose a reason for hiding this comment

jlebon commented Apr 3, 2025

jcapiitao left a comment

Choose a reason for hiding this comment

jcapiitao commented Apr 3, 2025

jcapiitao commented Apr 3, 2025

jlebon commented Apr 3, 2025

jbtrystram commented Apr 3, 2025

jlebon commented Apr 4, 2025

dustymabe commented Apr 4, 2025

openshift-ci bot commented Apr 4, 2025

openshift-ci-robot commented Apr 4, 2025

jlebon commented Apr 4, 2025

openshift-ci bot commented Apr 4, 2025

openshift-ci bot commented Apr 4, 2025

NO-JIRA: Switch layered build to `treefile-apply`, drain `get-ocp-repo.sh` #1780

NO-JIRA: Switch layered build to `treefile-apply`, drain `get-ocp-repo.sh` #1780

jlebon commented Mar 28, 2025 •

edited

Loading

jlebon commented Mar 31, 2025 •

edited

Loading