feat: move patch apply logic to patchable #1032

dervoeti · 2025-03-18T11:53:00Z

Description

This is a draft PR to solve #677. I replaced the apply_patches.sh script with Patchable for Zookeeper and Druid.

We decided in https://github.com/stackabletech/decisions/issues/44 that all sources should be mirrored on Github (to be implemented by #1011).

My idea was that we can just reuse the existing checkout logic, since it does exactly what we want: checkout a repo at a specific commit and apply a bunch of .patch files on top of it. In this PR, the original Apache repositories are used, those would need to be replaced by our mirror repos.

I added an optional flag --images_repo_root to manually specify a custom root for the images repo instead of relying on the auto-detection that Patchable currently does. We could also just use the auto-detection, but then we'd probably need to run git init && touch .patchable before running the patchable checkout command inside the container, which is a bit awkward. The extra flag isn't super pretty either though and I couldn't get around using PathBuf in the implementation, hence I made this a draft PR to serve as a PoC. This is just one way we could replace the patch scripts with Patchable. Maybe this can be solved in a more elegant way, I didn't find one yet though.

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

- [ ] Changes are OpenShift compatible
- [ ] All added packages (via microdnf or otherwise) have a comment on why they are added
- [ ] Things not downloaded from Red Hat repositories should be mirrored in the Stackable repository and downloaded from there
- [ ] All packages should have (if available) signatures/hashes verified
- [ ] Add an entry to the CHANGELOG.md file
- [ ] Integration tests ran successfully

TIP: Running integration tests with a new product image

The image can be built and uploaded to the kind cluster with the following commands:

bake --product <product> --image-version <stackable-image-version>
kind load docker-image <image-tagged-with-the-major-version> --name=<name-of-your-test-cluster>

See the output of bake to retrieve the image tag for <image-tagged-with-the-major-version>.

…into feat/move-patch-apply-logic-to-patchable

rust/patchable/src/main.rs

stackable-base/Dockerfile

rust/patchable/src/main.rs

druid/Dockerfile

zookeeper/Dockerfile

Co-authored-by: Natalie Klestrup Röijezon <[email protected]>

dervoeti · 2025-03-21T14:07:45Z

@nightkr It's ready for another round of feedback 🙂 Once it's in a good shape I'll start adapting the patch processes of the other products.

nightkr

Largely LGTM

zookeeper/Dockerfile

…into feat/move-patch-apply-logic-to-patchable

dervoeti · 2025-03-31T10:51:36Z

@nightkr I migrated all Dockerfiles now and tested all builds using the latest versions of the product, worked fine.

…into feat/move-patch-apply-logic-to-patchable

druid/Dockerfile

hbase/Dockerfile

hive/Dockerfile

superset/Dockerfile

trino/Dockerfile

…der stage to final stage

…en commit plugin bug

nightkr

LGTM if it builds for you.

* wip * Update druid/Dockerfile Co-authored-by: Natalie Klestrup Röijezon <[email protected]> * fix: remove unnecessary check / shadow repo root var * fix: druid src path * fix: druid src path * feat: introduce stackable-devel image * fix: use PathBuf in ProductVersionContext * chore: align zookeeper patch directory structure * fix: stackable-devel dnf and shell config * chore: switch patch process in other products * fix: hive build * fix: trino build * fix: spark build * chore: make hadolint happy * fix: remove hbase intermediate sources / remove unnecessary and operator * fix: permissions in patchable build process * chore: remove unnecessary curl command in build process * chore: move adding of JMX config and start-metastore script from builder stage to final stage * chore: remove git repo in trino and hbase-operator-tools to avoid maven commit plugin bug --------- Co-authored-by: Natalie Klestrup Röijezon <[email protected]>

* build nifi-opa-plugin from source with workaround * add changelog entry * add patch * build nifi-opa-plugin from branch feat/reworked-opa-response * Update nifi/Dockerfile Co-authored-by: Malte Sander <[email protected]> * chore(⏲): Use fast download mirrors (#1061) * chore: Use fast download mirrors * chore: Show download progress bar * chore: Improve gpg key warning message * chore: Show upload progress bar Thanks to <https://bashupload.com/how_to_upload_progress_curl> * Apply suggestions from code review Co-authored-by: Techassi <[email protected]> * chore: Make output more consistent --------- Co-authored-by: Techassi <[email protected]> * chore(⏲): Add missing tool update tasks to templates (#1062) * chore(issue_templates): Use YY.M.X placeholders * chore(issue_templates): Remove tasklists :sob: * chore(issue_templates): Add missing tool update tasks for cyclonedx and auditable * feat: move patch apply logic to patchable (#1032) * wip * Update druid/Dockerfile Co-authored-by: Natalie Klestrup Röijezon <[email protected]> * fix: remove unnecessary check / shadow repo root var * fix: druid src path * fix: druid src path * feat: introduce stackable-devel image * fix: use PathBuf in ProductVersionContext * chore: align zookeeper patch directory structure * fix: stackable-devel dnf and shell config * chore: switch patch process in other products * fix: hive build * fix: trino build * fix: spark build * chore: make hadolint happy * fix: remove hbase intermediate sources / remove unnecessary and operator * fix: permissions in patchable build process * chore: remove unnecessary curl command in build process * chore: move adding of JMX config and start-metastore script from builder stage to final stage * chore: remove git repo in trino and hbase-operator-tools to avoid maven commit plugin bug --------- Co-authored-by: Natalie Klestrup Röijezon <[email protected]> * fix: add missing patchable config for Kafka 3.8.0 (#1065) * fix(pr-template): Fix the rendering of the list (#1073) * feat: make image namespace a workflow input (default sdp) (#1072) * feat: make image namespace a workflow input (default sdp) * feat: add registry namespace input to build workflows and remove default * ci(mirror): Include image name and version in run-name (#1089) * ci(mirror): Include image and version in run-name * chore: Use colon * fix: spark connect client Harbor credentials (#1088) * fix: spark connect client Harbor credentials (#1092) * feat: connect client image includes JupyterLab (#1071) * feat: install demo dependencies * spark-connect-client is now built directly off of spark-k8s * run pre-commit hooks * fix shellcheck sc2102 * chore: Bump tools ahead of 25.7.0 (#1090) * chore(jmx_exporter): Bump products to use 1.2.0 * chore(tools): Bump kubectl (1.33.0 and yq (4.45.2) * chore(cyclonedx-bom): Bump to 6.0.0 * chore: Update changelog * chore: Changelog formatting * chore(java): Add JDK 24 (#1097) * chore(java): Add JDK 24 * chore: Update changelog * chore(vector): Bump to 0.46.1 (#1098) * chore(vector): Bump to 0.46.1 * chore(vector): Bump products to use 0.46.1 * chore(zookeeper): Remove 3.9.2 (#1093) * chore: Changelog formatting * chore(zookeeper): Remove 2.9.2 * chore: Update changelog * chore: Update changelog * chore(ubi-rust-builders): Update container images ahead of Stackable Release 25.7.0 (#1091) * chore(ubi-rust-builders): Remove ubi8-rust-builder * chore(ubi-rust-builders): Update base image and use protoc 30.2 * Apply suggestions from code review Co-authored-by: Techassi <[email protected]> Co-authored-by: Sebastian Bernauer <[email protected]> --------- Co-authored-by: Techassi <[email protected]> Co-authored-by: Sebastian Bernauer <[email protected]> * chore(opa): Update versions ahead of 25.7.0 (#1103) * chore(opa): Remove 0.67.1 * chore(opa): Remove legacy bundle-builder * chore(opa): Bump ubi9 base image * chore(opa): Add 1.4.2 * fix(opa): Manually install Go version NOTE: The dnf package was 1.23.6 and OPA required 1.23.8 NOTE: I tried making the version paramaterized, but bake wouldn't allow it (it worked fine with `docker build` and `docker build xbuild`). * ci(mirror): Add golang * chore(opa): Use mirrored golang image * chore(opa): Parameterise golang container version * chore: Update changelog * fix: Bump vector to 0.46.1 for java-base:24 (#1104) * chore(superset): Add 4.1.2 (#1102) * chore(superset): Add 4.1.2 * fix(superset): Remove invalid parts of the file * chore: Update changelog * feat(nifi): Add nifi-iceberg-bundle (#1060) * feat(nifi): Add nifi-iceberg-bundle * changelog * Add SBOM to final image * Use version 0.0.1 * hadolint * Bump to 0.0.2 * Bump to 0.0.3 * fix(nifi): Delete correct intermediate folder (#1106) * fix(nifi): Delete correct intermediate folder * changelog * fix Dockerfile * build nifi opa plugin from tag * Update CHANGELOG.md Co-authored-by: Malte Sander <[email protected]> * fix hadolint * Update nifi/Dockerfile Co-authored-by: Malte Sander <[email protected]> --------- Co-authored-by: Malte Sander <[email protected]> Co-authored-by: Nick <[email protected]> Co-authored-by: Techassi <[email protected]> Co-authored-by: Lukas Krug <[email protected]> Co-authored-by: Natalie Klestrup Röijezon <[email protected]> Co-authored-by: Razvan-Daniel Mihai <[email protected]> Co-authored-by: Sebastian Bernauer <[email protected]>

dervoeti added 2 commits March 17, 2025 15:03

wip

a2509c3

Merge branch 'main' of https://github.com/stackabletech/docker-images …

ee787fe

…into feat/move-patch-apply-logic-to-patchable

dervoeti requested a review from nightkr March 18, 2025 11:53

nightkr reviewed Mar 18, 2025

View reviewed changes

dervoeti and others added 4 commits March 20, 2025 13:33

Update druid/Dockerfile

48e0345

Co-authored-by: Natalie Klestrup Röijezon <[email protected]>

fix: remove unnecessary check / shadow repo root var

21fbd88

fix: druid src path

5fa3620

fix: druid src path

75f8990

dervoeti force-pushed the feat/move-patch-apply-logic-to-patchable branch from 2d20b52 to 1efa478 Compare March 21, 2025 12:34

feat: introduce stackable-devel image

bdc6dfd

dervoeti force-pushed the feat/move-patch-apply-logic-to-patchable branch from 1efa478 to bdc6dfd Compare March 21, 2025 12:35

dervoeti added 3 commits March 21, 2025 13:41

fix: use PathBuf in ProductVersionContext

597ec11

chore: align zookeeper patch directory structure

0293728

fix: stackable-devel dnf and shell config

455e2fa

dervoeti requested a review from nightkr March 21, 2025 14:06

nightkr reviewed Mar 21, 2025

View reviewed changes

zookeeper/Dockerfile Outdated Show resolved Hide resolved

dervoeti added 7 commits March 24, 2025 18:42

chore: switch patch process in other products

70360f2

fix: hive build

1b02ecf

Merge branch 'main' of https://github.com/stackabletech/docker-images …

abd2a55

…into feat/move-patch-apply-logic-to-patchable

fix: trino build

de74bde

fix: spark build

e17b727

Merge branch 'main' of https://github.com/stackabletech/docker-images …

df556a5

…into feat/move-patch-apply-logic-to-patchable

chore: make hadolint happy

3019af4

dervoeti marked this pull request as ready for review March 31, 2025 10:49

dervoeti requested a review from nightkr March 31, 2025 10:49

dervoeti added 3 commits April 4, 2025 12:21

fix: remove hbase intermediate sources / remove unnecessary and operator

65a8ef9

chore: remove product sources in subshell

0a92081

fix: permissions in patchable build process

eb83aba

dervoeti added 3 commits April 9, 2025 15:06

Merge branch 'main' of https://github.com/stackabletech/docker-images …

237d275

…into feat/move-patch-apply-logic-to-patchable

Merge branch 'main' of https://github.com/stackabletech/docker-images …

92327a9

…into feat/move-patch-apply-logic-to-patchable

Merge branch 'main' of https://github.com/stackabletech/docker-images …

245f135

…into feat/move-patch-apply-logic-to-patchable

nightkr reviewed Apr 14, 2025

View reviewed changes

dervoeti added 3 commits April 14, 2025 13:58

chore: remove unnecessary curl command in build process

d11d061

chore: move adding of JMX config and start-metastore script from buil…

8dbf055

…der stage to final stage

chore: remove git repo in trino and hbase-operator-tools to avoid mav…

92b43bf

…en commit plugin bug

dervoeti requested a review from nightkr April 14, 2025 15:35

nightkr approved these changes Apr 24, 2025

View reviewed changes

dervoeti added this pull request to the merge queue Apr 24, 2025

Merged via the queue into main with commit 5bf5149 Apr 24, 2025
3 checks passed

dervoeti deleted the feat/move-patch-apply-logic-to-patchable branch April 24, 2025 13:59

This was referenced Apr 24, 2025

Unify patch system (in Dockerfiles) #677

Closed

fix: add missing patchable config for Kafka 3.8.0 #1065

Merged

Techassi mentioned this pull request Jun 10, 2025

chore: Update issue templates #1148

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat: move patch apply logic to patchable #1032

feat: move patch apply logic to patchable #1032

Uh oh!

dervoeti commented Mar 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dervoeti commented Mar 21, 2025

Uh oh!

nightkr left a comment

Uh oh!

Uh oh!

dervoeti commented Mar 31, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nightkr left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

feat: move patch apply logic to patchable #1032

feat: move patch apply logic to patchable #1032

Uh oh!

Conversation

dervoeti commented Mar 18, 2025

Description

Definition of Done Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dervoeti commented Mar 21, 2025

Uh oh!

nightkr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dervoeti commented Mar 31, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nightkr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!