ci: Introduce pre-commit hooks (#765)

* add pre-commit config

* ignore yaml list indentation style

* ignore long lines in python (because we want inline comments for version info)

* fix lint issues

fix python lint

* move actionlint.yaml to the right place

* add shellcheck pre-commit hook

* replace flake8 with ruff

* remove flake8 config

* ci: add pre-commit checks on pull request

* ci: disable pr-commit action until after the workflow is registered

* ci: rename reviewdog workflow to be grouped with other PR workflows

* fix lint issues

Author: NickLarsenNZ

.github/pull_request_template.md

@@ -2,12 +2,11 @@
 
 *Please add a description here.*
 
-
 ## Definition of Done Checklist
 
 - Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
 - Please make sure all these things are done and tick the boxes
- 
+
 ```[tasklist]
 - [ ] Changes are OpenShift compatible
 - [ ] All added packages (via microdnf or otherwise) have a comment on why they are added

.github/workflows/pr_pre-commit.yaml

@@ -0,0 +1,16 @@
+---
+name: pre-commit
+
+on:
+  pull_request:
+
+jobs:
+  pre-commit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
+      - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0
+        with:
+          python-version: '3.12'
+#       NOTE (@NickLarsenNZ): Disabled until the workflow is registered in Github Actions (after merge to main)
+#       - uses: pre-commit/action@2c7b3805fd2a0fd8c1884dcaebf91fc102a13ecd # v3.0.1

.github/workflows/reviewdog.yaml renamed to .github/workflows/pr_reviewdog.yaml

@@ -1,4 +1,3 @@
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]
-

.gitignore

@@ -18,3 +18,7 @@ MD013:
 MD024:
   # Only check sibling headings
   siblings_only: true
+
+# MD033/no-inline-html Inline HTML
+MD033:
+  allowed_elements: [details, summary, img]

.markdownlint.yaml

@@ -0,0 +1,52 @@
+---
+# See https://pre-commit.com for more information
+fail_fast: false
+
+exclude: \.patch$
+
+# See https://pre-commit.com/hooks.html for more hooks
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.2.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: detect-aws-credentials
+        args: ["--allow-missing-credentials"]
+      - id: detect-private-key
+
+  - repo: https://github.com/adrienverge/yamllint
+    rev: v1.26.3
+    hooks:
+      - id: yamllint
+
+  - repo: https://github.com/igorshubovych/markdownlint-cli
+    rev: v0.31.1
+    hooks:
+      - id: markdownlint
+
+  - repo: https://github.com/koalaman/shellcheck-precommit
+    rev: v0.10.0
+    hooks:
+      - id: shellcheck
+        args: ["--severity=info"]
+
+  # WARNING (@NickLarsenNZ): Nix users need to install ruff first.
+  # If you do not, you will need to delete the cached ruff binary shown in the
+  # error message
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.5.1
+    hooks:
+      # Run the linter.
+      - id: ruff
+      # Run the formatter.
+      - id: ruff-format
+
+  - repo: local
+    hooks:
+      - id: update-readme-badges
+        name: update-readme-badges
+        language: system
+        entry: .scripts/update_readme_badges.sh
+        stages: [commit, merge-commit, manual]
+        pass_filenames: false

.pre-commit-config.yaml

@@ -22,9 +22,9 @@
 ]
 
 for product in conf.products:
-    product_name = product['name']
+    product_name = product["name"]
     if product_name not in PRODUCTS:
         continue
 
-    for version in product['versions']:
+    for version in product["versions"]:
         print(f"{product_name}#{version['product']}")

.scripts/enumerate-product-versions.py

@@ -38,7 +38,7 @@ for arch in "${ARCHITECTURES[@]}"; do
   # tini does not currently publish signatures or SBOMs
   # renaming binary because original file name has no version
   echo "Downloading TINI"
-  curl --fail -Ls -o "tini-$VERSION-$arch" "https://github.com/krallin/tini/releases/download/v$VERSION/tini-$arch" 
+  curl --fail -Ls -o "tini-$VERSION-$arch" "https://github.com/krallin/tini/releases/download/v$VERSION/tini-$arch"
 
   echo "Uploading to Nexus"
   curl --fail -u "$NEXUS_USER:$NEXUS_PASSWORD" --upload-file "tini-$VERSION-$arch" 'https://repo.stackable.tech/repository/packages/tini/'

.scripts/upload_new_tini_version.sh

@@ -1,3 +1,5 @@
+# shellcheck disable=all
+#
 # Don't do this on your machine, as we need to sign the mirrored image, which should be done by a GitHub CI.
 # Use the action "Mirror container image" in the docker-images repo for that.
 # You can e.g. pull the image from

.scripts/upload_sig_storage_driver.sh

@@ -10,3 +10,4 @@ rules:
     check-keys: false
   comments:
     min-spaces-from-content: 1  # Needed due to https://github.com/adrienverge/yamllint/issues/443
+  indentation: disable

.yamllint.yaml

@@ -2,7 +2,7 @@
 
 This repository contains Dockerfiles and scripts to build base images for use within Stackable.
 
-<!-- start:badges: autogenerated by ./update_readme_badges.sh -->
+<!-- start:badges: autogenerated by .scripts/update_readme_badges.sh -->
 | | | | |
 | -: | -: | -: | -: |
 | [![Build Airflow]][dev_airflow.yaml] | [![Build Druid]][dev_druid.yaml] | [![Build Hadoop]][dev_hadoop.yaml] | [![Build HBase]][dev_hbase.yaml] |
@@ -95,7 +95,7 @@ USER 1000:1000
 ENTRYPOINT ["/stackable-zookeeper-operator"]
 ```
 
-<!-- start:links: autogenerated by ./update_readme_badges.sh -->
+<!-- start:links: autogenerated by .scripts/update_readme_badges.sh -->
 [Build Airflow]: https://github.com/stackabletech/docker-images/actions/workflows/dev_airflow.yaml/badge.svg
 [dev_airflow.yaml]: https://github.com/stackabletech/docker-images/actions/workflows/dev_airflow.yaml
 [Build Druid]: https://github.com/stackabletech/docker-images/actions/workflows/dev_druid.yaml/badge.svg

README.md

@@ -311,7 +311,7 @@ if [[ -n "${_PIP_ADDITIONAL_REQUIREMENTS=}" ]] ; then
     >&2 echo "         the container starts, so it is onlny useful for testing and trying out"
     >&2 echo "         of adding dependencies."
     >&2 echo
-    pip install --root-user-action ignore --no-cache-dir ${_PIP_ADDITIONAL_REQUIREMENTS}
+    pip install --root-user-action ignore --no-cache-dir "${_PIP_ADDITIONAL_REQUIREMENTS}"
 fi
 
 

airflow/stackable/utils/entrypoint.sh

@@ -2,7 +2,7 @@
 # This wrapper script allows commands that are embedded in environment variables
 # to be invoked to carry out actions when the main container is complete.
 # N.B. for airflow services (scheduler, webserver, worker) this will not be relevant
-# as the service has to be terminated, but for pods spawned and terminated by the 
+# as the service has to be terminated, but for pods spawned and terminated by the
 # kubernetes executor this allows us to stop other containers, such as vector.
 eval "$_STACKABLE_PRE_HOOK"
 

airflow/stackable/utils/run-airflow.sh

@@ -198,4 +198,4 @@
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.

druid/licenses/DRUID_LICENSE

@@ -198,4 +198,4 @@
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.

druid/licenses/PROMETHEUS_EMITTER_LICENSE

@@ -40,4 +40,4 @@ e) to display the Original Work publicly.
 
 15) Right to Use. You may use the Original Work in all ways not otherwise restricted or conditioned by this License or by law, and Licensor promises not to interfere with or be responsible for such uses by You.
 
-16) Modification of This License. This License is Copyright © 2005 Lawrence Rosen. Permission is granted to copy, distribute, or communicate this License without modification. Nothing in this License permits You to modify this License as applied to the Original Work or to Derivative Works. However, You may modify the text of this License and copy, distribute or communicate your modified version (the "Modified License") and apply it to other original works of authorship subject to the following conditions: (i) You may not indicate in any way that your Modified License is the "Open Software License" or "OSL" and you may not use those names in the name of your Modified License; (ii) You must replace the notice specified in the first paragraph above with the notice "Licensed under <insert your license name here>" or with a notice of your own that is not confusingly similar to the notice in this License; and (iii) You may not claim that your original works are open source software unless your Modified License has been approved by Open Source Initiative (OSI) and You comply with its license review and certification process.
+16) Modification of This License. This License is Copyright © 2005 Lawrence Rosen. Permission is granted to copy, distribute, or communicate this License without modification. Nothing in this License permits You to modify this License as applied to the Original Work or to Derivative Works. However, You may modify the text of this License and copy, distribute or communicate your modified version (the "Modified License") and apply it to other original works of authorship subject to the following conditions: (i) You may not indicate in any way that your Modified License is the "Open Software License" or "OSL" and you may not use those names in the name of your Modified License; (ii) You must replace the notice specified in the first paragraph above with the notice "Licensed under <insert your license name here>" or with a notice of your own that is not confusingly similar to the notice in this License; and (iii) You may not claim that your original works are open source software unless your Modified License has been approved by Open Source Initiative (OSI) and You comply with its license review and certification process.

druid/licenses/STACKABLE_LICENSE

@@ -8,11 +8,13 @@ fi
 
 WHATAMI="$1"
 CONFDIR="$(cd "$2" && pwd)"
+# shellcheck disable=SC1091 # java-util is not available at this scripts source
 JAVA_BIN="$(source ./bin/java-util && get_java_bin_dir)"
 if [ -z "$JAVA_BIN" ]; then
   >&2 echo "Could not find java - please run ./bin/verify-java to confirm it is installed."
   exit 1
 fi
-exec "$JAVA_BIN"/java `cat "$CONFDIR"/jvm.config | xargs` \
+# shellcheck disable=SC2046 # unsure how this `cat | xargs` works, and this script might rely on discouraged behaviour
+exec "$JAVA_BIN"/java $(cat "$CONFDIR"/jvm.config | xargs) \
   -cp "$CONFDIR":"./lib/*" \
-  org.apache.druid.cli.Main server "$WHATAMI"
+  org.apache.druid.cli.Main server "$WHATAMI"

druid/stackable/bin/run-druid

@@ -198,4 +198,4 @@
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.

hadoop/licenses/HADOOP_LICENSE

@@ -198,4 +198,4 @@
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.

hadoop/licenses/JMX_EXPORTER_LICENSE

@@ -40,4 +40,4 @@ e) to display the Original Work publicly.
 
 15) Right to Use. You may use the Original Work in all ways not otherwise restricted or conditioned by this License or by law, and Licensor promises not to interfere with or be responsible for such uses by You.
 
-16) Modification of This License. This License is Copyright © 2005 Lawrence Rosen. Permission is granted to copy, distribute, or communicate this License without modification. Nothing in this License permits You to modify this License as applied to the Original Work or to Derivative Works. However, You may modify the text of this License and copy, distribute or communicate your modified version (the "Modified License") and apply it to other original works of authorship subject to the following conditions: (i) You may not indicate in any way that your Modified License is the "Open Software License" or "OSL" and you may not use those names in the name of your Modified License; (ii) You must replace the notice specified in the first paragraph above with the notice "Licensed under <insert your license name here>" or with a notice of your own that is not confusingly similar to the notice in this License; and (iii) You may not claim that your original works are open source software unless your Modified License has been approved by Open Source Initiative (OSI) and You comply with its license review and certification process.
+16) Modification of This License. This License is Copyright © 2005 Lawrence Rosen. Permission is granted to copy, distribute, or communicate this License without modification. Nothing in this License permits You to modify this License as applied to the Original Work or to Derivative Works. However, You may modify the text of this License and copy, distribute or communicate your modified version (the "Modified License") and apply it to other original works of authorship subject to the following conditions: (i) You may not indicate in any way that your Modified License is the "Open Software License" or "OSL" and you may not use those names in the name of your Modified License; (ii) You must replace the notice specified in the first paragraph above with the notice "Licensed under <insert your license name here>" or with a notice of your own that is not confusingly similar to the notice in this License; and (iii) You may not claim that your original works are open source software unless your Modified License has been approved by Open Source Initiative (OSI) and You comply with its license review and certification process.

hadoop/licenses/STACKABLE_LICENSE

@@ -198,4 +198,4 @@
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.

hbase/licenses/HBASE_LICENSE

@@ -198,4 +198,4 @@
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.

hbase/licenses/JMX_EXPORTER_LICENSE

@@ -40,4 +40,4 @@ e) to display the Original Work publicly.
 
 15) Right to Use. You may use the Original Work in all ways not otherwise restricted or conditioned by this License or by law, and Licensor promises not to interfere with or be responsible for such uses by You.
 
-16) Modification of This License. This License is Copyright © 2005 Lawrence Rosen. Permission is granted to copy, distribute, or communicate this License without modification. Nothing in this License permits You to modify this License as applied to the Original Work or to Derivative Works. However, You may modify the text of this License and copy, distribute or communicate your modified version (the "Modified License") and apply it to other original works of authorship subject to the following conditions: (i) You may not indicate in any way that your Modified License is the "Open Software License" or "OSL" and you may not use those names in the name of your Modified License; (ii) You must replace the notice specified in the first paragraph above with the notice "Licensed under <insert your license name here>" or with a notice of your own that is not confusingly similar to the notice in this License; and (iii) You may not claim that your original works are open source software unless your Modified License has been approved by Open Source Initiative (OSI) and You comply with its license review and certification process.
+16) Modification of This License. This License is Copyright © 2005 Lawrence Rosen. Permission is granted to copy, distribute, or communicate this License without modification. Nothing in this License permits You to modify this License as applied to the Original Work or to Derivative Works. However, You may modify the text of this License and copy, distribute or communicate your modified version (the "Modified License") and apply it to other original works of authorship subject to the following conditions: (i) You may not indicate in any way that your Modified License is the "Open Software License" or "OSL" and you may not use those names in the name of your Modified License; (ii) You must replace the notice specified in the first paragraph above with the notice "Licensed under <insert your license name here>" or with a notice of your own that is not confusingly similar to the notice in this License; and (iii) You may not claim that your original works are open source software unless your Modified License has been approved by Open Source Initiative (OSI) and You comply with its license review and certification process.

hbase/licenses/STACKABLE_LICENSE

@@ -1,2 +1,3 @@
+---
 rules:
   - pattern: ".*"

hbase/stackable/jmx/master.yaml

@@ -1,2 +1,3 @@
+---
 rules:
   - pattern: ".*"

hbase/stackable/jmx/regionserver.yaml

@@ -1,2 +1,3 @@
+---
 rules:
   - pattern: ".*"

hbase/stackable/jmx/restserver.yaml

@@ -1,6 +1,6 @@
 versions = [
     {
         "product": "0.0.1-SNAPSHOT",
-        "java-base": "17"
-    }
+        "java-base": "17",
+    },
 ]

hello-world/versions.py

@@ -198,4 +198,4 @@
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.

hive/licenses/HIVE_LICENSE

@@ -198,4 +198,4 @@
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.

hive/licenses/JMX_EXPORTER_LICENSE

@@ -40,4 +40,4 @@ e) to display the Original Work publicly.
 
 15) Right to Use. You may use the Original Work in all ways not otherwise restricted or conditioned by this License or by law, and Licensor promises not to interfere with or be responsible for such uses by You.
 
-16) Modification of This License. This License is Copyright © 2005 Lawrence Rosen. Permission is granted to copy, distribute, or communicate this License without modification. Nothing in this License permits You to modify this License as applied to the Original Work or to Derivative Works. However, You may modify the text of this License and copy, distribute or communicate your modified version (the "Modified License") and apply it to other original works of authorship subject to the following conditions: (i) You may not indicate in any way that your Modified License is the "Open Software License" or "OSL" and you may not use those names in the name of your Modified License; (ii) You must replace the notice specified in the first paragraph above with the notice "Licensed under <insert your license name here>" or with a notice of your own that is not confusingly similar to the notice in this License; and (iii) You may not claim that your original works are open source software unless your Modified License has been approved by Open Source Initiative (OSI) and You comply with its license review and certification process.
+16) Modification of This License. This License is Copyright © 2005 Lawrence Rosen. Permission is granted to copy, distribute, or communicate this License without modification. Nothing in this License permits You to modify this License as applied to the Original Work or to Derivative Works. However, You may modify the text of this License and copy, distribute or communicate your modified version (the "Modified License") and apply it to other original works of authorship subject to the following conditions: (i) You may not indicate in any way that your Modified License is the "Open Software License" or "OSL" and you may not use those names in the name of your Modified License; (ii) You must replace the notice specified in the first paragraph above with the notice "Licensed under <insert your license name here>" or with a notice of your own that is not confusingly similar to the notice in this License; and (iii) You may not claim that your original works are open source software unless your Modified License has been approved by Open Source Initiative (OSI) and You comply with its license review and certification process.