Build packages in CI #218
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,120 @@ | ||||||||||||||||||
| --- | ||||||||||||||||||
| name: Build packages | ||||||||||||||||||
|
|
||||||||||||||||||
| 'on': | ||||||||||||||||||
| push: | ||||||||||||||||||
| branches: [main] | ||||||||||||||||||
| paths: | ||||||||||||||||||
| - '*.sh' | ||||||||||||||||||
| pull_request: | ||||||||||||||||||
| paths: | ||||||||||||||||||
| - '*.sh' | ||||||||||||||||||
|
Comment on lines
+8
to
+11
There was a problem hiding this comment. 🧩 Analysis chain🌐 Web query:
💡 Result: Use the workflow-level path filter with a recursive globstar: '/.sh'. Example (runs on push/pull_request when any .sh file anywhere in the repo changes): on: push: paths: - '/.sh' Notes that explain common “doesn’t match nested files” issues: 1) Don’t use '.sh' alone if you need nested matching. '.sh' matches only in the workflow path root (top level), while '/.sh' matches recursively. 2) Quote the pattern strings to avoid YAML/parse issues (GitHub docs recommend escaping/quoting special characters in glob patterns). 3) If you need to exclude certain .sh files, combine include patterns and exclude patterns. GitHub requires at least one positive pattern if you use '!' patterns, and you can alternatively use paths-ignore. Example with excludes: on: push: paths: - '/.sh' - '!/scripts/' Authoritative reference: GitHub’s Workflow syntax docs for path filters show that paths uses glob patterns with '*' and '' and that the workflow runs if at least one path matches. Citations:
🏁 Script executed: # Search for all shell scripts in the repository
find . -name "*.sh" -type f 2>/dev/null | head -20Repository: ShipSoft/shipdist Length of output: 350
At lines 8 and 11, this pattern misses shell scripts in subdirectories (such as Use the recursive globstar pattern instead: Suggested fix push:
branches: [main]
paths:
- - '*.sh'
+ - '**/*.sh'
pull_request:
paths:
- - '*.sh'
+ - '**/*.sh'📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||
| schedule: | ||||||||||||||||||
| # Weekly rebuild on Monday at 04:00 UTC | ||||||||||||||||||
| - cron: '0 4 * * 1' | ||||||||||||||||||
| workflow_dispatch: | ||||||||||||||||||
| inputs: | ||||||||||||||||||
| package: | ||||||||||||||||||
| description: Package to build | ||||||||||||||||||
| default: FairShip | ||||||||||||||||||
| defaults: | ||||||||||||||||||
| description: Defaults to use | ||||||||||||||||||
| default: release | ||||||||||||||||||
|
|
||||||||||||||||||
| concurrency: | ||||||||||||||||||
| group: build-${{ github.ref }} | ||||||||||||||||||
| cancel-in-progress: true | ||||||||||||||||||
|
Comment on lines
+24
to
+26
There was a problem hiding this comment. Concurrency group may cancel unrelated workflow_dispatch runs. Multiple Suggested improvement concurrency:
- group: build-${{ github.ref }}
+ group: build-${{ github.ref }}-${{ inputs.package || 'FairShip' }}-${{ inputs.defaults || 'release' }}
cancel-in-progress: true📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||
|
|
||||||||||||||||||
| env: | ||||||||||||||||||
| REMOTE_STORE: /cvmfs/ship-nightlies.cern.ch/main/sw | ||||||||||||||||||
| WRITE_STORE: s3://ship-packages | ||||||||||||||||||
| WORK_DIR: /opt/ship | ||||||||||||||||||
| DEFAULTS: release | ||||||||||||||||||
|
|
||||||||||||||||||
| jobs: | ||||||||||||||||||
| build: | ||||||||||||||||||
| name: Build ${{ inputs.package || 'FairShip' }} | ||||||||||||||||||
| runs-on: [self-hosted, alma9, x86_64] | ||||||||||||||||||
|
There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
# Verify whether actionlint config exists and whether custom labels are declared.
fd -i 'actionlint.yaml|actionlint.yml' .github
rg -n -C2 'self-hosted-runner|labels|alma9|x86_64' .github --iglob '*actionlint*.y*ml'Repository: ShipSoft/shipdist Length of output: 1784 🏁 Script executed: #!/bin/bash
# List .github directory structure
find .github -type f -name "*.yml" -o -name "*.yaml" | head -20
# Search for actionlint config
find .github -type f \( -name ".actionlintrc*" -o -name "actionlint*" \) 2>/dev/null
# Check the workflow file around lines 36 and 74
grep -n "runs-on\|alma9\|x86_64" .github/workflows/build.yml 2>/dev/null | head -20Repository: ShipSoft/shipdist Length of output: 214 🏁 Script executed: #!/bin/bash
# Search for actionlint config at repository root and subdirectories
find . -maxdepth 2 -type f \( -name ".actionlintrc*" -o -name "actionlint.yaml" -o -name "actionlint.yml" \) 2>/dev/null
# Check if actionlint config exists anywhere
ls -la .github/ | grep -i actionlint
# Get more context around line 36 and 81 in build.yml
sed -n '30,40p' .github/workflows/build.yml
sed -n '75,85p' .github/workflows/build.ymlRepository: ShipSoft/shipdist Length of output: 615 🌐 Web query:
💡 Result: To make actionlint accept custom self-hosted runner labels (including ones like AlmaLinux 9 x86_64), configure actionlint.yaml under .github/ with self-hosted-runner.labels. 1) Create .github/actionlint.yaml (or .github/actionlint.yml) Example: self-hosted-runner: labels: - alma9 - x86_64 # If your actual runner labels include dynamic suffixes, use glob patterns. # Examples (adjust to your real label strings): # - alma9-* # - *-x86_64 2) Where these config fields apply - actionlint reads the configuration file from actionlint.yaml / actionlint.yml placed in the .github directory. - self-hosted-runner.labels is “Labels of self-hosted runner in array of strings”. actionlint treats each entry as a pattern (glob syntax supported by Go’s path.Match), and matches these patterns against labels used in runs-on. 3) Why this fixes the “label … is unknown” error - actionlint’s runner-label rule validates that labels used in runs-on are known; for custom self-hosted labels you must add them to self-hosted-runner.labels in actionlint.yaml. If you tell me the exact label name(s) you use for your AlmaLinux 9 x86_64 runners (as they appear in the GitHub Runner “Labels” setting and in runs-on), I can suggest the tightest glob patterns to cover them. Citations:
Create actionlint configuration for custom self-hosted runner labels. Lines 36 and 81 use custom self-hosted runner labels self-hosted-runner:
labels:
- alma9
- x86_64Without this configuration, actionlint will fail CI validation. 🧰 Tools🪛 actionlint (1.7.12)[error] 36-36: label "alma9" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2025-vs2026", "windows-2022", "windows-11-arm", "ubuntu-slim", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "macos-latest", "macos-latest-xlarge", "macos-latest-large", "macos-26-intel", "macos-26-xlarge", "macos-26-large", "macos-26", "macos-15-intel", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xlarge", "macos-14-large", "macos-14", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file (runner-label) [error] 36-36: label "x86_64" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2025-vs2026", "windows-2022", "windows-11-arm", "ubuntu-slim", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "macos-latest", "macos-latest-xlarge", "macos-latest-large", "macos-26-intel", "macos-26-xlarge", "macos-26-large", "macos-26", "macos-15-intel", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xlarge", "macos-14-large", "macos-14", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file (runner-label) 🤖 Prompt for AI Agents |
||||||||||||||||||
| container: | ||||||||||||||||||
| image: registry.cern.ch/ship/gha-runner:latest | ||||||||||||||||||
| volumes: | ||||||||||||||||||
| - /cvmfs:/cvmfs:ro | ||||||||||||||||||
| - shipbuild:/opt/ship | ||||||||||||||||||
| env: | ||||||||||||||||||
| AWS_ACCESS_KEY_ID: ${{ secrets.S3_ACCESS_KEY_ID }} | ||||||||||||||||||
| AWS_SECRET_ACCESS_KEY: ${{ secrets.S3_SECRET_ACCESS_KEY }} | ||||||||||||||||||
| S3_ENDPOINT_URL: https://s3.cern.ch | ||||||||||||||||||
| steps: | ||||||||||||||||||
| - uses: actions/checkout@v4 | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: Install uv and aliBuild | ||||||||||||||||||
| run: | | ||||||||||||||||||
| curl -LsSf https://astral.sh/uv/install.sh | sh | ||||||||||||||||||
| echo "$HOME/.local/bin" >> "$GITHUB_PATH" | ||||||||||||||||||
| "$HOME/.local/bin/uvx" --from alibuild aliBuild version | ||||||||||||||||||
|
Comment on lines
+50
to
+54
There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: cat -n .github/workflows/build.yml | sed -n '45,60p'Repository: ShipSoft/shipdist Length of output: 835 🏁 Script executed: # Check for lock files, requirements, and version specifications
find . -type f \( -name "*.lock" -o -name "requirements*.txt" -o -name "pyproject.toml" -o -name ".python-version" \) | head -20
# Check for other workflow files to see if they pin versions
ls -la .github/workflows/ 2>/dev/null
# Search for any UV_VERSION or ALIBUILD_VERSION env vars
rg -i "UV_VERSION|ALIBUILD_VERSION" .Repository: ShipSoft/shipdist Length of output: 436 🏁 Script executed: cat -n ./uv.sh
# Also check if build.yml calls this script or specifies versions elsewhere
grep -n "uv\|version\|ALIBUILD" .github/workflows/build.ymlRepository: ShipSoft/shipdist Length of output: 1438 Pin Lines 51 and 53 install floating latest versions of
🤖 Prompt for AI Agents |
||||||||||||||||||
|
|
||||||||||||||||||
| - name: Configure S3 credentials | ||||||||||||||||||
| run: | | ||||||||||||||||||
| echo "::add-mask::${AWS_ACCESS_KEY_ID}" | ||||||||||||||||||
| echo "::add-mask::${AWS_SECRET_ACCESS_KEY}" | ||||||||||||||||||
| printf 'host_base = s3.cern.ch\nhost_bucket = %%(bucket)s.s3.cern.ch\naccess_key = %s\nsecret_key = %s\n' \ | ||||||||||||||||||
| "$AWS_ACCESS_KEY_ID" "$AWS_SECRET_ACCESS_KEY" > ~/.s3cfg | ||||||||||||||||||
|
Comment on lines
+60
to
+61
There was a problem hiding this comment. Harden At Line 54, Suggested fix printf 'host_base = s3.cern.ch\nhost_bucket = %%(bucket)s.s3.cern.ch\naccess_key = %s\nsecret_key = %s\n' \
"$AWS_ACCESS_KEY_ID" "$AWS_SECRET_ACCESS_KEY" > ~/.s3cfg
+ chmod 600 ~/.s3cfg📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||
|
|
||||||||||||||||||
| - name: Build | ||||||||||||||||||
| run: | | ||||||||||||||||||
| PACKAGE="${{ inputs.package || 'FairShip' }}" | ||||||||||||||||||
| DEFAULTS="${{ inputs.defaults || env.DEFAULTS }}" | ||||||||||||||||||
| WRITE_STORE_FLAG="" | ||||||||||||||||||
| if [ "${{ github.event_name }}" != "pull_request" ]; then | ||||||||||||||||||
| WRITE_STORE_FLAG="--write-store ${WRITE_STORE}" | ||||||||||||||||||
| fi | ||||||||||||||||||
| uvx --from alibuild aliBuild build "$PACKAGE" \ | ||||||||||||||||||
| --defaults "$DEFAULTS" \ | ||||||||||||||||||
| --remote-store "${REMOTE_STORE}" \ | ||||||||||||||||||
| $WRITE_STORE_FLAG \ | ||||||||||||||||||
| --work-dir "$WORK_DIR" \ | ||||||||||||||||||
| -c . \ | ||||||||||||||||||
| -j "$(nproc)" | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: Print build log on failure | ||||||||||||||||||
| if: failure() | ||||||||||||||||||
| run: | | ||||||||||||||||||
| echo "Searching for build logs in $WORK_DIR/BUILD/" | ||||||||||||||||||
| ls -la "$WORK_DIR/BUILD/" 2>&1 || true | ||||||||||||||||||
| for log in "$WORK_DIR"/BUILD/*-latest/log; do | ||||||||||||||||||
| if [ -f "$log" ]; then | ||||||||||||||||||
| echo "::group::Build log: $log" | ||||||||||||||||||
| cat "$log" | ||||||||||||||||||
| echo "::endgroup::" | ||||||||||||||||||
| fi | ||||||||||||||||||
| done | ||||||||||||||||||
|
|
||||||||||||||||||
| build-container: | ||||||||||||||||||
| name: Build container image | ||||||||||||||||||
| needs: build | ||||||||||||||||||
| if: github.event_name == 'push' || startsWith(github.ref, 'refs/tags/') | ||||||||||||||||||
| runs-on: [self-hosted, alma9, x86_64] | ||||||||||||||||||
| steps: | ||||||||||||||||||
| - uses: actions/checkout@v4 | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: Determine image tag | ||||||||||||||||||
| id: tag | ||||||||||||||||||
| run: | | ||||||||||||||||||
| if [[ "$GITHUB_REF" == refs/tags/* ]]; then | ||||||||||||||||||
| echo "tag=${GITHUB_REF#refs/tags/}" >> "$GITHUB_OUTPUT" | ||||||||||||||||||
| echo "latest=true" >> "$GITHUB_OUTPUT" | ||||||||||||||||||
| else | ||||||||||||||||||
| echo "tag=main-${GITHUB_SHA::8}" >> "$GITHUB_OUTPUT" | ||||||||||||||||||
| echo "latest=false" >> "$GITHUB_OUTPUT" | ||||||||||||||||||
| fi | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: Build and push | ||||||||||||||||||
| run: | | ||||||||||||||||||
| IMAGE=registry.cern.ch/ship/ship-sim | ||||||||||||||||||
| TAG="${{ steps.tag.outputs.tag }}" | ||||||||||||||||||
| podman build -t "$IMAGE:$TAG" container/ | ||||||||||||||||||
| podman push "$IMAGE:$TAG" | ||||||||||||||||||
| if [ "${{ steps.tag.outputs.latest }}" = "true" ]; then | ||||||||||||||||||
| podman tag "$IMAGE:$TAG" "$IMAGE:latest" | ||||||||||||||||||
| podman push "$IMAGE:latest" | ||||||||||||||||||
| fi | ||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| FROM almalinux:9-minimal | ||
|
|
||
| RUN microdnf install -y \ | ||
| https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest.noarch.rpm && \ | ||
| microdnf install -y cvmfs fuse environment-modules && \ | ||
| microdnf clean all | ||
|
|
||
| COPY cvmfs-ship.conf /etc/cvmfs/default.d/ship.conf | ||
|
|
||
| COPY entrypoint.sh /entrypoint.sh | ||
| RUN chmod +x /entrypoint.sh | ||
|
|
||
| RUN useradd -m -u 1000 shipuser | ||
| USER shipuser | ||
|
|
||
| ENTRYPOINT ["/entrypoint.sh"] | ||
| CMD ["/bin/bash"] | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| CVMFS_REPOSITORIES=ship.cern.ch | ||
| CVMFS_HTTP_PROXY=DIRECT |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| #!/bin/bash | ||
| set -e | ||
|
Check notice on line 2 in container/entrypoint.sh
|
||
|
|
||
| # Source environment modules | ||
| source /etc/profile.d/modules.sh 2>/dev/null || true | ||
|
|
||
| # Load FairShip environment from CVMFS if available | ||
| if [ -d /cvmfs/ship.cern.ch ]; then | ||
| alienv_output=$(/cvmfs/ship.cern.ch/bin/alienv printenv FairShip/latest 2>&1) | ||
| if ! eval "$alienv_output" 2>/dev/null; then | ||
| echo "Warning: failed to load FairShip environment from CVMFS" >&2 | ||
| echo "$alienv_output" >&2 | ||
| fi | ||
|
Comment on lines
+9
to
+13
There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: # Find and read the entrypoint.sh file
find . -name "entrypoint.sh" -type fRepository: ShipSoft/shipdist Length of output: 86 🏁 Script executed: # Check if we can locate the file in container directory
fd "entrypoint.sh" . --type fRepository: ShipSoft/shipdist Length of output: 86 🏁 Script executed: # List directory structure to understand the repo layout
git ls-files | grep -E "(container|entrypoint)" | head -20Repository: ShipSoft/shipdist Length of output: 131 🏁 Script executed: cat -n ./container/entrypoint.shRepository: ShipSoft/shipdist Length of output: 610 🏁 Script executed: # Verify bash set -e behavior with command substitution in assignment
bash <<'EOF'
set -e
echo "Testing set -e with failed command substitution in assignment..."
var=$(exit 1)
echo "After assignment, var='$var'"
echo "Script continued - set -e did NOT exit"
EOF
echo "Exit code: $?"Repository: ShipSoft/shipdist Length of output: 138 🏁 Script executed: # Test what happens when eval receives mixed stderr
bash <<'EOF'
set -e
echo "Testing eval with mixed stderr..."
# Simulate alienv failing and outputting to stderr + stdout
output=$({ echo "export VAR=value"; echo "error message" >&2; exit 0; } 2>&1)
echo "Output variable: '$output'"
echo "Now evaluating..."
eval "$output" 2>/dev/null || echo "Eval failed"
echo "VAR is: $VAR"
EOFRepository: ShipSoft/shipdist Length of output: 188 🏁 Script executed: # Test the exact pattern from entrypoint.sh more carefully
bash <<'EOF'
set -e
if [ -d /tmp ]; then
# Simulate alienv failure - outputs to stderr AND exits with non-zero
alienv_output=$(/bin/sh -c 'echo "error: something failed" >&2; exit 1' 2>&1)
echo "After command sub, alienv_output='$alienv_output'"
echo "Command substitution completed - set -e did NOT exit"
if ! eval "$alienv_output" 2>/dev/null; then
echo "Warning: failed to load FairShip environment from CVMFS" >&2
echo "$alienv_output" >&2
fi
fi
echo "Script completed successfully"
EOF
echo "Final exit code: $?"Repository: ShipSoft/shipdist Length of output: 79 🏁 Script executed: # Verify: does the suggested fix actually work better?
bash <<'EOF'
set -e
echo "Testing suggested fix pattern..."
if alienv_output=$(/bin/sh -c 'echo "error: something failed" >&2; exit 1' 2>&1); then
echo "Command succeeded, output: $alienv_output"
if ! eval "$alienv_output"; then
echo "Warning: failed to evaluate FairShip environment from CVMFS" >&2
fi
else
echo "Warning: failed to load FairShip environment from CVMFS" >&2
echo "Output was: $alienv_output"
fi
echo "Script completed"
EOF
echo "Exit code: $?"Repository: ShipSoft/shipdist Length of output: 232 Prevent premature exit before your warning path executes. At Line 9, when Use Suggested fix if [ -d /cvmfs/ship.cern.ch ]; then
- alienv_output=$(/cvmfs/ship.cern.ch/bin/alienv printenv FairShip/latest 2>&1)
- if ! eval "$alienv_output" 2>/dev/null; then
- echo "Warning: failed to load FairShip environment from CVMFS" >&2
- echo "$alienv_output" >&2
- fi
+ if alienv_output=$(/cvmfs/ship.cern.ch/bin/alienv printenv FairShip/latest); then
+ if ! eval "$alienv_output"; then
+ echo "Warning: failed to evaluate FairShip environment from CVMFS" >&2
+ fi
+ else
+ echo "Warning: failed to load FairShip environment from CVMFS" >&2
+ fi
fi🤖 Prompt for AI Agents |
||
| fi | ||
|
|
||
| exec "$@" | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
Repository: ShipSoft/shipdist
Length of output: 88
🏁 Script executed:
# Read the build.yml file with line numbers cat -n .github/workflows/build.ymlRepository: ShipSoft/shipdist
Length of output: 3930
Tag pushes won't trigger workflow due to branch restriction on push event.
The
pushtrigger is restricted tobranches: [main](line 6), preventing tag pushes from initiating the workflow. While thebuild-containerjob includes tag handling at lines 88-94 with the conditionstartsWith(github.ref, 'refs/tags/')(line 80), it depends on thebuildjob (line 79), which never executes on tag events.Suggested fix
push: branches: [main] + tags: + - '*' paths: - '**/*.sh'📝 Committable suggestion
🤖 Prompt for AI Agents