Skip to content

Align stencil oracles and sibling backends to verbatim PolyBench/C 4.2.1 #410

Align stencil oracles and sibling backends to verbatim PolyBench/C 4.2.1

Align stencil oracles and sibling backends to verbatim PolyBench/C 4.2.1 #410

Workflow file for this run

name: CI
# Seven parallel CPU runners, each a self-contained slice of the pipeline (was one ~200-min
# sequential runner). Shared toolchain + base pip come from
# ./.github/actions/setup; each job adds only the extra it needs, so ONLY frameworks-pluto pays for
# the Pluto build, ONLY mpi pays for gt4py + the MPICH/mpi4py source build.
#
# BALANCE is measured, never assumed. Run 30288151626 had translators at 40m32s against mpi's
# 5m09s, so its two small phases moved to the idle mpi runner. Run 30480407159 then measured the
# three-way shard of what was left at 4m + 8m + 9m -- three runners and a splitting script to move
# a 21m job off a critical path that e2e[pythran] holds at 31m, i.e. no wall clock bought and two
# runners spent. Merged back to one; the script is gone. Shard only what is ON the critical path.
#
# 2026-08-11: mpi + dace-numeric + hf-export fused into one job (see the mpi section below). Run
# 31478271044 measured mpi at 11m51s, dace-numeric at 16m18s -- run 31023526869 last measured
# hf-export, when it runs, at 2m37s -- ~31m sequential, none of the three on the critical path. The
# same run measured something sharper than any one leg's own time: of eleven jobs, only THREE start
# when the run is created (integration, frameworks-pluto, mpi, all at 09:34:06); dace-numeric queued
# 22m for a free slot, port-fidelity 33m, container-image 61m, unit 102m, translators 105m -- three
# concurrent slots, not eleven, so past three, job COUNT drives total wall clock more than any one
# job's own duration. Fusing three short/idle legs into one drops a whole queue wave
# (ceil(11/3)=4 -> ceil(9/3)=3). frameworks-pluto stays separate: its own range is 15m05s-28m18s
# across the same two runs, it is the only runner building Pluto from source (its own LLVM-17
# toolchain, unrelated to MPI/OpenMPI or DaCe's per-kernel C++ builds), and a bad day on both sides
# of that union risks the same 45m+ single-runner budget this fusion exists to avoid -- reversing an
# isolation the file already chose once (see e2e-native/e2e-pythran above) for a saving this size.
#
# The e2e sweep (tests/test_e2e_numerical.py) is partitioned by BACKEND via
# HPCAGENT_BENCH_E2E_BACKENDS so no single runner carries all ~80 min of it and Pluto stays isolated:
# numba+jax on unit, pluto on frameworks-pluto, c+cpp+fortran on e2e-native, pythran on
# e2e-pythran (pythran isolated -- it is the memory/time pig) -- all seven backends covered
# exactly once. Unset (a local run) still sweeps the full set.
#
# Within a job every phase after setup carries ``if: ${{ !cancelled() }}`` so one phase failing
# still surfaces the rest (the job stays red). Phase 0 (format) and setup stay hard gates.
#
# Runners:
# unit format + structure + native emit + agent-bench + hf + e2e[numba,jax]
# translators numpyto op suite
# frameworks-pluto codegen frameworks + fp16 + e2e[pluto] (ONLY Pluto build)
# e2e-native integration sweep (CLI->DB->plot) + e2e[c,cpp,fortran]
# (native heavy-compile slice, -n auto)
# e2e-pythran e2e[pythran] (pythran isolated -- ~7 GB/compile, -n 2)
# container-image build + smoke-launch the agent-bench cpu container image
# mpi the whole MPI/distributed track (descriptor + driver codegen + launch +
# scoring + halo + sizing) on MPICH + a source-built mpi4py (ONLY MPI build),
# plus port fidelity + benchmark reference validation (gt4py), plus (fused,
# 2026-08-11) the DaCe corpus vs numpy (dace-numeric, a C++ build per kernel, one
# subprocess each, -n 4) and the HF dataset build/publish (hf-export)
#
# GPU frameworks (triton, tvm-gpu) self-skip here and run in agentbench-gpu.yml.
on:
push:
branches: [ main, extended, ci-fix ]
pull_request:
branches: [ main, extended, ci-fix ]
merge_group:
branches: [ main, extended, ci-fix ]
concurrency:
group: ${{github.workflow}}-${{github.ref}}
# Cancel superseded runs on a branch, never on the default branch. A push to main that lands while
# the previous run is still going used to kill it, and a killed run takes its FINDINGS with it: a
# -std= break on the mpi wrapper had already been reported by one run and was discarded before
# anyone read it, so the same breakage had to be rediscovered a push later. Branch runs keep the
# old behaviour, where the newer commit is the only one anybody wants an answer about.
cancel-in-progress: ${{github.ref != 'refs/heads/main'}}
jobs:
# ===========================================================================================
# unit -- fast categories: the format gate, the structural firewall, native emit, agent-bench,
# the HF export, and the cheap e2e backends (numba/jax).
# ===========================================================================================
unit:
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-ci') }}
name: unit (format + structure + agent-bench + e2e[numba,jax])
runs-on: ubuntu-latest
timeout-minutes: 90
env:
# Job-level so every pytest here collects; --cov-report= suppresses the per-job
# render (the combine job owns the only report).
PYTEST_ADDOPTS: "--cov=hpcagent_bench --cov-append --cov-report="
steps:
- uses: actions/checkout@v4
with:
# Full history so format's merge-base diff against the target branch resolves.
fetch-depth: 0
# The KernelBench originals are a submodule; without this the provenance test can only
# skip, which is indistinguishable from a mapping that stopped resolving.
submodules: recursive
- uses: actions/setup-python@v5
with:
python-version: '3.12'
# ccache -- emitted kernels are byte-identical across runs (deterministic emit, no
# embedded paths or timestamps), so every CI run recompiles the very same TUs.
# compilers.yaml invokes bare `gcc`/`g++`/`gfortran`, so putting the shim dir first on
# PATH routes the whole build through the cache without touching any build code.
- name: Setup -- ccache + TBB
run: |
sudo apt-get update -qq
# libtbb-dev is not optional for the cpp_isopar column: libstdc++ picks its parallel
# <execution> backend per TU with __has_include(<tbb/tbb.h>), so with the headers absent
# the par/par_unseq policies compile and run SEQUENTIALLY under a parallel name -- a
# silently wrong measurement, not a build failure. languages.stdpar_link_flags asks the
# compiler that same question before appending -ltbb, so installing it here is what
# actually turns the column parallel.
sudo apt-get install -y --no-install-recommends ccache libtbb-dev
echo "/usr/lib/ccache" >> "$GITHUB_PATH"
echo "CCACHE_DIR=$HOME/.ccache" >> "$GITHUB_ENV"
CCACHE_DIR="$HOME/.ccache" ccache --set-config max_size=1G
# The CPU baselines compile -march=native and ccache hashes that flag literally,
# so an object cached on a wider-ISA runner runs here as SIGILL (exit 132). Key
# the cache on the ISA the compiler would actually select.
cpu_key=$(awk -F': ' '/^(model name|flags)/ {print $2}' /proc/cpuinfo |
head -2 | sha256sum | cut -c1-12)
# An empty key would silently restore the old cross-runner behaviour.
test -n "$cpu_key" || { echo "could not derive a CPU key" >&2; exit 1; }
echo "CPU_KEY=$cpu_key" >> "$GITHUB_ENV"
- name: Setup -- restore the compile cache
uses: actions/cache@v4
with:
path: ~/.ccache
key: ccache-${{ github.job }}-${{ env.CPU_KEY }}-${{ github.run_id }}
restore-keys: ccache-${{ github.job }}-${{ env.CPU_KEY }}-
# Zero AFTER the restore: the cache carries its own counters back, so zeroing earlier
# would make the end-of-job stats cumulative instead of this run's hit rate.
- name: Setup -- reset the ccache counters
run: ccache --zero-stats
# Phase 0 runs BEFORE the heavy setup so a formatting slip fails in seconds. yapf /
# fprettify / clang-format are pip wheels (the clang-format wheel ships the binary).
- name: Phase 0 -- format check (changed files @ 120 cols)
run: |
python -m pip install --upgrade pip
python -m pip install yapf fprettify clang-format
base="${{ github.event.pull_request.base.ref || 'main' }}"
git fetch --no-tags origin "$base"
python scripts/check_format.py --base "origin/$base"
- uses: ./.github/actions/setup
- name: Phase 1 -- unit sweep (every test file no other phase claims)
if: ${{ !cancelled() }}
# DISCOVERED, not enumerated. `ls` decides what runs and .github/dedicated_tests.txt
# records the exceptions, so a new test file is covered the moment it lands. The reverse
# -- filenames typed into this workflow -- drifted badly: 144 test files existed, 44 were
# named anywhere in CI, and 94 never ran, including the guards written for the regressions
# they were meant to catch. tests/test_ci_coverage.py fails if the two ever disagree.
# -n auto balances the files across the runner's cores at runtime, which beats any static
# split of a set whose costs span three orders of magnitude.
#
# CAPPED, like every other heavy phase. Uncapped, this is what killed run 30454516782: the
# sweep reached 91% with zero failures, the runner was then terminated, and because a job
# cancellation is not a step failure the FIVE phases below never ran at all. A step timeout
# is a failure, so `!cancelled()` keeps them running and the job reports what it knows.
timeout-minutes: 40
run: |
excl=$(mktemp)
grep -vE '^\s*(#|$)' .github/dedicated_tests.txt > "$excl"
files=$(ls tests/test_*.py | grep -vxF -f "$excl")
test -n "$files" || { echo 'unit sweep selected no files'; exit 1; }
echo "$files" | tr ' ' '\n' | wc -l | xargs echo "sweeping files:"
python -m pytest -q -p no:cacheprovider -rfEs -n auto --timeout=600 $files
- name: Phase 2 -- translations (C / C++ / Fortran emit + compile + validate)
if: ${{ !cancelled() }}
run: |
python -m pytest -q -p no:cacheprovider -rfEs tests/test_native_autogen.py
- name: Phase 3 -- agent-bench (tools + no-op/BLAS optimizers, both modes, in-proc judge)
if: ${{ !cancelled() }}
# Per-step cap: a hung/slow phase fails THIS step (a step timeout is a failure,
# not a job cancellation), so the ``!cancelled()`` phases below still run instead
# of the whole job dying at the 90-min job timeout and skipping 5 + 7.
timeout-minutes: 25
run: |
python -m pytest -q -p no:cacheprovider -rfEs --timeout=600 \
tests/test_agent_tools.py \
tests/test_noop_optimizer.py \
tests/test_blas_optimizer.py \
tests/test_agent_service.py \
tests/test_token_usage.py \
tests/test_harbor_adapter.py \
tests/test_packaging.py \
tests/test_optimizer_plugin.py \
tests/test_agent_bench.py
- name: Phase 5 -- e2e numerical sweep [numba, jax] @ S
if: ${{ !cancelled() }}
# Bounded like Phase 3 so a wedged worker fails this step (leaving Phase 7 to run),
# not the job. --timeout is the per-test backstop; the oracle already SIGKILLs its
# own forked jax/native children, so this only catches an in-process numba hang.
#
# 35 -> 55: the old budget was set against a 200-port kernelbench subtrack and an
# UNINSTRUMENTED run. Both moved. The subtrack is now 239 (+19.5%) and coverage is on
# job-wide, which is deliberate here -- unlike Phase 2c this phase drives real library
# code, so its coverage is signal and cannot just be switched off. Last green run took
# 26:01; it reached 93% before the runner killed it at 35:00. This is a budget correction
# for work that was added on purpose, not headroom for a hang -- the per-test --timeout=600
# is still what catches that.
timeout-minutes: 55
env:
HPCAGENT_BENCH_E2E_BACKENDS: "numba,jax"
run: |
python -m pytest -q -p no:cacheprovider -rfEs -n auto --timeout=600 tests/test_e2e_numerical.py
# Hit rate is the whole point of the cache -- surface it so a regression to 0% is visible.
- name: ccache statistics
if: always()
run: ccache --show-stats
# if: always() -- a job that went red still covered lines on the way, and the total is more
# honest with them than without. if-no-files-found: ignore for a job that failed before pytest.
- name: Upload coverage data
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-unit
path: .coverage*
include-hidden-files: true
if-no-files-found: ignore
# ===========================================================================================
# integration -- the @pytest.mark.integration tests, which BUILD AND RUN real artifacts.
#
# Its own job because it is the only phase here with that cost profile, and sharing a budget
# with the unit phases is what made it fail: as a step of `unit` it hit the 25-minute STEP
# ceiling (run 30990017840), which is a step FAILURE, so the job went red and dragged the
# downstream `coverage` job red with it -- three reds on the run page, one cause, zero test
# failures. Splitting gives it the whole job budget instead of the tail of somebody else's.
#
# Deliberately NOT `needs: [unit]`: a Phase 1 unit failure must not hide what the integration
# tests would have said. Only the PUBLISH step is gated on both (see the hf-export phase in mpi).
# ===========================================================================================
integration:
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-ci') }}
name: integration (build/run real artifacts)
runs-on: ubuntu-latest
timeout-minutes: 90
env:
PYTEST_ADDOPTS: "--cov=hpcagent_bench --cov-append --cov-report="
# tests/test_dace_openblas_link.py builds a pinned OpenBLAS with USE_OPENMP=1 (~10 min at
# -j4) and skips the build when the install is already there -- cached below, not rebuilt.
OPTARENA_BLAS_CACHE: ${{ github.workspace }}/.openblas-cache
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- uses: actions/setup-python@v5
with:
python-version: '3.12'
# Same ccache + TBB rationale as `unit`; the key is namespaced by github.job, so this job
# warms its own cache rather than contending for that one.
- name: Setup -- ccache + TBB
run: |
sudo apt-get update -qq
sudo apt-get install -y --no-install-recommends ccache libtbb-dev
echo "/usr/lib/ccache" >> "$GITHUB_PATH"
echo "CCACHE_DIR=$HOME/.ccache" >> "$GITHUB_ENV"
CCACHE_DIR="$HOME/.ccache" ccache --set-config max_size=1G
cpu_key=$(awk -F': ' '/^(model name|flags)/ {print $2}' /proc/cpuinfo |
head -2 | sha256sum | cut -c1-12)
test -n "$cpu_key" || { echo "could not derive a CPU key" >&2; exit 1; }
echo "CPU_KEY=$cpu_key" >> "$GITHUB_ENV"
- name: Setup -- restore the compile cache
uses: actions/cache@v4
with:
path: ~/.ccache
key: ccache-${{ github.job }}-${{ env.CPU_KEY }}-${{ github.run_id }}
restore-keys: ccache-${{ github.job }}-${{ env.CPU_KEY }}-
- name: Setup -- reset the ccache counters
run: ccache --zero-stats
# Only the INSTALL tree (51 MB); the 332 MB source tree is not needed once it exists. Keyed
# by CPU_KEY as well as the tag: the build is not DYNAMIC_ARCH, so a cache restored onto a
# different runner uarch would SIGILL rather than fail honestly.
- name: Setup -- restore the from-source OpenBLAS (USE_OPENMP=1)
uses: actions/cache@v4
with:
path: ${{ github.workspace }}/.openblas-cache/openblas-openmp/install
key: openblas-openmp-v0.3.29-${{ runner.os }}-${{ env.CPU_KEY }}
- uses: ./.github/actions/setup
- name: Phase 6 -- integration-marked tests (build/run real artifacts)
if: ${{ !cancelled() }}
# -m integration auto-collects EVERY @pytest.mark.integration test (present and
# future) UNDER THE PATHS LISTED HERE, so a new one is covered the moment it is
# marked -- but only if it lives in one of them. Both suites are listed because the
# translators tree has its own conftest and is otherwise never scanned by this job.
# `make test` excludes this marker for a fast local loop; CI is where they must run.
#
# Capped well under the job ceiling so a wedged build still fails as a STEP, leaving the
# coverage upload below to run -- the cap is a hang detector, not the budget it used to be.
timeout-minutes: 70
run: |
python -m pytest -q -p no:cacheprovider -rfEs --timeout=900 -m integration \
tests/ hpcagent_bench/numpy_translators/tests/
- name: ccache statistics
if: always()
run: ccache --show-stats
- name: Upload coverage data
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-integration
path: .coverage*
include-hidden-files: true
if-no-files-found: ignore
# ===========================================================================================
# port-fidelity -- can DaCe still read what the generator emits? A question about the CORPUS
# rather than about the harness, and it compiles nothing, so it gets its own cheap runner. The
# torch-agreement phase (do the ML ports still mean what they were ported from) lands here too.
# ===========================================================================================
port-fidelity:
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-ci') }}
name: port-fidelity (dace frontend reads the generated corpus)
runs-on: ubuntu-latest
# Phase 8 (50) + Phase 8b (25) + checkout/setup/torch, which is ~10 on a warm cache.
timeout-minutes: 100
env:
PYTEST_ADDOPTS: "--cov=hpcagent_bench --cov-append --cov-report="
steps:
- uses: actions/checkout@v4
with:
# third_party/KernelBench holds the upstream models Phase 8b compares against;
# checked out here so adding that phase needs no runner change.
submodules: recursive
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- uses: ./.github/actions/setup
- name: Setup -- CPU torch (the models Phase 8b compares against)
# The dep is DECLARED, in requirements/torch-cpu.txt, rather than spelled inline here --
# that file carries the CPU index and the reason it is not in optional.txt (which
# containers/cpu.def installs, so an entry there would pull torch into the agent image).
# Retried like every other install here: runners hit transient PyPI read errors on wheels
# this size, and the phase self-skips without torch, so a network blip would otherwise turn
# 215 comparisons into skips. `-m torch_agreement` below plus -rfEs makes that loud.
run: |
pip_retry() { for i in 1 2 3 4 5; do python -m pip install "$@" && return 0; echo "pip attempt $i failed; retrying in $((i*5))s"; sleep $((i*5)); done; return 1; }
pip_retry -r requirements/torch-cpu.txt
- name: Phase 8 -- the DaCe frontend still reads what the generator emits
# Parse only: to_sdfg(simplify=False) never invokes a C++ compiler, so the whole corpus is
# affordable on a runner with no toolchain beyond python. The gate is a RATCHET on the
# known refusals -- a new one fails, and one that starts parsing fails too, so the list
# can only shrink. *_dace.py is gitignored, so the test emits what it needs first.
#
# No -n: the sweep is ONE test that already forks a subprocess per program, so xdist would
# shard nothing. The concurrency lives in the test (PARSE_WORKERS). Measured 2026-08-08 on
# the full 625-program corpus: 4.3 min to emit + 45 min to parse serially -- which is why
# this step timed out at 40 twice on 08-07 -- against 4.3 + 20 at two workers. Five REFUSED
# entries are `hang`, 180 s of pure timeout each, and serially they alone are 15 minutes.
# 50 rather than 25 because a hosted runner is slower than the machine that was measured.
if: ${{ !cancelled() }}
timeout-minutes: 50
run: |
python -m pytest -q -p no:cacheprovider -rfsxX -m dace_frontend \
--timeout=2400 tests/test_dace_frontend_validity.py
- name: Phase 8b -- the ML ports still compute what their PyTorch models compute
# The numpy reference is the oracle for every backend, so a port that drifted from the
# KernelBench model it came from grades every submission against the wrong answer while
# staying green. 215 of the 250 ports run against their upstream model at preset S on CPU;
# the other 35 are pinned in UNALIGNED, a RATCHET like the frontend gate above -- a port
# that stops agreeing fails, and a pinned one that becomes comparable fails too.
# --maxfail=10 so a translator-wide break reports ten named ports instead of 215.
# -m torch_agreement pairs the marker with the path, like Phase 8 above: the path says
# which file, the marker says which class of test, so a cheap unmarked helper test added
# to that file later does not silently inherit this phase's torch requirement. -rfEs
# prints skip REASONS, which is what keeps a torch wheel that failed to install from
# reading as a green run instead of as 250 named skips.
if: ${{ !cancelled() }}
timeout-minutes: 25
run: |
python -m pytest -q -p no:cacheprovider -rfEs --maxfail=10 -m torch_agreement \
--timeout=600 tests/test_kernelbench_torch_agreement.py
- name: Upload the generated DaCe corpus
# *_dace.py is gitignored and re-emitted per run, so a kernel that only fails here cannot be
# diffed against the local emit that passes without the files THIS runner produced.
if: failure()
uses: actions/upload-artifact@v4
with:
name: dace-corpus-portfid
path: hpcagent_bench/benchmarks/**/*_dace.py
retention-days: 7
if-no-files-found: ignore
- name: Upload coverage data
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-port-fidelity
path: .coverage*
include-hidden-files: true
if-no-files-found: ignore
# ===========================================================================================
# translators -- the numpyto_* translator's own suite (~640 tests), sharded three ways by file.
# Port fidelity + the benchmark reference validators moved to the mpi runner (see BALANCE above).
# ===========================================================================================
translators:
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-ci') }}
name: translators (numpyto op suite)
runs-on: ubuntu-latest
timeout-minutes: 90
# ONE runner. It was three, cost-sharded, until run 30480407159 measured the result: 4m + 8m +
# 9m. Three runners and a splitting mechanism to move a 21m job off a critical path set by
# e2e[pythran] at 31m -- so the sharding bought no wall clock at all and cost two runners.
env:
# Job-level so every pytest here collects; --cov-report= suppresses the per-job
# render (the combine job owns the only report).
PYTEST_ADDOPTS: "--cov=hpcagent_bench --cov-append --cov-report="
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
# ccache -- emitted kernels are byte-identical across runs (deterministic emit, no
# embedded paths or timestamps), so every CI run recompiles the very same TUs.
# compilers.yaml invokes bare `gcc`/`g++`/`gfortran`, so putting the shim dir first on
# PATH routes the whole build through the cache without touching any build code.
- name: Setup -- ccache + TBB
run: |
sudo apt-get update -qq
# libtbb-dev is not optional for the cpp_isopar column: libstdc++ picks its parallel
# <execution> backend per TU with __has_include(<tbb/tbb.h>), so with the headers absent
# the par/par_unseq policies compile and run SEQUENTIALLY under a parallel name -- a
# silently wrong measurement, not a build failure. languages.stdpar_link_flags asks the
# compiler that same question before appending -ltbb, so installing it here is what
# actually turns the column parallel.
sudo apt-get install -y --no-install-recommends ccache libtbb-dev
echo "/usr/lib/ccache" >> "$GITHUB_PATH"
echo "CCACHE_DIR=$HOME/.ccache" >> "$GITHUB_ENV"
CCACHE_DIR="$HOME/.ccache" ccache --set-config max_size=1G
# The CPU baselines compile -march=native and ccache hashes that flag literally,
# so an object cached on a wider-ISA runner runs here as SIGILL (exit 132). Key
# the cache on the ISA the compiler would actually select.
cpu_key=$(awk -F': ' '/^(model name|flags)/ {print $2}' /proc/cpuinfo |
head -2 | sha256sum | cut -c1-12)
# An empty key would silently restore the old cross-runner behaviour.
test -n "$cpu_key" || { echo "could not derive a CPU key" >&2; exit 1; }
echo "CPU_KEY=$cpu_key" >> "$GITHUB_ENV"
- name: Setup -- restore the compile cache
uses: actions/cache@v4
with:
path: ~/.ccache
key: ccache-${{ github.job }}-${{ env.CPU_KEY }}-${{ github.run_id }}
restore-keys: ccache-${{ github.job }}-${{ env.CPU_KEY }}-
# Zero AFTER the restore: the cache carries its own counters back, so zeroing earlier
# would make the end-of-job stats cumulative instead of this run's hit rate.
- name: Setup -- reset the ccache counters
run: ccache --zero-stats
- uses: ./.github/actions/setup
- name: Phase 2a -- numpy-translator op suite
# ~640 tests: per-op faithfulness plus feature-fix and corpus kernels, each emitted to
# C/C++/Fortran (+ numba/pythran/jax where the test opts in), compiled, validated vs
# numpy. A couple of full-kernel cases (cloudsc, velocity_tendencies) run ~5 min, hence
# --timeout=900. xdist balances the files across the runner's cores at runtime, which no
# static split can beat on a set whose per-file cost spans three orders of magnitude.
if: ${{ !cancelled() }}
timeout-minutes: 60
run: |
python -m pytest -q -p no:cacheprovider -rfsxX -n auto --timeout=900 \
hpcagent_bench/numpy_translators/tests/
# Hit rate is the whole point of the cache -- surface it so a regression to 0% is visible.
- name: ccache statistics
if: always()
run: ccache --show-stats
# ===========================================================================================
# frameworks-pluto -- the ONLY runner that builds Pluto. Codegen frameworks (cc/llvm/tvm/pluto)
# + fp16, plus the pluto slice of the e2e sweep (pluto is opt-in in run_kernel, named here).
# ===========================================================================================
# if: always() -- a job that went red still covered lines on the way, and the total is more
# honest with them than without. if-no-files-found: ignore for a job that failed before pytest.
- name: Upload coverage data
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-translators
path: .coverage*
include-hidden-files: true
if-no-files-found: ignore
frameworks-pluto:
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-ci') }}
name: frameworks + pluto (codegen backends + fp16 + e2e[pluto])
runs-on: ubuntu-latest
timeout-minutes: 120
env:
# bondhugula/pluto @ 0.12.0-33-gdc46216 -- pin for reproducibility, bump deliberately.
PLUTO_COMMIT: dc462163c8b4fc97d378a4d245d1a64741cb4111
# Job-level so every pytest here collects; --cov-report= suppresses the per-job
# render (the combine job owns the only report).
PYTEST_ADDOPTS: "--cov=hpcagent_bench --cov-append --cov-report="
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
# ccache -- emitted kernels are byte-identical across runs (deterministic emit, no
# embedded paths or timestamps), so every CI run recompiles the very same TUs.
# compilers.yaml invokes bare `gcc`/`g++`/`gfortran`, so putting the shim dir first on
# PATH routes the whole build through the cache without touching any build code.
- name: Setup -- ccache + TBB
run: |
sudo apt-get update -qq
# libtbb-dev is not optional for the cpp_isopar column: libstdc++ picks its parallel
# <execution> backend per TU with __has_include(<tbb/tbb.h>), so with the headers absent
# the par/par_unseq policies compile and run SEQUENTIALLY under a parallel name -- a
# silently wrong measurement, not a build failure. languages.stdpar_link_flags asks the
# compiler that same question before appending -ltbb, so installing it here is what
# actually turns the column parallel.
sudo apt-get install -y --no-install-recommends ccache libtbb-dev
echo "/usr/lib/ccache" >> "$GITHUB_PATH"
echo "CCACHE_DIR=$HOME/.ccache" >> "$GITHUB_ENV"
CCACHE_DIR="$HOME/.ccache" ccache --set-config max_size=1G
# The CPU baselines compile -march=native and ccache hashes that flag literally,
# so an object cached on a wider-ISA runner runs here as SIGILL (exit 132). Key
# the cache on the ISA the compiler would actually select.
cpu_key=$(awk -F': ' '/^(model name|flags)/ {print $2}' /proc/cpuinfo |
head -2 | sha256sum | cut -c1-12)
# An empty key would silently restore the old cross-runner behaviour.
test -n "$cpu_key" || { echo "could not derive a CPU key" >&2; exit 1; }
echo "CPU_KEY=$cpu_key" >> "$GITHUB_ENV"
- name: Setup -- restore the compile cache
uses: actions/cache@v4
with:
path: ~/.ccache
key: ccache-${{ github.job }}-${{ env.CPU_KEY }}-${{ github.run_id }}
restore-keys: ccache-${{ github.job }}-${{ env.CPU_KEY }}-
# Zero AFTER the restore: the cache carries its own counters back, so zeroing earlier
# would make the end-of-job stats cumulative instead of this run's hit rate.
- name: Setup -- reset the ccache counters
run: ccache --zero-stats
- uses: ./.github/actions/setup
- name: Setup -- apache-tvm (tvm_cpu framework)
# apache-tvm's libtvm_runtime.so resolves tvm::ffi::* out of the SEPARATE apache-tvm-ffi
# wheel. Installing only apache-tvm loads to
# "undefined symbol: _ZN3tvm3ffi4json9StringifyE...", which the suite then reports as
# "tvm import failed" and SKIPS -- a broken install wearing an absent one's clothes.
# Both pinned to the pair this repo develops against.
run: python -m pip install apache-tvm==0.25.0rc0 apache-tvm-ffi==0.1.12
- name: Setup -- build Pluto (polyhedral source-to-source) against LLVM/clang 17
# HARD step (proven recipe -- containers/pluto.Dockerfile builds the same way): a build
# regression fails the job loudly, not silently.
run: |
set -euo pipefail
# clang-17 feeds pet via --with-clang-prefix; libgmp/flex/bison/texinfo/libltdl
# build isl/cloog/candl; pkg-config + libyaml-dev are pet's configure deps.
sudo apt-get install -y --no-install-recommends \
clang-17 llvm-17 llvm-17-dev libclang-17-dev \
autoconf automake libtool pkg-config libgmp-dev libyaml-dev flex bison texinfo libltdl-dev
# pet prefers the monolithic -lclang-cpp, but Ubuntu ships only libclang-cpp.so.17
# (no unversioned .so), so pet falls back to individual -lclang* libs and misses
# -lclangASTMatchers. Add the symlink so pet links the one monolithic lib (all
# clang symbols, ast_matchers included).
sudo ln -sf /usr/lib/llvm-17/lib/libclang-cpp.so.17 /usr/lib/llvm-17/lib/libclang-cpp.so
# cloog unconditionally builds doc/cloog.pdf via texi2dvi (wants a full TeX we do
# not need); shadow texi2dvi with a no-op that just creates its -o target.
printf '%s\n' '#!/bin/sh' \
'out=; while [ $# -gt 0 ]; do [ "$1" = "-o" ] && { out=$2; shift; }; shift; done' \
'[ -n "$out" ] && : > "$out"; exit 0' | sudo tee /usr/local/bin/texi2dvi >/dev/null
sudo chmod +x /usr/local/bin/texi2dvi
git clone --recursive https://github.com/bondhugula/pluto.git "$RUNNER_TEMP/pluto"
cd "$RUNNER_TEMP/pluto"
git checkout "$PLUTO_COMMIT"
git submodule update --init --recursive
./autogen.sh
./configure --with-clang-prefix=/usr/lib/llvm-17 \
CC=clang-17 CXX=clang++-17 CXXFLAGS='-std=c++17 -include cstdint'
make -j"$(nproc)"
sudo make install # polycc + pluto -> /usr/local/bin
sudo ldconfig
# Smoke-test the built polycc on a trivial affine scop -- proves it RUNS, not just
# that it installed. (`polycc --version` forwards to the pluto binary, which exits
# non-zero on "no input file", so it is NOT a usable check under set -e.) Multidim
# array so pet extracts the SCoP; output must be non-empty.
printf '%s\n' \
'#include <stdint.h>' \
'void mm(const int64_t N, double (*restrict A)[N], double (*restrict B)[N], double (*restrict C)[N]) {' \
'#pragma scop' \
' for (int64_t i=0;i<N;i++) for (int64_t j=0;j<N;j++) for (int64_t k=0;k<N;k++) C[i][j]+=A[i][k]*B[k][j];' \
'#pragma endscop' \
'}' > "$RUNNER_TEMP/smoke.c"
( cd "$RUNNER_TEMP" && polycc --pet smoke.c -o smoke_out.c && test -s smoke_out.c )
echo "pluto OK -- polycc built + transforms a scop: $(command -v polycc)"
- name: Phase 4 -- codegen frameworks + fp16 (build + run + validate a kernel)
# cc/llvm/tvm_cpu/pluto and fp16-via-jax run here; polly (needs a Polly-enabled clang)
# and the GPU frameworks self-skip when their toolchain/GPU is absent. -rfEs surfaces skips.
if: ${{ !cancelled() }}
env:
HPCAGENT_BENCH_TVM_NOTUNE: "1"
HPCAGENT_BENCH_TVM_METASCHEDULE_TRIALS: "2"
run: |
python -m pytest -q -p no:cacheprovider -rfEs --timeout=900 \
tests/test_frameworks.py \
tests/test_fp16.py \
tests/test_default_datatype_tolerance.py
- name: Phase 5 -- e2e numerical sweep [pluto] @ S
if: ${{ !cancelled() }}
env:
HPCAGENT_BENCH_E2E_BACKENDS: "pluto"
run: |
python -m pytest -q -p no:cacheprovider -rfEs -n auto tests/test_e2e_numerical.py
# Phase 5d -- the optimization-report e2e. HERE and nowhere else: this is the ONLY runner that
# builds Pluto, and the test needs polycc AND dace in one process (dace comes from the shared
# setup, so this job is the only one that has both). Two kernels at preset S, single node, no
# MPI, no container. -rfEs so a toolchain skip is visible rather than a silent pass -- the test's
# own skip predicates are explicit (polycc on PATH, dace importable), never a bare except.
- name: Phase 5d -- opt-report + disassembly e2e [dace, pluto] @ S
if: ${{ !cancelled() }}
timeout-minutes: 25
run: |
python -m pytest -q -p no:cacheprovider -rfEs --timeout=1200 tests/test_opt_reports_e2e.py
# Hit rate is the whole point of the cache -- surface it so a regression to 0% is visible.
- name: ccache statistics
if: always()
run: ccache --show-stats
# ===========================================================================================
# e2e-native -- the c/cpp/fortran slice of the e2e sweep (heavy-compile, but bounded). Split off
# from pythran: ubuntu-latest is 4 vCPU / 16 GB and the old combined job died at exit 143 ("runner
# has received a shutdown signal" -- the runner VM was reclaimed). c/cpp/fortran compiles each peak
# at a few hundred MB (vs a single pythran compile at ~7 GB / 11 min), so with pythran gone this
# slice runs at full -n auto (4) well under 16 GB. tests/test_e2e_numerical.py reads
# HPCAGENT_BENCH_E2E_BACKENDS and run_kernel(only_backends=...) emits ONLY the requested backends, so a
# job that omits pythran never invokes a pythran compile.
# ===========================================================================================
# if: always() -- a job that went red still covered lines on the way, and the total is more
# honest with them than without. if-no-files-found: ignore for a job that failed before pytest.
- name: Upload coverage data
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-frameworks-pluto
path: .coverage*
include-hidden-files: true
if-no-files-found: ignore
e2e-native:
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-ci') }}
name: e2e[c,cpp,fortran] + integration sweep
runs-on: ubuntu-latest
timeout-minutes: 75
env:
# Job-level so every pytest here collects; --cov-report= suppresses the per-job
# render (the combine job owns the only report).
PYTEST_ADDOPTS: "--cov=hpcagent_bench --cov-append --cov-report="
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
# ccache -- emitted kernels are byte-identical across runs (deterministic emit, no
# embedded paths or timestamps), so every CI run recompiles the very same TUs.
# compilers.yaml invokes bare `gcc`/`g++`/`gfortran`, so putting the shim dir first on
# PATH routes the whole build through the cache without touching any build code.
- name: Setup -- ccache + TBB
run: |
sudo apt-get update -qq
# libtbb-dev is not optional for the cpp_isopar column: libstdc++ picks its parallel
# <execution> backend per TU with __has_include(<tbb/tbb.h>), so with the headers absent
# the par/par_unseq policies compile and run SEQUENTIALLY under a parallel name -- a
# silently wrong measurement, not a build failure. languages.stdpar_link_flags asks the
# compiler that same question before appending -ltbb, so installing it here is what
# actually turns the column parallel.
sudo apt-get install -y --no-install-recommends ccache libtbb-dev
echo "/usr/lib/ccache" >> "$GITHUB_PATH"
echo "CCACHE_DIR=$HOME/.ccache" >> "$GITHUB_ENV"
CCACHE_DIR="$HOME/.ccache" ccache --set-config max_size=1G
# The CPU baselines compile -march=native and ccache hashes that flag literally,
# so an object cached on a wider-ISA runner runs here as SIGILL (exit 132). Key
# the cache on the ISA the compiler would actually select.
cpu_key=$(awk -F': ' '/^(model name|flags)/ {print $2}' /proc/cpuinfo |
head -2 | sha256sum | cut -c1-12)
# An empty key would silently restore the old cross-runner behaviour.
test -n "$cpu_key" || { echo "could not derive a CPU key" >&2; exit 1; }
echo "CPU_KEY=$cpu_key" >> "$GITHUB_ENV"
- name: Setup -- restore the compile cache
uses: actions/cache@v4
with:
path: ~/.ccache
key: ccache-${{ github.job }}-${{ env.CPU_KEY }}-${{ github.run_id }}
restore-keys: ccache-${{ github.job }}-${{ env.CPU_KEY }}-
# Zero AFTER the restore: the cache carries its own counters back, so zeroing earlier
# would make the end-of-job stats cumulative instead of this run's hit rate.
- name: Setup -- reset the ccache counters
run: ccache --zero-stats
- uses: ./.github/actions/setup
- name: Phase 4b -- integration sweep (CLI -> DB -> plot, numpy + native autopar)
# Lands HERE, on the lightest native runner (~14 min of its 75), because this is the
# job that already owns the auto-generated native compile+validate path -- the same
# chain the sweep drives end to end -- and clang (the polly autopar leg) comes from
# the shared setup action, so the job pays no new setup. unit (23.5) and e2e-pythran
# (30.9, at budget) have no room; translators has no native toolchain rationale.
# ~25 s serial: 18 numpy kernels (~5 s) + 3 emitted+compiled polly kernels + one plot.
# -n1 (no xdist): the module-scoped fixture drives BOTH sweeps into ONE hpcagent_bench.db,
# and splitting its tests across workers would re-run the whole fixture per worker.
if: ${{ !cancelled() }}
timeout-minutes: 15
run: |
python -m pytest -q -p no:cacheprovider -rfEs --timeout=1800 tests/test_integration_sweep.py
- name: Phase 5 -- e2e numerical sweep [c, cpp, fortran] @ S, fp64
if: ${{ !cancelled() }}
# pythran removed -> peak a few GB -> full -n auto (4) safe under 16 GB.
timeout-minutes: 60
env:
HPCAGENT_BENCH_E2E_BACKENDS: "c,cpp,fortran"
run: |
python -m pytest -q -p no:cacheprovider -rfEs -n auto --timeout=600 tests/test_e2e_numerical.py
- name: Phase 5b -- e2e numerical sweep [c, cpp, fortran] @ S, fp32
# The SAME corpus at fp32. Not redundant with the fp64 phase above: fp64 emits with an empty
# precision string, which makes numpyto_common.ir.apply_precision a no-op, so the phase above
# never exercises the precision-lowering path at ALL -- the float dtype remap over arrays /
# scalars / locals, KernelIR.float_precision (the emitter's default for a temp missing from
# local_dtypes), and each backend's narrow-float spelling (C ``float``, Fortran ``real(4)``,
# whose kind matching -std=f2018 enforces strictly). This runner has the headroom: the fp64
# phase is ~8 min of the job's 75.
# fp32 ONLY: fp16 keeps ~3 decimal digits, so a whole-corpus fp16 sweep would fail on honest
# numeric drift rather than emit bugs -- fp16 stays a targeted, fp16-safe kernel list in
# tests/test_fp16.py. Verified green over all 349 kernels x c/cpp/fortran before gating.
if: ${{ !cancelled() }}
timeout-minutes: 60
env:
HPCAGENT_BENCH_E2E_BACKENDS: "c,cpp,fortran"
HPCAGENT_BENCH_E2E_PRECISION: "fp32"
run: |
python -m pytest -q -p no:cacheprovider -rfEs -n auto --timeout=600 tests/test_e2e_numerical.py
# The KernelBench subtrack is EXCLUDED from the sweep above (test_e2e_numerical's
# UNGATED_SUBTRACKS), so without this nothing in CI would notice a translator change halving
# what those 250 ports lower to. It asserts a floor on the COUNT, not per kernel -- so the
# floor being STALE is the same blind spot: it sat at 121 while 192 actually lowered, leaving
# 71 kernels free to regress green. Raise it whenever a measurement says it moved.
- name: Phase 5c -- kernelbench translation ratchet [c] @ S
if: ${{ !cancelled() }}
timeout-minutes: 45
run: |
python -m pytest -q -p no:cacheprovider -rfEs --timeout=2400 \
tests/test_kernelbench_translation.py
# Hit rate is the whole point of the cache -- surface it so a regression to 0% is visible.
- name: ccache statistics
if: always()
run: ccache --show-stats
# ===========================================================================================
# e2e-pythran -- the pythran slice of the e2e sweep, isolated because pythran is the memory/time
# pig: a single pythran compile peaks ~7 GB / 11 min while c/cpp/fortran compiles are hundreds of
# MB. Folding it into the native job overran the 16 GB runner (exit 143). Same partition mechanism
# as e2e-native: HPCAGENT_BENCH_E2E_BACKENDS="pythran" so run_kernel(only_backends=...) emits pythran only.
# ===========================================================================================
# if: always() -- a job that went red still covered lines on the way, and the total is more
# honest with them than without. if-no-files-found: ignore for a job that failed before pytest.
- name: Upload coverage data
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-e2e-native
path: .coverage*
include-hidden-files: true
if-no-files-found: ignore
e2e-pythran:
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-ci') }}
name: e2e[pythran]
runs-on: ubuntu-latest
timeout-minutes: 75
env:
# Job-level so every pytest here collects; --cov-report= suppresses the per-job
# render (the combine job owns the only report).
PYTEST_ADDOPTS: "--cov=hpcagent_bench --cov-append --cov-report="
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
# ccache -- emitted kernels are byte-identical across runs (deterministic emit, no
# embedded paths or timestamps), so every CI run recompiles the very same TUs.
# compilers.yaml invokes bare `gcc`/`g++`/`gfortran`, so putting the shim dir first on
# PATH routes the whole build through the cache without touching any build code.
- name: Setup -- ccache + TBB
run: |
sudo apt-get update -qq
# libtbb-dev is not optional for the cpp_isopar column: libstdc++ picks its parallel
# <execution> backend per TU with __has_include(<tbb/tbb.h>), so with the headers absent
# the par/par_unseq policies compile and run SEQUENTIALLY under a parallel name -- a
# silently wrong measurement, not a build failure. languages.stdpar_link_flags asks the
# compiler that same question before appending -ltbb, so installing it here is what
# actually turns the column parallel.
sudo apt-get install -y --no-install-recommends ccache libtbb-dev
echo "/usr/lib/ccache" >> "$GITHUB_PATH"
echo "CCACHE_DIR=$HOME/.ccache" >> "$GITHUB_ENV"
CCACHE_DIR="$HOME/.ccache" ccache --set-config max_size=1G
# The CPU baselines compile -march=native and ccache hashes that flag literally,
# so an object cached on a wider-ISA runner runs here as SIGILL (exit 132). Key
# the cache on the ISA the compiler would actually select.
cpu_key=$(awk -F': ' '/^(model name|flags)/ {print $2}' /proc/cpuinfo |
head -2 | sha256sum | cut -c1-12)
# An empty key would silently restore the old cross-runner behaviour.
test -n "$cpu_key" || { echo "could not derive a CPU key" >&2; exit 1; }
echo "CPU_KEY=$cpu_key" >> "$GITHUB_ENV"
- name: Setup -- restore the compile cache
uses: actions/cache@v4
with:
path: ~/.ccache
key: ccache-${{ github.job }}-${{ env.CPU_KEY }}-${{ github.run_id }}
restore-keys: ccache-${{ github.job }}-${{ env.CPU_KEY }}-
# Zero AFTER the restore: the cache carries its own counters back, so zeroing earlier
# would make the end-of-job stats cumulative instead of this run's hit rate.
- name: Setup -- reset the ccache counters
run: ccache --zero-stats
- uses: ./.github/actions/setup
- name: Phase 5 -- e2e numerical sweep [pythran] @ S
if: ${{ !cancelled() }}
# -n 1: pythran ALONE at -n 2 STILL got the runner VM reclaimed ("runner has received a
# shutdown signal", exit 143) -- two concurrent ~7 GB template instantiations exhaust the
# 16 GB box, so isolating pythran was necessary but not sufficient. One compile at a time,
# paired with the oracle's COMPILE_MEMORY_CAP_GB rlimit on the compiler, keeps a runaway
# compile a per-kernel skip:unsupported:compile instead of a dead runner. One backend at
# -n 1 still fits the cap (the four-backend job did ~1000 items at -n 2 in under 4 min).
timeout-minutes: 60
env:
HPCAGENT_BENCH_E2E_BACKENDS: "pythran"
run: |
python -m pytest -q -p no:cacheprovider -rfEs -n 1 --timeout=600 tests/test_e2e_numerical.py
# Hit rate is the whole point of the cache -- surface it so a regression to 0% is visible.
- name: ccache statistics
if: always()
run: ccache --show-stats
# ===========================================================================================
# container-image -- build + smoke-launch the agent-bench CPU container image (docker build ->
# docker save -> apptainer build -> exec). Split off from the sweep so the heavy docker+apptainer
# build neither contends for the pythran runner's RAM nor serializes behind the native sweep.
# ===========================================================================================
# if: always() -- a job that went red still covered lines on the way, and the total is more
# honest with them than without. if-no-files-found: ignore for a job that failed before pytest.
- name: Upload coverage data
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-e2e-pythran
path: .coverage*
include-hidden-files: true
if-no-files-found: ignore
container-image:
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-ci') }}
name: agent-bench container image (build + launch)
runs-on: ubuntu-latest
timeout-minutes: 120
env:
# Job-level so every pytest here collects; --cov-report= suppresses the per-job
# render (the combine job owns the only report).
PYTEST_ADDOPTS: "--cov=hpcagent_bench --cov-append --cov-report="
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- uses: ./.github/actions/setup
- name: Setup -- Apptainer (sudoless install script)
run: |
# Install apptainer WITHOUT root, via its unprivileged install script -- the same rootless
# path our HPC targets (no sudo) use -- exercised through our shipped
# `hpcagent-bench-install-apptainer` wrapper (hpcagent_bench/containers.py). rpm2cpio + cpio: the
# installer unpacks an rpm. uidmap: newuidmap/newgidmap that --fakeroot uses to map this
# user's subuid range.
sudo apt-get install -y rpm2cpio cpio uidmap
hpcagent-bench-install-apptainer "$HOME/apptainer"
echo "$HOME/apptainer/bin" >> "$GITHUB_PATH"
# Sudoless apptainer needs unprivileged user namespaces for BOTH --fakeroot build and
# plain exec. Where the host already permits them -- local dev, HPC, a permissive runner --
# nothing below runs sudo. Only a *clamped* runner (ubuntu-24.04's AppArmor default) needs
# the one-time host knob, so probe first and flip it solely when blocked.
if unshare --user --map-root-user true 2>/dev/null; then
echo "unprivileged userns already permitted -- no host change, fully sudoless"
else
echo "unprivileged userns blocked on this runner -- enabling for the job VM (provisioning only)"
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
fi
- name: Phase 6a -- build + smoke-test the cpu agent-bench image (unified OCI -> SIF, gating)
# GATES now. UNIFIED build: one OCI Dockerfile (containers/hpcagent_bench.Dockerfile, HW=cpu) is the
# single recipe; apptainer builds the SIF FROM that OCI image, no .def, no --fakeroot (the
# conversion runs no %post). We go through a `docker save` TAR + `docker-archive:` rather than
# `docker-daemon://` so apptainer never needs Docker-daemon/socket access -- the archive path is
# daemon-agnostic and the robust way to hand a locally-built OCI image to apptainer.
# UNVERIFIED end-to-end on the heavy image; the docker-save -> docker-archive -> SIF -> exec
# MECHANISM is validated locally with a minimal image. Revert = `apptainer build --fakeroot
# hpcagent_bench-cpu.sif containers/cpu.def`.
if: ${{ !cancelled() }}
run: |
# Reclaim disk FIRST. The unified image (torch + dace + jax + the HPC stack) is multi-GB
# and this step keeps THREE copies live at once -- the docker layers, the `docker save`
# tar, and the converted SIF -- while a stock runner has only ~14 GB free. Without this
# the build itself succeeds and `docker save` dies on "no space left on device". These
# preinstalled toolchains are never used by this job.
sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc /usr/local/.ghcup || true
df -h / | tail -1
docker build -f containers/hpcagent_bench.Dockerfile --build-arg HW=cpu -t hpcagent_bench:cpu .
docker save hpcagent_bench:cpu -o hpcagent_bench-cpu.tar
# The docker layers are dead once the tar exists; drop them before the SIF (a third copy).
docker image rm hpcagent_bench:cpu >/dev/null 2>&1 || true
docker builder prune -af >/dev/null 2>&1 || true
apptainer build hpcagent_bench-cpu.sif docker-archive:hpcagent_bench-cpu.tar
# ...and the tar is dead once the SIF exists; Phase 6b only needs the SIF.
rm -f hpcagent_bench-cpu.tar
df -h / | tail -1
# The cpu image must import its numeric deps and carry the CPU-only torch build: a
# bare ``torch`` would drag in the ~2 GB CUDA stack (see cpu.def). Assert the ``+cpu``
# local version.
apptainer exec hpcagent_bench-cpu.sif python3 -c "import numpy, scipy, dace, jax, numba, pythran, xgboost, h5py, netCDF4; import torch; assert '+cpu' in torch.__version__, 'cpu image has non-CPU torch: ' + torch.__version__; print('cpu image OK: deps import, torch', torch.__version__)"
# The agent-linkable numeric libs (containers/LIBRARIES.md) must be present: HPTT is
# SOURCE-built (build-hptt.sh, the fragile step -- guard header + .so), the rest are apt.
apptainer exec hpcagent_bench-cpu.sif sh -c 'set -e; test -e /usr/local/include/hptt.h; test -e /usr/local/lib/libhptt.so; for l in libhwy libsleef libhwloc liblapacke; do ldconfig -p | grep -q "$l" || { echo "missing $l"; exit 1; }; done; echo "cpu image native libs OK: hptt (src) + hwy + sleef + hwloc + lapacke"; for b in perf numactl heaptrack likwid-topology papi_avail strace objdump; do command -v $b >/dev/null 2>&1 || { echo "missing $b"; exit 1; }; done; echo "cpu image perf tools OK: perf numactl heaptrack likwid papi strace objdump";'
- name: Phase 6b -- container functionality (sudoless two-container launch, gating)
# GATES now. The launch is sudoless -- unprivileged userns supplies the container isolation,
# no sudo -- so a broken judge<->agent path reds the build instead of being swallowed.
if: ${{ !cancelled() }}
env:
HPCAGENT_BENCH_JUDGE_SIF: ${{ github.workspace }}/hpcagent_bench-cpu.sif
run: |
python -m pytest -q -p no:cacheprovider -rfEs --timeout=900 tests/test_container_launch.py
# ===========================================================================================
# mpi -- the whole MPI / distributed-residency track on its own runner (the "5th runner"): the
# pure-numpy data-distribution + descriptor tests, the C-driver / stub codegen, the ABI + wire
# checks, the oversubscribed multi-rank launch + scoring e2e, the halo exchange, and the sizing
# (strong/weak) tests. MPICH is the track default (ABI-compatible with cray-mpich for host
# replacement on Ault; only libfabric/CXI injected on Alps), and mpi4py is source-built against
# it. The launch/e2e tests ``mpirun --oversubscribe`` on the CI cores and self-skip cleanly when
# MPI_Init cannot bootstrap; the GPU-device e2e tests self-skip with no CUDA/nvcc.
#
# Fused here 2026-08-11 (see the top-of-file note for the measurement): dace-numeric (the
# generated DaCe corpus vs numpy, a C++ build per kernel) and hf-export (build, and on a main
# push publish, the HuggingFace dataset) run as trailing phases on this same runner. `needs:
# [unit, integration]` is hf-export's own gate, carried over unchanged in effect: it decides the
# PUBLISH sub-step only -- the mpi and dace-numeric phases above run regardless, on `!cancelled()`
# like every other phase here, same as they did as their own ungated jobs.
# ===========================================================================================
# if: always() -- a job that went red still covered lines on the way, and the total is more
# honest with them than without. if-no-files-found: ignore for a job that failed before pytest.
- name: Upload coverage data
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-container-image
path: .coverage*
include-hidden-files: true
if-no-files-found: ignore
mpi:
# !cancelled(): the mpi/dace-numeric phases below must still run and report even when unit or
# integration went red (they did before this fusion, as independent jobs with no needs at all).
# needs: [unit, integration]: schedules this job only after both finish -- unavoidable once
# their `result` has to gate the hf-export PUBLISH sub-step below -- and is the same wait
# hf-export's own job already paid as its own job.
if: ${{ !cancelled() && !contains(github.event.pull_request.labels.*.name, 'no-ci') }}
needs: [unit, integration]
name: mpi + dace-numeric + hf-export (distributed track, DaCe-vs-numpy corpus, HF dataset)
runs-on: ubuntu-latest
# ~11m51s (mpi) + ~16m18s (dace-numeric) + ~2m37s (hf-export), measured on runs 31478271044 /
# 31023526869 -- ~31m sequential. 75 leaves more than 2x that, with room above the two explicit
# phase caps below (30 + 30) on a bad day for either.
timeout-minutes: 75
env:
# OpenMPI on a GH-Actions runner has no high-speed fabric (no InfiniBand / UCX device), so pin
# the simple self + shared-memory transports and disable UCX's VFS probe -- otherwise MPI_Init
# selects a missing transport and the REAL multi-rank launches (Phase 3) fail or hang. The
# rmaps oversubscribe knob lets R ranks share the 4 CI cores (belt-and-suspenders with the
# launcher's mpirun --oversubscribe).
OMPI_MCA_pml: ob1
OMPI_MCA_btl: self,vader
OMPI_MCA_rmaps_base_oversubscribe: "1"
UCX_VFS_ENABLE: "n"
# Job-level so every pytest here collects; --cov-report= suppresses the per-job
# render (the combine job owns the only report). hf-export's phase runs no pytest, so this
# is simply unused there, same as it already is during every non-pytest setup step above.
PYTEST_ADDOPTS: "--cov=hpcagent_bench --cov-append --cov-report="
steps:
- uses: actions/checkout@v4
with:
# hf-export's phase needs this (KernelBench provenance); harmless extra for the mpi and
# dace-numeric phases, which never checked submodules out on their own.
submodules: recursive
- uses: actions/setup-python@v5
with:
python-version: '3.12'
# ccache -- emitted kernels are byte-identical across runs (deterministic emit, no
# embedded paths or timestamps), so every CI run recompiles the very same TUs.
# compilers.yaml invokes bare `gcc`/`g++`/`gfortran`, so putting the shim dir first on
# PATH routes the whole build through the cache without touching any build code.
- name: Setup -- ccache + TBB
run: |
sudo apt-get update -qq
# libtbb-dev is not optional for the cpp_isopar column: libstdc++ picks its parallel
# <execution> backend per TU with __has_include(<tbb/tbb.h>), so with the headers absent
# the par/par_unseq policies compile and run SEQUENTIALLY under a parallel name -- a
# silently wrong measurement, not a build failure. languages.stdpar_link_flags asks the
# compiler that same question before appending -ltbb, so installing it here is what
# actually turns the column parallel.
sudo apt-get install -y --no-install-recommends ccache libtbb-dev
echo "/usr/lib/ccache" >> "$GITHUB_PATH"
echo "CCACHE_DIR=$HOME/.ccache" >> "$GITHUB_ENV"
CCACHE_DIR="$HOME/.ccache" ccache --set-config max_size=1G
# The CPU baselines compile -march=native and ccache hashes that flag literally,
# so an object cached on a wider-ISA runner runs here as SIGILL (exit 132). Key
# the cache on the ISA the compiler would actually select.
cpu_key=$(awk -F': ' '/^(model name|flags)/ {print $2}' /proc/cpuinfo |
head -2 | sha256sum | cut -c1-12)
# An empty key would silently restore the old cross-runner behaviour.
test -n "$cpu_key" || { echo "could not derive a CPU key" >&2; exit 1; }
echo "CPU_KEY=$cpu_key" >> "$GITHUB_ENV"
- name: Setup -- restore the compile cache
uses: actions/cache@v4
with:
path: ~/.ccache
key: ccache-${{ github.job }}-${{ env.CPU_KEY }}-${{ github.run_id }}
restore-keys: ccache-${{ github.job }}-${{ env.CPU_KEY }}-
# Zero AFTER the restore: the cache carries its own counters back, so zeroing earlier
# would make the end-of-job stats cumulative instead of this run's hit rate.
- name: Setup -- reset the ccache counters
run: ccache --zero-stats
- uses: ./.github/actions/setup
- name: Setup -- OpenMPI + a source-built mpi4py (real multi-rank on the CI runner)
# OpenMPI, not MPICH: MPICH's Hydra bootstraps as SINGLETON worlds on a GH-Actions runner
# (PMI/hwloc can't come up), so `mpiexec -n 4` yields four world-size-1 jobs and every
# multi-rank launch/scoring/correctness test could only self-skip or fail. OpenMPI's
# `mpirun --oversubscribe -n R` launches R REAL ranks on the 2 CI cores, so those tests
# actually RUN and pass -- no capability-gap skip. (cray-mpich ABI parity is for the Alps
# deployment, orthogonal to CI test correctness.) libopenmpi-dev ships mpicc + mpirun;
# python3-dev + --no-binary source-build mpi4py against THIS OpenMPI so its ABI matches the
# mpicc-built C driver.
run: |
sudo apt-get update && sudo apt-get install -y --no-install-recommends openmpi-bin libopenmpi-dev python3-dev
MPICC=mpicc python -m pip install --no-binary=mpi4py mpi4py
- name: Phase 1 -- data distribution + descriptor (pure numpy, no launcher)
if: ${{ !cancelled() }}
run: |
python -m pytest -q -p no:cacheprovider -rfEs --timeout=600 \
tests/test_mpi_descriptor.py \
tests/test_mpi_scatter_gather_roundtrip.py \
tests/test_mpi_wire.py \
tests/test_mpi_sizing.py \
tests/test_mpi_prompt.py
- name: Phase 2 -- driver / stub codegen + ABI shape (mpicc.mpich -c compile checks)
if: ${{ !cancelled() }}
run: |
python -m pytest -q -p no:cacheprovider -rfEs --timeout=600 \
tests/test_mpi_driver_codegen.py \
tests/test_mpi_halo.py
- name: Phase 3 -- oversubscribed multi-rank launch + scoring + correctness e2e
# Real MPI launches: run WITHOUT pytest-xdist so R oversubscribed ranks per test do not
# also contend with parallel test workers on the few CI cores. --timeout is the per-test
# backstop; the launch helpers self-skip when no MPI bootstraps.
if: ${{ !cancelled() }}
timeout-minutes: 30
run: |
python -m pytest -q -p no:cacheprovider -rfEs --timeout=900 \
tests/test_mpi_call.py \
tests/test_mpi_drivers_launch.py \
tests/test_mpi_correctness_oracle.py \
tests/test_mpi_scoring.py \
tests/test_noop_mpi_optimizer.py
# Moved off the translators runner, which was the workflow's critical path at 40m32s while
# this one finished in 5m09s. Neither phase needs MPI; they need a runner with headroom.
- name: Setup -- gt4py (Phase 2c fv3 reference DSL)
if: ${{ !cancelled() }}
run: python -m pip install gt4py
- name: Phase 2b -- port fidelity (numpy port vs original C/C++/Fortran reference)
if: ${{ !cancelled() }}
run: |
python -m pytest -q -p no:cacheprovider -rfEs --timeout=600 tests/ports/
- name: Phase 2c -- benchmark reference validation (numpy vs naive loop / GT4Py DSL / physics)
if: ${{ !cancelled() }}
env:
# NO COVERAGE on this phase, deliberately. Every file it measures lives under
# hpcagent_bench/benchmarks/, which [tool.coverage.run] omit excludes from the report --
# so instrumenting it buys exactly nothing and costs the job.
#
# `omit` stops LINE tracing, not the per-call dispatch: sys.settrace fires on every call
# event even for files it will not record. This phase is call-dominated (the cloudsc
# branch test alone makes 4.4M calls), so it pays that dispatch 4.4M times for data that
# is then discarded. Measured locally: 8.28 s bare against >1500 s instrumented -- killed
# at 1500 s without finishing, so >181x and a floor, not a figure. In CI the same
# 745 tests went from 183.57 s to 736 s once coverage landed, pushing the heaviest test
# past --timeout=600 -- which is why this job has been red for three consecutive runs.
#
# COVERAGE_CORE=sysmon is NOT the way out here: coverage 7.13.5 refuses it whenever
# `branch = true` on Python < 3.14 ("sys.monitoring can't measure branches in this
# version") and again for concurrency=, then warns and silently falls back to the C
# tracer. Measured, not assumed: COVERAGE_CORE=sysmon on that same test ran 1500 s and was
# killed -- identical to the unset run. It reads as a fix and changes nothing.
PYTEST_ADDOPTS: ""
run: |
# Discover the whole tree rather than listing files: the xsbench / gromacs / lavamd
# reference suites sat outside an explicit list here and so never ran in CI at all.
python -m pytest -q -p no:cacheprovider -rfEs --import-mode=importlib -n auto --timeout=600 \
hpcagent_bench/benchmarks/
# --- dace-numeric phase (fused 2026-08-11; formerly its own job -- see the banner above) ---
- name: The generated DaCe corpus still computes what numpy computes @ S
if: ${{ !cancelled() }}
# One subprocess per kernel: DaCe's parse state is process-global, and a kernel whose
# frontend wedges or whose generated code segfaults must cost that kernel a verdict rather
# than take the sweep with it.
#
# -n 4, not auto: every worker forks a C++ build, so the limit is memory rather than cores.
# Each worker gets its own per-kernel build folder (keys are unique) and they SHARE one
# build cache -- measured, a cold cmake configure is 5.06s against 0.72s warm, which is the
# difference between a ten-minute job and a forty-minute one.
#
# --maxfail=20 so a translator-wide break reports twenty named kernels, not three hundred.
timeout-minutes: 30
env:
# NOT /tmp: it is tmpfs, and exhausting it fails builds in a way that reads as a kernel defect.
HPCAGENT_BENCH_DACE_BUILD_ROOT: ${{ runner.temp }}/dace_numeric
run: |
rm -rf "$HPCAGENT_BENCH_DACE_BUILD_ROOT"
# Regenerate ONCE, before xdist starts. *_dace.py is gitignored, so collection emits what
# a fresh checkout lacks -- and every worker would do that independently, four processes
# writing the same files through one fingerprint cache. Doing it here leaves the workers'
# own autogen.ensure a pure cache hit, so there is no race to reason about.
python -c "from tests.test_dace_frontend_validity import generated_programs; \
print(len(generated_programs()), 'dace programs ready')"
python -m pytest -q -p no:cacheprovider -rfEs -n 4 -m dace_numeric \
--maxfail=20 --timeout=900 tests/test_dace_numeric_agreement.py
# Hit rate is the whole point of the cache -- surface it so a regression to 0% is visible.
# One combined step now: it reports the mpi phases' + dace-numeric's compiles together, same
# ~/.ccache dir either way, since both used to print it only for their own half.
- name: ccache statistics
if: always()
run: ccache --show-stats
- name: Upload the generated DaCe corpus
# Same reason as port-fidelity's copy: the corpus this runner emitted is the only way to tell
# a generator difference from an environment difference when a kernel is red here and green
# locally at the same dace SHA.
if: failure()
uses: actions/upload-artifact@v4
with:
name: dace-corpus-numeric
path: hpcagent_bench/benchmarks/**/*_dace.py
retention-days: 7
if-no-files-found: ignore
# --- hf-export phase (fused 2026-08-11; formerly its own job -- see the banner above) ---
- name: Phase 7 -- export (+ publish on a main push) the HuggingFace dataset
# export-hf always writes the parquet artifact -- unconditionally, like every other phase in
# this job, on `!cancelled()` rather than on the mpi/dace-numeric phases above succeeding.
# PUBLISH is the one thing that still keeps hf-export's original all-or-nothing gate: it
# fires only on a push to main with HF_TOKEN + HF_DATASET_REPO set AND both `needs` green --
# never publish a dataset from a run where unit or integration went red.
if: ${{ !cancelled() }}
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
HF_DATASET_REPO: ${{ vars.HF_DATASET_REPO }}
run: |
pip install -r requirements/hf.txt
PUSH=""
if [ "${{ github.event_name }}" = "push" ] && [ "${{ github.ref }}" = "refs/heads/main" ] \
&& [ -n "$HF_DATASET_REPO" ] && [ -n "$HF_TOKEN" ] \
&& [ "${{ needs.unit.result }}" = "success" ] && [ "${{ needs.integration.result }}" = "success" ]; then
PUSH="--push $HF_DATASET_REPO"
fi
python -m hpcagent_bench.cli export-hf --selector all --out hpcagent_bench_hf.parquet $PUSH
- name: Phase 7 -- upload the HF dataset as a workflow artifact
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v4
with:
name: hpcagent_bench-hf-dataset
path: hpcagent_bench_hf.parquet
if-no-files-found: ignore
# ===========================================================================================
# gpu -- DISABLED extra runner (blocked). GPU codegen tests (tvm-gpu / triton) belong on a real
# GPU box; this job is the home for them but stays OFF (if: false) until one is provisioned.
# On the CPU runners those cases self-skip (no CUDA / no triton), so nothing is lost meanwhile.
# To enable: flip ``if: false`` to a self-hosted GPU runner label.
# ===========================================================================================
# if: always() -- a job that went red still covered lines on the way, and the total is more
# honest with them than without. if-no-files-found: ignore for a job that failed before pytest.
- name: Upload coverage data
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-mpi
path: .coverage*
include-hidden-files: true
if-no-files-found: ignore
gpu:
if: false
name: gpu (tvm-gpu / triton codegen) [disabled -- no GPU runner]
runs-on: ubuntu-latest
timeout-minutes: 120
env:
# Job-level so every pytest here collects; --cov-report= suppresses the per-job
# render (the combine job owns the only report).
PYTEST_ADDOPTS: "--cov=hpcagent_bench --cov-append --cov-report="
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- uses: ./.github/actions/setup
- name: Setup -- GPU codegen frameworks (tvm + triton)
run: python -m pip install apache-tvm==0.25.0rc0 apache-tvm-ffi==0.1.12 triton
- name: GPU frameworks + fp16 (tvm-gpu / triton targets)
run: |
python -m pytest -q -p no:cacheprovider -rfEs --timeout=900 \
tests/test_frameworks.py tests/test_fp16.py
# if: always() -- a job that went red still covered lines on the way, and the total is more
# honest with them than without. if-no-files-found: ignore for a job that failed before pytest.
- name: Upload coverage data
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-gpu
path: .coverage*
include-hidden-files: true
if-no-files-found: ignore
# ===========================================================================================
# coverage -- the ONE total. Each job above uploads its raw .coverage* data; nothing renders a
# report per job, because eight partial percentages are not a coverage figure and reading them
# as one is the mistake this job exists to prevent.
#
# Runs when a job went RED -- a red run's coverage is the most useful moment to look at it -- but
# not when the run was CANCELLED. always() fires on cancellation too, and then no job has uploaded
# anything, so the combine step hard-errors with "no coverage data was uploaded" and the run page
# shows a coverage FAILURE whose actual cause was somebody pushing again. `coverage combine` needs
# relative_files (pyproject) since the container job's paths differ from the runner's.
# ===========================================================================================
coverage:
if: ${{!cancelled()}}
name: coverage (combined total)
# dace-numeric folded into mpi (2026-08-11); its coverage rides mpi's single combined upload now.
needs: [unit, integration, translators, frameworks-pluto, e2e-native, e2e-pythran, container-image, mpi, port-fidelity]
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install coverage
run: python -m pip install coverage
- name: Download every job's coverage data
uses: actions/download-artifact@v4
with:
pattern: coverage-*
path: coverage-data
# NOT merge-multiple: every job uploads its data as `.coverage`, so flattening them into
# one directory makes seven files race for one path. The winner became the "total" and
# the other six were discarded -- 59.96% on one green run, 13.44% on the next. Here two
# extractions interleaved instead and left a torn SQLite file, which is the only reason
# the defect ever announced itself. One subdirectory per artifact, so no collision.
- name: Combine and report
run: |
# pipefail, because every command below is piped into `tee`: without it the pipeline's
# status is tee's, a torn or malformed coverage database exits 0, and the only check that
# then fires is the file-count one -- which reports a partial merge and sends the reader
# after the wrong cause entirely.
set -o pipefail
shopt -s nullglob dotglob
files=(coverage-data/*/.coverage*)
if [ ${#files[@]} -eq 0 ]; then
echo "::error::no coverage data was uploaded by any job -- the total cannot be computed"
exit 1
fi
coverage combine "${files[@]}" 2>&1 | tee combine.log
# Silent partial combines are what hid the collision through every green run: a total
# built from one job of seven still prints a plausible percentage.
grep -q "Combined ${#files[@]} file" combine.log || {
echo "::error::combine consumed fewer than ${#files[@]} files -- the total is not a total"
exit 1
}
coverage report --precision=2 | tee coverage.txt
coverage xml -o coverage.xml
coverage html -d htmlcov
{
echo '## Total coverage'
echo
echo '```'
tail -n 25 coverage.txt
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- name: Upload the combined report
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-report
path: |
coverage.xml
coverage.txt
htmlcov/
if-no-files-found: warn