[Github][RFC] Add workflow to diff codegen on llvm-test-suite

[lukel97]

A common task when reviewing PRs in the LLVM subproject is checking out the PR locally, building it, running it on some benchmarks e.g. llvm-test-suite, and comparing the codegen against some known version.

The process is fairly laborious so this PR adds a GitHub workflow to automate it. It's triggered by commenting "/test-suite" on a PR. The workflow will kick off, build clang with the head and base of the PR, build the benchmarks in llvm-test-suite for several configurations with each version of clang, compute the diff in the output assembly via the tdiff.py script, and then report back with the diffs in a comment.

Here's an example on my fork where you can see the diff of a codegen change in the RISC-V backend: lukel97#7 (comment)

At the moment it's very simple but could be fleshed out later. Currently it builds llvm-test-suite for handful of common configurations, some cross-compiled:

• -target aarch64-linux-gnu -march=armv9-a -O3
• -target riscv64-linux-gnu -march=rva23u64 -O3
• -target x86_64-linux-gnu -O3

We could eventually extend this to accept arbitrary targets and flags in the comment. It would also be nice to support LTO diffs in future but we will need to add some extra support in llvm-test-suite to extract the asm during the link step.

It also just comments a link to download the codegen diffs, but could eventually also include some output from the ./utils/compare.py script about e.g. changes in code size or statistics. For now, those results are just uploaded as an artifact.

In terms of worker resources, running it on the free GitHub hosted workers is good enough. Building Clang takes a while, over an hour, but building the test-suite only takes around 10 minutes. But we could stick it on something beefier if we wanted the feedback to be faster.

This workflow requires the PR to be mergeable, as it wants to get the diff of the "mergeability" commit that GitHub generates for each PR. That way the diff is always between the latest version of the base branch and the PR, not the base branch at the time the PR was created.

[llvmbot]

@llvm/pr-subscribers-backend-risc-v

@llvm/pr-subscribers-github-workflow

Author: Luke Lau (lukel97)

Changes

A common task when reviewing PRs in the LLVM subproject is checking out the PR locally, building it, running it on some benchmarks e.g. llvm-test-suite, and comparing the codegen against some known version.

The process is fairly laborious so this PR adds a GitHub workflow to automate it. It's triggered by commenting "/test-suite" on a PR. The workflow will kick off, build clang with the head and base of the PR, build the benchmarks in llvm-test-suite for several configurations with each version of clang, compute the diff in the output assembly via the tdiff.py script, and then report back with the diffs in a comment.

Here's an example on my fork where you can see the diff of a codegen change in the RISC-V backend: lukel97#5 (comment)

At the moment it's very simple but could be fleshed out later. Currently it builds llvm-test-suite for handful of common configurations, some cross-compiled:

• -target aarch64-linux-gnu -march=armv9-a -O3
• -target riscv64-linux-gnu -march=rva23u64 -O3
• -target x86_64-linux-gnu -O3

We could eventually extend this to accept arbitrary targets and flags in the comment. It would also be nice to support LTO diffs in future but we will need to add some extra support in llvm-test-suite to extract the asm during the link step.

It also just comments a link to download the codegen diffs, but could eventually also include some output from the ./utils/compare.py script about e.g. changes in code size or statistics. For now, those results are just uploaded as an artifact.

In terms of worker resources, running it on the free GitHub hosted workers is good enough. Building Clang takes a while, over an hour, but building the test-suite only takes around 10 minutes. But we could stick it on something beefier if we wanted the feedback to be faster.

This workflow requires the PR to be mergeable, as it wants to get the diff of the "mergeability" commit that GitHub generates for each PR. That way the diff is always between the latest version of the base branch and the PR, not the base branch at the time the PR was created.

---

Full diff: https://github.com/llvm/llvm-project/pull/190010.diff

1 Files Affected:

• (added) .github/workflows/test-suite.ll (+148)

diff --git a/.github/workflows/test-suite.ll b/.github/workflows/test-suite.ll
new file mode 100644
index 0000000000000..0d9c81fc74083
--- /dev/null
+++ b/.github/workflows/test-suite.ll
@@ -0,0 +1,148 @@
+# When /test-suite is commented on a PR, checks out the PR, builds clang and
+# then the test-suite in several configurations. It then checks out the base of
+# the PR, builds clang and the test-suite again, and then uploads the diff of
+# the codegen.
+
+name: Diff test-suite codegen
+
+on:
+  issue_comment:
+    types:
+      - created
+
+jobs:
+  test-suite:
+    name: Build and diff
+    runs-on: ubuntu-24.04
+    permissions:
+      issues: write
+    if: >-
+      !startswith(github.event.comment.body, '<!--IGNORE-->') &&
+      github.event.issue.pull_request && contains(github.event.comment.body, '/test-suite')
+    steps:
+      - id: get-pr
+        uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
+        with:
+          script: |
+            const { data: pr } = await github.rest.pulls.get({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              pull_number: context.payload.issue.number
+            })
+            if (!pr.mergeable)
+              await github.rest.issues.createComment({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                body: "Can't diff PR, PR isn't mergeable"
+              })
+            return pr
+      - if: ${{ !fromJSON(steps.get-pr.outputs.result).mergeable }}
+        run: exit 1
+      - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
+        with:
+          script: |
+            github.rest.reactions.createForIssueComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              comment_id: context.payload.comment.id,
+              content: '+1'
+            })
+      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+        with:
+          ref: ${{ fromJSON(steps.get-pr.outputs.result).merge_commit_sha }}
+          repository: ${{ fromJSON(steps.get-pr.outputs.result).head.repo.full_name }}
+          fetch-depth: 2
+      - run: |
+          echo "HEAD_SHA=$(git rev-parse HEAD)" >> $GITHUB_ENV
+          echo "BASE_SHA=$(git rev-parse HEAD^)" >> $GITHUB_ENV
+      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+        with:
+          repository: llvm/llvm-test-suite
+          path: llvm-test-suite
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y cmake ninja-build libc6-dev-{arm64,riscv64}-cross libgcc-14-dev-{arm64,riscv64}-cross libstdc++-14-dev-{arm64,riscv64}-cross
+      - name: Configure Clang
+        run: cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD='AArch64;X86;RISCV' -DLLVM_ENABLE_PROJECTS='clang;lld' -DLLVM_APPEND_VC_REV=OFF llvm -GNinja
+      - name: Build Clang @ head
+        run: ninja -C build
+      - name: Configure and build test-suite @ head
+        run: |
+          cat << EOF > rva23u64.cmake
+          set(CMAKE_SYSTEM_NAME Linux)
+          set(CMAKE_C_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang)
+          set(CMAKE_CXX_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang++)
+          set(CMAKE_C_COMPILER_TARGET riscv64-linux-gnu)
+          set(CMAKE_CXX_COMPILER_TARGET riscv64-linux-gnu)
+          set(CMAKE_C_FLAGS_INIT "-march=rva23u64 -save-temps=obj")
+          set(CMAKE_CXX_FLAGS_INIT "-march=rva23u64 -save-temps=obj")
+          set(CMAKE_SYSTEM_PROCESSOR riscv64)
+          set(CMAKE_LINKER_TYPE LLD)
+          EOF
+          cat << EOF > armv9-a.cmake
+          set(CMAKE_SYSTEM_NAME Linux)
+          set(CMAKE_C_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang)
+          set(CMAKE_CXX_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang++)
+          set(CMAKE_C_COMPILER_TARGET aarch64-linux-gnu)
+          set(CMAKE_CXX_COMPILER_TARGET aarch64-linux-gnu)
+          set(CMAKE_C_FLAGS_INIT "-march=armv9-a -save-temps=obj")
+          set(CMAKE_CXX_FLAGS_INIT "-march=armv9-a -save-temps=obj")
+          set(CMAKE_SYSTEM_PROCESSOR arm64)
+          set(CMAKE_LINKER_TYPE LLD)
+          EOF
+          cat << EOF > x86_64.cmake
+          set(CMAKE_C_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang)
+          set(CMAKE_CXX_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang++)
+          set(CMAKE_C_FLAGS_INIT "-save-temps=obj")
+          set(CMAKE_CXX_FLAGS_INIT "-save-temps=obj")
+          set(CMAKE_LINKER_TYPE LLD)
+          EOF
+          build_llvm_test_suite () {
+            cmake -B build.$1 -C cmake/caches/O3.cmake --toolchain $2 -DTEST_SUITE_BENCHMARKING_ONLY=ON -DTEST_SUITE_RUN_BENCHMARKS=OFF -GNinja
+            ninja -C build.$1
+            $GITHUB_WORKSPACE/build/bin/llvm-lit build.$1 -o results.$1.json
+          }
+          build_llvm_test_suite rva23u64-O3-b rva23u64.cmake
+          build_llvm_test_suite armv9-a-O3-b armv9-a.cmake
+          build_llvm_test_suite x86_64-O3-b x86_64.cmake
+        working-directory: llvm-test-suite
+      - name: Build test-suite @ base
+        run: git checkout $BASE_SHA && ninja -C build
+      - name: Configure and build test-suite @ base
+        run: |
+          build_llvm_test_suite () {
+            cmake -B build.$1 -C cmake/caches/O3.cmake --toolchain $2 -DTEST_SUITE_BENCHMARKING_ONLY=ON -DTEST_SUITE_RUN_BENCHMARKS=OFF -GNinja
+            ninja -C build.$1
+            $GITHUB_WORKSPACE/build/bin/llvm-lit build.$1 -o results.$1.json
+          }
+          build_llvm_test_suite rva23u64-O3-a rva23u64.cmake
+          build_llvm_test_suite armv9-a-O3-a armv9-a.cmake
+          build_llvm_test_suite x86_64-O3-a x86_64.cmake
+        working-directory: llvm-test-suite
+      - run: |
+          mkdir diffs
+          ./utils/tdiff.py -a build.rva23u64-O3-a -b build.rva23u64-O3-b -s all > diffs/rva23u64-O3.diff || true
+          ./utils/tdiff.py -a build.armv9-a-O3-a -b build.armv9-a-O3-b -s all > diffs/armv9-a-O3.diff || true
+          ./utils/tdiff.py -a build.x86_64-O3-a -b build.x86_64-O3-b -s all > diffs/x86_64-O3.diff || true
+        working-directory: llvm-test-suite
+      - uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f #v7.0.0
+        id: upload-diffs
+        with:
+          name: diffs
+          path: llvm-test-suite/diffs
+      - uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f #v7.0.0
+        with:
+          name: results
+          path: llvm-test-suite/results*.json
+      - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
+        env:
+          DIFF_URL: ${{ steps.upload-diffs.outputs.artifact-url }}
+        with:
+          script: |
+            github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: context.issue.number,
+              body: `test-suite diff from ${process.env.BASE_SHA}...${process.env.HEAD_SHA}: ${process.env.DIFF_URL}`
+            })

[boomanaiden154]

Is it possible to refactor the CMake caches you're creating out into separate files?

[lukel97]

Is it possible to refactor the CMake caches you're creating out into separate files?

Done in ba5f906, but one thing to watch out for is that we need to do another checkout of the repository to make sure we use the scripts/cmake files in the upstream repository, not the PR which is untrusted: af377f0

The CodeQL workflow picked this up. But it still seems to complain about using the ./utils/tdiff.py scripts etc. in llvm-test-suite. I think that's a false positive though, since they're always from the main branch of llvm/llvm-test-suite and not e.g. the PR

[petrhosek]

I'd consider moving these options to a cache file or use one of existing ones from https://github.com/llvm/llvm-project/tree/main/clang/cmake/caches.

[lukel97]

Moved to a dedicated cache file in fc12789

[petrhosek]

We should consider building a custom Docker image to avoid having to install these packages on every run. I also think it'd be preferable to use compiler-rt and libc++ over libgcc and libstdc++ (and eventually also LLVM libc).

[lukel97]

The docker image makes sense, do you want me to do that as a follow up? FWIW these packages are pretty light so far, it takes about 19 seconds to install: https://github.com/lukel97/llvm-project/actions/runs/23885053232/job/69646132003#step:9:1

Agreed it would be nice to eventually use compiler-rt/libc++. But there aren't any *-cross packages for these on Ubuntu AFAICT. I guess we would need to build these as part of a runtimes build? But I haven't yet explored cross compiling those yet, and I'm not sure how much longer it would take the workflow to run

[petrhosek]

I'd build them in the same build using the runtimes build, but that can be also done in a follow up change. It's definitely going to take longer than 19 seconds since you need to run CMake and Ninja multiple times but I still think it's desirable since that way you can also measure the impact of runtime changes.

[boomanaiden154]

A custom docker image for this might be a bit heavyweight. Although it wouldn't hurt if we use the CI container to take advantage of the faster toolchain there.

[boomanaiden154]

I'm assuming this is to avoid some interaction with existing automation? Can you add a comment?

[lukel97]

I copied this from issue-release-workflow.yml which has the /cherry-pick command. I'm not actually sure what it's for. But I figured we should probably just restrict it to comments that start with a slash command anyway so removed it in aa01aa3

[boomanaiden154]

Why do we need to check this out separately instead of just pulling it from the main checkout?

[lukel97]

The checkout on line 53 can be from an arbitrary ref on an arbitrary fork. So a malicious actor could change the scripts in .github/workflows/test-suite/ to be whatever, something that steals env vars or something, which this workflow would then clone and execute later on.

The separate checkout here makes sure we always take the scripts from llvm/llvm-project, and on the main branch. So as long as no one publishes malicious scripts there we should be ok

[boomanaiden154]

They already basically have full RCE privileges though through CMake?

If we want to improve the security here, we should be writing out the comment as a file and using the issue-write workflow so we don't need GITHUB_TOKEN to have permissions to write comments.

[lukel97]

Ah good point, sounds like issue-write.yml is exactly what I'm looking for. Changed to use it in 9a18f4a

[boomanaiden154]

You'll need to precommit the update in issue-write.yml as the workflow definition is always taken from the repository's main branch.

I'll approve a PR doing that if you want to put one up.

[lukel97]

This workflow definition is taken from the default branch too, I've set my own forks default branch to this PR for testing. Do you still want me to precommit it?

[boomanaiden154]

https://llvm.org/docs/CIBestPractices.html#top-level-read-permissions here too.

[lukel97]

Done in 6fccd7b

[boomanaiden154]

A custom docker image for this might be a bit heavyweight. Although it wouldn't hurt if we use the CI container to take advantage of the faster toolchain there.

[boomanaiden154]

Can you look at trying to use the CI container for this workflow? It should be about the same, just have a (significantly) faster default toolchain and also enable the use of precompiled headers given we use clang instead of gcc.

[lukel97]

Done in ad54fac. Kicked off a test run with it here: https://github.com/lukel97/llvm-project/actions/runs/23910646008

[lukel97]

Wow the build is almost twice as fast. The only issue is that it looks like the cross compile builds are failing because the version of CMake is too old. It looks like it's version 3.28 on the image but we need at least 3.29 since the toolchain files use CMAKE_LINKER_TYPE. The default github ubuntu runner has 3.31: https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md#tools.

Are you able to update the CMake version in the image or should we try and install a newer version in this workflow?

[lukel97]

Updated the CMake version in this image for now in cc3e4c6

[boomanaiden154]

We could, but ubuntu 26.04 also comes out in a week or two, so I think we can just wait until then to upgrade.

[boomanaiden154]

Why is this changing beyond adding the new workflow?

[lukel97]

Was meaning to give an explanation for this but was just waiting for a test run to finish.

It turns out that on issue_comment workflows, the head branch which is used to fetch the PR number below is actually set to the default branch (main), not the branch of the PR that was commented on. So the issue workflow fails.

To fix that it looks like we have to manually plumb the PR number through an artefact: https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#using-data-from-the-triggering-workflow

The diff got rendered pretty badly here, but the change is that the graphql query is now wrapped in the if (!pr_number) conditional.

[boomanaiden154]

This looks reasonably good at this point. Let's split off the issue-write changes into a separate PR.

[boomanaiden154]

Can you add a TODO (assign it to me if you want) to drop this once we're on 26.04?

[lukel97]

Done in 9aa7930

[tstellar]

Why do we need a newer cmake version here?

[lukel97]

We need it specifically for CMAKE_LINKER_TYPE=LLD which is only available in 3.29. The cross compilation toolchain files need it so we can just use LLD instead of having to install several target specific linkers

[boomanaiden154]

We should probably enable ccache here on the assumption that had and base are going to be pretty similar?

[lukel97]

Probably not the llvm-test-suite build because the compiler will always be different anyway. But I've added sccache to the LLVM build in a37a98e

[lukel97]

Ping for this and #192205

[lukel97]

Gentle ping

[lukel97]

Ping

[tstellar]

Why do you need this step, isn't all the PR information available via the context?

[lukel97]

Unfortunately we only get the issue object fields so it's missing PR-specific things like the head ref, merge commit, mergability status etc. E.g. if you check the documentation on the payload object, the issue.pull_request field only contains a handful of URLs: https://docs.github.com/en/webhooks/webhook-events-and-payloads#issue_comment

FWIW I was also surprised it was this awkward to get the PR info out of the event

[tstellar]

Hi, I really like this workflow idea, but it is a little bit of security risk. Could we limit this so that only people with commit access can trigger the workflow? Could you do something similar to this: https://github.com/llvm/llvm-project/pull/196769/changes#diff-25cf99555bf126ee82e57adcc61fc11b0d490fc2f12f10de105acd5d841fbe2cR49-R61