Working around the obstacles to publishing official Docker base images

[chriskilding]

Many people have requested that GitHub publishes its base images for Actions runners, so that they can reuse them rather than building runners from scratch.

Here we encounter problem 1: Those requests usually end up being closed due to software licensing problems (#176 (comment)).

In the time since those requests were made, the devs have made it possible to run the build scripts ourselves with Packer. This has helped people who are deploying their runners on Azure.

Here we encounter problem 2: many of us who want to use self-hosted runners aren't on Azure, or want/need to use Docker-based runners via a project like actions-runner-controller. As @catthehacker explained in #2320 (comment), it's prohibitively difficult to build the project directly into Docker images due to the use of systemd.

Rather than going around the loop again, I propose we take a step back and think of a more high-level way of achieving what we want, which avoids the license and technological constraints.

What we care about is that:

• We can arrive at a set of Docker images for runners which are functionally equivalent to the official GitHub Actions runners.
• We can do this in an easy automated way.
• We can do this without spending too much time building them.

Therefore, if we can't directly use GitHub's official Actions runner images, the next best thing would be an executable specification / compatibility check / test suite, which we can use to ensure that our own base images contain all the relevant software that the official images do.

The image docs at https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu2004-README.md are not executable themselves, but would provide the foundation for writing such a test suite.

Thoughts?

[chriskilding]

My own situation, for reference: my company is currently trialling Actions. It can only use Actions on self-hosted infrastructure (in part because our enterprise Artifactory is only accessible inside our corporate network, and therefore the Actions must also be inside our network to see it). We're currently deploying self-hosted runners on EKS using actions-runner-controller. The minimal runner definitions from that project omit a lot of things which are present in GitHub-hosted runners, so many standard Actions fail (or need extra configuration) when run on the self-hosted runners.

This is very jarring for developers who are used to the convenience of GitHub-hosted runners. Therefore we need a way to achieve functional parity with them, without spending a lot of time manually building our own Docker runner images from scratch.

[catthehacker]

Therefore, if we can't directly use GitHub's official Actions runner images, the next best thing would be an executable specification / compatibility check / test suite, which we can use to ensure that our own base images contain all the relevant software that the official images do.

I'm unsure why you want to create tests and what kind of tests, but all software is tested already and those scripts can be re-used.

[chriskilding]

These test scripts are a good start. Couple of things:

1. Are these tests complete as they stand? I.e. if I ran them against a Docker image that I made myself, and it passed the tests, would this guarantee it was compatible with any Action that works on GitHub's standard runners?
2. What would be the best way to package these into a readily consumable artifact, such that it would be trivial for downstream teams to use it to test their Docker runner images?

[catthehacker]

1. It would guarantee that the software is there.
2. If they build it from repo, they have tests scripts and those are executed during build.

[chriskilding]

The loose analogy I'm going for is the Technical Compatibility Kit (TCK) for Java. This allows someone to write their own JDK, run the TCK against it, and if it passes the TCK then they have pretty good confidence that it can run any typical Java program.

If GitHub can't publish its own runner images, the next best thing is to let us verify that our runner images will work with any typical Action. In other words a TCK for Actions. That's the kind of level the test suite for the runner would have to get to. We'll try out the tests as they stand with our self-hosted runner and see how far we can get with them.