Skip to content

ci: add caching to speed up tutor-based CI workflow - #802

Open
asadali145 with Copilot wants to merge 13 commits into
mainfrom
copilot/improve-tutor-environment-steps
Open

ci: add caching to speed up tutor-based CI workflow#802
asadali145 with Copilot wants to merge 13 commits into
mainfrom
copilot/improve-tutor-environment-steps

Conversation

Copilot AI commented May 19, 2026

Copy link
Copy Markdown
Contributor

What are the relevant tickets?

N/A

Description (What does it do?)

Adds several caching layers to .github/workflows/ci.yml to avoid rebuilding the Tutor environment from scratch on every push. The biggest time sinks were the Docker image build (~15–20 min), the edx-platform clone (~5–10 min), and Tutor config generation (~9–15 min).

Changes:

  1. Resolve edx-platform tip SHA early — a cheap git ls-remote call (no download) gives the current HEAD SHA for the branch; used as cache key across all subsequent caches.

  2. Cache built dist/ packages — keyed by a hash of all files under src/**, pyproject.toml, and uv.lock (including static assets and templates); skips uv build --all-packages on hits.

  3. Cache pip packages — caches ~/.cache/pip keyed by branch + OS; speeds up repeated pip install tutor>=… calls.

  4. Shallow-clone edx-platform + cache the clone dir — replaces the full two-step clone+checkout with a single git clone --depth=1 --branch=… and caches the directory keyed by branch+SHA; skips the clone entirely on hits.

  5. Docker image cache via ghcr.io — before building, tries to pull ghcr.io/mitodl/openedx-dev-cache:<branch-sha>. On hit, retags the pulled image and skips tutor images build openedx-dev. On miss, builds normally then pushes to ghcr.io for future runs using tutor config printvalue DOCKER_IMAGE_OPENEDX_DEV to resolve the correct image name. Push is conditioned on push events only (never on pull_request runs). The GHCR namespace is hardcoded via env.GHCR_CACHE_OWNER: mitodl so fork PRs always pull from the upstream org's cache.

  6. Cache Tutor config directory — caches ~/.local/share/tutor and ~/.local/share/tutor-main keyed by Tutor version + branch; skips tutor config save entirely on hits (the docker-compose env files are already present from the cache).

  7. Generate edx-platform egg-info on the host runner — runs pip install --no-deps -e /path/to/edx-platform after the edx-platform is cloned/restored. When edx-platform is bind-mounted into the Tutor container, the image's Open_edX.egg-info/ directory is overwritten by the host checkout (which has no egg-info). Without the egg-info, pkg_resources cannot read the edx-platform's lms.djangoapp entry points, so get_plugin_apps(ProjectType.LMS) fails to discover apps like content_libraries, causing a RuntimeError at Django startup. Previously, tutor dev init (via mounted-directories.sh) regenerated this egg-info inside a running container; this step reproduces that regeneration in ~5 seconds on the host without starting any containers.

Estimated savings per run on cache hit:

Step Before After
Clone edx-platform ~5–10 min (full) < 1 min (shallow) or skip
Build Docker image ~15–20 min skip
Generate Tutor config ~9–15 min ~5 s or skip
Tutor pip install ~1–2 min < 30 s

How can this be tested?

  1. Open a PR and observe the CI run: the first run will populate all caches.
  2. Push another commit to the same PR without changing edx-platform or the plugin source — subsequent runs should skip the Docker build, the edx-platform clone, and tutor config save.
  3. Verify the ghcr.io/mitodl/openedx-dev-cache package is created in the org's packages after the first run on the main branch.

Additional Context

  • The packages: write permission is declared at the job level (GitHub Actions does not support conditional job-level permissions). However, it is never exercised during pull_request runs: both the GHCR login step and the Docker push step are conditioned on github.event_name == 'push'. PRs pull from GHCR using anonymous access (the cache package is public-readable).
  • tutor dev launch -I --skip-build was replaced by tutor config save because the former starts Docker containers (MongoDB, MySQL, etc.) that write data files owned by root. Those root-owned files caused tar: Permission denied errors in the post-cache step, preventing the Tutor config cache from ever being saved. tutor config save generates all needed docker-compose files in ~5 seconds without starting any containers.
  • The edx-platform's Open_edX.egg-info/ is generated on the host runner rather than inside a container. The egg-info directory is a portable text-format metadata directory; pkg_resources inside the container reads it correctly via the bind mount regardless of which Python generated it. The egg-info is also captured in the edx-platform directory cache, so on cache hits it is already present.
  • The Tutor config cache key omits the edx-platform SHA (unlike the Docker image and edx-platform clone caches). The generated config files depend only on the Tutor version and branch, not on the specific edx-platform commit, so tying the key to the SHA would cause unnecessary invalidations on every new master commit.
  • The Docker image pushed to ghcr.io will accumulate over time. Consider configuring a package retention policy in GitHub settings to clean up old image tags automatically.
  • actions/cache@v4 (SHA-pinned) and docker/login-action@v3 (SHA-pinned) are the only new actions added.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR speeds up the Tutor-based integration test workflow by adding multiple caches (Python build artifacts, pip cache, edx-platform checkout, Tutor config, and a Docker image cache in GHCR) so repeated CI runs don’t rebuild everything from scratch.

Changes:

  • Resolve the edx-platform branch tip SHA up front and use it to key subsequent caches.
  • Cache dist/ artifacts, pip downloads, and the edx-platform checkout to reduce repeated network/build work.
  • Add a GHCR-backed Docker image cache and cache Tutor config dirs to potentially skip image builds and tutor dev launch.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/workflows/ci.yml Outdated
@asadali145

Copy link
Copy Markdown
Contributor

@copilot fix the failing integration tests in https://github.com/mitodl/open-edx-plugins/actions/runs/26108947537/job/76780542597, look for the reason and fix it.

Copilot AI commented May 19, 2026

Copy link
Copy Markdown
Contributor Author

@copilot fix the failing integration tests in https://github.com/mitodl/open-edx-plugins/actions/runs/26108947537/job/76780542597, look for the reason and fix it.

Fixed in 2895b45. The failure was in the "Push Docker image to registry cache" step:

Error response from daemon: No such image: overhangio/openedx:21.0.6-main-dev

The workflow was constructing the dev image name by appending -dev to DOCKER_IMAGE_OPENEDX (overhangio/openedx:21.0.6-mainoverhangio/openedx:21.0.6-main-dev), but Tutor actually builds and names the dev image as openedx-dev:21.0.6-main — a completely different naming scheme. The fix replaces the manual construction with tutor config printvalue DOCKER_IMAGE_OPENEDX_DEV, which returns the correct local image name that Tutor actually created.

@asadali145

Copy link
Copy Markdown
Contributor

@copilot CI checks are still taking about 25 minutes on master branch. This includes about 15m for the tutor launch step. Looks like there is still some issue? Look at the latest CI run and see what can be fixed/improved.

Copilot AI commented May 20, 2026

Copy link
Copy Markdown
Contributor Author

@copilot CI checks are still taking about 25 minutes on master branch. This includes about 15m for the tutor launch step. Looks like there is still some issue? Look at the latest CI run and see what c...

Investigated and fixed in 396604e. Two root causes were found:

Root cause 1 — Tutor config cache never saved due to permission errors:
tutor dev launch -I --skip-build starts Docker containers (MongoDB, MySQL, etc.) which write data files owned by root into ~/.local/share/tutor[-main]/data/. The Post-cache step's tar couldn't read those files (Permission denied), so the cache was never stored and tutor dev launch re-ran every time (9–15 min).

Root cause 2 — Cache key included edx-platform SHA:
The master branch gets new commits many times per day, changing the SHA and invalidating the cache key even if the save had worked.

Fixes:

  1. Replaced tutor dev launch -I --skip-build; tutor dev stop with tutor config save — generates the docker-compose files the test runner needs in ~5 seconds, without starting any containers (so no root-owned data files, cache saves cleanly).
  2. Removed the edx-platform SHA from the Tutor config cache key. The generated config files (docker-compose templates, etc.) don't depend on edx-platform's specific commit — only on the Tutor version and branch. The cache now persists across edx-platform tip changes.

On the next run after a cache miss, "Generate Tutor config" will take ~5 seconds and save cleanly. Subsequent runs will skip it entirely via cache hit.

@asadali145

Copy link
Copy Markdown
Contributor

@copilot fix the pre-commit checks

Copilot AI commented May 20, 2026

Copy link
Copy Markdown
Contributor Author

@copilot fix the pre-commit checks

Fixed in b2f5f29. Two pre-commit hooks were failing:

  1. yamlfmt — reformatted the long tutor-config cache key line (auto-fixed by the hook itself).
  2. actionlint (shellcheck SC2086) — several unquoted shell variables:
    • All >> $GITHUB_OUTPUT redirections → >> "$GITHUB_OUTPUT"
    • The docker compose command's $DIRECTORY, $DEV, and $PWD variables → properly quoted; also expanded that long single-line command to multi-line for readability.

@asadali145

Copy link
Copy Markdown
Contributor

@copilot now tests are failing with missing apps, I think tutor launch cannot be replaced with tutor config save. R&D on how tutor works with open edx and then make required changes

Copilot AI commented May 20, 2026

Copy link
Copy Markdown
Contributor Author

@copilot now tests are failing with missing apps, I think tutor launch cannot be replaced with tutor config save. R&D on how tutor works with open edx and then make required changes

Fixed in 276723d.

Root cause: tutor dev launch runs tutor dev init, which executes the mounted-directories.sh script inside the LMS container. This script detects that edx-platform is bind-mounted (by checking for the missing bindmount-canary file) and runs pip install -e . to regenerate Open_edX.egg-info/ in the mounted directory.

Without this egg-info, pkg_resources cannot find the edx-platform's entry points. The edx-platform registers content_libraries (and many other apps) as LMS Django apps via the lms.djangoapp entry point group. The get_plugin_apps(ProjectType.LMS) call in lms/envs/common.py reads from these entry points to populate INSTALLED_APPS. Without the egg-info, content_libraries is never added to INSTALLED_APPS, causing the RuntimeError.

Fix: Added a step after Add edx-platform tutor mounts that runs pip install --no-deps -e /path/to/edx-platform on the host runner. This creates the Open_edX.egg-info/ directory in the edx-platform directory, which is then available inside the container via the bind mount. No container startup needed — this is equivalent to what mounted-directories.sh does but runs in ~5 seconds on the host instead of the ~9 minutes that tutor dev launch took.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/workflows/ci.yml Outdated
@asadali145
asadali145 force-pushed the copilot/improve-tutor-environment-steps branch from 9fd7b9e to 6becfaa Compare July 29, 2026 13:14
@asadali145

Copy link
Copy Markdown
Contributor

@claude review this PR

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

.github/workflows/ci.yml:108

  • The cache key is split across two YAML lines, which introduces whitespace into the literal cache key and makes it harder to reason about cache hits/misses. Keep the key on a single line so the key value is exactly what it appears to be.
        key: tutor-config-${{ steps.tutor-version.outputs.version_tag }}-${{ steps.edx-sha.outputs.branch_slug
          }}

Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/workflows/ghcr-cleanup.yml Outdated
@asadali145

Copy link
Copy Markdown
Contributor

Investigated Copilot's latest review comments

.github/workflows/ci.yml:135 and :171-174 (GHCR login/push steps not gated on github.event_name == 'push') — both comments point at the same underlying tradeoff, and it's a real one that needs a team call, not just a fix:

  • github.event_name == 'push': the GHCR login/push steps only ever run once code has landed on main. Every PR run — including same-repo branches — can only pull an existing cache image, never write one. This is the safer option, but it's also what caused the "Docker image cache never works for PR runs" problem fixed a few iterations back (PR runs always cache-missed and ate the full ~13min image rebuild).
  • github.repository_owner == env.GHCR_CACHE_OWNER (current state): true for every PR targeting this repo, fork or not — it doesn't distinguish forks by itself. What actually protects against forks is a separate, GitHub-enforced rule: for pull_request (not pull_request_target) runs where the head repo differs from the base repo, GitHub automatically forces secrets.GITHUB_TOKEN to read-only regardless of the workflow's declared permissions: block — so a forked PR's push attempt just fails harmlessly.
  • Where the real exposure is: PRs from branches that live inside this repo (like this one) don't get that fork downgrade, so packages: write is genuinely live for the whole job — cloning edx-platform, uv build --all-packages, running pytest, etc. — not just at the two gated steps. That's scoped to people who already have push access to this repo (not the public), but it does mean a compromised dependency pulled in during a PR run could push an arbitrary image into ghcr.io/mitodl/openedx-dev-cache.

The tradeoff: event_name == 'push' is safer (write token only active post-merge) but reintroduces the ~13min-per-PR-run regression. repository_owner == mitodl (current state) fixes that performance problem but keeps a live write-scoped token active for the full duration of same-repo-branch PR runs.

Looping in the team to make the call here — no code changes made pending that discussion.

@asadali145

Copy link
Copy Markdown
Contributor

Hi @blarghmatey @rhysyngsun,

Could you please take a look at these findings by Claude in our current implementation to speed up the CI? I am looking for suggestion on what should we do?

The integration-tests job held packages:write while running PR code and
three sources of upstream code (edx-platform's setup.py, Tutor from git
main, the image's pip installs). docker/login-action leaves the
credential in ~/.docker/config.json for the remainder of the job, so the
write scope was live during the image build and the test run, not just
at the push step.

Split the build and push into ci-image-cache.yml, which runs nightly on
a schedule and touches no PR code. integration-tests now only pulls, and
falls back to a local build when the cache misses.

Rebuilding nightly also bounds image staleness. The openedx-dev image
pins the Python dependencies of the release branch it was built against,
and a pinned Tutor range is stable for months, so a cache populated only
on demand would drift arbitrarily far from what the branch requires.

Both workflows derive the cache tag through a shared composite action;
if they disagreed on the tag the cache would silently never hit.

Drop the Tutor config cache: the step it skipped takes one second, and
restoring config over a later `tutor mounts add` risked masking the
mount.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Snt95k6UYxUndphyNBey7S
@blarghmatey

Copy link
Copy Markdown
Member

Resolving the packages: write question: split the image build into its own workflow

Pushed in 581cf18. This takes the third option from the discussion above — neither event_name == 'push' nor repository_owner == mitodl — and removes the tradeoff instead of picking a side.

Where the exposure actually is

The framing in the earlier comment is right about the fork mechanics, but understates the surface. docker/login-action writes the credential into ~/.docker/config.json, and it stays there for the rest of the job — through tutor images build and through the 10-minute test run, until the post-step cleans up at job end. Meanwhile integration-tests deliberately executes upstream code from three sources: pip install --no-deps -e .../edx-platform runs edx-platform's setup.py on the runner, the master leg installs Tutor unpinned from git main, and the image build pulls an upstream Dockerfile with unpinned pip installs. So it isn't only "a compromised dependency could" — running third-party code alongside a write-scoped registry credential is the normal path through this job.

That's what makes the choice feel forced. But tutor images build openedx-dev needs none of it: the image is built from Tutor's own templates and does not embed the edx-platform source (that's bind-mounted at run time). It doesn't need the repo checked out, the plugins built, edx-platform cloned, or the tests run.

What changed

New .github/workflows/ci-image-cache.yml — the only workflow with packages: write. Nightly cron, plus workflow_dispatch, plus push-to-main when the Tutor install logic changes. Matrix over edx_branch; each leg installs Tutor, builds openedx-dev, pushes to GHCR. It runs no PR-authored code and never checks out edx-platform. Off the critical path, so its ~13 min per leg costs no PR latency (and nothing in runner minutes on a public repo).

ci.ymlpackages: writepackages: read. GHCR login is now continue-on-error: true, the push step is gone, and the local build is demoted to a fallback for cache misses that never publishes what it builds. Fork PRs and cold caches still work, just slowly.

New .github/actions/setup-tutor — a composite action that installs Tutor and derives version_tag / branch_slug / cache_image_tag, shared by both workflows. This is load-bearing rather than tidiness: if the two workflows ever computed the tag differently the cache would silently never hit, and that failure mode is invisible in a green build.

Dropped the Tutor config cache. Worth checking the numbers before defending it — in run 30455146710 Generate Tutor config took 1 second. The cache saved a second while creating the hazard Copilot flagged at line 106: restoring cached config over a later tutor mounts add. tutor config save now runs unconditionally inside the composite action.

ghcr-cleanup.yml — moved to daily at 05:00 UTC, an hour behind the refresh job, since each nightly push now orphans the version it replaces.

A second reason for the nightly rebuild

This one hasn't come up in the thread and I think it matters more than the security question.

The openedx-dev image pins the Python dependencies of whatever release branch it was built against. On release/teak and release/ulmo the Tutor version range is stable for months, so under the current design a single image would be reused across that entire window while the release branches keep moving underneath it. The edx-platform source is bind-mounted so it stays current, but the site-packages inside the image don't — you'd eventually be testing current platform code against months-old dependencies. Rebuilding nightly bounds that drift to a day.

Two things to check before merging

  1. The openedx-dev-cache package is private. Anonymous auth against ghcr.io/v2/mitodl/openedx-dev-cache returns DENIED, so the "fork PRs pull anonymously" line in the PR description isn't true today — every cache hit so far has been an authenticated pull by an upstream PR run, and fork PRs are eating the full rebuild. I kept the (now read-only) login in ci.yml so this works either way, but flipping the package to public in org settings is what gets fork PRs onto the cache.

  2. ci-image-cache.yml can't run until it's on main. Scheduled and dispatched workflows only run from the default branch, so this PR's CI can't exercise it. The tag format is unchanged, though, so the images already in GHCR should still hit — expect teak/ulmo to pull as before, and master to fall back to a build if Tutor main has moved since the last push. After merge, workflow_dispatch it once to confirm before relying on it.

Also worth noting on the review comments

  • The ghcr-cleanup.yml cron indentation comment is a false positive. A block sequence at the same column as its parent key is valid YAML, it's what this repo's yamlfmt hook produces (see on.pull_request.branches in ci.yml), and yaml.safe_load parses it to {'schedule': [{'cron': ...}]}. No change needed.
  • Several other Copilot comments are stale — the restore-keys ones and "cache tag omits Tutor version" were fixed in later commits.
  • The run_edx_integration_tests.sh changes (pip install uv instead of the curl installer, uv pip install for plugin installs) are unrelated to caching and worth a deliberate look rather than passing through on a CI-perf PR.

Where this leaves the numbers

Unchanged from the last run: ~12 min per leg on a cache hit, versus 45–52 min on main. The image build is no longer the bottleneck — the ~10 min test step is, and no amount of caching touches it. If more is needed, that's the next thing to look at.

The test step guessed the Tutor root from the matrix branch: tutor-main
for master, tutor otherwise. That spelling comes from __app__, which only
gains the -main suffix while tutor's main branch carries a non-empty
__version_suffix__. Tutor cut v22.0.0 from main today and reset the
suffix to "", so __app__ became plain "tutor", the root moved, and the
master leg died on a missing env/local/docker-compose.yml.

`tutor config printroot` reports the real location, and the compose
project name follows from its basename, so neither has to be predicted.
This also drops the hardcoded /home/runner prefix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Snt95k6UYxUndphyNBey7S
@blarghmatey

Copy link
Copy Markdown
Member

CI result on 581cf18, and a Tutor v22 breakage that is not from this PR

release/teak passed end to end — composite action → read-only pull → cache hit → tutor mounts add → tests, with no packages: write anywhere in the job. That's the restructure confirmed working.

master failed, release/ulmo was cancelled by fail-fast. Neither is caused by the split. Details below, since two separate upstream events landed today.

1. master: Tutor's app directory moved out from under the hardcoded path

open /home/runner/.local/share/tutor-main/env/local/docker-compose.yml: no such file or directory

The test step picked the Tutor root from the matrix branch — tutor-main for master, tutor otherwise. That -main suffix isn't a property of the branch; it comes from Tutor's __app__, which only gains it while main carries a non-empty __version_suffix__:

if __version_suffix__:
    __version__ += "-" + __version_suffix__
    __app__ += "-" + __version_suffix__

Tutor cut v22.0.0 from main today (2026-08-05) and reset __version_suffix__ to "", so __about__.py on main now resolves __app__ to plain tutor. The root moved to ~/.local/share/tutor and the hardcoded path stopped existing.

This logic is unchanged from main — the PR only reindented it. But it blocks the branch, so fixed in 40935ad by asking Tutor rather than predicting: tutor config printroot gives the real root, and the compose project name follows from its basename. The master special case is gone, along with the hardcoded /home/runner prefix.

Expect this to resurface whenever upstream flips the suffix back, which is why the fix derives the value instead of tracking it.

2. main is separately red today, for an unrelated upstream reason

The two runs on main today (31009612954, 31006714198) failed inside the image build:

E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/l/linux/linux-libc-dev_5.15.0-187.197_amd64.deb  404  Not Found
ERROR: process "/bin/sh -c apt update && apt install -y build-essential curl git language-pack-en" did not complete successfully: exit code: 100

Ubuntu dropped a package version the image's apt index still references. Nothing to do with this PR — but worth noting that a warm image cache makes this class of failure survivable: teak sailed past it here by pulling a prebuilt image, while main rebuilds every time and takes the hit. That's an argument for the nightly refresh beyond raw speed.

3. The ulmo cache miss was correct behaviour

release/ulmo rebuilt rather than pulling, which is the fallback doing its job: Tutor 21.0.9 shipped 2026-08-04, and the cached ulmo image was built 2026-07-29 against 21.0.8, so version_tag changed and the cache properly invalidated.

This is the gap ci-image-cache.yml closes. Today the first PR run after any Tutor patch release eats the full ~13 min rebuild; with the nightly refresh that window is at most a day, and PR authors stop paying for upstream's release cadence.

Unrelated, but timely

Tutor v22.0.0 is the Verawood release, out today. I've opened #841 to retire Teak from the matrix and add Verawood — including the detail that the branch → Tutor mapping needs an explicit >=22.0.0,<23.0.0 case, because Ulmo is currently the catch-all else and Verawood would silently install Tutor 21 against a Verawood platform.

master has failed twice on an upstream version skew, and both times
fail-fast cancelled release/ulmo mid-test, so there is still no signal on
whether ulmo passes. The legs test independent Open edX branches and one
breaking upstream says nothing about the rest.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Snt95k6UYxUndphyNBey7S
@blarghmatey

Copy link
Copy Markdown
Member

The printroot fix worked. master now fails on a real upstream skew, plus a latent bug in the test script

Run 31020983674: release/teak green again, master got all the way into the test run (the compose file resolved, containers started, pytest executed — the previous failure is fixed), and then failed on something else entirely.

What master fails on now

Every plugin dies at collection with the same shape of error:

ImportError: cannot import name 'CoursewareAccessChecksRequested' from 'openedx_filters.learning.filters'
  (/openedx/venv/lib/python3.12/site-packages/openedx_filters/learning/filters.py)
ImportError: cannot import name 'CourseStartDateValidationFailed' from 'openedx_filters.learning.filters'

Neither name appears anywhere in this repo — these are edx-platform master importing from openedx_filters. It's a version skew:

openedx-filters
edx-platform master (the mounted source) 3.8.0
release/verawood (what Tutor v22 builds) 3.4.1

Tutor's main branch stopped being a nightly today: v22.0.0 was cut from it, which is the Verawood release. So the master leg now builds a Verawood image and bind-mounts master source into it. The source needs filter classes that shipped in 3.5.0–3.8.0; the image has 3.4.1.

This is the same root cause as the tutor-main path breakage — Tutor main is temporarily a release rather than a nightly — and it should resolve on its own once upstream restores the nightly suffix and resumes tracking master. It is not caused by this PR: the last green master leg was 2026-08-03, before v22.0.0 existed.

The latent bug this exposes

run_edx_integration_tests.sh already tries to install edx-platform's own pins:

if [ $CI ]; then
	pip install -r ./requirements/edx/testing.txt

That would resolve the skew — master's testing.txt pulls openedx-filters==3.8.0. But it has never run. CI is set on the runner, not inside the container, and docker compose run doesn't forward it, so the guard is always false. The image's pins have therefore always had to match the mounted source by luck, which is exactly why a Tutor/platform divergence is fatal rather than self-correcting.

Worth fixing deliberately rather than as a drive-by, since making it fire changes what gets installed and adds minutes to every run. Options:

  1. Pass -e CI=true in the docker compose run invocation so the existing branch works as intended.
  2. Install just the skewed constraint rather than all of testing.txt.
  3. Leave it and accept that the master leg tracks Tutor's image pins, not the platform's.

I'd want a maintainer's call on that — it's orthogonal to caching, and (1) has real runtime cost.

Also pushed: fail-fast: false (4ee888c)

master has failed twice now, and both times it cancelled release/ulmo mid-test, so there is still no answer to whether ulmo passes. The legs test independent Open edX branches; one breaking upstream says nothing about the others. Small change, but without it this PR can't be evaluated while master is broken upstream.

Where the PR stands

The caching restructure itself is verified: teak is green end to end through the composite action, read-only pull, cache hit, and test run, with no packages: write anywhere in the job. The two remaining failures are both upstream Tutor v22 fallout, one of which I've fixed (printroot) and one of which needs the decision above.

@asadali145
asadali145 requested a lite review from Copilot August 6, 2026 09:21
@asadali145
asadali145 marked this pull request as ready for review August 6, 2026 09:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Suppressed comments (7)

.github/workflows/ci-image-cache.yml:53

  • steps: is followed by step entries at the wrong indentation level. As written, the step list is not under steps: and the workflow will fail to parse.
    steps:
    - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6

    - name: Set up Tutor
      id: tutor

.github/workflows/ci-image-cache.yml:63

  • The remaining step entries in this workflow also need to be indented under steps: (currently they are aligned with steps:). This prevents the workflow from being valid YAML for Actions.
    - name: Log in to GitHub Container Registry
      uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
      with:
        registry: ghcr.io
        username: ${{ github.actor }}

.github/workflows/ci-image-cache.yml:73

  • The push step needs to be indented under steps: as well; otherwise the workflow is invalid.
    - name: Push image to GHCR cache
      env:
        LOCAL_IMAGE: ${{ steps.tutor.outputs.openedx_dev_image }}
        CACHE_IMAGE: ${{ steps.tutor.outputs.cache_image_tag }}
      run: |

.github/workflows/ghcr-cleanup.yml:21

  • steps: is followed by step entries at the wrong indentation level; the list items must be nested under steps: or GitHub Actions will fail to parse the workflow.
    steps:
    - name: Delete untagged openedx-dev-cache versions
      uses: actions/delete-package-versions@25ad4af5be03aef10e0b3376b248688b7a44c40b # v5
      with:
        package-name: openedx-dev-cache

.github/workflows/ghcr-cleanup.yml:30

  • This step also needs to be indented under steps:; otherwise the workflow is invalid YAML for Actions.
    - name: Trim old tagged openedx-dev-cache versions
      uses: actions/delete-package-versions@25ad4af5be03aef10e0b3376b248688b7a44c40b # v5
      with:
        package-name: openedx-dev-cache
        package-type: container

.github/workflows/ci.yml:83

  • The PR description mentions caching the Tutor config directory (~/.local/share/tutor*) and skipping tutor config save on cache hits, but the workflow always runs tutor config save inside the setup-tutor composite action and there is no actions/cache step for the Tutor root. Either update the PR description to match the implementation, or add the missing cache restore/save so repeated runs can skip regenerating config.
    - name: Set up Tutor
      id: tutor
      uses: ./.github/actions/setup-tutor
      with:
        edx_branch: ${{ matrix.edx_branch }}
        ghcr_owner: ${{ env.GHCR_CACHE_OWNER }}

run_edx_integration_tests.sh:68

  • if [ $CI ]; then will error (and exit due to set -e) when CI is unset, making the script fail in non-CI runs. Use a safe parameter expansion check so the conditional behaves correctly when CI is missing.
echo "===== Installing uv ====="
pip install uv

if [ $CI ]; then

Comment thread .github/workflows/ci-image-cache.yml
Comment thread .github/workflows/ghcr-cleanup.yml
asadali145 and others added 2 commits August 10, 2026 13:29
Update the checkout pin comment to the exact tag (v6.0.2) it resolves
to, and repin delete-package-versions to the underlying commit SHA
instead of the annotated tag object's SHA.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants