RUE-1473: measure local CFG epoch payloads #318

Workflow file for this run

.github/workflows/performance-collect.yml at 512adaa

	# Compiler-performance collection (ADR-0067 Phase 5, RUE-1188).
	#
	# Measures each platform's collection epoch on every trunk commit and appends
	# the raw run objects to the `performance-data-v1` orphan branch.
	#
	# Two properties this workflow exists to guarantee:
	#
	# Writes are serialized. `index.json` is read-modify-write, so two collectors
	# publishing at once would lose one of them. A single non-cancelling writer
	# group and one publish job give the branch exactly one writer at a time.
	#
	# Only raw observations are stored. Indexes, dispersion, chart data, and
	# summaries are rebuilt from these records at site build time and never written
	# here. Anything derived that landed on the branch would eventually disagree
	# with the records it came from, and the stored copy would be believed.
	#
	# A series that stops advancing is a failure. A run that cannot enter its
	# series fails this workflow rather than warning, because the only thing worse
	# than a broken measurement is a broken measurement that reports success
	# (RUE-1258).
	name: Performance collection

	on:
	push:
	branches: [trunk]
	workflow_dispatch:

	# Measurement jobs only read. The publish job re-declares the write it needs.
	permissions:
	contents: read

	# Never cancel in progress. A cancelled measurement wastes a runner; a cancelled
	# publish can leave the branch describing a run object it does not contain.
	# Queueing is the point, not a limitation.
	concurrency:
	group: performance-collection
	cancel-in-progress: false

	jobs:
	measure:
	strategy:
	fail-fast: false
	matrix:
	include:
	- os: ubuntu-24.04
	platform: x86_64-linux
	image: ubuntu-24.04
	- os: ubuntu-24.04-arm
	platform: aarch64-linux
	image: ubuntu-24.04
	- os: macos-15
	platform: aarch64-macos
	image: macos-15
	runs-on: ${{ matrix.os }}
	name: measure (${{ matrix.platform }})
	# macOS takes 99 samples of the startup probe by calibrated policy, and the
	# corpus is expected to grow.
	timeout-minutes: 120
	steps:
	- uses: actions/checkout@v6

	- name: Select Xcode 16.2
	if: runner.os == 'macOS'
	run: sudo xcode-select -s /Applications/Xcode_16.2.app

	- name: Install dotslash
	uses: facebook/install-dotslash@v2

	- name: Build the compiler and the runner
	run: \|
	./buck2 build //crates/rue:rue //crates/rue-bench:rue-bench \
	--target-platforms //platforms:release

	- name: Measure
	env:
	# Declared, not detected: a run must never be able to claim quietly
	# that it satisfied an environment policy it did not.
	RUE_BENCH_RUNNER_LABEL: github-hosted
	RUE_BENCH_RUNNER_IMAGE: ${{ matrix.image }}
	run: \|
	set -euo pipefail
	mkdir -p collected
	RUE="$(scripts/rue-bin --target-platforms //platforms:release)"
	BENCH="$(./buck2 build //crates/rue-bench:rue-bench \
	--target-platforms //platforms:release --show-simple-output 2>/dev/null \| tail -1)"

	# Exit 3 is "written but not appendable". The evidence is worth
	# keeping either way, so the status is recorded here and judged after
	# the artifact has uploaded.
	#
	# No --epoch: the manifest marks its own collection epoch. A number
	# here would have to be edited alongside the manifest whenever the
	# next epoch is declared, and the two would eventually disagree about
	# what is being collected.
	set +e
	"$BENCH" \
	--manifest performance/manifest.toml \
	--platform "${{ matrix.platform }}" \
	--compiler "$RUE" \
	--commit "${{ github.sha }}" \
	--repo-root . \
	--out "collected/${{ matrix.platform }}.json"
	status=$?
	set -e
	echo "runner exit status: $status"
	if [ "$status" -eq 2 ]; then
	echo "::error::the runner could not produce a run object"
	exit 1
	fi
	echo "status=$status" > "collected/${{ matrix.platform }}.status"

	- name: Upload the run object
	# Always, so a run that cannot enter its series is still kept as
	# evidence when the next step fails the job.
	if: always()
	uses: actions/upload-artifact@v6
	with:
	name: run-${{ matrix.platform }}
	path: collected/
	retention-days: 30

	# RUE-1258. A run that cannot enter its series is the series stopping,
	# which is the failure this workflow exists to surface rather than a
	# warning to scroll past. Ten days of a frozen dashboard, hiding a 7x
	# compile-time regression, went unnoticed behind the warning this
	# replaces.
	#
	# Checked after the upload so the raw record still reaches the publish
	# job: storing the observation is right even when publishing a point from
	# it is not.
	- name: Fail if the run cannot enter its series
	run: \|
	set -euo pipefail
	status="$(cut -d= -f2 "collected/${{ matrix.platform }}.status")"
	if [ "$status" -eq 3 ]; then
	echo "::error::the run measured at this commit cannot enter its series;" \
	"the published series stops advancing here. Declare the next epoch in" \
	"performance/manifest.toml, marking it \`collection = true\`."
	exit 1
	fi
	if [ "$status" -eq 4 ]; then
	echo "::error::the valid run has been retained as evidence, but it exceeds" \
	"the reviewed fresh-process non-regression ratchet in performance/manifest.toml."
	exit 1
	fi

	publish:
	needs: measure
	# Runs even when a platform failed: a partial sweep still has valid runs
	# worth storing, and a platform that is persistently broken should be
	# visible in the data rather than absent from it.
	if: always()
	runs-on: ubuntu-24.04
	name: publish
	permissions:
	contents: write
	steps:
	- uses: actions/checkout@v6

	- name: Download run objects
	uses: actions/download-artifact@v6
	with:
	path: incoming
	merge-multiple: true

	- name: Append to performance-data-v1
	env:
	GIT_AUTHOR_NAME: rue-performance-collector
	GIT_AUTHOR_EMAIL: noreply@github.com
	GIT_COMMITTER_NAME: rue-performance-collector
	GIT_COMMITTER_EMAIL: noreply@github.com
	run: \|
	set -euo pipefail
	shopt -s nullglob

	runs=(incoming/*.json)
	if [ ${#runs[@]} -eq 0 ]; then
	echo "no run objects were produced; nothing to publish"
	exit 0
	fi

	# A worktree keeps the data branch entirely separate from the source
	# checkout, so a stray source file can never be committed to it.
	git fetch origin performance-data-v1 --depth=1 2>/dev/null \|\| true
	if git rev-parse --verify origin/performance-data-v1 >/dev/null 2>&1; then
	git worktree add ../data origin/performance-data-v1
	git -C ../data switch -c performance-data-v1 \|\| \
	git -C ../data switch performance-data-v1
	else
	# First run bootstraps the branch. Orphan, so it shares no history
	# with trunk: the data is an independent artifact, not a fork of the
	# source, and nothing on it should ever be diffed against a commit.
	echo "bootstrapping the performance-data-v1 orphan branch"
	git worktree add --detach ../data
	git -C ../data checkout --orphan performance-data-v1
	git -C ../data rm -rf . >/dev/null 2>&1 \|\| true
	mkdir -p ../data/runs
	printf '{"runs":[],"schema_version":1}\n' > ../data/index.json
	fi

	mkdir -p ../data/runs
	[ -f ../data/index.json ] \|\| printf '{"runs":[],"schema_version":1}\n' > ../data/index.json

	python3 scripts/publish-performance-runs.py \
	--incoming incoming \
	--data-root ../data

	cd ../data
	if git diff --quiet && git diff --cached --quiet && [ -z "$(git status --porcelain)" ]; then
	echo "nothing new to publish"
	exit 0
	fi
	git add -A
	git commit -m "Collect performance runs for ${GITHUB_SHA:0:12}"
	# Retried rather than forced: a lost update here silently drops
	# someone else's run object.
	for attempt in 1 2 3 4 5; do
	if git push origin performance-data-v1; then
	exit 0
	fi
	echo "push rejected (attempt $attempt); rebasing onto the remote"
	git fetch origin performance-data-v1
	git rebase origin/performance-data-v1
	sleep $((attempt * 5))
	done
	echo "::error::could not publish after 5 attempts"
	exit 1

Navigation Menu

Uh oh!

RUE-1473: measure local CFG epoch payloads #318

Workflow file

RUE-1473: measure local CFG epoch payloads #318

Uh oh!

Workflow file for this run