CI runner scalability: back-pressure, concurrency limits, and horizontal scaling

[dlorenc]

Background

The current CI architecture works well for a single repo with low commit rates but has several failure modes under load. This issue tracks the known gaps and proposed fixes.

Current architecture limits

1. Outbox dispatcher — sequential delivery

processOutboxBatch claims up to 50 events per 5-second poll and delivers them sequentially (no concurrency within a batch). Each delivery has a 10-second HTTP timeout.

Worst-case throughput: 50 events × 10s = 500s to drain one batch. A burst of 100 commits means events wait minutes before the ci-runner even sees them.

Fix: deliver the batch concurrently with a bounded goroutine pool (e.g. 10 workers).

---

2. ci-runner — unbounded goroutine fan-out, single pod

Every incoming webhook immediately calls runAsync in a new goroutine. There is no semaphore, no queue, no back-pressure. With replicas: 1 and 4 CPU / 4 GB shared with buildkitd:

• 3–4 concurrent runs: fine (golang:1.25 cached, lightweight builds)
• 5–10 concurrent runs: memory pressure as buildkitd holds snapshots for each concurrent solve
• 10+ concurrent runs: near-certain OOM → pod restarts → all in-flight runs lost with no terminal status

Fix (most impactful): a concurrency semaphore in the webhook handler — reject with 429 when at capacity so the outbox retries later with backoff. Something like:

const maxConcurrentRuns = 4
sem := make(chan struct{}, maxConcurrentRuns)

// in webhook handler:
select {
case sem <- struct{}{}:
    go func() {
        defer func() { <-sem }()
        runAsync(...)
    }()
default:
    http.Error(w, "too busy", http.StatusTooManyRequests)
}

The outbox already has exponential backoff on non-2xx responses, so a 429 naturally queues the job for retry.

---

3. buildkitd — no per-solve parallelism cap

BuildKit handles concurrent solves but has no per-solve resource limits. A single large image pull can starve all concurrent solves on the instance.

Fix: set --oci-worker-parallelism=N (e.g. 4) to queue excess solves rather than attempt them all simultaneously.

---

4. No horizontal scaling

deploy/k8s/ci-runner.yaml has replicas: 1 with no HPA. Since each pod has its own buildkitd sidecar, horizontal scaling works cleanly — more pods = more independent build capacity.

Fix: add a HorizontalPodAutoscaler scaling on CPU utilization. The internal LoadBalancer already distributes across pods. The in-memory run registry (gap #5 below) needs to be resolved first.

---

5. Run registry — in-process memory, lost on pod restart

In-flight run IDs are stored in a sync.Map-like struct in the ci-runner process. A pod restart (OOM, rolling deploy) silently drops all running jobs. Callers polling GET /run/{run_id} get 404 with no explanation.

Fix: persist run state to docstore as a check run in "running" status (or a dedicated run table). This also enables cross-pod GET /run/{run_id} lookups once HPA is in place.

---

6. Docstore — capped at max-instances=1

--max-instances=1 in deploy.yml means all CI check POSTs, status polls, and webhook deliveries queue behind one Cloud Run instance. Probably not the first failure point (~100 req/s capacity) but blocks full horizontal scaling of the ci-runner.

Fix: raise --max-instances. The outbox uses FOR UPDATE SKIP LOCKED so multiple instances are safe.

---

Priority order

Fix	Impact	Effort
Concurrency semaphore + 429 in ci-runner	Prevents OOM, enables clean back-pressure	Small
--oci-worker-parallelism on buildkitd	Prevents snapshot memory exhaustion	Trivial
Parallel outbox delivery	Eliminates delivery backlog under burst	Small
Persist run state to docstore	Survives restarts, unblocks HPA	Medium
HPA on ci-runner deployment	True horizontal scale-out	Small (after run state fix)
Raise docstore max-instances	Unblocks at-scale check result writes	Trivial

The concurrency semaphore is the highest-leverage single change — it provides back-pressure that lets every other component breathe and is ~20 lines of code.

Acceptance criteria (full fix)

• ci-runner rejects webhooks with 429 when maxConcurrentRuns is reached
• buildkitd started with --oci-worker-parallelism=4 (or configurable)
• Outbox batch delivered concurrently (bounded pool)
• Run state persisted to docstore; GET /run/{run_id} works after pod restart
• HPA added to deploy/k8s/ci-runner.yaml
• Docstore --max-instances raised to ≥ 3
• Load test: 20 rapid commits → all check runs eventually resolve, no OOM