GitHub mirror: sync main branch commits to a git repository via octo-sts

[dlorenc]

Overview

Implement a live GitHub mirror: every commit that lands on a docstore repo's main branch is immediately pushed as a git commit to a configured GitHub repository. Authentication uses octo-sts — the mirror exchanges a GCP workload identity OIDC token for a short-lived GitHub installation token, avoiding long-lived credentials entirely.

---

Trigger

Subscribe to the com.docstore.commit.created event (via the webhook eventing system from issue #41) filtered to the main branch. On each event, mirror the commit to the configured GitHub repo.

---

Mirror flow

For each commit.created event on main:

1. Authenticate — exchange the Cloud Run workload identity OIDC token for a short-lived GitHub token via octo-sts:

   POST https://octo-sts.app/token
   {
     "scope": "owner/repo",
     "permissions": {"contents": "write"}
   }
   Authorization: Bearer <OIDC token>
   

   Token is valid for ~10 minutes; fetch fresh on each mirror operation.

2. Fetch content — download the commit's full tree from docstore via GET /repos/{name}/-/archive?branch=main&at={sequence} (the bulk archive endpoint from issue CI runner: BuildKit-based check execution on Cloud Run gen2 #75).

3. Commit to GitHub — push the tree as a git commit using the GitHub API:

   • Use the Git Data API (POST /repos/{owner}/{repo}/git/trees, POST /repos/{owner}/{repo}/git/commits, PATCH /repos/{owner}/{repo}/git/refs/heads/main)
   • Preserve docstore commit metadata in the git commit message:

     {docstore commit message}
     
     docstore-repo: {repo}
     docstore-sequence: {sequence}
     docstore-author: {author}
     docstore-branch: main
     

   • Parent SHA is the previous mirror commit (tracked in state — see below).

4. Track state — store the last mirrored (sequence → git SHA) mapping persistently so the mirror can resume after restarts without re-replaying history.

---

Architecture

Option A: Mirror service in the docstore server

Add a MirrorManager to the docstore server that subscribes to the in-process broker (SSE fan-out) and mirrors on commit.created. State stored in a new mirror_config DB table.

Option B: Standalone mirror binary

A separate cmd/mirror binary that:

• Registers a webhook subscription with docstore for commit.created on main
• Exposes POST /webhook (same pattern as ci-runner)
• Runs mirror logic on each delivery

Recommendation: Option B — keeps the mirror concern out of the core server, follows the ci-runner pattern, deployable independently on Cloud Run.

---

Configuration

Flag / Env var	Description
--docstore-url	Base URL of the docstore server
--repo	Docstore repo to mirror (e.g. default/docstore)
--github-repo	Target GitHub repo (owner/repo)
--github-branch	Target branch (default main)
OCTO_STS_AUDIENCE	octo-sts audience (typically the GitHub repo URL)
--webhook-secret	HMAC secret for incoming docstore webhooks
--runner-url	Public URL of this mirror service (for webhook registration)
--state-file	Path to persist last-mirrored sequence + git SHA (default /tmp/mirror-state.json)

For Cloud Run: use a GCS-backed state file or a small DB table instead of a local file (instance may be replaced).

---

octo-sts setup

1. Install the octo-sts GitHub App on the target repo
2. Create .octo-sts/policy.yaml in the target GitHub repo granting the Cloud Run service account's identity contents: write access
3. No long-lived secrets needed — auth is fully OIDC-based

---

Conflict handling

If the target GitHub repo has diverged (e.g. direct commits were pushed), force-push the mirror commit. The target repo is owned by the mirror; direct commits should not be made.

---

Initial sync

On first run (no state), replay all commits on main from sequence 1 to current HEAD in order, creating one git commit per docstore commit. This may take a few minutes for repos with long history.

---

State persistence

For Cloud Run (stateless instances): store {sequence, git_sha} in either:

• A GCS object (gs://bucket/mirror-state/{repo}.json) — simplest
• A new mirror_state DB table in docstore — more reliable

Start with GCS; the flag --state-bucket enables it.

---

Acceptance Criteria

• cmd/mirror binary with POST /webhook handler
• Auto-registers webhook subscription with docstore on startup
• octo-sts token exchange on each mirror operation (fresh token per commit)
• Git tree created via GitHub API matching docstore tree at that sequence
• Docstore commit metadata preserved in git commit message
• Parent SHA tracked and used for incremental commits
• Initial sync from sequence 1 on first run
• State persisted to GCS (configurable bucket) or local file
• go test ./... passes (unit tests with httptest stubs for GitHub API and octo-sts)

Dependencies

• Eventing system (Eventing: CloudEvents delivery via SSE, webhooks, and Pub/Sub #41 Milestone 1) ✓
• Archive endpoint (CI runner: BuildKit-based check execution on Cloud Run gen2 #75) ✓
• octo-sts app installed on target GitHub repo (ops setup)