|
| 1 | +--- |
| 2 | +name: local-platform-e2e |
| 3 | +description: Stand up benchmarks-platform locally (Postgres + MinIO + ClickHouse in docker) and run a real @benchsdk/cli benchmark against it, with no cloud or provider credentials. Use when testing @benchsdk/client / @benchsdk/cli against the platform end to end, or when debugging benchmark reporting, worker planning, artifacts, or dashboard results locally. |
| 4 | +--- |
| 5 | + |
| 6 | +# Local end-to-end: @benchsdk/cli ↔ benchmarks-platform |
| 7 | + |
| 8 | +Goal: exercise upsert benchmark → create run → planWorkers → claim → heartbeat → |
| 9 | +task_results → artifact upload → complete → dashboard, with zero external |
| 10 | +credentials. |
| 11 | + |
| 12 | +## 1. Infra (docker) |
| 13 | + |
| 14 | +```bash |
| 15 | +docker run -d --name pg -p 5432:5432 -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=bench postgres:16 |
| 16 | +docker run -d --name minio -p 9000:9000 -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \ |
| 17 | + quay.io/minio/minio server /data |
| 18 | +docker run -d --name ch -p 8123:8123 -e CLICKHOUSE_PASSWORD=chpass clickhouse/clickhouse-server:25.6 |
| 19 | +docker run --rm --network host --entrypoint sh quay.io/minio/mc -c \ |
| 20 | + "mc alias set local http://127.0.0.1:9000 minioadmin minioadmin && mc mb -p local/bench" |
| 21 | +``` |
| 22 | + |
| 23 | +- Use ClickHouse **>= 25.x**: 24.8 fails `ch:migrate` with |
| 24 | + `TTL expression result column should have DateTime or Date type, but has DateTime64(3,'UTC')`. |
| 25 | +- The importer passes `clickhouse_settings: { date_time_input_format: 'best_effort' }` |
| 26 | + per insert, so a default-configured server works. If you hit |
| 27 | + `Cannot parse input: expected '"' before: 'Z"...'` (older code), work around it |
| 28 | + server-side, and remember to REMOVE the override before verifying an importer |
| 29 | + fix — otherwise the server setting masks it: |
| 30 | + ```bash |
| 31 | + docker exec ch bash -c 'mkdir -p /etc/clickhouse-server/users.d && printf "<clickhouse><profiles><default><date_time_input_format>best_effort</date_time_input_format></default></profiles></clickhouse>" > /etc/clickhouse-server/users.d/besteffort.xml' |
| 32 | + docker restart ch |
| 33 | + # verify which mode is actually active: |
| 34 | + curl -s "http://127.0.0.1:8123/?user=default&password=chpass" \ |
| 35 | + --data-binary "SELECT value FROM system.settings WHERE name='date_time_input_format'" |
| 36 | + ``` |
| 37 | + |
| 38 | +## 2. benchmarks-platform `.env.local` |
| 39 | + |
| 40 | +Point `TIGRIS_*` at MinIO — the events and artifacts routes require S3 or they |
| 41 | +return 502 on every batch. `region: "auto"` + presigned PUT works with MinIO. |
| 42 | + |
| 43 | +``` |
| 44 | +DATABASE_URL=postgresql://postgres:postgres@127.0.0.1:5432/bench |
| 45 | +DATABASE_URL_UNPOOLED=postgresql://postgres:postgres@127.0.0.1:5432/bench |
| 46 | +CLICKHOUSE_URL=http://127.0.0.1:8123 |
| 47 | +CLICKHOUSE_DATABASE=default |
| 48 | +CLICKHOUSE_USER=default |
| 49 | +CLICKHOUSE_PASSWORD=chpass |
| 50 | +ADMIN_API_KEY=local-admin-key |
| 51 | +BETTER_AUTH_SECRET=<openssl rand -hex 32> |
| 52 | +BETTER_AUTH_URL=http://localhost:3000 |
| 53 | +TIGRIS_ACCESS_KEY_ID=minioadmin |
| 54 | +TIGRIS_SECRET_ACCESS_KEY=minioadmin |
| 55 | +TIGRIS_STORAGE_ENDPOINT=http://127.0.0.1:9000 |
| 56 | +TIGRIS_BUCKET=bench |
| 57 | +``` |
| 58 | + |
| 59 | +Then: |
| 60 | +```bash |
| 61 | +npm run db:migrate |
| 62 | +CLICKHOUSE_URL=http://127.0.0.1:8123 CLICKHOUSE_USER=default CLICKHOUSE_PASSWORD=chpass npm run ch:migrate |
| 63 | +npm run dev |
| 64 | +``` |
| 65 | +`ch:migrate` does **not** read `.env.local`; without exported vars it silently |
| 66 | +prints "CLICKHOUSE_URL not set; skipping". |
| 67 | + |
| 68 | +## 3. Mandatory seed (otherwise every run creation 500s on an FK) |
| 69 | + |
| 70 | +`app/api/v1/benchmarks/[slug]/runs/route.ts` hardcodes a default org and user id |
| 71 | +for every created run. Insert those exact rows: |
| 72 | + |
| 73 | +```sql |
| 74 | +INSERT INTO "user" (id,name,email,email_verified) |
| 75 | +VALUES ('mxYI5c90QNkRPhvuc2HvNHy5mM7MG7jG','David Tice','david@example.com',true); |
| 76 | +INSERT INTO organization (id,name,slug,created_at,owner_id) |
| 77 | +VALUES ('zMrSfAyEyVJ2eKIxgoBZtrMvEd6a78ad','ComputeSDK','computesdk',now(),'mxYI5c90QNkRPhvuc2HvNHy5mM7MG7jG'); |
| 78 | +INSERT INTO member (id,organization_id,user_id,role,created_at) |
| 79 | +VALUES ('mem1','zMrSfAyEyVJ2eKIxgoBZtrMvEd6a78ad','mxYI5c90QNkRPhvuc2HvNHy5mM7MG7jG','owner',now()); |
| 80 | +``` |
| 81 | +Re-read those constants before seeding — they may change. |
| 82 | + |
| 83 | +As of the org-scoped-auth change (`lib/api/api-auth.ts` `requireApiAuth`), this seed is |
| 84 | +only needed for the **admin-key** path: an org-scoped key supplies the run's |
| 85 | +`organizationId` itself and `userId` from the key's `created_by`. Admin-key runs with no |
| 86 | +`organizationId` in the body still fall back to those two hardcoded ids. |
| 87 | + |
| 88 | +## 3b. Minting org-scoped `bp_` API keys locally |
| 89 | + |
| 90 | +The org API-key HTTP route is **session**-scoped and answers `{"error":"Unauthenticated"}` |
| 91 | +to the admin key, so keys can only be created from the dashboard UI or directly. For |
| 92 | +scripted multi-tenant tests, mint them with the platform's own generator so the sha256 |
| 93 | +hash/prefix/lastFour match what `verifyApiKey()` expects: |
| 94 | + |
| 95 | +```ts |
| 96 | +// scripts/e2e-seed-orgs.ts (throwaway), run with: |
| 97 | +// npx tsx --env-file=.env.local scripts/e2e-seed-orgs.ts |
| 98 | +import { generateApiKey } from "@/lib/api-keys"; |
| 99 | +import { apiKey, member, organization, user } from "@/db/auth-schema"; |
| 100 | +const g = generateApiKey(); // g.plaintext is the bp_<prefix>_<secret> to send |
| 101 | +await db.insert(apiKey).values({ |
| 102 | + id, organizationId, name, prefix: g.prefix, hashedKey: g.hashedKey, |
| 103 | + lastFour: g.lastFour, createdBy: someUserId, createdAt: new Date(), |
| 104 | + revokedAt: null, expiresAt: null, // set these to test revoked/expired → 401 |
| 105 | +}); |
| 106 | +``` |
| 107 | +Each org needs `user` + `organization` (`owner_id`) + `member` rows first. To view an |
| 108 | +org's runs in the dashboard, add a `member` row for your dashboard user in that org — |
| 109 | +dashboard auth is session-based and completely separate from API keys. |
| 110 | + |
| 111 | +## 4. Dashboard access |
| 112 | + |
| 113 | +Sign up via `POST /api/auth/sign-up/email` (email+password is enabled), then add |
| 114 | +a `member` row for that user in the seeded org, and sign in at |
| 115 | +`http://localhost:3000/signin`. Run pages live at |
| 116 | +`/{orgSlug}/benchmarks/{benchmarkSlug}/runs/{runId}` (+ `/workers`). |
| 117 | + |
| 118 | +## 5. Getting results into ClickHouse locally |
| 119 | + |
| 120 | +`@vercel/queue.send()` fails locally (swallowed as a warning), so nothing |
| 121 | +imports automatically. Trigger it by hand — the cron route accepts the admin key |
| 122 | +when `CRON_SECRET` is unset: |
| 123 | +```bash |
| 124 | +curl -s "http://localhost:3000/api/cron/import-clickhouse?limit=50" -H "Authorization: Bearer local-admin-key" |
| 125 | +``` |
| 126 | +Check `imported`/`failed`/`failureSamples` in the response. |
| 127 | + |
| 128 | +## 6. Running a benchmark with no provider credentials |
| 129 | + |
| 130 | +Build first (`packages/*/dist` is not committed): `pnpm install && pnpm -r --filter "./packages/**" build`. |
| 131 | + |
| 132 | +Write a throwaway bench inside the repo (untracked, e.g. `e2e-local/local.bench.ts`) |
| 133 | +so pnpm workspace resolution finds `@benchsdk/cli`, with a fake participant: |
| 134 | + |
| 135 | +```ts |
| 136 | +import { defineBenchmark, runBenchmark } from '@benchsdk/cli'; |
| 137 | +const config = defineBenchmark({ |
| 138 | + benchmarkSlug: 'e2e-local', benchmarkName: 'E2E', iterations: 4, concurrency: 1, |
| 139 | + task: async (ctx) => { await ctx.step('create', () => new Promise(r => setTimeout(r, 50))); }, |
| 140 | +}); |
| 141 | +runBenchmark(config, [{ name: 'local', requiredEnvVars: [] } as any], process.argv.slice(2)); |
| 142 | +``` |
| 143 | + |
| 144 | +Run it: |
| 145 | +```bash |
| 146 | +BENCHMARKS_PLATFORM_URL=http://localhost:3000 COMPUTESDK_ADMIN_API_KEY=local-admin-key \ |
| 147 | + npx tsx e2e-local/local.bench.ts --iterations 4 --concurrency 2 |
| 148 | +``` |
| 149 | +`BENCHMARKS_PLATFORM_URL` is the **root** URL (the runner appends `/api/v1`). |
| 150 | + |
| 151 | +The runner reads `COMPUTESDK_ADMIN_API_KEY ?? COMPUTESDK_API_KEY`, so to prove the master key |
| 152 | +is not needed, pass an org `bp_` key as `COMPUTESDK_API_KEY` and strip the admin vars from the |
| 153 | +child env (`env -u COMPUTESDK_ADMIN_API_KEY -u ADMIN_API_KEY …`). Note the "View at:" URL the |
| 154 | +CLI prints uses `BENCHMARKS_PLATFORM_ORG_SLUG` (default `computesdk`), **not** the org that owns |
| 155 | +the key, so with an org key the printed link may 404 — set that env var to the key's org slug. |
| 156 | + |
| 157 | +## 6b. Probing tenant isolation |
| 158 | + |
| 159 | +With benchmarks-platform#47, all `app/api/v1/benchmarks/**` routes take either the global |
| 160 | +`ADMIN_API_KEY` or an org `bp_` key. Expected shapes when a key from org B addresses org A's run: |
| 161 | +`403 {"error":"API key is not authorized for this organization"}` (`getScopedRun` / |
| 162 | +`requireRunAccess`), `404` for an unknown run/benchmark, `401 Invalid API key` for |
| 163 | +revoked/expired/malformed tokens, `401 API key required` with no header. Listings |
| 164 | +(`GET /benchmarks/:slug/runs` and `/results`) narrow by `organizationScope(auth)`. |
| 165 | + |
| 166 | +When probing the **heartbeat** in-process cache, the body must carry `currentStep`, all four |
| 167 | +`progressDone/InFlight/Errors/Total` fields **and** `concurrency` (top-level, not nested) |
| 168 | +or the cache is never populated and your "cache poisoning" probe proves nothing. A |
| 169 | +cache-served beat is recognisable because the response's `worker` object is minimal (no |
| 170 | +`status`/`benchmarkId` columns). Route timing metadata (`cacheCoalesced`) is only logged |
| 171 | +for requests slower than 1000 ms, so don't rely on the dev-server log for this. |
| 172 | + |
| 173 | +## 7. Useful probes when testing lifecycle behaviour |
| 174 | + |
| 175 | +- `psql` is not installed on the host — use `docker exec pg psql -U postgres -d bench ...`. |
| 176 | +- To catch transient run statuses (`planned` → `in_progress` → `completed`), poll |
| 177 | + in a background subshell **while** the bench runs, then `sort -u` the samples: |
| 178 | + ```bash |
| 179 | + (for i in $(seq 1 60); do docker exec pg psql -U postgres -d bench -tA -c \ |
| 180 | + "select r.status||'|'||p.status||'|'||w.status from ..."; sleep 0.4; done > /tmp/poll.txt) & |
| 181 | + ``` |
| 182 | + Note `exec`'s `&` backgrounds the whole `cd X && ...` chain — pass `workdir` |
| 183 | + instead of a leading `cd`, or the foreground command runs in the wrong dir. |
| 184 | +- To force a failed run, make the harness task throw for one taskIndex; the CLI |
| 185 | + calls `failWorker` when any task fails (runner.ts), which is what drives the |
| 186 | + run to `failed`. |
| 187 | +- Worker release / re-claim and oversized event batches are easiest to drive with |
| 188 | + raw curl/python against the v1 API using the admin key; the events body needs |
| 189 | + `{type:'task_results', sequenceNumber, isFinal, attemptId, records:[...]}`. |
| 190 | + |
| 191 | +## 8. Known sharp edges (verify before blaming your setup) |
| 192 | + |
| 193 | +- `--group-by round`: `targetConcurrency` is read by the platform as |
| 194 | + tasks-per-worker, so the runner must send the full `schedule.length`. If it |
| 195 | + sends 1, only one task is planned (worker range 0-0) while every record is |
| 196 | + still accepted — results look right but progress/ranges do not. |
| 197 | +- `benchmark_run_executions.status` should roll up from worker status |
| 198 | + (`in_progress` on first claim, `completed`/`failed` on the last worker). If a |
| 199 | + finished run still reads `planned`, the rollup is broken — the dashboard badge |
| 200 | + derives status separately and will hide it, so check Postgres and |
| 201 | + `/progress.run.status` vs `/progress.summary.status`, which must agree. |
| 202 | +- Barriers: custom step names work off the `concurrency` samples the heartbeat |
| 203 | + sends, but `worker.ready` and `sandbox.live` are derived from the worker's |
| 204 | + `progress_in_flight` column — for those you must call |
| 205 | + `reporter.setProgress({ inFlight: N, ... })` first or the barrier sees |
| 206 | + `active=0` and hangs until timeout. |
| 207 | +- `client.releaseWorker()` → 200 puts the worker back to `pending` and the |
| 208 | + attempt to `released`; a re-claim then gets `attemptNumber: 2`. A duplicate-key |
| 209 | + error on `unique(worker_id, attempt_number)` means the claim route hardcoded |
| 210 | + attempt number 1. |
| 211 | +- Body limits: the events route allows 32 MiB (`MAX_EVENT_BODY_BYTES`), all other |
| 212 | + v1 routes 1 MiB. 5,000 records × 3 steps ≈ 3.15 MiB, so full-size SDK batches |
| 213 | + should be 202. If you see 413 on events, the raised cap is not wired up. Always |
| 214 | + check `benchmark_event_batches.status = 'persisted'` too — a 202 only means the |
| 215 | + body was accepted. |
| 216 | +- Org-scoped `bp_…` keys on benchmark v1 routes arrived with |
| 217 | + benchmarks-platform#47 (see §3b/§6b). On a platform checkout without it those |
| 218 | + routes are admin-only and every `bp_…` key 401s there, even though the same key |
| 219 | + works on `/api/v1/organizations/...` — check which behaviour your checkout has |
| 220 | + before debugging a 401. |
| 221 | + |
| 222 | +## Devin Secrets Needed |
| 223 | + |
| 224 | +None. This entire flow runs offline with local docker containers and a |
| 225 | +self-chosen `ADMIN_API_KEY`. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?v=4&size=40){kind=avatar}
0 commit comments