ci: unbreak benchmark result commits after the pnpm migration - #265
Conversation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Contributor License AgreementAll contributors are covered by a CLA. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| git config user.name "github-actions[bot]" | ||
| git config user.email "github-actions[bot]@users.noreply.github.com" | ||
| git add package.json package-lock.json results.svg *_tti.svg pricing.svg results/ | ||
| git add package.json results.svg *_tti.svg pricing.svg results/ |
There was a problem hiding this comment.
🟡 Weekly dependency refreshes are silently thrown away instead of being recorded
The updated dependency lock file is no longer included in the automatic commit (git add package.json ... at .github/workflows/sandbox-tti-benchmarks.yml:302) after the weekly run refreshes dependencies, so the refreshed versions are discarded and the repository keeps recording old ones.
Impact: Scheduled runs benchmark newer provider SDK versions than the repository records, and the dependency refresh never lands, so the recorded versions drift permanently out of date.
Why removing the lock-file pathspec loses the pnpm update result
The collect jobs run pnpm update on schedule events (.github/workflows/sandbox-tti-benchmarks.yml:164-169), which rewrites pnpm-lock.yaml in the working tree. Before the pnpm migration the equivalent npm update output was persisted via git add package.json package-lock.json .... The PR removes the now-nonexistent package-lock.json pathspec but does not substitute pnpm-lock.yaml, so the modified lock file stays unstaged. Combined with the new git rebase --autostash, the dirty lock file is stashed/reapplied and then dropped when the runner is torn down. The same omission applies to .github/workflows/browser-benchmarks.yml:226, .github/workflows/browser-throughput-benchmarks.yml:273, .github/workflows/snapshot-fork-benchmarks.yml:284 and .github/workflows/storage-benchmarks.yml:260.
| git add package.json results.svg *_tti.svg pricing.svg results/ | |
| git add package.json pnpm-lock.yaml results.svg *_tti.svg pricing.svg results/ |
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Good catch — staged pnpm-lock.yaml in all 5 workflows so the scheduled pnpm update result is recorded again.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sandbox Dax Benchmark Results
|
AI Gateway Benchmark Results
View full run · SVG available as a build artifact |
1 participant
Summary
No benchmark results have been committed since 2026-07-24, so computesdk.com/benchmarks/* is stale. Two independent breakages in the "Commit and push" step, both hit again in today's scheduled runs:
package-lock.json, but 5 workflows still stage it:git addon a missing pathspec is fatal (fatal: pathspec 'package-lock.json' did not match any files), killing sandbox-tti, storage, browser, browser-throughput and snapshot-fork before the push.error: cannot rebase: You have unstaged changesbecause the run leaves other tracked files dirty. Every rebase in the retry loop now autostashes:Applied to all workflows with the retry loop so the sandbox-* ones don't reintroduce it.
Not addressed here: the 11 sandbox-* workflows registered in #? on 2026-07-29 (memory, disk, system, cpu-node, pgbench, network-*, dns, download, latency, realworld) fail earlier —
pnpm run build:bundlesandpnpm run generate-<mode>-svgdon't exist inpackage.json— which is a separate fix.Link to Devin session: https://app.devin.ai/sessions/707b6c6fe3c24b5caf22cdb19eafedf2
Requested by: @HeyGarrison