Skip to content

ci: post benchmark diff report as PR comment #31

Description

@riccardovietri

Summary

When a PR modifies physics or synthesis logic, reviewers currently have no automatic signal about whether the benchmark numbers changed. Add a CI step that runs the three benchmark scenarios, diffs the key metrics against the base branch, and posts the delta as a PR comment.

Goal

Make numeric regressions visible in the PR review interface without requiring reviewers to check out and run examples locally.

Acceptance criteria

  • GitHub Actions workflow step that runs python examples/decision_memo_examples.py and extracts key metrics (Tj at recommended flow, risk level, min/recommended LPM, margin remaining)
  • Step runs on both the PR branch and the merge base, diffs the structured output
  • Posts a table comment to the PR like:
## Benchmark Delta (PR vs base)

| Scenario          | Metric              | Base   | PR     | Δ      |
|-------------------|---------------------|--------|--------|--------|
| H100 parallel     | Tj_rec (°C)         | 67.4   | 67.4   | 0.0    |
| H100 series       | Tj_hottest (°C)     | 73.6   | 73.6   | 0.0    |
| B200-proxy        | min_lpm             | 6.2    | 6.2    | 0.0    |
  • Comment is updated (not duplicated) on subsequent pushes to the same PR
  • Comment is skipped if no benchmark-relevant files changed (physics.py, decision_report.py, schemas.py)

Notes

  • Only fires on PRs, not on pushes to main
  • Should use a bot token or GITHUB_TOKEN for posting comments
  • Related to snapshot tests issue — both are regression-detection mechanisms, but this one surfaces the delta in the PR view without requiring a snapshot approval step

Context

Relevant files: .github/workflows/, examples/decision_memo_examples.py

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions