Skip to content

CHANGELOG.md

Latest commit

 

History

History
34 lines (29 loc) · 1.78 KB

CHANGELOG.md

File metadata and controls

34 lines (29 loc) · 1.78 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog.

[released] - 2026-07-14

Added

  • Adaptive performance regression testing for benchmarks: metrics from historical runs (stored in S3 Parquet) are used to compute statistical baselines and detect regressions automatically.
  • New _compute_baselines() function using median + MAD (k=3.0, scaled MAD with an explicit minimum absolute margin: max(min_margin, k * scaled_MAD)) for robust threshold computation.
  • New _check_reference_performance() function to compare tracked metrics against computed baselines using upper_limit and lower_limit keys.
  • New metrics.parquet_metrics module (load_scenario_metrics, load_recent_scenario_metrics_map) for loading per-scenario metric history from Parquet on S3 with optimized column projection and filter pushdown.
  • max_age_days parameter in history loading helpers to discard stale historical runs.
  • github_issue_handler now includes ScenarioRunInfo and PerformanceRegressionInfo for formatting benchmark and regression issue markdown.
  • PerformanceRegressionInfo now includes scenario metadata (definition link, backend, contacts) and presents a cost-trend table with mermaid charts.
  • PerformanceRegressionInfo now accepts an optional BenchmarkScenario to enrich regression issues with scenario definition links and contact information.
  • Weekly regression-testing.yml workflow that reads S3 Parquet history and opens or updates GitHub issues for detected cost regressions.
  • Unit tests for regression metrics in test_regression_metrics.py covering threshold computation, baseline computation, violation detection, and issue body rendering.

Fixed