[tx] Add separate JIT vs post-JIT timing measurement to memory benchmark#1021
[tx] Add separate JIT vs post-JIT timing measurement to memory benchmark#1021pcmoritz merged 4 commits intoNovaSky-AI:mainfrom
Conversation
The benchmark now sends multiple requests per test: the first request triggers JIT compilation and subsequent requests measure actual post-JIT runtime. This provides clearer insight into compilation overhead vs steady-state performance. - Add jit_e2e_sec field to capture first request (JIT) time - Rename client_e2e_sec to post_jit_e2e_sec for clarity - Add --num-measurement-iters flag (default: 3) for post-JIT iterations - Update CSV output and summary tables to show both metrics Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
The pull request successfully introduces separate timing measurements for JIT compilation and post-JIT runtime, providing a clearer understanding of performance characteristics. The changes involve updating the BenchmarkConfig and TestResult data structures, modifying the test execution logic to perform warmup and multiple measurement iterations, and updating output formats (CLI summary and CSV) to reflect the new metrics. The addition of the --num-measurement-iters CLI flag makes the benchmark more configurable.
Address PR feedback: - Rename parameter to match CLI flag name for consistency - Remove default value since it's always passed from caller - Extract common timing logic into _run_timed_requests helper method - Use random tokens for each request Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
ed5d1ee to
cf65e60
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
The pull request successfully introduces separate timing measurements for JIT compilation and post-JIT runtime, significantly enhancing the memory benchmark's insights. The changes are well-integrated across the configuration, data structures, execution logic, and reporting. The addition of _run_timed_requests effectively encapsulates the new measurement methodology, and the updated output formats provide clear visibility into the new metrics.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Summary
_run_timed_requestshelper methodChanges
jit_e2e_secfield to capture first request (JIT) timeclient_e2e_sectopost_jit_e2e_secfor clarity--num-measurement-itersflag (default: 3) for post-JIT iterations