Improvements to throughput benchmark - #502
Conversation
tlwillke
commented
Jul 18, 2025
tlwillke
commented
- Warmup now uses random vectors instead of a split of the test vectors
- In addition to Avg QPS, can report Median QPS and Max QPS
- Number of warmup and test passes is configurable
- Overloaded QueryExecutor to accept random vector queries
…nd max QPS options.
MarkWolters
left a comment
MarkWolters
left a comment
There was a problem hiding this comment.
Looks good, I would just like to understand why we are modifying this process to use a randomly generated vector rather than the set of query vectors from the dataset.
| IntStream.range(0, totalQueries) | ||
| .parallel() | ||
| .forEach(k -> { | ||
| // Generate a random vector |
There was a problem hiding this comment.
If the associated ConfiguredSystem has a Dataset already why are we generating a random vector rather than using the queryVector? I understand this works with the change to QueryExecutor, I'm just not clear on why we are making this change?
There was a problem hiding this comment.
There are often very few queries in a dataset. Best practice is to not use the same vectors for warmup that you use for testing (due to potential caching, fitting, etc.). Using the query set reduces the split available for testing. Now, we are using random, since it really doesn't matter what you run through for warmup.
* Improved throughput benchmark, adding random vector warmups, median and max QPS options. * Added std dev measure.
2 participants
