Problem
Recall in msmarco-v2-vector relies on a precalculated ground-truth file (queries-recall.json.bz2) containing true top-k nearest neighbors computed via brute-force over the full corpus.
When running with a dataset subset, the ground-truth document IDs may not exist in the index, producing meaningless recall numbers. There is currently no way to get valid recall metrics for a subset.
Proposal
When a subset of the dataset is indexed, the track should recompute brute-force ground truth dynamically (e.g. as part of the benchmark run) instead of using the precalculated file, and fall back to the existing file for full-corpus runs.
Problem
Recall in msmarco-v2-vector relies on a precalculated ground-truth file (queries-recall.json.bz2) containing true top-k nearest neighbors computed via brute-force over the full corpus.
When running with a dataset subset, the ground-truth document IDs may not exist in the index, producing meaningless recall numbers. There is currently no way to get valid recall metrics for a subset.
Proposal
When a subset of the dataset is indexed, the track should recompute brute-force ground truth dynamically (e.g. as part of the benchmark run) instead of using the precalculated file, and fall back to the existing file for full-corpus runs.