Release Notes for 4.0.0-rc.9

docs/release notes/4.0.0-RC.9/617.docs.md

@@ -0,0 +1,4 @@
+This includes:
+
+- A set of tutorials covering basic JVector concepts
+- A guide for running benchmarks

docs/release notes/4.0.0-RC.9/625.testing.md

@@ -0,0 +1,93 @@
+## Summary
+
+This PR introduces YAML-driven benchmark compute + reporting selection, and adds run-scoped logging artifacts (`sys_info.json`, `dataset_info.csv`, `experiments.csv`) so Grid benchmark runs are reproducible and easier to analyze offline. Reporting/benchmark policy is now run-level (via `yaml-configs/run.yml`), while dataset YAML files are simplified to dataset-tuned construction/search settings only.
+
+---
+
+## Key changes
+
+### Run-level policy file: `yaml-configs/run.yml`
+
+- Added a new run-level config file that defines:
+    - **benchmark computation policy** (`benchmarks`: what to compute)
+    - **console projection** (`console`: subset + named metrics)
+    - **experiments logging policy** (`logging`: selection + sink type + run metadata)
+- Dataset YAMLs are now focused on **dataset-tuned parameters only** (construction/search grids). Reporting/benchmark controls are no longer expected in dataset YAMLs.
+
+### Config versioning clarity
+
+- Renamed the old vague YAML `version` field to **`onDiskIndexVersion`** (still validated against `OnDiskGraphIndex.CURRENT_VERSION`).
+- Introduced **`yamlSchemaVersion`** (starting at **1**) to version YAML schema independently of on-disk index header format.
+- Added deprecated back-compat for legacy files that still use `version: 6`.
+
+### Compute vs display vs logging separation
+
+- Benchmarks specify **what is computed**; console/logging are **projections** only.
+- Added stable `Metric.key` identifiers for all benchmark outputs and telemetry so selection and CSV columns are robust (no header-string matching).
+- Enforced:
+    - **strict subset validation** for benchmark-stat selections (must be computed)
+    - **best-effort warnings** for runtime-unavailable telemetry/metrics (omitted from output/CSV, with warnings)
+
+### Run-scoped artifacts and logging
+
+- When logging is enabled, each BenchYAML invocation creates a run directory under:
+    - `logging/<run_id>/`
+- The run directory contains (see examples below):
+    - `sys_info.json` — host/OS/JVM/SIMD/threading/memory + stable `system_id`
+    - `dataset_info.csv` — one row per dataset used in the run
+    - `experiments.csv` — flat, append-only table of fixed params + selected output-key columns
+- `experiments.csv` schema stability:
+    - fixed columns are defined centrally in `ExperimentsSchemaV1`
+    - output-key columns are an ordered union across all topK scenarios encountered in the run
+    - non-applicable / missing values are written as **empty CSV fields** (easy for pandas/matplotlib)
+
+### Plumbing / ergonomics
+
+- Added `RunArtifacts` to centralize:
+    - run setup + validation
+    - dataset registration (`dataset_info.csv`)
+    - row logging (`experiments.csv`)
+- Updated BenchYAML and HelloVectorWorld to use `run.yml` + `RunArtifacts` (no manual wiring of compute/display/logging maps).
+- Refactored Grid signatures to take a single `RunArtifacts` object instead of many reporting parameters.
+- Legacy callers continue to work via the legacy Grid overload + `RunArtifacts.disabled()`.
+
+### Legacy compatibility
+
+- Legacy YAML files **without** `yamlSchemaVersion` are treated as **schema 0**:
+    - parsed leniently (unknown fields ignored)
+    - emit a **deprecation warning**
+    - optional extraction of legacy `search.benchmarks` is supported for legacy behavior when `run.yml` is absent
+- AutoBenchYAML is not migrated to run.yml yet; it continues to work with legacy configs.
+
+---
+
+## Behavior
+
+- **Logging enabled** (`run.yml` has `logging.type`):
+    - BenchYAML writes run artifacts under `logging/<run_id>/`
+    - prints a startup message pointing to the run directory and how to delete it to reclaim space
+    - appends experiment rows as configurations execute
+- **Logging disabled** (no `logging` section or blank `type` in `run.yml`):
+    - run artifacts are not created
+    - `RunArtifacts` becomes a no-op via `RunArtifacts.disabled()`
+- Console output continues to work via run-level selection; dataset configs tune only construction/search parameters.
+
+---
+
+## Notes / limitations
+
+- `experiments.csv` uses `Metric.key` strings as column names for benchmark outputs/telemetry; missing values are empty CSV fields.
+- `sys_info.json` SIMD reports the configured vector width via `io.github.jbellis.jvector.vector_bit_size` (with `simd_config_present`), avoiding brittle runtime probing.
+- Threading in `sys_info.json` distinguishes build executor parallelism vs parallel streams (common FJP), explaining why build/query values can differ.
+- This PR intentionally does not overhaul per-index build time attribution (e.g., cache hits have no build time by design); deeper build-timing correctness is left for a future pass.
+- Dataset YAML breaking changes are intentional: `version` → `onDiskIndexVersion`, plus removal of dataset-level benchmarks/console/logging fields in favor of `run.yml`.
+
+## Example of run.yml and yaml-config for dataset
+<img width="905" height="849" alt="ada002-100k yml" src="https://github.com/user-attachments/assets/8c5db3e7-78b8-4618-93cd-4dbc29703b5b" />
+<img width="801" height="778" alt="run yml" src="https://github.com/user-attachments/assets/cc849f22-2674-463d-9223-d0c018148038" />
+
+## Examples of artifacts (sys_info, dataset_info, experiments)
+
+<img width="1501" height="555" alt="experiments csv" src="https://github.com/user-attachments/assets/9eb382a8-86cc-4877-97e1-4fdcdc09cbe8" />
+<img width="1101" height="106" alt="dataset_info csv" src="https://github.com/user-attachments/assets/73597f6d-f7c5-4ba5-b4e0-bdb93661df4b" />
+<img width="518" height="774" alt="sys_info json" src="https://github.com/user-attachments/assets/45238e20-159c-4f1b-99cf-352a212ec077" />

docs/release notes/4.0.0-RC.9/631.testing.md

@@ -0,0 +1,6 @@
+This PR fixes a resource leak where BenchmarkDiagnostics (and its DiskUsageMonitor/NIO WatchService) was created without being closed in several benchmark execution paths. Over repeated runs/config sweeps this leaked OS watch resources/file handles and could fail with “User limit of inotify instances reached.”
+
+Changes:
+- Wrap BenchmarkDiagnostics usage in try-with-resources in QueryTester and Grid run paths so it is always closed.
+- Refactor ThroughputBenchmark to create BenchmarkDiagnostics within runBenchmark using try-with-resources (instead of storing it as a field), ensuring proper cleanup.
+- Remove temporary debugging counters/prints once closure was verified.

docs/release notes/4.0.0-RC.9/634.testing.md

@@ -0,0 +1,21 @@
+This PR expands the capability of the DiskUsageMonitor in the BenchYAML and autoBenchYAML testing applications to be able to monitor more than a single directory and to provide detailed information on disk used and number of new files for each directory monitored, as well as a summary of the total bytes used and number of new files across all monitored directories.
+
+Although this allows DiskUsageMonitor to monitor an arbitrary number of directories in practice as of this PR it is used to monitor the work directory and the cache directory which contains indexes that may be reused for testing purposes.
+
+example rollup summary at index construction time:
+```
+Disk Usage Summary Graph Index Build:
+  [testDirectory]:
+    Total Disk Used: 1.61 GB
+    Total Files: 1
+    Net Change: 1.61 GB, +1 files
+  [indexCache]:
+    Total Disk Used: 0 B
+    Total Files: 0
+    Net Change: 0 B, +0 files
+  [Overall Total]:
+    Total Disk Used: 1.61 GB
+    Total Files: 1
+    Net Change: 1.61 GB, +1 files
+Index build time: 956.781921 seconds
+```

docs/release notes/4.0.0-RC.9/636.testing.md

@@ -0,0 +1,37 @@
+## Summary
+
+This PR adds quantization timing telemetry to BenchYAML reporting (console + experiments.csv), and makes it selectable from `run.yml`. The new metrics capture quantizer **compute** and **encoding** time for both **index construction** and **search-time vector encoding**, with stable `Metric.key` identifiers.
+
+## Key changes
+
+### Quantization timing metrics (new)
+- Emit quant timing metrics as `Metric` outputs:
+    - **Index construction:** `construction.index_quant_time_s.<QT>.(compute_time_s|encoding_time_s)`
+    - **Search-time:** `search.search_quant_time_s.<QT>.(compute_time_s|encoding_time_s)`
+- Per-quantizer keys are emitted for PQ/BQ; NVQ reports **compute only** (NVQ encoding is intentionally not measured).
+- Human-readable headers are phase-tagged to disambiguate:
+    - `Idx PQ compute (s)`, `Idx PQ encoding (s)`, `Search PQ compute (s)`, etc.
+
+### Run-level selection support (prefix expansion)
+- Added prefix-style named metric selection in `run.yml`:
+    - `metrics.construction.index_quant_time_s` → expands to all `construction.index_quant_time_s.*` metrics
+    - `metrics.construction.search_quant_time_s` → expands to all `search.search_quant_time_s.*` metrics
+- Selection + application now supports “prefix families” of metrics (similar to `recall_at_{topK}` expansion).
+
+### Logging fixes (experiments.csv)
+- Updated logging schema planning to include prefix-selected metrics so quant timing columns appear in `experiments.csv` when selected via run.yml.
+- CSV column keys remain `Metric.key` strings; missing/non-applicable values remain empty fields.
+
+### Console ergonomics
+- Added 2-line header rendering with a max column width cap so wide metric names wrap cleanly.
+- Manual header breaks are supported via `\n` in the `Metric.header` string (useful for specific columns like Avg QPS).
+
+### Runtime warnings (availability)
+- Preserved “warn once” behavior for missing exact-key metrics (e.g., index build time on cache hit).
+- Quant timing warnings were adjusted so we do not warn about NVQ encoding (not measured by design).
+
+## Notes / limitations
+- PQ compute time is recorded when the PQ compressor is computed (cache miss). On cache hits, compute time is intentionally blank (no compute performed).
+- NVQ encoding time is intentionally not measured; NVQ reports compute only.
+- Prefix selection is used for quant timing because concrete per-quantizer keys are only known once the run produces metrics.
+  <img width="1222" height="422" alt="quantization_metrics_logging" src="https://github.com/user-attachments/assets/2f90b2ad-aab1-4523-ac4c-4de3745700e7" />

docs/release notes/4.0.0-RC.9/637.fix.md

@@ -0,0 +1,16 @@
+This PR makes dataset loading more robust in the face of file corruption errors.
+It also indirects dataset access through DataSetInfo, caching the attached DataSet locally, and enabling access via getDataSet(). This will vastly speed up testing flows which are uneccesssarily processing datasets into memory right when the actual data is not yet accessed.
+
+[UPDATE]
+
+Due to a request for fixing some longer-standing issues, the scope of this PR has increased moderately:
+
+* DataSet loader formats which can not provide metadata (VSF, ...) on their own now pull details from dataset_metadata.yml for VSF. If none such is available, an error should be thrown.
+
+* Addtionally, the contract type (DataSetProperties) is now a layer of requirements added onto the DataSetInfo type, including an indicator for dataset preprocessing aspects (zero vectors, dupes) and normalization.
+
+* surefire tests were disabled in examples. enabled, and fixed up a unit test
+
+* DataSetProperties is a carrier type in the new DataSetInfo contract
+
+After this change, any dataset which is loaded where the VSF can't be resolved from some definitive source will cause an error to be thrown, as it should.

docs/release notes/4.0.0-RC.9/641.testing.md

@@ -0,0 +1,2 @@
+This extends the v1 experiments.csv schema with build_compressor and search_compressor columns so logged runs record the resolved compressor configurations used during index build and search. Values use the compressor’s printable toString() (e.g., ProductQuantization[...]) and are CSV-escaped to preserve commas. Search compressor logging is null-safe when compression is disabled (cv == null).
+<img width="1400" height="345" alt="build-and-search-compressor-info" src="https://github.com/user-attachments/assets/cda7634c-b5f6-4f98-9fc3-1d91955f630b" />

docs/release notes/4.0.0-RC.9/644.docs.md

@@ -0,0 +1,3 @@
+This PR adds links to the introductory tutorials in the main README. The existing step-by-step guide is moved out to it's own file, considering that it's rather cryptic for new users to decipher. I elected not to remove it entirely since it still contains useful information that isn't (yet) documented elsewhere.
+
+As a result the README is now significantly more concise.

docs/release notes/4.0.0-RC.9/645.testing.md

@@ -0,0 +1,51 @@
+## Investigation Complete: Root Cause Found and Fixed
+
+### Key Findings
+
+#### 1. The "Recall Degradation" Was Actually a Test Harness Bug
+Test results showed:
+- **BenchYAML + fusedGraph: No** → recall: 0.65
+- **BenchYAML + fusedGraph: Yes** → recall: 0.65
+- **AutoBenchYAML + fusedGraph: No** → recall: 0.77 ← **ANOMALY**
+- **AutoBenchYAML + fusedGraph: Yes** → recall: 0.65
+
+The anomaly was that AutoBenchYAML with fusedGraph:No produced artificially **high** recall (0.77), not that FusedPQ was producing low recall. All configurations using FusedPQ correctly produced consistent recall of 0.65.
+
+#### 2. Root Cause: Missing Vector Encoding in `runAllAndCollectResults`
+**Location:** `jvector-examples/src/main/java/io/github/jbellis/jvector/example/Grid.java`, line 833
+
+**The Bug:**
+```java
+CompressedVectors cvArg = (searchCompressorObj instanceof CompressedVectors) ? (CompressedVectors) searchCompressorObj : null;
+```
+
+This line attempted to cast a `VectorCompressor` (ProductQuantization) to `CompressedVectors`, which always failed, resulting in `cvArg = null`. When `cvArg` is null and fusedGraph is disabled, the `ConfiguredSystem.scoreProviderFor` method falls back to **exact scoring using uncompressed vectors** (line 1084 / 1094), which artificially inflates recall.
+
+**The Fix:**
+```java
+// Encode vectors for reranking if a compressor is provided
+CompressedVectors cvArg;
+if (searchCompressorObj == null) {
+    cvArg = null;
+} else {
+    cvArg = searchCompressorObj.encodeAll(ds.getBaseRavv());
+}
+```
+
+This fix properly encodes the vectors using the compressor, matching the behavior of `runOneGraph` (the correct implementation used by BenchYAML).
+
+### Impact
+- **Before fix:** AutoBenchYAML with fusedGraph:No was using exact (uncompressed) scoring, giving misleadingly high recall
+- **After fix:** AutoBenchYAML will now correctly use PQ-compressed vectors for approximate scoring during graph traversal, with NVQ reranking, producing consistent recall across all test harnesses
+### Verification
+After applying this fix:
+- AutoBenchYAML + fusedGraph:No → recall: ~0.65 (matching other configurations)
+- All test harnesses will produce consistent recall measurements
+- The perceived "recall degradation with high dimensionality" will disappear, as it was never a real issue with FusedPQ
+
+### Additional Notes
+The investigation also examined the FusedPQ implementation thoroughly and confirmed that:
+- The storage and retrieval of fused PQ data is correct
+- The scoring mathematics are correct
+- The SIMD implementations are correct
+- FusedPQ works correctly for all dimensionalities when properly configured

docs/release notes/4.0.0-RC.9/647.testing.md

@@ -0,0 +1,37 @@
+## Summary
+
+This PR optimizes `AccuracyMetrics` to improve the performance and readability of recall and precision measurements. The primary focus is shifting from $O(N^2)$ list-scanning to $O(N)$ Set-based lookups and reducing object allocation overhead by interacting directly with primitive `NodeScore` arrays.
+
+## Key changes
+
+### Logic cleanup and "dead code" removal
+- **Unused Signature Removal:** Removed the redundant `topKCorrect(List, List, ...)` signature, consolidating logic into a single private method that operates directly on `SearchResult`.
+- **Branch Elimination:** Removed the unreachable `if (gtView.size() > retrieved.size())` branch. Due to existing guard clauses (`kGT <= kRetrieved` and `kRetrieved <= retrieved.size()`), this condition was mathematically impossible.
+- **Parity Preservation:** Preserved specific, argument-focused exception messaging to ensure clear feedback for invalid `k` parameters.
+
+### Performance optimizations
+- **Algorithmic Shift:** Replaced $O(N)$ `List.contains()` calls with $O(1)$ `HashSet.contains()`. This moves the core intersection logic from quadratic to linear time complexity.
+- **AP Nested Scan Removal:** In `averagePrecisionAtK`, replaced the $O(i)$ `subList(0, i).contains(p)` duplicate check with a `HashSet` lookup. This transforms the AP calculation from an $O(K^2)$ operation to $O(K)$.
+- **Allocation Reduction:** Eliminated intermediate boxing and `ArrayList` creation in the `SearchResult` path. The logic now iterates directly over the `NodeScore[]` array, significantly reducing GC pressure during large benchmark runs.
+- **Loop Efficiency:** Replaced Stream API calls with manual `for` loops in high-frequency paths to avoid the object overhead and "setup tax" of the Stream API.
+- **HashSet Pre-Sizing:** Sets are now initialized with explicit capacities ($K / 0.75$) to prevent expensive internal rehashing.
+
+### Test coverage
+- **Functional Validation:** Verified Recall, AP, and MAP calculations against hand-calculated results to ensure mathematical parity with standard IR definitions.
+- **Duplicate Handling:** Confirmed that duplicate IDs in search results are handled via the "seen" Set, preventing artificial score inflation while maintaining $O(1)$ lookup speed.
+- **Exception Parity:** Explicitly verified that `IllegalArgumentException` strings remain 1:1 identical to the original implementation to prevent breaking downstream error-parsing.
+
+## Performance results
+
+Benchmarks were conducted using 10,000 queries on the `gecko-100k` dataset (768 dimensions).
+
+| Metric | Original Time | Optimized Time | Delta |
+| :--- | :--- | :--- | :--- |
+| **recall@10** | **~8ms**| ~same | - |
+| **recall@100** | **~58ms** | **~25ms** | **~57% reduction** |
+| **AP@100 / MAP@100** | **~62ms** | **~26ms** | **~58% reduction** |
+
+While the benefit is negligible for $K=10$, the optimization becomes critical as $K$ increases. The reduction in AP measurement time is particularly significant as it eliminates the $O(K^2)$ complexity previously caused by nested sublist scans.
+
+## Notes / limitations
+- **Autoboxing:** While this PR eliminates intermediate `List` allocations, it still utilizes `HashSet<Integer>`. Further gains could be achieved using primitive-specific collections (like `IntHashSet`) if necessary in the future.