HNSW using the linear package #3691

normen662 · 2025-10-22T17:21:01Z

This PR implements the HNSW paper using the recently introduced linear package together with RaBitQ.

alecgrieser

This has obviously taken a bit of time, but this is part one of the review. It covers:

The HNSW class and core algorithm
The Node and NodeKind classes

Still yet to look at are:

The StorageAdapter and implementations
Change sets
Any of the changes to the linear and RaBitQ packages
All tests

As hopefully is clear in the review, a lot of what's in it are requests for clarification. Some of these should probably turn into comments.

I also think that it would be good to take another look at the teamscale findings. Most of those are also pretty minor, but it would be good to try to conform a bit more to them. I'm less concerned about things like method length, nesting, or number of parameters (especially for private methods), but it would be nice to take another look at them.

Overall, I think the approach makes sense, through. Nice!

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/HNSW.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/AbstractNode.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/AbstractStorageAdapter.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/AccessInfo.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/AggregatedVector.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/BaseNeighborsChangeSet.java

fdb-extensions/src/main/java/com/apple/foundationdb/linear/AffineOperator.java

fdb-extensions/src/main/java/com/apple/foundationdb/linear/VectorOperator.java

fdb-extensions/src/test/java/com/apple/foundationdb/async/hnsw/HNSWHelpersTest.java

fdb-extensions/src/test/java/com/apple/foundationdb/async/hnsw/HNSWTest.java

alecgrieser

Okay, this adds more to the review, in particular focusing on the storage serialization/deserialization. I still have:

The changes to the other packages
Tests
Looking at updates since the last review

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/AccessInfo.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/StorageAdapter.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/AbstractStorageAdapter.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/InliningStorageAdapter.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/CompactStorageAdapter.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/StorageTransform.java

fdb-extensions/src/main/java/com/apple/foundationdb/linear/AffineOperator.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/Config.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/HNSW.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/StorageAdapter.java

ScottDugas · 2025-11-03T17:43:57Z

@normen662 @alecgrieser @MMcM
Teamscale is currently not reporting back to github
https://fdb.teamscale.io/activity/merge-requests/foundationdb-fdb-record-layer/FoundationDB%2Ffdb-record-layer%2F3691

Looking at the coverage report from the actions, I think the test gaps in teamscale are incorrect, but I would trust the findings, at least mostly.
You can see the summaries for changed files, which is pretty helpful for the new ones: https://github.com/FoundationDB/fdb-record-layer/actions/runs/19036192615

alecgrieser

I think I overall like what's being done with Transform. I did leave one comment about a usage pattern that is a bit surprising, if not understandable. I looked at the tests, and they seem like a good set of basic high-level tests. I'm not sure off the top of my head what improvements I'd like to see, but it does seem like we should stress it a bit more. It may also be the kind of thing where if we took the current version and then devised more interesting testing strategies, that would be fine

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/ResultEntry.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/AccessInfo.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/InliningStorageAdapter.java

fdb-extensions/src/test/java/com/apple/foundationdb/async/hnsw/HNSWTest.java

MMcM

A few mores dates.

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/InliningNode.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/InliningStorageAdapter.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/InsertNeighborsChangeSet.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/NeighborsChangeSet.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/Node.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/OnReadListener.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/OnWriteListener.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/StorageAdapter.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/package-info.java

fdb-extensions/src/test/java/com/apple/foundationdb/async/hnsw/HNSWTest.java

alecgrieser

Mostly LGTM. The only serious thing is the question raised about the StorageTransform change that went in

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/InliningStorageAdapter.java

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/StorageTransform.java

fdb-extensions/src/test/java/com/apple/foundationdb/async/hnsw/DataRecordsTest.java

fdb-extensions/src/test/java/com/apple/foundationdb/async/hnsw/HNSWTest.java

alecgrieser

This LGTM. There are failing tests, which I think are all from using hasToString instead of doesNotHaveToString in the latest update in a few places. If that's all that's wrong, and correcting that results in a passing PRB, I think this is good to merge

fdb-extensions/src/test/java/com/apple/foundationdb/async/hnsw/DataRecordsTest.java

alecgrieser mentioned this pull request Oct 23, 2025

vector basics; matrixes, serialization and half-precision floating point support #3677

Merged

normen662 force-pushed the hnsw-on-linear branch 4 times, most recently from 4cfef2a to 3a04055 Compare October 24, 2025 16:49

normen662 requested a review from alecgrieser October 28, 2025 19:17

normen662 added the enhancement New feature or request label Oct 28, 2025

normen662 force-pushed the hnsw-on-linear branch 3 times, most recently from 8fcb3a6 to fc7994f Compare October 29, 2025 13:15

alecgrieser requested changes Oct 29, 2025

View reviewed changes

MMcM approved these changes Oct 30, 2025

View reviewed changes

normen662 force-pushed the hnsw-on-linear branch 2 times, most recently from 267e633 to 1963a0a Compare October 30, 2025 15:41

alecgrieser requested changes Oct 30, 2025

View reviewed changes

MMcM requested changes Oct 30, 2025

View reviewed changes

fdb-extensions/src/main/java/com/apple/foundationdb/async/hnsw/StorageAdapter.java Show resolved Hide resolved

normen662 force-pushed the hnsw-on-linear branch 6 times, most recently from 0a243ee to bedfd30 Compare November 3, 2025 13:24

alecgrieser reviewed Nov 3, 2025

View reviewed changes

normen662 force-pushed the hnsw-on-linear branch 3 times, most recently from d6bc44c to e83340d Compare November 4, 2025 19:26

MMcM requested changes Nov 4, 2025

View reviewed changes

normen662 force-pushed the hnsw-on-linear branch from e83340d to 0971796 Compare November 5, 2025 08:41

normen662 added 19 commits November 5, 2025 14:46

rebase fallout

bf8e33b

addressing some comments from FoundationDB#3677

b262f29

changes to StorageAdapter needed for vector samples

e452392

adding affine operator

57425f8

adding storageTransformOperator

40f09ac

HNSW uses rabitq and affine operators

9baa4fd

save point

3a8b78e

rabitq works inside of HNSW

9c2ee9b

rabitq works inside of HNSW

6bd8e8c

adressing comments and simplifying code

b534e2c

adressing comments and simplifying code

e232ed8

adressing comments and simplifying code

03bc73a

adressing comments and simplifying code

fb7c1d2

inverting the apply-invertedApply

f601317

Transformed

1c91c00

addressing more comments

d2e4ef3

last round of addressing comments

e13762d

updating copyright years

809d55f

done

b0b03ff

normen662 force-pushed the hnsw-on-linear branch from 26d135a to b0b03ff Compare November 5, 2025 13:46

MMcM approved these changes Nov 5, 2025

View reviewed changes

alecgrieser requested changes Nov 5, 2025

View reviewed changes

alecgrieser approved these changes Nov 5, 2025

View reviewed changes

normen662 force-pushed the hnsw-on-linear branch from a35f7d9 to 3f25776 Compare November 5, 2025 18:54

adding one more test

2dc59a9

normen662 force-pushed the hnsw-on-linear branch from 3f25776 to 2dc59a9 Compare November 5, 2025 19:42

alecgrieser approved these changes Nov 5, 2025

View reviewed changes

normen662 merged commit 69c8839 into FoundationDB:main Nov 5, 2025
8 checks passed

normen662 mentioned this pull request Nov 6, 2025

HNSW (Hierarchical Navigable Small Worlds) implementation as fdb-extension #3598

Closed

HNSW using the linear package #3691

HNSW using the linear package #3691

Uh oh!

Conversation

normen662 commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alecgrieser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alecgrieser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ScottDugas commented Nov 3, 2025

Uh oh!

alecgrieser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MMcM left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alecgrieser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

normen662 commented Oct 22, 2025 •

edited

Loading