Fix DiskANN: Change to diskannpy #612
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently, an old implementation of the diskann python bindings is used. Microsoft has an updated implementation called diskannpy available now. Replace the old diskann module with a new version that uses the StaticMemoryIndex from diskannpy. Add corresponding config and dockerfile. Also reduce the complexity of the previous code by removing unnecessary functions, leading to a minimal version comparable to the hnswlib module.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, Diskann is disabled due to various issues, e.g. AVX512 dependence. The code used for the Diskann source and the Python bindings is from a branch of the diskann repository which has not been updated since 2021. Instead, diskannpy should be used, which are the official bindings and part of the main branch of Diskann.
Refactor the diskann module to use the StaticMemoryIndex from the diskannpy library. Simplify Dockerfile and configuration, make the module minimal to reduce complexity, similar to the hnswlib module. Remove PQ since its primarily useful for indices that are stored on disk or take very long to build, and trades off recall [1].
[1] https://microsoft.github.io/DiskANN/docs/python/latest/diskannpy.html