Replies: 3 comments
-
Can you provide the actual query string for the two different queries? |
Beta Was this translation helpful? Give feedback.
-
FYI - I believe AWS MemoryDB is not using the open source https://github.com/valkey-io/valkey-search/ implementation for KNN search. So you might have limited engagement here in the Valkey discussion board, since the details of their implementation is proprietary |
Beta Was this translation helpful? Give feedback.
-
Also, |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi I'm using MemoryDB in AWS, with Valkey 7.3 as the underlying engine
I've got a vector index with 400k vectors - the vectors have a dimension of 1024
I'm running hybrid searches in my app, I've got 1 Tag field that I use to filter the results.
In my vec index, there is a dataset of 50k vec associated with a specific owner TAG, when I run a search query with that specific owner as a filter value, the query time spikes, for other owner values, the query time is very low, as expected.
output of a profile cmd
I ran 2 queries, one simple FT search with filter and one KNN search with a filter applied
Simple:
[[b'tag.count', 52768], [b'sync.time', 1], [b'query.time', 148263], [b'result.count', 52768], [b'result.time', 0]]
[[b'tag.count', 463], [b'sync.time', 1], [b'query.time', 4717], [b'result.count', 463], [b'result.time', 0]]
Hybrid:
[[b'tag.count', 0], [b'sync.time', 1], [b'query.time', 405819], [b'vector.count', 10], [b'vector.time', 405815], [b'result.count', 10], [b'result.time', 0]]
[[b'tag.count', 463], [b'sync.time', 1], [b'query.time', 697], [b'vector.count', 10], [b'vector.time', 695], [b'lowpass.count', 1], [b'result.count', 10], [b'result.time', 0]]
This is the output of my ft.info command
10) (integer) 1000
11) current_capacity
12) (integer) 430804
13) block_size
14) (integer) 1024
I'd like to understand why the filter is not being applied before the KNN search, and causing a massive spike in query time,
Accuracy is very important for me, so I'm reluctant to move to HNSW index
Is there anything I can do, besides moving to HNSW index?
Beta Was this translation helpful? Give feedback.
All reactions