-
Notifications
You must be signed in to change notification settings - Fork 492
PGVectorStore: use helpers to convert between nodes and storage #2232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
PGVectorStore: use helpers to convert between nodes and storage #2232
Conversation
…ps can be rehydrated
|
@llamaindex/autotool
@llamaindex/community
@llamaindex/core
@llamaindex/env
@llamaindex/experimental
llamaindex
@llamaindex/node-parser
@llamaindex/readers
@llamaindex/tools
@llamaindex/wasm-tools
@llamaindex/workflow
@llamaindex/anthropic
@llamaindex/assemblyai
@llamaindex/aws
@llamaindex/clip
@llamaindex/cohere
@llamaindex/deepinfra
@llamaindex/deepseek
@llamaindex/discord
@llamaindex/excel
@llamaindex/fireworks
@llamaindex/google
@llamaindex/groq
@llamaindex/huggingface
@llamaindex/jinaai
@llamaindex/mistral
@llamaindex/mixedbread
@llamaindex/notion
@llamaindex/ollama
@llamaindex/openai
@llamaindex/perplexity
@llamaindex/portkey-ai
@llamaindex/replicate
@llamaindex/together
@llamaindex/vercel
@llamaindex/vllm
@llamaindex/voyage-ai
@llamaindex/xai
@llamaindex/astra
@llamaindex/azure
@llamaindex/chroma
@llamaindex/elastic-search
@llamaindex/firestore
@llamaindex/milvus
@llamaindex/mongodb
@llamaindex/pinecone
@llamaindex/postgres
@llamaindex/qdrant
@llamaindex/supabase
@llamaindex/upstash
@llamaindex/weaviate
commit: |
As described in #2225, when nodes are persisted to Postgresql and then re-hydrated with
query, they do not retainrelationshipkey/value pair. That means nodes lose their relationship to theirSOURCEdocument. This behavior is (as described in my comment in #2225) specific to the PGVectorStore class; SimpleVectorStore and WeaviateVectorStore properly retain `relationship.The root of the bug is that PGVectorStore doesn't use
nodeToMetadatato dehydrate the node to a single dict and doesn't usemetadataDictToNodeto re-hydrate it. Instead, PGVectorStore just stores the node'smetadata(ignoring the rest of the node data, discarding it).This PR uses
nodeToMetadataandmetadataDictToNode, imitating WeaviateVectorStore. Now, it works for me -- I can dehydrate and rehydrate nodes, and have them retain their relationships.Closes #2225
Before, incorrect value of the
metadatafield in Postgres -- all of these keys are arbitrary user-specific metadata, nothing LlamaIndexTS-related exceptcreate_date:After, with this PR:
I have not added tests with this PR because I'm bad at unit tests and there aren't any tests that I see for PGVectorStore. If required, I will futz around with Copilot and try to get some meaningful tests that pass. Let me know!