Skip to content

Conversation

@jeremybmerrill
Copy link
Contributor

As described in #2225, when nodes are persisted to Postgresql and then re-hydrated with query, they do not retain relationship key/value pair. That means nodes lose their relationship to their SOURCE document. This behavior is (as described in my comment in #2225) specific to the PGVectorStore class; SimpleVectorStore and WeaviateVectorStore properly retain `relationship.

The root of the bug is that PGVectorStore doesn't use nodeToMetadata to dehydrate the node to a single dict and doesn't use metadataDictToNode to re-hydrate it. Instead, PGVectorStore just stores the node's metadata (ignoring the rest of the node data, discarding it).

This PR uses nodeToMetadata and metadataDictToNode, imitating WeaviateVectorStore. Now, it works for me -- I can dehydrate and rehydrate nodes, and have them retain their relationships.

Closes #2225

Before, incorrect value of the metadata field in Postgres -- all of these keys are arbitrary user-specific metadata, nothing LlamaIndexTS-related except create_date:

{
    "fn": "filings_raw/0000023217-2024Q1-10-Q-cag20231126_10q.html",
    "cik": "23217",
    "font-family": "\"Times New Roman\"",
    "create_date": "2025-09-25T01:00:52.659Z",
    ...
}

After, with this PR:

{
    "fn": "filings_raw/0000023217-2024Q1-10-Q-cag20231126_10q.html",
    "cik": "23217",
    "doc_id": "d515c040-7ff9-482a-88f9-2e1c68f7e4d2",
    "_node_type": "TextNode",
    "ref_doc_id": "d515c040-7ff9-482a-88f9-2e1c68f7e4d2",
    "create_date": "2025-10-26T01:17:29.250Z",
    "document_id": "d515c040-7ff9-482a-88f9-2e1c68f7e4d2",
    "font-family": "\"Times New Roman\"",
    "_node_content": "{\"id_\":\"1400530c-ab43-4791-a1a5-ad5479ab848f\",\"excludedEmbedMetadataKeys\":[\"\",\"Unnamed: 0\",\"cik\",\"fn\",\"classification\",\"paragraph_index\",\"font-size\",\"font-family\",\"font-style\",\"font-weight\",\"line-height\",\"text-align\",\"width\",\"margin-bottom\",\"margin-top\",\"text-indent\",\"vertical-align\",\"color\",\"text_len\",\"pct_numbers\"],\"excludedLlmMetadataKeys\":[],\"relationships\":{\"SOURCE\":{\"nodeId\":\"d515c040-7ff9-482a-88f9-2e1c68f7e4d2\",\"metadata\":{\"\":\"1825\",\"Unnamed: 0\":\"1325\",\"cik\":\"23217\",\"fn\":\"filings_raw/0000023217-2024Q1-10-Q-cag20231126_10q.html\",\"classification\":\"body\",\"paragraph_index\":\"405\",\"font-size\":\"10pt\",\"font-family\":\"\\\"Times New Roman\\\"\",\"font-style\":\"\",\"font-weight\":\"\",\"line-height\":\"\",\"text-align\":\"justify\",\"width\":\"\",\"margin-bottom\":\"\",\"margin-top\":\"\",\"text-indent\":\"27pt\",\"vertical-align\":\"\",\"color\":\"\",\"text_len\":\"478.0\",\"pct_numbers\":\"0.0\"},\"hash\":\"K+SksiqFAH0gXsK3F0isP0/kPjXWT3/9F/dJJybN4sE=\"}},\"text\":\"\",\"textTemplate\":\"\",\"metadataSeparator\":\"\n\",\"startCharIdx\":0,\"endCharIdx\":478,\"type\":\"TEXT\",\"hash\":\"mAn2tHceWJ8PrhydvePeEfPrDdK571IaOqwY5tPukBs=\"}",
   ... 
}

I have not added tests with this PR because I'm bad at unit tests and there aren't any tests that I see for PGVectorStore. If required, I will futz around with Copilot and try to get some meaningful tests that pass. Let me know!

@changeset-bot
Copy link

changeset-bot bot commented Oct 26, 2025

⚠️ No Changeset found

Latest commit: 7b3b848

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@pkg-pr-new
Copy link

pkg-pr-new bot commented Oct 26, 2025

Open in StackBlitz

@llamaindex/autotool

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/autotool@2232

@llamaindex/community

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/community@2232

@llamaindex/core

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/core@2232

@llamaindex/env

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/env@2232

@llamaindex/experimental

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/experimental@2232

llamaindex

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/llamaindex@2232

@llamaindex/node-parser

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/node-parser@2232

@llamaindex/readers

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/readers@2232

@llamaindex/tools

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/tools@2232

@llamaindex/wasm-tools

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/wasm-tools@2232

@llamaindex/workflow

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/workflow@2232

@llamaindex/anthropic

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/anthropic@2232

@llamaindex/assemblyai

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/assemblyai@2232

@llamaindex/aws

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/aws@2232

@llamaindex/clip

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/clip@2232

@llamaindex/cohere

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/cohere@2232

@llamaindex/deepinfra

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/deepinfra@2232

@llamaindex/deepseek

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/deepseek@2232

@llamaindex/discord

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/discord@2232

@llamaindex/excel

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/excel@2232

@llamaindex/fireworks

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/fireworks@2232

@llamaindex/google

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/google@2232

@llamaindex/groq

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/groq@2232

@llamaindex/huggingface

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/huggingface@2232

@llamaindex/jinaai

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/jinaai@2232

@llamaindex/mistral

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/mistral@2232

@llamaindex/mixedbread

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/mixedbread@2232

@llamaindex/notion

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/notion@2232

@llamaindex/ollama

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/ollama@2232

@llamaindex/openai

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/openai@2232

@llamaindex/perplexity

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/perplexity@2232

@llamaindex/portkey-ai

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/portkey-ai@2232

@llamaindex/replicate

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/replicate@2232

@llamaindex/together

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/together@2232

@llamaindex/vercel

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/vercel@2232

@llamaindex/vllm

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/vllm@2232

@llamaindex/voyage-ai

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/voyage-ai@2232

@llamaindex/xai

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/xai@2232

@llamaindex/astra

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/astra@2232

@llamaindex/azure

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/azure@2232

@llamaindex/chroma

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/chroma@2232

@llamaindex/elastic-search

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/elastic-search@2232

@llamaindex/firestore

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/firestore@2232

@llamaindex/milvus

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/milvus@2232

@llamaindex/mongodb

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/mongodb@2232

@llamaindex/pinecone

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/pinecone@2232

@llamaindex/postgres

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/postgres@2232

@llamaindex/qdrant

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/qdrant@2232

@llamaindex/supabase

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/supabase@2232

@llamaindex/upstash

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/upstash@2232

@llamaindex/weaviate

npm i https://pkg.pr.new/run-llama/LlamaIndexTS/@llamaindex/weaviate@2232

commit: 7b3b848

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PGVectorStore and column external_id

1 participant