[content-hash 4/5] feat(cli): add leann migrate-ids to convert existing index#337
Open
raoabinav wants to merge 4 commits into
Open
[content-hash 4/5] feat(cli): add leann migrate-ids to convert existing index#337raoabinav wants to merge 4 commits into
raoabinav wants to merge 4 commits into
Conversation
Sub-PR 1 of 5 from the plan in yichuan-w#329. Purely additive — no behavior change for any caller, existing index loaders ignore the field. Writes a new `passage_id_scheme: "sequential"` field into the .meta.json produced by both build_index and build_index_from_arrays. Bumps version to "1.1" for human-inspectable schema tracking (no code reads version today, so the bump is safe). Module-level constants PASSAGE_ID_SCHEME_SEQUENTIAL / _CONTENT_HASH document the value space; the content-hash scheme itself ships in sub-PR 2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sub-PR 2 of 5 from yichuan-w#329. Builds on yichuan-w#330 (which added the meta.json field). New behavior: - `LeannBuilder(..., passage_id_scheme="content-hash")` makes add_text() key passages by sha256(text)[:16] instead of insertion index. Stable across file moves, reorderings, and re-runs of the same corpus. - `leann build --id-scheme content-hash` exposes it at the CLI. - Default unchanged ("sequential"). Existing indexes continue to work identically; no migration triggered. Identical-text chunks collide (same hash). For this sub-PR the second occurrence overwrites the first in the offset map — that's the dedup behavior I'd want by default. A `--preserve-duplicates` escape hatch can land later if needed (see the open question in yichuan-w#329). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sub-PR 3 of 5 from yichuan-w#329. Builds on yichuan-w#330 / yichuan-w#331. Two changes: 1. `LeannCLI._make_incremental_builder` now reads the existing index's `passage_id_scheme` from meta.json and uses that, ignoring any conflicting `--id-scheme` on the args (with a note printed). Otherwise an update command on a content-hash index would mix sequential IDs into a hash-keyed passages.jsonl and break lookups. 2. `LeannSearcher` exposes `self.passage_id_scheme` so consumers can introspect; defaults to "sequential" for older indexes that don't record it (pre-yichuan-w#330). No behavior change for fresh builds — the CLI's --id-scheme still controls which scheme a brand-new index gets. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 20, 2026
…nt-hash Sub-PR 4 of 5 from yichuan-w#329. Builds on the reader-side support in sub-PR 3. `leann migrate-ids <index>` rewrites an existing sequential-ID index to use sha256(text)[:16] keys. Purely a Python-side rewrite — FAISS labels stay valid, only the string IDs they map to change. Four artifacts get rewritten atomically (via .migrate sibling files + shutil.move): - .passages.jsonl : new "id" field per line - .passages.idx : new offset map keyed by new IDs (deduplicates collisions) - .ids.txt : new label → ID mapping (preserves FAISS label count) - .meta.json : passage_id_scheme = "content-hash", version = "1.1" Collisions (identical-text chunks) get deduped — the later occurrence wins in the offset map. ids.txt still has one line per FAISS label so the graph keeps working; duplicate labels just point to the same passage. Interactive confirmation required unless --yes is passed. --dry-run reports the collision count without writing anything. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
raoabinav
force-pushed
the
feat/leann-migrate-ids
branch
from
d3566f4 to
fccb6ab
Compare
raoabinav
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sub-PR 4 of 5 from #329. Stacks on the reader-side support in the prior sub-PR.
leann migrate-ids <index>rewrites an existing sequential-ID index to usesha256(text)[:16]keys. Purely a Python-side rewrite — FAISS labels stay valid, only the string IDs they map to change. Four artifacts get rewritten atomically (via.migratesibling files +shutil.move):.passages.jsonl— new"id"field per line.passages.idx— new offset map keyed by new IDs (deduplicates collisions).ids.txt— new label → ID mapping (preserves FAISS label count).meta.json—passage_id_scheme = "content-hash",version = "1.1"Collisions (identical-text chunks) get deduped — the later occurrence wins in the offset map.
.ids.txtstill has one line per FAISS label so the graph keeps working; duplicate labels just point to the same passage.Interactive confirmation required unless
--yesis passed.--dry-runreports the collision count without writing anything.Content-hash passage IDs train (#329)
passage_id_schemefield in meta.json--id-scheme content-hashbuilder + CLI flagleann migrate-idscommand