[content-hash 3/5] feat: incremental updates honor existing index's passage_id_scheme#336
Open
raoabinav wants to merge 3 commits into
Open
[content-hash 3/5] feat: incremental updates honor existing index's passage_id_scheme#336raoabinav wants to merge 3 commits into
raoabinav wants to merge 3 commits into
Conversation
Sub-PR 1 of 5 from the plan in yichuan-w#329. Purely additive — no behavior change for any caller, existing index loaders ignore the field. Writes a new `passage_id_scheme: "sequential"` field into the .meta.json produced by both build_index and build_index_from_arrays. Bumps version to "1.1" for human-inspectable schema tracking (no code reads version today, so the bump is safe). Module-level constants PASSAGE_ID_SCHEME_SEQUENTIAL / _CONTENT_HASH document the value space; the content-hash scheme itself ships in sub-PR 2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sub-PR 2 of 5 from yichuan-w#329. Builds on yichuan-w#330 (which added the meta.json field). New behavior: - `LeannBuilder(..., passage_id_scheme="content-hash")` makes add_text() key passages by sha256(text)[:16] instead of insertion index. Stable across file moves, reorderings, and re-runs of the same corpus. - `leann build --id-scheme content-hash` exposes it at the CLI. - Default unchanged ("sequential"). Existing indexes continue to work identically; no migration triggered. Identical-text chunks collide (same hash). For this sub-PR the second occurrence overwrites the first in the offset map — that's the dedup behavior I'd want by default. A `--preserve-duplicates` escape hatch can land later if needed (see the open question in yichuan-w#329). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sub-PR 3 of 5 from yichuan-w#329. Builds on yichuan-w#330 / yichuan-w#331. Two changes: 1. `LeannCLI._make_incremental_builder` now reads the existing index's `passage_id_scheme` from meta.json and uses that, ignoring any conflicting `--id-scheme` on the args (with a note printed). Otherwise an update command on a content-hash index would mix sequential IDs into a hash-keyed passages.jsonl and break lookups. 2. `LeannSearcher` exposes `self.passage_id_scheme` so consumers can introspect; defaults to "sequential" for older indexes that don't record it (pre-yichuan-w#330). No behavior change for fresh builds — the CLI's --id-scheme still controls which scheme a brand-new index gets. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
raoabinav
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sub-PR 3 of 5 from #329. Stacks on #330 (meta field) and #331 (builder).
Two changes:
LeannCLI._make_incremental_buildernow reads the existing index'spassage_id_schemefrom meta.json and uses that, ignoring any conflicting--id-schemeon the args (with a note printed). Otherwise an update command on a content-hash index would mix sequential IDs into a hash-keyedpassages.jsonland break lookups.LeannSearcherexposesself.passage_id_schemeso consumers can introspect; defaults to"sequential"for older indexes that don't record it (pre-[content-hash 1/5] refactor: record passage_id_scheme in meta.json #330).No behavior change for fresh builds — the CLI's
--id-schemestill controls which scheme a brand-new index gets.Content-hash passage IDs train (#329)
passage_id_schemefield in meta.json--id-scheme content-hashbuilder + CLI flagleann migrate-idscommand (next)