Skip to content

Refine the Flash tree with merge - #375

Merged
rejojer merged 3 commits into
mainfrom
fix/default-merge
Aug 2, 2026
Merged

Refine the Flash tree with merge#375
rejojer merged 3 commits into
mainfrom
fix/default-merge

Conversation

@rejojer

@rejojer rejojer commented Aug 2, 2026

Copy link
Copy Markdown
Member

Three changes to how the Flash tree is refined.

1. The default path merges instead of thinning

Both operations collapse subtrees whose structure does not earn its keep. Merge
subsumes thinning and is better on the two things that matter.

Thinning collapses a subtree spanning 1 page, or 2 pages on a tree of more than
20 nodes. Merge collapses when S(v) <= tree_cost(v), and any node with children
costs at least R + 1 = 2, so everything thinning would collapse is collapsed by
merge as well. Merge additionally collapses larger subtrees that do not beat a
linear scan, on a principled threshold rather than a node-count heuristic.

Thinning also discarded the headings of a collapsed subtree. Merge keeps them on
the parent as key_items. On the 2023 annual report thinning lost 55 titles;
merge loses none.

utils.page_level_thinning is retained as legacy with no callers.

Measured over the nine documents in examples/documents/:

Property Result
Merge survivors are a subset of thinning survivors 9/9 documents, plus 3000 randomised trees
Titles preserved exact multiset, 0 lost
Page coverage no regression; pre-existing front-matter gaps unchanged
validate() after merge 0 new issues on every document
One bottom-up pass is a fixpoint second pass is a no-op; merge_tree idempotent
Max leaf span (summary prompt size) unchanged on 7/9, +1 page on 2
Node count drops, e.g. annual report 212 to 181, so ~15% fewer summary calls

2. Same-page merge under --optimize (@张鸣天)

Retrieval is page-granular, so frontier siblings covering identical pages cannot
be told apart: an agent routed to any of them reads the same text. They now
collapse into one node whose title is the union of theirs, which a leaf summary
call rewrites when the node is large enough to earn one. Deterministic, no LLM.

This runs inside --optimize, so --optimize merge now does strictly more than
the default rather than producing the same tree.

3. Internal marker no longer reaches the output

Same-page merge tags a node with _same_page to tell summary generation the
title was synthesised and may be rewritten. That marker was stripped on the two
page_index_flash paths but not on optimize_tree() or the
python3 -m pageindex.tree_optimize CLI, so both emitted it into their trees. It
is now stripped at those exits too, while optimize() still carries it
internally so summaries can retitle.

rejojer and others added 3 commits August 2, 2026 17:56
@rejojer rejojer changed the title Replace thinning with merge Refine the Flash tree with merge Aug 2, 2026
@rejojer
rejojer merged commit 2a29ac5 into main Aug 2, 2026
4 checks passed
@rejojer
rejojer deleted the fix/default-merge branch August 2, 2026 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants