Preserve position gaps in graph phrase queries#152931
Open
Incheonkirin wants to merge 1 commit into
Open
Conversation
match_phrase drops position holes (left by stop filters, nori_part_of_speech, or other token-removing filters) when the analysis chain produces a token graph and slop is 0, in two places: * createSpanQuery added the gap one clause early, before the previous term instead of between it and the current one. This is the misplaced SpanGap reported in elastic#86021. * analyzeGraphPhrase ignored position increments at articulation states without a side path and assembled the outer query with a plain SpanNearQuery constructor, so gaps between segments were lost entirely. Documents whose indexed positions contain the hole can fail to match at slop 0, including documents containing the exact source string of the query. Fix both sites using the position increments already preserved by GraphTokenStreamFiniteStrings, and visit the gap clauses of the outer span query so the clause count stays consistent with a full walk of the final query. This matches the behavior of the slop>0 finite-strings path and of non-graph phrase analysis, which both keep holes. Relates to elastic#28838
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
match_phrasedrops position holes when a token graph reaches the slop=0phrase path with holes left by token-removing filters, for example
synonym_graphfollowed bystop, or nori decompounding combined withnori_part_of_speech. Documents whose indexed positions contain the hole canfail to match at slop 0, including documents containing the exact source
string of the query.
There are two distinct defects in
MatchQueryParser:createSpanQueryemits the gap one clause early. In the token loop,addGapfor token i runs beforeaddClausefor token i-1, so the gaplands before the previous term instead of after it. This is the misplaced
SpanGapvisible in the_validateoutput of match_phrase queries miss documents containing stop words in synonyms #86021.The articulation-point loop in
analyzeGraphPhrasenever readsPositionIncrementAttributefor states without a side path, and assemblesthe outer query with
new SpanNearQuery(clauses, 0, true), which cannotcarry gaps between segments. The hole is lost entirely.
Reproduction (scenario from #86021)
Index analyzer
standard + stop(of), search analyzerstandard + synonym_graph("fbi, federal bureau of investigation") + stop(of).A document containing
federal bureau of investigationis indexed atpositions
federal=0 bureau=1 investigation=3. Query built before thischange:
The gap sits between
federalandbureau, so bothmatch_phrase("fbi")and
match_phrase("federal bureau of investigation")miss that document.After this change:
The second defect fires without synonyms, which is how I hit this. Korean
nori_tokenizer(decompound_mode=mixed)emits compound tokens(
positionLength > 1, so the stream is a graph) andnori_part_of_speechremoves particles (holes). A
match_phrasequery containing the exact sourcetext of a document then returns nothing at slop 0: for the source string
보험계약대출이율(positions보험계약@0(len2) 보험@0 계약@1 대출@2 율@4,particle
이removed at position 3), the query requires율adjacent to대출and misses the document. The new yamlRestTest inanalysis-noricovers this end to end and fails with 0 hits without this change. Raising
slopmasks the bug only becauseanalyzeGraphPhraseswitches to thefinite-strings path, which preserves holes; the slop=0 span path was the only
place they were lost.
Changes
createSpanQuery: emit the pending clause before the gap, so the gap sitsbetween the correct terms.
analyzeGraphPhrase: read the position increment of the tokens at eacharticulation state and add gaps through a
SpanNearQuery.Builderfor theouter query. The gap clauses of the built query are visited so the clause
count accumulated during construction stays consistent with a full walk of
the final query (
AbstractQueryBuilder#toQueryasserts this).Both changes only consume position increments that
GraphTokenStreamFiniteStringsalready preserves, and align the slop=0 spanpath with
QueryBuilder#analyzePhrase(non-graph streams produce"foo ? bar") and with this class's own slop>0 path.Out of scope
When the stop filter runs before the synonym filter, the failure happens
inside the analysis chain: synonym rules cannot match across holes, and
expansions emit stop words the index never contains (LUCENE-8137). Query
construction cannot repair that.
testGraphPhraseStopBeforeSynonymOutOfScopedocuments this boundary and passes with and without this change.
Relates to #28838. Resolves the scenario reported in #86021 (closed as a
duplicate of #28838).
Testing
MatchQueryBuilderTests: gap placement within, after,and before a side path with canned graph token streams, the match_phrase queries miss documents containing stop words in synonyms #86021 and
Bug: When using graph synonym and stop token filter together #28838-shaped scenarios with real
SynonymGraphFilter+StopFilterchains, and the out-of-scope boundary.
analysis-norireproducing the no-synonym case endto end (fails with 0 hits before the fix).
:server:testand:plugins:analysis-nori:yamlRestTestpass with thechange.