Skip to content

Preserve position gaps in graph phrase queries#152931

Open
Incheonkirin wants to merge 1 commit into
elastic:mainfrom
Incheonkirin:fix/graph-phrase-position-gaps
Open

Preserve position gaps in graph phrase queries#152931
Incheonkirin wants to merge 1 commit into
elastic:mainfrom
Incheonkirin:fix/graph-phrase-position-gaps

Conversation

@Incheonkirin

Copy link
Copy Markdown
Contributor

match_phrase drops position holes when a token graph reaches the slop=0
phrase path with holes left by token-removing filters, for example
synonym_graph followed by stop, or nori decompounding combined with
nori_part_of_speech. Documents whose indexed positions contain the hole can
fail to match at slop 0, including documents containing the exact source
string of the query.

There are two distinct defects in MatchQueryParser:

  1. createSpanQuery emits the gap one clause early. In the token loop,
    addGap for token i runs before addClause for token i-1, so the gap
    lands before the previous term instead of after it. This is the misplaced
    SpanGap visible in the _validate output of match_phrase queries miss documents containing stop words in synonyms #86021.

  2. The articulation-point loop in analyzeGraphPhrase never reads
    PositionIncrementAttribute for states without a side path, and assembles
    the outer query with new SpanNearQuery(clauses, 0, true), which cannot
    carry gaps between segments. The hole is lost entirely.

Reproduction (scenario from #86021)

Index analyzer standard + stop(of), search analyzer
standard + synonym_graph("fbi, federal bureau of investigation") + stop(of).
A document containing federal bureau of investigation is indexed at
positions federal=0 bureau=1 investigation=3. Query built before this
change:

spanOr([fbi, spanNear([federal, SpanGap(:1), bureau, investigation], 0, true)])

The gap sits between federal and bureau, so both match_phrase("fbi")
and match_phrase("federal bureau of investigation") miss that document.
After this change:

spanOr([fbi, spanNear([federal, bureau, SpanGap(:1), investigation], 0, true)])

The second defect fires without synonyms, which is how I hit this. Korean
nori_tokenizer(decompound_mode=mixed) emits compound tokens
(positionLength > 1, so the stream is a graph) and nori_part_of_speech
removes particles (holes). A match_phrase query containing the exact source
text of a document then returns nothing at slop 0: for the source string
보험계약대출이율 (positions 보험계약@0(len2) 보험@0 계약@1 대출@2 율@4,
particle removed at position 3), the query requires adjacent to
대출 and misses the document. The new yamlRestTest in analysis-nori
covers this end to end and fails with 0 hits without this change. Raising
slop masks the bug only because analyzeGraphPhrase switches to the
finite-strings path, which preserves holes; the slop=0 span path was the only
place they were lost.

Changes

  • createSpanQuery: emit the pending clause before the gap, so the gap sits
    between the correct terms.
  • analyzeGraphPhrase: read the position increment of the tokens at each
    articulation state and add gaps through a SpanNearQuery.Builder for the
    outer query. The gap clauses of the built query are visited so the clause
    count accumulated during construction stays consistent with a full walk of
    the final query (AbstractQueryBuilder#toQuery asserts this).

Both changes only consume position increments that
GraphTokenStreamFiniteStrings already preserves, and align the slop=0 span
path with QueryBuilder#analyzePhrase (non-graph streams produce
"foo ? bar") and with this class's own slop>0 path.

Out of scope

When the stop filter runs before the synonym filter, the failure happens
inside the analysis chain: synonym rules cannot match across holes, and
expansions emit stop words the index never contains (LUCENE-8137). Query
construction cannot repair that. testGraphPhraseStopBeforeSynonymOutOfScope
documents this boundary and passes with and without this change.

Relates to #28838. Resolves the scenario reported in #86021 (closed as a
duplicate of #28838).

Testing

match_phrase drops position holes (left by stop filters,
nori_part_of_speech, or other token-removing filters) when the
analysis chain produces a token graph and slop is 0, in two places:

* createSpanQuery added the gap one clause early, before the previous
  term instead of between it and the current one. This is the
  misplaced SpanGap reported in elastic#86021.
* analyzeGraphPhrase ignored position increments at articulation
  states without a side path and assembled the outer query with a
  plain SpanNearQuery constructor, so gaps between segments were lost
  entirely.

Documents whose indexed positions contain the hole can fail to match
at slop 0, including documents containing the exact source string of
the query.

Fix both sites using the position increments already preserved by
GraphTokenStreamFiniteStrings, and visit the gap clauses of the outer
span query so the clause count stays consistent with a full walk of
the final query. This matches the behavior of the slop>0
finite-strings path and of non-graph phrase analysis, which both keep
holes.

Relates to elastic#28838
@elasticsearchmachine elasticsearchmachine added needs:triage Requires assignment of a team area label v9.5.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Jul 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contributor Pull request authored by a developer outside the Elasticsearch team needs:triage Requires assignment of a team area label v9.5.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants