RAG: keep chunking-excluded types out of keyword retrieval and sources - #125
Merged
Conversation
#124) Image content is excluded from chunking, but the exclusion only existed on the indexing side: the BM25 keyword leg still ranked Images by title, so they were cited as sources with an empty snippet while the answer could not know anything about them (the exact pollution the exclusion was meant to prevent). - search_keyword: mirror EXCLUDED_PORTAL_TYPES in the filter queries. - build_sources: a parent without any context chunk contributed nothing to the answer and is no longer cited. - SolrExtraConditions.from_encoded: an empty string (a bare extra_conditions= URL parameter from the results page URL rewrite) means no conditions, instead of logging an invalid-base64 warning on every request.
danalvrz
approved these changes
Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #124. Found by @danalvrz during acceptance testing (kitconcept.intranet ticket 570, finding 2).
Image content is excluded from chunking, but the exclusion only existed on the indexing side: the BM25 keyword leg still ranked Images by title, so they were cited as sources with an empty snippet while the answer could not know anything about them — the exact pollution the exclusion was meant to prevent.
search_keyword: mirrorsEXCLUDED_PORTAL_TYPESin the filter queries, so both retrieval legs agree with the chunking policy.build_sources: a parent without any context chunk contributed nothing to the answer and is no longer cited.SolrExtraConditions.from_encoded("")treats a bareextra_conditions=URL parameter (the results page URL rewrite produces those) as no conditions instead of logging an invalid-base64 warning on every request.Verified live on the intranet corpus with "Was ist der Strategieprozess DFNT 2030?" (each workspace has a matching "Vorschaubild" preview image — the exact reported pattern): before, the Image was cited with an empty snippet and the answer claimed the topic was undocumented; after, no Image/empty-snippet sources, and the answer improved because the freed context slot went to a real document.
Backend suite: 733 passed, 3 skipped.