Fix: search limit for items with same ids, different collections#461
Draft
jonhealy1 wants to merge 3 commits into
Draft
Fix: search limit for items with same ids, different collections#461jonhealy1 wants to merge 3 commits into
jonhealy1 wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Technical Context
/searchwith?limitfails with same item ID in different collections #392 (and resolves downstream issue/searchwith?limitfails with same item ID in different collections stac-fastapi-pgstac#315)token=next:...) now encode three sorting attributes instead of two.keyset_wherenatively accommodates old token shapes by gracefully treating missing third elements asNULL. Snapshot text assertions verifying rawORDER BYstrings inside test runners will require updates to match the new sorting sequence.Description
Summary of Changes
This PR addresses a critical keyset pagination failure that occurs when two separate collections contain an item with the exact same
idand identical sorting metadata (e.g., matchingdatetimetimestamps).The core fix is implemented entirely within
keyset_sortkeys(_search jsonb):(datetime, id)to a globally unique triplet:(datetime, collection, id).collectionelement before the itemid. Because PgSTAC tables are physically partitioned by collection identifier, this arrangement allows the PostgreSQL query planner to perform partition pruning, preserving high-throughput performance over millions of rows.CASEhandler inside the dynamic expression builder to mapfield = 'collection'safely to table column targets (i.collection), preventing it from being dropped as an unregisteredNULLexpression.Rationale
In the STAC specification, an item
idis only guaranteed to be unique within its parent collection catalog. When executing multi-collection searches, paginating past matching items previously forced a strict tuple inequality check in PostgreSQL:(datetime, id) < (token_datetime, token_id)When evaluating duplicate IDs on Page 2, this statement evaluates to
FALSE(e.g.,(X, Y) < (X, Y)). The database mistakenly concludes that no remaining records exist, collapsing the response payload to0 featuresand corrupting the subsequentprevtoken link calculations. Introducing the collection partition key ensures stable, deterministic row differentiation.Area for Reviewers to Focus On
src/pgstac/sql/002d_keyset.sql: Verify that the hardcoded string fallback for core tracking identifiers (id,collection) successfully prevents registry lookup misses.src/pgstac/tests/pgtap/004_search.sql: Review the programmatic pgTAP integration test. It leveragesjsonb_path_query_firstto pull and feed the base64 token dynamically between pages to evaluate forward/reverse stability against a seeded duplicate item name without relying on brittle, hardcoded strings.Checklist
README.mdto reflect any new environment variables, configuration changes, or breaking updates.AI tool usage