fix: refresh partition materialized views concurrently to avoid ACCESS EXCLUSIVE locks

[fabito]

Related Issue(s)

Fixes #311

Description

update_partition_stats() and create_partition() refresh the partitions and partition_steps materialized views with a blocking REFRESH MATERIALIZED VIEW, which takes an ACCESS EXCLUSIVE lock. Since search() reads partition_steps on every call, this causes two failure modes in production:

1. Deadlocks on the primary between searches and partition maintenance, as reported in Deadlock between search requests and ingestion process #311 — the deadlock victim is always the search, surfacing as HTTP 500s to API consumers.
2. Query cancellations on hot-standby / Aurora readers: the ACCESS EXCLUSIVE lock is WAL-logged (rmgr: Standby, desc: LOCK) and replayed on replicas, where it cancels any in-flight search still holding a relation lock after max_standby_streaming_delay with canceling statement due to conflict with recovery — User was holding a relation lock for too long.

We hit mode 2 in production (pgstac 0.9.11 on Aurora PostgreSQL 17.7, reader endpoint serving stac-fastapi-pgstac): with pgstac.use_queue=true and a scheduled run_queued_queries() drainer, every busy drain emitted ACCESS EXCLUSIVE locks on the matviews and deterministically cancelled long-running multi-farm searches on the reader. Log correlation showed every cancellation burst landing inside a busy drainer run, exactly max_standby_streaming_delay after the standby lock record arrived.

Change

• Add the missing unique index on partition_steps (name) (partitions already has one on (partition)), which REFRESH ... CONCURRENTLY requires. name is the partition's table name, so it is inherently unique.
• Add refresh_partition_matviews(), which refreshes both views with REFRESH MATERIALIZED VIEW CONCURRENTLY — this takes only an EXCLUSIVE lock, which does not conflict with readers' ACCESS SHARE locks, eliminating both the deadlock and the standby-cancellation paths. It falls back to a blocking refresh (previous behavior, with a NOTICE) for the cases CONCURRENTLY cannot handle, e.g. a never-populated materialized view.
• Call the helper from both refresh sites (update_partition_stats(), create_partition()).

Answering the open questions from the issue thread:

• Can REFRESH ... CONCURRENTLY run inside plpgsql / the queue procedure? Yes — verified on PostgreSQL 17: unlike CREATE INDEX CONCURRENTLY, concurrent matview refresh is a single-transaction operation and works both inside plpgsql functions and via EXECUTE in a procedure that uses transaction control (run_queued_queries()). The CONCURRENTLY-stripping in the queue runner is unaffected: it operates on queued query text (needed for CREATE INDEX CONCURRENTLY, which genuinely cannot run there), not on statements inside function bodies.
• First refresh after creation? Both matviews are created populated (AS SELECT without WITH NO DATA); the fallback covers restore-style edge cases where a view exists unpopulated.

Trade-off: a concurrent refresh does more work than a blocking one (temp-table diff against the unique index). Both views have one row per partition, so the extra cost is negligible relative to the ANALYZE already performed by update_partition_stats().

Notes on generated files and the incremental migration

pgstac--unreleased.sql and pgstac.sql were regenerated with scripts/stageversion. The pgstac--0.9.11--unreleased.sql incremental is deliberately not regenerated here, and two pre-existing issues surfaced while testing it (both reproduce on clean main, independent of this change):

1. The committed incremental was last regenerated in Item fragments #448 and is missing feat(search): v0.10 keyset search consolidation #456's objects — scripts/test --migrations on main fails post-upgrade with function pgstac.jsonb_common_paths_agg(jsonb) does not exist.
2. Regenerating it with scripts/stageversion produces an upgrade script that fails with cannot drop table search_wheres because other objects depend on it — the pgpkg-generated drop doesn't account for dependent objects, so the regenerated incremental appears to need hand-editing.

Since both predate this PR, I've left the incremental as-is rather than bundle a large, partly-broken regeneration into an unrelated fix — happy to help with it separately.

scripts/test --pgtap passes: 360 assertions, including new checks for the partition_steps (name) unique index and the refresh_partition_matviews() function.

Type of change

• Bug fix (non-breaking change which fixes an issue)

Checklist

• Update tests (pgtap: unique index on partition_steps, helper function exists)
• Update CHANGELOG