Skip to content

fix: contain a raising RLS policy to the subscription it came from - #2096

Open
hamodywe wants to merge 1 commit into
supabase:mainfrom
hamodywe:fix/isolate-subscription-rls-errors
Open

fix: contain a raising RLS policy to the subscription it came from#2096
hamodywe wants to merge 1 commit into
supabase:mainfrom
hamodywe:fix/isolate-subscription-rls-errors

Conversation

@hamodywe

Copy link
Copy Markdown

Closes #2093.

The problem

realtime.apply_rls checks each subscription against the table's policies with a bare

execute 'execute walrus_rls_stmt' into subscription_has_access;

Nothing catches an error there. A policy that raises on one subscription's stored claims — a malformed claim cast to uuid is enough — propagates out of apply_rls and takes down the entire list_changes call, so every subscriber on the tenant loses the batch, including subscriptions on other roles and other tables whose own policies cannot fail.

It is silent from every angle an operator would look at: sockets connect, realtime.subscription looks correct, the slot stays active. And the changes are not replayed — confirmed_flush_lsn advances during decoding even though the statement errored, so each affected batch is dropped for good. That is what turned this into a ~2.5 hour project-wide outage in #2093.

The change

The per-subscription check goes into its own block. A subscription whose policy raises is disqualified on its own and collected into errored_role_sub_ids; every other subscription is evaluated and delivered normally.

Those subscriptions then get a row of their own carrying Error 500: Internal Server Error, RLS policy evaluation failed, with only schema, table and type in the payload — the same shape the existing 400 and 401 rows use — so a policy that could not be evaluated never ends up disclosing the row it was meant to gate. Silently dropping the subscription instead would have turned a loud tenant-wide outage into a silent per-subscriber one, which is the part of #2093 that made it expensive. Happy to cut that half if you would rather keep the change minimal.

The warning follows the pattern realtime.send already uses (WarnSendingBroadcastMessage), with the subscription id in it so the poisoned row can be found. Note that list_changes runs with SET log_min_messages TO 'fatal', so in the polling path this reaches the client rather than the server log; the errors array is the reliable signal there.

Everything else in the function is byte-identical to 20260709120000_fix_apply_rls_filter_role_leak.

Verification

Elixir side, in subscriptions_test.exs: a poisoned subscription alongside a healthy one on the same role, and alongside a subscription on a different role. Asserts the healthy subscriptions still receive the record, the poisoned one receives the error row without a record key, and the batch is consumed rather than left behind.

I could not run mix test locally — this is a Windows box with no Elixir toolchain and the suite provisions tenant databases through its own Docker backend. So I verified the SQL directly instead, on supabase/postgres:17.6.1.127 with the tenant migrations applied, and the tests are written to run on CI:

  • Reproduced first. Three subscriptions on one table (two anon, one authenticated with using (true)), one anon carrying app_metadata.app_id as the string "null". Pre-fix: ERROR: invalid input syntax for type uuid: "null" at execute walrus_rls_stmt, and all three subscriptions get nothing.

  • After the change, same scenario:

               subscription_ids            |                               errors                               | delivered_details
    ----------------------------------------+--------------------------------------------------------------------+-------------------
     {11111111-...} (anon, healthy)          | {}                                                                 | hello
     {22222222-...} (anon, poisoned)         | {"Error 500: Internal Server Error, RLS policy evaluation failed"} |
     {33333333-...} (authenticated)          | {}                                                                 | hello
    
  • No behaviour change when nothing raises. Ran a scenario covering INSERT/UPDATE/DELETE, a selected_columns subscription, two roles, and a table with no primary key (the 400 path) against both versions of the function and diffed the output: identical apart from the generated ids.

  • Cost of the extra subtransaction, interleaved A/B on the same box, 100 apply_rls calls per run, best of 5, 100 subscriptions on one record:

    round before after
    1 250.9 ms 260.6 ms
    2 259.1 ms 266.3 ms
    3 252.6 ms 261.3 ms

    About 3%, roughly 1 µs per subscription evaluation. If that is too much for the hot path, the alternative is to keep the loop unguarded and re-run it with per-subscription isolation only after something raises — zero cost normally, at the price of duplicating the loop. I went with the simpler version because this is a correctness fix in a function that is already hard to read; say the word and I will switch it.

realtime.apply_rls evaluates each subscription's policy with an unguarded
`execute walrus_rls_stmt`. A policy that raises - a malformed claim cast to
uuid is enough - propagates out of apply_rls and takes down the whole
list_changes call, so every subscriber on the tenant loses the batch, not
only the one whose claims are bad. The slot's confirmed_flush advances during
decoding, so those changes are dropped rather than replayed on the next poll.

Wrap the per-subscription check in its own block. A subscription whose policy
raises is disqualified on its own and reported back with `Error 500`, carrying
only schema, table and action - the same shape as the existing 400/401 rows -
so a policy that could not be evaluated never discloses the row it was meant
to gate.

Closes supabase#2093
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

apply_rls: one subscription whose RLS evaluation raises silently kills postgres_changes delivery for ALL subscribers

1 participant