[Proposal] Sync streams

[rkistner]

Background

The PowerSync sync system was initially designed to sync your entire set of data up-front, and keep it up-to-date in a consistent way. This is great for offline-first applications, where you do need the entire set of data available. However, there are many cases where there is too much data to sync everything, so we need mechanisms to only sync the data that the user will need right now.

Status

2025-09-01:

• Initial server-side support is released in version 1.15.x of the PowerSync Service - see Protocol changes for sync streams #307 / New unified syntax #313.
• Core client-side support is available in v0.4.5 of powersync-sqlite-core - see Sync streams powersync-sqlite-core#112.

Client SDK support is pending. See:

• Sync streams powersync.dart#317
• Sync streams powersync-js#707
• Sync streams powersync-kotlin#257

Current workarounds

Client parameters is one workaround currently available. It can be used to specify the data the user needs, for example a project_ids array.

An issue is that it becomes difficult in client applications to track and build these arrays. This is especially true in web applications where the user may have multiple tabs open, with each tab needing different sets of data.

Proposal

To handle these use cases, we are overhauling sync rules to "sync streams". A developer may specify multiple stream definitions, and a user may subscribe to each stream one or more times, with different sets of parameters.

An Example

Streams can be defined similar to sync rules:

config:
  edition: 2 # see https://github.com/powersync-ja/powersync-service/discussions/348
streams:
  issue:
    query: select * from issues where id = subscription.parameters() ->> 'id' 
  issue_comments:
    query: select * from comments where issue_id = subscription.parameters() ->> 'id' 

Then on the client:

db.syncStream('issue', {id: issueId1}).subscribe()
db.syncStream('issue_comments', {id: issueId1}).subscribe()

// Can subscribe multiple times with different parameters:
db.syncStream('issue', {id: issueId2}).subscribe()

Subscriptions can be unsubscribed directly, or automatically when the app / tab closes. Each has a configurable ttl that it stays active after unsubscribing, to effectively keep a warm cache of recently-accessed data.

Subscriptions are designed to be automatically managed by UI components, for example with React hooks:

function IssueComments(props: {id: string}) {
  const db = usePowerSync();
  const commentStream = db.syncStream('issue_comments', {id: props.id});
  const { data, loading } = useQuery('select * from comments where issue_id = ?' [props.id], { streams: [commentStream] });
  // Render data here
}

The above example will automatically subscribe to the issue_comments stream with the relevant issue id when the component is first rendered. When the component is unmounted, the subscription is cancelled. This doesn't mean that data is immediately deleted, however. Instead, a TTL for subscriptions ensures that if the component becomes active again in the near future, all data will be there already.

Stream definition syntax

Sync streams will be supported alongside the legacy Sync Rules / bucket definitions for the foreseeable future, although we'll recommend migrating to sync streams once it's stable.

The syntax is as follows:

streams:
  <stream_name>:
    query: string # similar to data queries in bucket definitions
    auto_subscribe: boolean # true to sync this stream by default, false (default) if clients should explicitly subscribe
    priority: number # bucket priority, same as in bucket definitions
    accept_potentially_dangerous_queries: boolean # silence warnings on dangerous queries, same as in bucket definitions

Supported syntax in the query is slightly different from bucket definitions. Where bucket definitions had separate parameter and data queries, sync streams only have a single query. Instead of parameter queries, sync streams support a limited form of subqueries, for example:

-- use auth and subscription parameters directly in the query
select * from issues where id = subscription.parameters() ->> 'id' and owner_id = auth.user_id()

-- "in (subquery)" replaces parameter queries:
select * from comments where issue_id in (select id from issues where owner_id = auth.user_id())

To a large extent, the same underlying bucket system is used, and the same functionality is available as before with parameter queries, while being closer to plain SQL.

Accessing parameters is slightly different from bucket definitions:

subscription.parameters() -- all parameters for the specific subscription, as JSON
subscription.parameter('key') -- short-hand for getting a specific parameter
auth.jwt() -- JWT token payload, as JSON
auth.parameter('key') -- short-hand for getting a specific token payload parameter
auth.user_id() -- same as auth.parameter('sub')
connection.parameters() -- parameters specified on the connection level (equivalent to client parameters in the legacy Sync Rules system), as JSON
connection.parameter('key') -- short-hand for getting a specific connection parameter

Client-side syntax

The specific syntax will depend on the SDK, but in general each SDK should support:

• db.syncStream(name, [params]) returns a SyncStream instance, which can be used to subscribe() to that stream.
• calling subscribe() on a SyncStream returns a SyncStreamSubscription, giving you access to waitForFirstSync() and unsubscribe() on that subscription.
• the SyncStatus interface includes a list of SyncSubscriptionDefinitions, describing all streams that the client is subscribed to (either due to an explicit subscriptions or because the stream has auto_subscribe: true). That interface also reports per-stream download progress.
• Each sync stream has a ttl (time-to-live). After calling unsubscribe(), or when the page/app closes, the stream will keep on syncing for the ttl duration, effectively acting as a cache. The client will support specifying the ttl, and a method for ignoring the ttl and deleting the data as soon as possible.

Protocol changes

The sync/stream request format is extended to support specifying which streams to sync.
The response format includes info for associating buckets with sync streams, to allow tracking progress for individual streams.

The bucket data and checkpoint system remains unchanged: Each sync stream just maps to one or more buckets, which are synced the same as before. This means current clients can sync streams with auto_subscribe without any changes.

The specific protocol additions are described in #307.

[cahofmeyr]

Each has a configurable ttl that it stays active after unsubscribing, to effectively keep a warm cache of recently-accessed data.

Is there a default TTL?

[simolus3]

Yes, if no TTL is set in the subscribe() call, it defaults to 24h.