Summary
The chat client reports its entire rendered message transcript to the server every time a message settles. Over a long conversation this sends roughly O(n²) bytes across a session, and each snapshot can be large because it also carries retained HTML dependencies. We should reduce this, likely via an incremental (delta/append) report protocol — but the fix has an architectural trade-off worth discussing before implementing.
Background: how message reporting works today
Since the chat UI moved to React, the browser is the source of truth for the user-facing conversation (what's actually displayed, including any app-applied transformations). The server no longer keeps its own copy of displayed messages; instead the client reports them and the server treats that report as authoritative.
Mechanically:
- Whenever the set of settled (non-streaming) messages changes, the client serializes all of them and sends the result to the server as a Shiny input,
input$<id>_messages (tagged shinychat.messages).
- Builder:
buildMessagesSnapshot() in js/src/chat/state.ts — maps every settled message to { role, segments, attachments?, htmlDeps? }.
- Trigger: the
reportSnapshot effect in js/src/chat/ChatApp.tsx (fires on state.messages changes) and a manual co-send on submit.
- Transport:
sendMessagesSnapshot() in js/src/transport/shiny-transport.ts.
- The server consumes the report as the full current list:
- Python —
Chat._reported_messages() / Chat.messages() in pkg-py/src/shinychat/_chat.py, plus bookmark serialization and history, read the input directly.
- R —
get_reported_messages() in pkg-r/R/chat_history.R; history's on_response() takes the full reported list and slices the newly-added tail using a ui_offset cursor before attaching UI to record nodes.
Streaming does not trigger a report per chunk (in-flight content lives in a separate state.streamingMessage field), so the report fires roughly once per settle point — not on every token. The cost below is about settle-point frequency, not streaming.
The problem
Every settle point re-sends the whole transcript:
- Turn 1 completes → send 1 message.
- Turn 2 completes → send 2 messages.
- …
- Turn n completes → send n messages.
Total ≈ 1 + 2 + … + n = O(n²) message-payloads over the session. Each payload is not tiny either: segments hold the full rendered content string of each message, and any message that embedded a widget/HTML output carries its htmlDeps (script/stylesheet references), which are re-sent inside every subsequent snapshot.
For typical short conversations this is fine. For long-lived or widget-heavy chats it means steadily growing per-message websocket traffic and server-side deserialization work that scales with the square of the conversation length.
Why it isn't a trivial fix
The obvious fix — have the client send only what's new since its last report — requires the server to accumulate reports into the current list. That reintroduces server-side state that the client-authoritative design deliberately removed (see #272, which deleted Python's _store_message accumulator). In particular, Chat.messages() currently returns the input value as-is; with a delta protocol it would have to read from an accumulator instead.
So this is a genuine design decision, not just an optimization: is a small, dumb append buffer on the server acceptable, given the goal of keeping the browser as the single source of truth?
Options to explore
- Append-delta protocol. Client sends only messages appended since its last acknowledged report, plus a base count; server maintains a thin append buffer and exposes the accumulated list to
.messages() / history. Simplest incremental model; reintroduces a minimal accumulator.
- Content-addressed messages. Give each message a stable id; unchanged messages are sent as
{ id } references rather than full bodies; the server merges by id into its record. Avoids re-sending bodies without a strict append assumption (tolerates edits/branching), at the cost of a merge step.
- Session-level
htmlDeps table. Deduplicate HTML dependencies across the session — send each dep once and reference it by index from messages. Orthogonal to 1/2 and stackable; helps the widget-heavy case specifically.
- Do nothing / cap. Accept the ceiling and instead document it, or cap transcript length. Lowest effort; punts the scaling problem.
Acceptance criteria
- Per-session client→server bytes for reporting grow ~linearly (not quadratically) with conversation length.
Chat.messages() (Python), bookmark serialization, and R history restore continue to reflect exactly what was displayed, including app-applied transformations.- Behavior is consistent across the Python and R implementations.
Context
Raised in review of #272 (making the browser the source of truth for chat messages). That PR intentionally left this as a follow-up to keep its scope focused on the source-of-truth change.
Summary
The chat client reports its entire rendered message transcript to the server every time a message settles. Over a long conversation this sends roughly O(n²) bytes across a session, and each snapshot can be large because it also carries retained HTML dependencies. We should reduce this, likely via an incremental (delta/append) report protocol — but the fix has an architectural trade-off worth discussing before implementing.
Background: how message reporting works today
Since the chat UI moved to React, the browser is the source of truth for the user-facing conversation (what's actually displayed, including any app-applied transformations). The server no longer keeps its own copy of displayed messages; instead the client reports them and the server treats that report as authoritative.
Mechanically:
input$<id>_messages(taggedshinychat.messages).buildMessagesSnapshot()injs/src/chat/state.ts— maps every settled message to{ role, segments, attachments?, htmlDeps? }.reportSnapshoteffect injs/src/chat/ChatApp.tsx(fires onstate.messageschanges) and a manual co-send on submit.sendMessagesSnapshot()injs/src/transport/shiny-transport.ts.Chat._reported_messages()/Chat.messages()inpkg-py/src/shinychat/_chat.py, plus bookmark serialization and history, read the input directly.get_reported_messages()inpkg-r/R/chat_history.R; history'son_response()takes the full reported list and slices the newly-added tail using aui_offsetcursor before attaching UI to record nodes.Streaming does not trigger a report per chunk (in-flight content lives in a separate
state.streamingMessagefield), so the report fires roughly once per settle point — not on every token. The cost below is about settle-point frequency, not streaming.The problem
Every settle point re-sends the whole transcript:
Total ≈ 1 + 2 + … + n = O(n²) message-payloads over the session. Each payload is not tiny either:
segmentshold the full rendered content string of each message, and any message that embedded a widget/HTML output carries itshtmlDeps(script/stylesheet references), which are re-sent inside every subsequent snapshot.For typical short conversations this is fine. For long-lived or widget-heavy chats it means steadily growing per-message websocket traffic and server-side deserialization work that scales with the square of the conversation length.
Why it isn't a trivial fix
The obvious fix — have the client send only what's new since its last report — requires the server to accumulate reports into the current list. That reintroduces server-side state that the client-authoritative design deliberately removed (see #272, which deleted Python's
_store_messageaccumulator). In particular,Chat.messages()currently returns the input value as-is; with a delta protocol it would have to read from an accumulator instead.So this is a genuine design decision, not just an optimization: is a small, dumb append buffer on the server acceptable, given the goal of keeping the browser as the single source of truth?
Options to explore
.messages()/ history. Simplest incremental model; reintroduces a minimal accumulator.{ id }references rather than full bodies; the server merges by id into its record. Avoids re-sending bodies without a strict append assumption (tolerates edits/branching), at the cost of a merge step.htmlDepstable. Deduplicate HTML dependencies across the session — send each dep once and reference it by index from messages. Orthogonal to 1/2 and stackable; helps the widget-heavy case specifically.Acceptance criteria
Chat.messages()(Python), bookmark serialization, and R history restore continue to reflect exactly what was displayed, including app-applied transformations.Context
Raised in review of #272 (making the browser the source of truth for chat messages). That PR intentionally left this as a follow-up to keep its scope focused on the source-of-truth change.