Skip to content

fix(starrocks): serialize result batches into a growable buffer#1106

Merged
mbrobbel merged 1 commit into
sirius-db:devfrom
mbrobbel:starrocks-thrift-result-buffer
Jul 14, 2026
Merged

fix(starrocks): serialize result batches into a growable buffer#1106
mbrobbel merged 1 commit into
sirius-db:devfrom
mbrobbel:starrocks-thrift-result-buffer

Conversation

@mbrobbel

@mbrobbel mbrobbel commented Jul 7, 2026

Copy link
Copy Markdown
Member

Result batches can be arbitrarily large; the fixed 64 KiB thrift channel truncated them, surfacing as a transport error at fetch_data. Serialize into a plain growable buffer instead.

Stack

All based on dev, so each diff includes the levels below it until they merge (review commits, or the last commit, for the per-level change):

  1. fix(starrocks): serialize result batches into a growable buffer #1106 ⬅️ this PR
  2. feat(starrocks): encode decimal/date/timestamp result columns #1107
  3. feat(starrocks): add fragment and plan dump debug aids to the CN #1108
  4. feat(starrocks): translate arithmetic, date, IN, CASE, and scalar calls #1109
  5. feat(starrocks): translate one-phase aggregation and sorts #1110
  6. feat(starrocks): translate hash and nested-loop joins #1111

🤖 Generated with Claude Code

Result batches can be arbitrarily large; the fixed 64 KiB thrift
channel truncated anything bigger, surfacing as a transport error on
the fetch_data path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@aocsa aocsa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small non-blocking suggestion: this change is easiest to regress accidentally during a future refactor because the buffer size is no longer explicit. Would you mind adding a test that serializes a TResultBatch larger than 64 KiB and asserts the output length exceeds that limit? That would lock in the growable-buffer behavior.

@mbrobbel
mbrobbel added this pull request to the merge queue Jul 14, 2026
Merged via the queue into sirius-db:dev with commit 3fc0e94 Jul 14, 2026
12 checks passed
@mbrobbel
mbrobbel deleted the starrocks-thrift-result-buffer branch July 14, 2026 15:26
kevkrist pushed a commit to kevkrist/sirius that referenced this pull request Jul 15, 2026
…-db#1107)

Encode decimal128, date32, and microsecond-timestamp result columns
(plus large/view string arrays) as MySQL text on the `fetch_data` path.



## Stack

All based on `dev`, so each diff includes the levels below it until they
merge (review commits, or the last commit, for the per-level change):

1. sirius-db#1106
2. sirius-db#1107 ⬅️ this PR
3. sirius-db#1108
4. sirius-db#1109
5. sirius-db#1110
6. sirius-db#1111

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants