tp: scan SQL queries into batches by LalitMaganti · Pull Request #7151 · google/perfetto

LalitMaganti · 2026-08-20T19:06:13Z

Queries which filter, join, group, or compute still need SQLite. SqlScan steps
a prepared statement and fills RowBatch instances instead of exposing the
row-at-a-time cursor to the rest of the pipeline.

Lineage decides how each result column is stored. Columns traced to a
dataframe come out flat at the proven type. Other columns use Variant because
SQLite may return a different type on each row.

Preparing, stepping, and rewinding the statement all follow the Source status
contract. Type validation is left to the operator which knows how a column
will be used.

github-actions · 2026-08-20T19:58:03Z

🎨 Perfetto UI Builds

6715b98 (2026-08-20 19:58 UTC): https://storage.googleapis.com/perfetto-ci-artifacts/gh-32410484882-1-ui/ui/index.html
d266934 (2026-08-20 20:08 UTC): https://storage.googleapis.com/perfetto-ci-artifacts/gh-32411531697-1-ui/ui/index.html
e8ef8e0 (2026-08-20 20:31 UTC): https://storage.googleapis.com/perfetto-ci-artifacts/gh-32413526288-1-ui/ui/index.html
356cfad (2026-08-20 20:53 UTC): https://storage.googleapis.com/perfetto-ci-artifacts/gh-32415492006-1-ui/ui/index.html
75fa366 (2026-08-21 00:26 UTC): https://storage.googleapis.com/perfetto-ci-artifacts/gh-32431600546-1-ui/ui/index.html
5864776 (2026-08-21 00:42 UTC): https://storage.googleapis.com/perfetto-ci-artifacts/gh-32432978658-1-ui/ui/index.html
e685b5f (2026-08-21 10:08 UTC): https://storage.googleapis.com/perfetto-ci-artifacts/gh-32470579746-1-ui/ui/index.html
6e16818 (2026-08-21 14:04 UTC): https://storage.googleapis.com/perfetto-ci-artifacts/gh-32489187430-1-ui/ui/index.html
23d8d5a (2026-08-21 14:45 UTC): https://storage.googleapis.com/perfetto-ci-artifacts/gh-32491876480-1-ui/ui/index.html
7ca7354 (2026-08-21 15:43 UTC): https://storage.googleapis.com/perfetto-ci-artifacts/gh-32498237086-1-ui/ui/index.html
3cc9e50 (2026-08-21 16:30 UTC): https://storage.googleapis.com/perfetto-ci-artifacts/gh-32502240355-1-ui/ui/index.html
750ee97 (2026-08-25 14:06 UTC): https://storage.googleapis.com/perfetto-ci-artifacts/gh-32856173459-1-ui/ui/index.html
96fc48d (2026-08-25 14:46 UTC): https://storage.googleapis.com/perfetto-ci-artifacts/gh-32858990397-1-ui/ui/index.html
12e57d9 (2026-08-25 15:10 UTC): https://storage.googleapis.com/perfetto-ci-artifacts/gh-32862945389-1-ui/ui/index.html

Queries which transform their input still need SQLite, but feeding its row cursor directly into the pipeline restores per-row overhead. Step a prepared statement into RowBatch instances. Emit dataframe-backed columns at their proven storage type and use Variant for columns whose type is unknown.

Each execution must prepare and validate a fresh result shape so rewinds clear latched SQLite errors and schema changes cannot silently alter batches.

SQL scan tests inspect the same selected RowBatch representation and should depend on the shared test-only readers.

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from ab96e6e to 74b28f5 Compare August 20, 2026 19:15

LalitMaganti force-pushed the dev/lalitm/exec-breaker branch 2 times, most recently from 409d137 to 843ed07 Compare August 20, 2026 19:19

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from 74b28f5 to d4cb8e6 Compare August 20, 2026 19:19

LalitMaganti force-pushed the dev/lalitm/exec-breaker branch from 843ed07 to a084d88 Compare August 20, 2026 19:46

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from d4cb8e6 to d02f73c Compare August 20, 2026 19:46

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from d02f73c to b6eea90 Compare August 20, 2026 19:58

LalitMaganti force-pushed the dev/lalitm/exec-breaker branch from a084d88 to 2320bd0 Compare August 20, 2026 19:58

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from b6eea90 to 44a3370 Compare August 20, 2026 20:18

LalitMaganti force-pushed the dev/lalitm/exec-breaker branch 2 times, most recently from 064d156 to b9c1217 Compare August 20, 2026 20:41

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch 2 times, most recently from 06f8c12 to 13f6c7a Compare August 21, 2026 00:00

LalitMaganti force-pushed the dev/lalitm/exec-breaker branch from b9c1217 to 057bce9 Compare August 21, 2026 00:00

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch 2 times, most recently from 8d4e967 to d9aec30 Compare August 21, 2026 00:31

LalitMaganti force-pushed the dev/lalitm/exec-breaker branch from 057bce9 to 6c78e59 Compare August 21, 2026 00:31

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch 2 times, most recently from 2ec568a to e68c1da Compare August 21, 2026 13:14

LalitMaganti changed the base branch from dev/lalitm/exec-breaker to dev/lalitm/exec-assert-column-type August 21, 2026 13:16

LalitMaganti changed the title ~~tp: read a pipeline's rows from a query~~ tp: read a pipeline's rows from a query, and follow their types Aug 21, 2026

LalitMaganti force-pushed the dev/lalitm/exec-assert-column-type branch from 62f4dcd to 0ff9bcb Compare August 21, 2026 13:34

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from e68c1da to a6d3f55 Compare August 21, 2026 13:34

LalitMaganti force-pushed the dev/lalitm/exec-assert-column-type branch from 0ff9bcb to 2f056ac Compare August 21, 2026 13:43

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from a6d3f55 to bbd04c4 Compare August 21, 2026 13:43

LalitMaganti force-pushed the dev/lalitm/exec-assert-column-type branch from 2f056ac to 98d4d5e Compare August 21, 2026 13:53

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from bbd04c4 to 909c7b2 Compare August 21, 2026 13:53

LalitMaganti force-pushed the dev/lalitm/exec-assert-column-type branch from 98d4d5e to 2298a36 Compare August 21, 2026 14:23

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from 909c7b2 to 9b3bb60 Compare August 21, 2026 14:23

LalitMaganti force-pushed the dev/lalitm/exec-assert-column-type branch from 2298a36 to a22a502 Compare August 21, 2026 15:12

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from 9b3bb60 to 8339ebd Compare August 21, 2026 15:12

LalitMaganti force-pushed the dev/lalitm/exec-assert-column-type branch from a22a502 to c6f181b Compare August 21, 2026 15:21

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from 8339ebd to 78e655e Compare August 21, 2026 15:21

LalitMaganti force-pushed the dev/lalitm/exec-assert-column-type branch from c6f181b to 129d977 Compare August 21, 2026 15:33

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from 78e655e to 84ccb5f Compare August 21, 2026 15:33

LalitMaganti force-pushed the dev/lalitm/exec-assert-column-type branch from 129d977 to 0b6d57b Compare August 21, 2026 16:18

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from 84ccb5f to cb1b703 Compare August 21, 2026 16:18

LalitMaganti force-pushed the dev/lalitm/exec-assert-column-type branch from 0b6d57b to f4294d4 Compare August 25, 2026 13:49

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from cb1b703 to 37ade6e Compare August 25, 2026 13:49

LalitMaganti force-pushed the dev/lalitm/exec-assert-column-type branch from f4294d4 to cf16806 Compare August 25, 2026 13:54

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from 37ade6e to 57032cb Compare August 25, 2026 13:54

LalitMaganti changed the base branch from dev/lalitm/exec-assert-column-type to dev/lalitm/exec-dataframe-scan August 25, 2026 14:21

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from 57032cb to db06cd9 Compare August 25, 2026 14:21

LalitMaganti changed the title ~~tp: read a pipeline's rows from a query, and follow their types~~ tp: read a pipeline's rows from SQL Aug 25, 2026

LalitMaganti changed the title ~~tp: read a pipeline's rows from SQL~~ tp: scan SQL queries into batches Aug 25, 2026

LalitMaganti force-pushed the dev/lalitm/exec-dataframe-scan branch from 868c5e4 to 3939806 Compare August 25, 2026 14:59

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from db06cd9 to 33deaa4 Compare August 25, 2026 14:59

LalitMaganti added 3 commits August 25, 2026 17:49

tp: make SQL scans safely replayable

cd05800

Each execution must prepare and validate a fresh result shape so rewinds clear latched SQLite errors and schema changes cannot silently alter batches.

tp: reuse executor batch test helpers

31e8cae

SQL scan tests inspect the same selected RowBatch representation and should depend on the shared test-only readers.

LalitMaganti force-pushed the dev/lalitm/exec-dataframe-scan branch from 3939806 to 29817f8 Compare August 25, 2026 16:50

LalitMaganti force-pushed the dev/lalitm/exec-sql-scan branch from 33deaa4 to 31e8cae Compare August 25, 2026 16:50

Navigation Menu

Uh oh!

tp: scan SQL queries into batches - #7151

tp: scan SQL queries into batches#7151
LalitMaganti wants to merge 3 commits into
dev/lalitm/exec-dataframe-scanfrom
dev/lalitm/exec-sql-scan

LalitMaganti commented Aug 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 20, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

LalitMaganti commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎨 Perfetto UI Builds

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

LalitMaganti commented Aug 20, 2026 •

edited

Loading

github-actions Bot commented Aug 20, 2026 •

edited

Loading