Skip to content

fix(querier): decode JSON columns for scalar and time-series results - #12555

Open
tushar-signoz wants to merge 1 commit into
mainfrom
tvats-json-column-scan
Open

fix(querier): decode JSON columns for scalar and time-series results#12555
tushar-signoz wants to merge 1 commit into
mainfrom
tvats-json-column-scan

Conversation

@tushar-signoz

Copy link
Copy Markdown
Contributor

Description

  • POST /api/v5/query_range returned HTTP 500 (JSON Scan value must be clickhouse.JSON or map[string]any) for any scalar or time_series ClickHouse query whose result contained a JSON column — on a body_v2 stack, select * from signoz_logs.logs_v2 is enough. Only readAsRaw routed JSON columns to a []byte scan target; readAsScalar and readAsTimeSeries scanned into chcol.JSON, whose Scan rejects the string ClickHouse sends us.
  • Hoists the scan-target choice and the JSON post-processing out of readAsRaw into scanTarget / decodeColumnValue, shared by all three readers.
  • Also unwraps Dynamic columns (a JSON path such as body_v2.level) from their chcol.Variant envelope, so consumers get 7 / "x" instead of an opaque struct.

Issues closed by this PR

Fixes https://github.com/SigNoz/engineering-pod/issues/5911

Additional Information

The issue suggested treating Dynamic like JSON on the bytes path. Probing the driver against ClickHouse 25.12 shows that would break it — Dynamic scans natively, and only its string-backed values survive a *[]byte target:

column native (reflect.New(ScanType())) *[]byte
JSON JSON Scan value must be…
Dynamic (String) Variant{"str", "String"}
Dynamic (Int64) Variant{1, "Int64"} converting Int64 to *[]uint8

So Dynamic keeps its native scan target and is unwrapped after the scan instead. The wire format is unchanged (Variant already marshalled to its underlying value), only the Go type improves.

Verified end to end against the devenv ClickHouse, same request on both binaries:

requestType main this branch
scalar 500 200, body_v2 decoded as an object
time_series 500 200
raw 200 200

Out of scope, but found while probing: a Map(String, JSON) column panics inside the driver (reflect.Value.SetMapIndex: value of type []uint8 is not assignable to type chcol.JSON), and consume runs in an errgroup goroutine, so the HTTP recovery middleware would not catch it. Nested-JSON containers can't be byte-scanned either, so that needs its own fix.

Share the raw reader's JSON scan target across all three readers, and unwrap
Dynamic columns from their chcol.Variant envelope.
@github-actions github-actions Bot added the bug Something isn't working label Aug 13, 2026
@tushar-signoz tushar-signoz self-assigned this Aug 14, 2026
Comment thread pkg/querier/consume.go
return &v
}
return reflect.New(colType.ScanType()).Interface()
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if this this is the best possible fix for the highlighted problem, can there other possible places where we might miss reading this []byte like in waterfall / flamegraph apis once we move attributes to json?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants