fix(profiler): sanitize BigQuery uniqueCount SQL for TIMESTAMP/DATE and STRUCT-subfield columns (#30152) by kshitij1439 · Pull Request #30538 · open-metadata/OpenMetadata

kshitij1439 · 2026-07-27T21:01:30Z

Description

On BigQuery, the uniqueCount metric generated invalid SQL and failed for:

Non-integer columns (TIMESTAMP, DATE).
Nested STRUCT subfield columns (e.g., customer.email).

This is distinct from #27256, which fixed a separate BYTES/binary-column type
mismatch in the same function. That fix did not resolve the TIMESTAMP/DATE or
STRUCT-subfield cases described here — confirmed still present on main as of
the issue being filed.

Root Cause

Type Mismatch (TIMESTAMP/DATE): UniqueCount.query built the outer COUNTIF using column(col.name, col.type). This caused SQLAlchemy to bind parameter 1 matching the column's native TIMESTAMP/DATE type, triggering BigQuery type conversion errors.
Dotted Path Resolution (STRUCT subfields): The outer COUNTIF used raw dotted identifiers (e.g. customer.email) while the subquery grouped and labeled using sanitized identifiers, resulting in unresolved column errors (400 Unrecognized name: customer) and invalid parameter names (customer.email_1).

Solution

Introduced a shared, untyped integer column alias VALUE_COUNT_ALIAS = "value_count".
Updated UniqueCount.query() to construct func.countif(column(VALUE_COUNT_ALIAS, Integer()) == 1).
Synchronized profiler_interface.py (_compute_query_metrics) and columnValuesToBeUnique.py (_run_results) to label grouped CTE subqueries with VALUE_COUNT_ALIAS.
Added unit tests compiling against BigQueryDialect for TIMESTAMP, DATE, and STRUCT columns.

Affected Modules

ingestion/src/metadata/profiler/metrics/static/unique_count.py
ingestion/src/metadata/profiler/interface/sqlalchemy/profiler_interface.py
ingestion/src/metadata/data_quality/validations/column/sqlalchemy/columnValuesToBeUnique.py
ingestion/tests/unit/observability/profiler/sqlalchemy/bigquery/test_unique_count_bigquery.py

Testing

Ran unit tests with pytest:

pytest ingestion/tests/unit/observability/profiler/sqlalchemy/bigquery/test_unique_count_bigquery.py
pytest ingestion/tests/unit/observability/profiler/ -q

Greptile Summary

Updates BigQuery unique-count query generation to use a shared integer count alias.

Labels grouped profiler and data-quality count expressions as value_count.
Makes the outer BigQuery COUNTIF reference that alias instead of the original typed or dotted column identifier.
Adds SQL-compilation coverage for TIMESTAMP, DATE, and nested STRUCT-subfield columns.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
ingestion/src/metadata/profiler/metrics/static/unique_count.py	Introduces the shared count alias and uses an integer-typed alias reference in BigQuery COUNTIF generation.
ingestion/src/metadata/profiler/interface/sqlalchemy/profiler_interface.py	Labels the grouped profiler count with the alias expected by the BigQuery outer expression.
ingestion/src/metadata/data_quality/validations/column/sqlalchemy/columnValuesToBeUnique.py	Keeps the validator’s grouped CTE producer and aggregate consumer synchronized on the shared alias.
ingestion/tests/unit/observability/profiler/sqlalchemy/bigquery/test_unique_count_bigquery.py	Adds compilation tests covering profiler and validator SQL for temporal and nested STRUCT-subfield columns.

Sequence Diagram

sequenceDiagram
    participant Caller as Profiler / Validator
    participant CTE as Grouped subquery
    participant BQ as BigQuery
    Caller->>CTE: COUNT(source_column) AS value_count
    CTE->>BQ: Group by source_column
    Caller->>BQ: "COUNTIF(value_count = 1)"
    BQ-->>Caller: uniqueCount

_{Reviews (5): Last reviewed commit: "fix(profiler): gracefully skip BigQuery ..." | Re-trigger Greptile}

…nd STRUCT-subfield columns (open-metadata#30152)

github-actions · 2026-07-27T21:01:42Z

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

Linked issue BigQuery profiler: uniqueCount emits invalid SQL for TIMESTAMP/DATE and nested STRUCT-subfield columns #30152 is not added to the Shipping project.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

github-actions · 2026-07-27T21:01:48Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

github-actions · 2026-07-27T21:09:09Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

kshitij1439 · 2026-07-27T21:30:13Z

Hi team 👋 this PR seems blocked because #30152 is on Roadmap, not Shipping.
Could someone move it or add skip-pr-checks? Tests pass locally, thanks! 🙏

github-actions · 2026-07-28T12:59:43Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

github-actions · 2026-07-29T12:51:18Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copilot

Pull request overview

This PR fixes BigQuery-specific SQL generation for the uniqueCount profiler metric (and the related uniqueness data-quality validation) by making the outer COUNTIF compare against a stable, integer-typed alias instead of reusing the source column’s name/type—resolving failures for TIMESTAMP/DATE columns and nested STRUCT subfield paths.

Changes:

Introduces a shared VALUE_COUNT_ALIAS = "value_count" and updates BigQuery UniqueCount.query() to use countif(column(value_count, Integer()) == 1).
Aligns the grouped subquery/CTE projections in SQAProfilerInterface and ColumnValuesToBeUniqueValidator to label the grouped count as value_count.
Adds BigQuery SQL-compilation unit tests covering TIMESTAMP, DATE, and dotted STRUCT-subfield column cases.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
ingestion/src/metadata/profiler/metrics/static/unique_count.py	Adds `VALUE_COUNT_ALIAS` and updates BigQuery `COUNTIF` to reference an integer-typed alias.
ingestion/src/metadata/profiler/interface/sqlalchemy/profiler_interface.py	Ensures the grouped subquery feeding BigQuery `uniqueCount` labels the count output with `VALUE_COUNT_ALIAS`.
ingestion/src/metadata/data_quality/validations/column/sqlalchemy/columnValuesToBeUnique.py	Labels grouped CTE counts with `VALUE_COUNT_ALIAS` and sums the correct CTE column to match the BigQuery `uniqueCount` expression.
ingestion/tests/unit/observability/profiler/sqlalchemy/bigquery/test_unique_count_bigquery.py	Adds BigQuery dialect SQL-compilation tests for temporal types and dotted `STRUCT` subfields.

+from unittest.mock import Mock, patch
+
+from sqlalchemy import Column, Date, DateTime, Integer, String, create_engine
+from sqlalchemy.orm import DeclarativeBase, Session
+from sqlalchemy_bigquery.base import BigQueryDialect
+


…my-bigquery not installed

github-actions · 2026-07-29T17:59:30Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

gitar-bot · 2026-07-29T17:59:46Z

Code Review ✅ Approved

Sanitizes BigQuery uniqueCount SQL for TIMESTAMP, DATE, and STRUCT-subfield columns by using an untyped integer column alias. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

`Compact`
`gitar display:verbose`

_{Was this helpful? React with 👍 / 👎 | Gitar | Powered by Gitar — free for open source}

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

ingestion/tests/unit/observability/profiler/sqlalchemy/bigquery/test_unique_count_bigquery.py:38

pytest.importorskip(...) is executed before several module-level imports, so the subsequent imports are no longer at the top of the file and will trigger Ruff/flake8 E402 (see similar pattern in ingestion/tests/unit/topology/database/test_vertica_type_mapping.py:16-24, which adds # noqa: E402 after importorskip). Consider moving the importorskip block to just before the optional sqlalchemy_bigquery import so only that import needs # noqa: E402.

import pytest

pytest.importorskip(
    "sqlalchemy_bigquery",
    reason="sqlalchemy-bigquery not installed — skipping BigQuery SQL compilation tests",
)

from unittest.mock import Mock, patch

from sqlalchemy import Column, Date, DateTime, Integer, String, create_engine
from sqlalchemy.orm import DeclarativeBase, Session
from sqlalchemy_bigquery.base import BigQueryDialect

fix(profiler): sanitize BigQuery uniqueCount SQL for TIMESTAMP/DATE a…

f6ab935

…nd STRUCT-subfield columns (open-metadata#30152)

kshitij1439 requested a review from a team as a code owner July 27, 2026 21:01

fix(profiler): resolve unused VALUE_COUNT_ALIAS import in BigQuery test

0ce5ca4

Merge branch 'main' into fix/30152-bigquery-unique-count

949f569

Merge branch 'main' into fix/30152-bigquery-unique-count

fc649ee

Copilot AI review requested due to automatic review settings July 29, 2026 12:50

Copilot started reviewing on behalf of kshitij1439 July 29, 2026 12:51 View session

Copilot AI reviewed Jul 29, 2026

View reviewed changes

fix(profiler): gracefully skip BigQuery uniqueCount tests if sqlalche…

ddb9819

…my-bigquery not installed

Copilot AI review requested due to automatic review settings July 29, 2026 17:59

Copilot started reviewing on behalf of kshitij1439 July 29, 2026 17:59 View session

Copilot AI reviewed Jul 29, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(profiler): sanitize BigQuery uniqueCount SQL for TIMESTAMP/DATE and STRUCT-subfield columns (#30152) - #30538

fix(profiler): sanitize BigQuery uniqueCount SQL for TIMESTAMP/DATE and STRUCT-subfield columns (#30152)#30538
kshitij1439 wants to merge 5 commits into
open-metadata:mainfrom
kshitij1439:fix/30152-bigquery-unique-count

kshitij1439 commented Jul 27, 2026 •

edited by greptile-apps Bot

Loading

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

kshitij1439 commented Jul 27, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 29, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Jul 29, 2026

Uh oh!

gitar-bot Bot commented Jul 29, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

kshitij1439 commented Jul 27, 2026 • edited by greptile-apps Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Root Cause

Solution

Affected Modules

Testing

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

github-actions Bot commented Jul 27, 2026

❌ PR checklist incomplete

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

kshitij1439 commented Jul 27, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

github-actions Bot commented Jul 29, 2026

Uh oh!

gitar-bot Bot commented Jul 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kshitij1439 commented Jul 27, 2026 •

edited by greptile-apps Bot

Loading