Skip to content

Support Databricks query tags in session properties#5794

Open
WhammyLeaf wants to merge 8 commits into
SQLMesh:mainfrom
trygforsikring:add-databricks-query-tags-session-properties
Open

Support Databricks query tags in session properties#5794
WhammyLeaf wants to merge 8 commits into
SQLMesh:mainfrom
trygforsikring:add-databricks-query-tags-session-properties

Conversation

@WhammyLeaf

@WhammyLeaf WhammyLeaf commented May 15, 2026

Copy link
Copy Markdown

Description

The Databricks SQL connector supports per-statement query_tags, but SQLMesh did not previously pass model/session-level session_properties.query_tags through to the connector. This change stores validated query tags for the active adapter session and forwards them to Databricks SQL connector executions.

Query tags are only applied on the Databricks SQL connector path. They are not passed when execution is routed through Spark session / Databricks Connect paths, since those cursors do not support the query_tags argument.

ISSUE: #5795

Test Plan

  • Added Databricks engine adapter tests covering:
    • forwarding session query tags to Databricks SQL connector executions
    • clearing query tags after the session ends
    • preserving explicit per-statement query_tags
    • accepting NULL query tag values
    • rejecting invalid session_properties.query_tags
    • not applying query tags to Spark session / Databricks Connect execution paths
  • Verified the installed Databricks SQL connector exposes Cursor.execute(..., query_tags=...).
  • Built the package wheel successfully.
  • Ran:
pytest -q -o addopts='' tests/core/engine_adapter/test_databricks.py
python -m pip wheel . --no-deps -w /tmp/sqlmesh-wheel-final

@WhammyLeaf WhammyLeaf force-pushed the add-databricks-query-tags-session-properties branch 3 times, most recently from ad8c237 to d9881ce Compare May 15, 2026 15:05
@WhammyLeaf

Copy link
Copy Markdown
Author

Will fix the failing style tests.

Signed-off-by: Christian Troelsen <christian.troelsen@tryg.dk>
@WhammyLeaf WhammyLeaf force-pushed the add-databricks-query-tags-session-properties branch from d9881ce to 17b3af7 Compare May 15, 2026 18:19
@WhammyLeaf

Copy link
Copy Markdown
Author

@izeigerman Perhaps you can have a look at this 😄 ?

@StuffbyYuki

Copy link
Copy Markdown
Collaborator

@WhammyLeaf Can you rebase your branch? Then I'll kick off the workflows and give it a review!

@WhammyLeaf

Copy link
Copy Markdown
Author

@WhammyLeaf Can you rebase your branch? Then I'll kick off the workflows and give it a review!

Should be done now 😄

@StuffbyYuki

Copy link
Copy Markdown
Collaborator

@WhammyLeaf Just gave it a review, and these are not blockers but curious your thoughts:

  • Load-time validation for query_tags in meta.py (consistent with query_label / authorization)
  • Model-level tests in test_model.py
  • Brief docs in databricks.md for MAP(...) syntax and SQL-connector-only limitation

Let me know if I'm missing anything!

@StuffbyYuki StuffbyYuki self-requested a review June 22, 2026 15:12
@WhammyLeaf

WhammyLeaf commented Jun 23, 2026

Copy link
Copy Markdown
Author

@WhammyLeaf Just gave it a review, and these are not blockers but curious your thoughts:

  • Load-time validation for query_tags in meta.py (consistent with query_label / authorization)
  • Model-level tests in test_model.py
  • Brief docs in databricks.md for MAP(...) syntax and SQL-connector-only limitation

Let me know if I'm missing anything!

@StuffbyYuki This sounds about right. I have added both load-time validation, model-level tests and documentation now 😄

@StuffbyYuki

Copy link
Copy Markdown
Collaborator

@WhammyLeaf Thanks for the changes! The last thing is the DCO checks, and once all CI tests pass I'll merge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants