Skip to content

[SPARK-56453][SS] Add ResolveEventTimeWatermark to HiveSessionStateBuilder#55312

Closed
ericm-db wants to merge 2 commits intoapache:masterfrom
ericm-db:watermark-hive-resolution
Closed

[SPARK-56453][SS] Add ResolveEventTimeWatermark to HiveSessionStateBuilder#55312
ericm-db wants to merge 2 commits intoapache:masterfrom
ericm-db:watermark-hive-resolution

Conversation

@ericm-db
Copy link
Copy Markdown
Contributor

@ericm-db ericm-db commented Apr 11, 2026

What changes were proposed in this pull request?

Add ResolveEventTimeWatermark to the extendedResolutionRules in HiveSessionStateBuilder.

BaseSessionStateBuilder already includes this rule, but HiveSessionStateBuilder overrides extendedResolutionRules entirely without including it. This causes UnresolvedEventTimeWatermark nodes (which have output = Nil) to persist through analysis when using the Hive session state, blocking column resolution in downstream operators such as JOIN conditions and WHERE/ORDER BY clauses that reference aliased STREAM WATERMARK relations.

Why are the changes needed?

This is a regression introduced by #52428 (SPARK-53477), which added ResolveEventTimeWatermark to BaseSessionStateBuilder.extendedResolutionRules but missed that HiveSessionStateBuilder overrides that list entirely. When a query uses STREAM(...) WATERMARK ... <alias> syntax with the Hive session state, the UnresolvedEventTimeWatermark node is never resolved, causing UNRESOLVED_COLUMN errors when referencing columns through the alias.

Does this PR introduce any user-facing change?

Yes. Streaming SQL queries using WATERMARK with table aliases now resolve correctly when using the Hive session state, fixing UNRESOLVED_COLUMN errors.

How was this patch tested?

Added WatermarkColumnResolutionSuite with three test cases:

  • STREAM JOIN with WATERMARK and alias resolves columns correctly
  • STREAM with WATERMARK and alias resolves columns in WHERE
  • STREAM with WATERMARK and alias resolves columns in ORDER BY

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.6)

@ericm-db ericm-db changed the title [SPARK-XXXXX][SS] Add ResolveEventTimeWatermark to HiveSessionStateBuilder [SPARK-56453][SS] Add ResolveEventTimeWatermark to HiveSessionStateBuilder Apr 11, 2026
…ilder

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ericm-db ericm-db force-pushed the watermark-hive-resolution branch from 49987e2 to eef83f0 Compare April 11, 2026 22:20
Copy link
Copy Markdown
Contributor

@anishshri-db anishshri-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm pending green CI

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants