[SPARK-56453][SS] Add ResolveEventTimeWatermark to HiveSessionStateBuilder#55312
Closed
ericm-db wants to merge 2 commits intoapache:masterfrom
Closed
[SPARK-56453][SS] Add ResolveEventTimeWatermark to HiveSessionStateBuilder#55312ericm-db wants to merge 2 commits intoapache:masterfrom
ericm-db wants to merge 2 commits intoapache:masterfrom
Conversation
…ilder Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
49987e2 to
eef83f0
Compare
anishshri-db
approved these changes
Apr 11, 2026
Contributor
anishshri-db
left a comment
There was a problem hiding this comment.
lgtm pending green CI
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add
ResolveEventTimeWatermarkto theextendedResolutionRulesinHiveSessionStateBuilder.BaseSessionStateBuilderalready includes this rule, butHiveSessionStateBuilderoverridesextendedResolutionRulesentirely without including it. This causesUnresolvedEventTimeWatermarknodes (which haveoutput = Nil) to persist through analysis when using the Hive session state, blocking column resolution in downstream operators such as JOIN conditions and WHERE/ORDER BY clauses that reference aliased STREAM WATERMARK relations.Why are the changes needed?
This is a regression introduced by #52428 (SPARK-53477), which added
ResolveEventTimeWatermarktoBaseSessionStateBuilder.extendedResolutionRulesbut missed thatHiveSessionStateBuilderoverrides that list entirely. When a query usesSTREAM(...) WATERMARK ... <alias>syntax with the Hive session state, theUnresolvedEventTimeWatermarknode is never resolved, causingUNRESOLVED_COLUMNerrors when referencing columns through the alias.Does this PR introduce any user-facing change?
Yes. Streaming SQL queries using
WATERMARKwith table aliases now resolve correctly when using the Hive session state, fixingUNRESOLVED_COLUMNerrors.How was this patch tested?
Added
WatermarkColumnResolutionSuitewith three test cases:Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.6)