Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark 3.5: Implement RewriteTablePath #11555

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

szehon-ho
Copy link
Collaborator

This is the implementation for #10920 (an action to prepare metadata for an Iceberg table for DR copy)

This has been used in production for awhile in our setup, although support for rewrite of V2 position delete is new. I performed the following cleanups while contributing it.

  • Made RewriteTableSparkAction code more functional (avoid using member variable on the action to track state)
  • Moved some RewriteTableSparkAction code to core Utll classes to avoid having to make some classes public as was previously done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant