[FLINK-39245][Iceberg] Add AWS Glue Catalog support for Iceberg pipeline connector#4314
Open
norrishuang wants to merge 9 commits intoapache:masterfrom
Open
[FLINK-39245][Iceberg] Add AWS Glue Catalog support for Iceberg pipeline connector#4314norrishuang wants to merge 9 commits intoapache:masterfrom
norrishuang wants to merge 9 commits intoapache:masterfrom
Conversation
- Add iceberg-aws and iceberg-aws-bundle dependencies to pom.xml - Add Glue catalog config options: type=glue, catalog-impl, io-impl, glue.id, glue.skip-archive, glue.skip-name-validation, client.region - Register new options in IcebergDataSinkFactory - Update TYPE and WAREHOUSE descriptions to include glue catalog - Add unit tests for Glue catalog DataSink creation - Include software.amazon.awssdk in shade plugin
…h EMR runtime The EMR-bundled iceberg-flink-runtime already includes GlueCatalog and AWS SDK. Shading iceberg-aws into our jar causes 'does not implement Catalog' error due to duplicate Catalog interface from different classloaders.
On EMR, iceberg-flink-runtime is already in Flink's lib/ dir (app classloader). Shading it into our connector jar causes the Catalog interface to be loaded from two different classloaders, resulting in ClassCastException: GlueCatalog (ChildFirstClassLoader) cannot be cast to Catalog (app loader) Solution: all iceberg deps are now provided, connector jar is 67KB (our code only).
…onments Keep the original compile scope and shade behavior for iceberg-flink-runtime so that hadoop/hive catalog users on non-EMR environments are not affected. Only iceberg-aws is added as provided scope.
Both iceberg-flink-runtime and iceberg-aws are provided scope. The iceberg runtime must be supplied by the deployment environment (e.g. EMR ships it in Flink lib/, non-EMR users add it manually). This avoids ClassLoader conflicts entirely.
norrishuang
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Adds AWS Glue Catalog support to the Flink CDC Iceberg pipeline sink by exposing Glue-related Iceberg catalog properties through catalog.properties.*.
Changes:
- Extend Iceberg sink options to document/configure Glue catalog (and related catalog/io impl properties).
- Register the new options in the Iceberg sink factory.
- Add unit tests for creating an Iceberg sink configuration targeting Glue.
- Adjust Iceberg connector module dependencies and shade plugin configuration.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-iceberg/src/main/java/org/apache/flink/cdc/connectors/iceberg/sink/IcebergDataSinkOptions.java | Adds Glue-related config options and updates option descriptions. |
| flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-iceberg/src/main/java/org/apache/flink/cdc/connectors/iceberg/sink/IcebergDataSinkFactory.java | Registers additional (Glue-related) options as optional. |
| flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-iceberg/src/test/java/org/apache/flink/cdc/connectors/iceberg/sink/IcebergDataSinkFactoryTest.java | Adds tests for Glue configuration paths (type=glue and catalog-impl). |
| flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-iceberg/pom.xml | Adds Iceberg AWS dependency and modifies scopes / shading configuration. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
...eberg/src/main/java/org/apache/flink/cdc/connectors/iceberg/sink/IcebergDataSinkOptions.java
Outdated
Show resolved
Hide resolved
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-iceberg/pom.xml
Show resolved
Hide resolved
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-iceberg/pom.xml
Outdated
Show resolved
Hide resolved
...g/src/test/java/org/apache/flink/cdc/connectors/iceberg/sink/IcebergDataSinkFactoryTest.java
Show resolved
Hide resolved
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Contributor
|
Hi @norrishuang, would you like to update the related doc for this, and it should reflect which additional jars need to be added. |
…ions to Iceberg connector docs
Author
The doc was updated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This closes FLINK-39245
Summary
Add AWS Glue Catalog support for the Iceberg pipeline sink connector. Previously, the Iceberg pipeline only supported
hadoopandhivecatalog types. This PR enables users to use AWS Glue Data Catalog as the Iceberg catalog by settingcatalog.properties.type: glue.Changes
New Configuration Options
catalog.properties.typegluein addition tohadoopandhivecatalog.properties.catalog-implorg.apache.iceberg.aws.glue.GlueCatalog)catalog.properties.io-implorg.apache.iceberg.aws.s3.S3FileIO)catalog.properties.glue.idcatalog.properties.glue.skip-archivecatalog.properties.glue.skip-name-validationcatalog.properties.client.regionFiles Modified
IcebergDataSinkOptions.java— Added Glue-related config options, updatedTYPEandWAREHOUSEdescriptionsIcebergDataSinkFactory.java— Registered new optional config optionsIcebergDataSinkFactoryTest.java— Added 2 test cases for Glue catalog creation (viatype=glueandcatalog-impl)pom.xml— Dependency and shade plugin adjustmentsUsage Example
How It Works
Iceberg's CatalogUtil.buildIcebergCatalog() natively supports type=glue and automatically loads org.apache.iceberg.aws.glue.GlueCatalog. This PR exposes the necessary configuration options through the Flink CDC pipeline config layer and ensures the Glue-related catalog properties are correctly passed through via the catalog.properties.* prefix.
Testing
Unit tests pass (6/6 in IcebergDataSinkFactoryTest)
Verified end-to-end on Amazon EMR (Flink 1.20, Iceberg 1.10.0-amzn) with MySQL CDC → Iceberg (Glue Catalog + S3)