Skip to content

[FLINK-29549]- Flink Glue Catalog integration #206

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

FranMorilloAWS
Copy link

Purpose of the change

For example: Implements the Table API for the Kinesis Source.

Verifying this change

Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment
  • Added unit tests
  • Manually verified by running the Kinesis connector on a local Flink cluster.

Significant changes

(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)

  • Dependencies have been added or upgraded
  • Public API has been changed (Public API is any class annotated with @Public(Evolving))
  • Serializers have been changed
  • New feature has been introduced
    • If yes, how is this documented? (not applicable / docs / JavaDocs / not documented)

@FranMorilloAWS FranMorilloAWS marked this pull request as ready for review May 7, 2025 10:18
@FranMorilloAWS FranMorilloAWS changed the title [FLINK-29549]- Updated Flink Glue Catalog integration [FLINK-29549] Updated Flink Glue Catalog integration May 7, 2025
@FranMorilloAWS FranMorilloAWS changed the title [FLINK-29549] Updated Flink Glue Catalog integration [FLINK-29549]- Flink Glue Catalog integration May 7, 2025
Copy link
Contributor

@Samrat002 Samrat002 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution. A few points to note:

It looks like most of the code has already been reviewed across multiple PRs.

Concerns:

  1. The current implementation exposes low-level Glue details (e.g., support for lowercase table names) directly through the Flink Catalog. Is this acceptable? I recommend raising this topic with the community. In my opinion, it would be better to encapsulate such Glue-specific behavior and avoid exposing it directly via the Flink Catalog interface. The catalog's behavior should remain consistent with other catalogs, with differences controlled via configuration only.

  2. Please consider implementing the basic configuration options defined in FLIP-277. If that’s not feasible in this PR, a fast follow-up would be valuable, especially for users relying on different credential modes.

  3. It seems that some .idea folder files have been committed. Please remove them from the PR.

Cheers,
Samrat

@@ -0,0 +1,2 @@
./target
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this gitignore file required

@@ -0,0 +1,10 @@
# Default ignored files
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this file required ?

@@ -0,0 +1,18 @@
<?xml version="1.0" encoding="UTF-8"?>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this file required ?

@@ -0,0 +1,15 @@
<?xml version="1.0" encoding="UTF-8"?>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do you need this file ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants