[AMORO-4127] Add mixed Flink 1.19 support#4141
Conversation
- Add v1.19/amoro-mixed-flink-1.19 and v1.19/amoro-mixed-flink-runtime-1.19 modules - Register new modules in parent pom.xml - Update documentation to include Flink 1.19 in supported version matrix - Add kind cluster config for local E2E testing Signed-off-by: Jiwon Park <jpark92@outlook.kr>
github-actions
bot
added
type:docs
| <dependencies> | ||
| <dependency> | ||
| <groupId>org.apache.amoro</groupId> | ||
| <artifactId>amoro-mixed-flink-common</artifactId> |
There was a problem hiding this comment.
Shared module is still hard-pinned to Flink/Iceberg/Paimon 1.18
There was a problem hiding this comment.
Good catch. I initially kept amoro-mixed-flink-common pinned to 1.18 to avoid potential runtime breakage on the v1.18 module, but after reviewing the Flink 1.19 release notes there are no breaking API changes — only deprecations scheduled for Flink 2.0. Will update.
There was a problem hiding this comment.
After attempting the update, I found an actual breaking change: Output.emitRecordAttributes(RecordAttributes) was added as an abstract method in Flink 1.19, and RecordAttributes itself doesn't exist in Flink 1.18. So bumping common to 1.19 breaks the v1.18 module at runtime.
To support both 1.18 and 1.19 simultaneously, the shared common module approach hits a limit here. Do you have a preferred way to handle this — e.g. keeping common pinned to 1.18 and overriding the affected class in the v1.19 module, or a different architecture?
There was a problem hiding this comment.
It is an empty output implementation. I think it may can just implement an empty method: emitRecordAttributes. HDYT @czy006
There was a problem hiding this comment.
Done — bumped amoro-mixed-flink-common to Flink/Iceberg/Paimon 1.19 and added an empty emitRecordAttributes to EMPTY_OUTPUT in MixedFormatWriter.
Flink 1.17/1.18 runtime compat is preserved: their Output interface doesn't have emitRecordAttributes, so it's never dispatched and RecordAttributes is never loaded. The project already used this "compile against newer, run on older" pattern (common was previously compiled against Flink 1.18 while also running on 1.17).
|
|
||
| ## Environment preparation | ||
| Download Flink and related dependencies, and download Flink 1.17/1.18 as needed. Taking Flink 1.17 as an example: | ||
| Download Flink and related dependencies, and download Flink 1.17/1.18/1.19 as needed. Taking Flink 1.18 as an example: |
There was a problem hiding this comment.
Update releated flink versions to flink 1.18?
There was a problem hiding this comment.
Some places still tells users to build the 1.17 runtime
There was a problem hiding this comment.
Good catch — unintentionally changed the example version from 1.17 to 1.18, which is out of scope for this PR. Reverted to 1.17.
There was a problem hiding this comment.
The remaining 1.17 references (build command, path, example script) are intentionally left as-is — updating them is part of the Flink 1.17 drop PR which will be submitted separately. Will be cleaned up there.
|
cc @czy006 |
Signed-off-by: Jiwon Park <jpark92@outlook.kr>
xxubai
changed the title
- Update flink.version from 1.18.1 to 1.19.1 - Replace iceberg-flink-1.18 with iceberg-flink-1.19 - Replace paimon-flink-1.18 with paimon-flink-1.19 - Add empty emitRecordAttributes() to EMPTY_OUTPUT in MixedFormatWriter to satisfy the new abstract method added in Flink 1.19 Output interface Flink 1.17/1.18 runtime compatibility is preserved via JVM lazy class loading: emitRecordAttributes is never dispatched in pre-1.19 runtimes since the method does not exist in their Output interface, so RecordAttributes is never loaded. Signed-off-by: Jiwon Park <jpark92@outlook.kr>
Flink 1.19 added a check that rejects dropping the currently active database (DropDatabaseOperation validates current catalog/database). Switch to default_catalog before dropping the test database to avoid the "Cannot drop a database which is currently in use" error. Signed-off-by: Jiwon Park <jpark92@outlook.kr>
Move dev/e2e/kind-config.yaml to a separate PR as requested during review. Signed-off-by: Jiwon Park <jpark92@outlook.kr>
|
I’ll merge this if there are no more comments in the next few days. |
3 participants
Why are the changes needed?
Close #4127.
Amoro's mixed-format Flink integration currently supports Flink 1.17 and 1.18. Iceberg's multi-engine support matrix includes Flink 1.19, and aligning Amoro's supported versions with Iceberg reduces friction for future Iceberg upgrades.
Brief change log
v1.19/amoro-mixed-flink-1.19module withpom.xmldepending oniceberg-flink-1.19andpaimon-flink-1.19v1.19/amoro-mixed-flink-runtime-1.19moduleamoro-format-mixed/amoro-mixed-flink/pom.xmlREADME.mdanddocs/_index.mdto include Flink 1.19.x in the supported version matrixdocs/engines/flink/flink-get-started.mdandusing-logstore.mdto cover Flink 1.19dev/e2e/kind-config.yamlfor local E2E cluster setupNote on
amoro-mixed-flink-common: The common module compiles against Flink 1.18 APIs (providedscope). All APIs used in common are confirmed compatible with Flink 1.19 — deprecated items (RichParallelSourceFunction,open(Configuration),getExecutionConfig()) are scheduled for removal in Flink 2.0 and continue to work in 1.19 with fallback support. No breaking API changes were identified between Flink 1.18 and 1.19 for the APIs used in this module.Note on CI: The existing CI workflows build all Flink version modules in a single pass (no per-version matrix); registering the new modules in
pom.xmlis sufficient for CI coverage.How was this patch tested?
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request
Documentation