Skip to content

[FLINK-38840][postgres] Postgres YAML connector supports emitting complete Table ID#4209

Merged
yuxiqian merged 4 commits intoapache:masterfrom
Mrart:FLINK-38840
Jan 7, 2026
Merged

[FLINK-38840][postgres] Postgres YAML connector supports emitting complete Table ID#4209
yuxiqian merged 4 commits intoapache:masterfrom
Mrart:FLINK-38840

Conversation

@Mrart
Copy link
Contributor

@Mrart Mrart commented Jan 4, 2026

Currently, Postgres YAML connector maps (schema, table) as Pipeline event Table ID, and the database part is missing.

We may add a new option to generate complete Table ID (database, schema, table). This could be useful when records are written into Kafka as JSON messages.

@Mrart Mrart marked this pull request as draft January 4, 2026 00:50
@github-actions github-actions bot added postgres-cdc-connector postgres-pipeline-connector docs Improvements or additions to documentation labels Jan 4, 2026
@Mrart Mrart marked this pull request as ready for review January 4, 2026 09:47
@Mrart
Copy link
Contributor Author

Mrart commented Jan 4, 2026

@yuxiqian PTAL

…plete Table ID

[FLINK-38840][postgres] Postgres YAML connector supports emitting complete Table ID

[FLINK-38840][postgres] Postgres YAML connector supports emitting complete Table ID

[FLINK-38840][postgres] Postgres YAML connector supports emitting complete Table ID
@loserwang1024
Copy link
Contributor

@Mrart
It seems that even we don't have table-id.include-database option but directly emitting complete Table ID. The compability of currently pipeline won't be broke. And kafka sink can add a new option to determine whether to print database. @yuxiqian , WDYT?

@yuxiqian
Copy link
Member

yuxiqian commented Jan 5, 2026

@Mrart It seems that even we don't have table-id.include-database option but directly emitting complete Table ID. The compability of currently pipeline won't be broke. And kafka sink can add a new option to determine whether to print database. @yuxiqian , WDYT?

I'm a little concerned about changing default Table ID format. It may affect transform & route table matching rules and schema registry states where Table IDs are used as key.

@Mrart
Copy link
Contributor Author

Mrart commented Jan 5, 2026

@Mrart It seems that even we don't have table-id.include-database option but directly emitting complete Table ID. The compability of currently pipeline won't be broke. And kafka sink can add a new option to determine whether to print database. @yuxiqian , WDYT?

We still need to control the output of the full tableID through configuration to ensure that it is compatible with production

Copy link
Member

@yuxiqian yuxiqian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes LGTM. Would @loserwang1024 like to take another look?

@Mrart
Copy link
Contributor Author

Mrart commented Jan 6, 2026

@loserwang1024 PTLK

@loserwang1024
Copy link
Contributor

LGTM

@yuxiqian yuxiqian merged commit 99c00f3 into apache:master Jan 7, 2026
62 of 63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants