Background
This together with the pramen_batchid, pramen_info_date, pramen_snapshotid column would allow pointing from data entries to snapshot id of the Iceberg table when the record was added or last modified.
The purpose:
pramen_batchid - uniquely defines a Pramen job. But it can have multiple writes to the same Iceberg table.pramen_info_date - the information date. Together with pramen_batchid determines a single write to an Iceberg table.pramen_snapshotid - uniquely defines a write to an Iceberg table. Corresponds 1:1 to Iceberg snapshot_id.
Feature
Add pramen_batchid, pramen_info_date, pramen_snapshotid property to Iceberg tables when writing to it.
Example [Optional]
--
Proposed Solution [Optional]
--
Background
This together with the
pramen_batchid,pramen_info_date,pramen_snapshotidcolumn would allow pointing from data entries to snapshot id of the Iceberg table when the record was added or last modified.The purpose:
pramen_batchid- uniquely defines a Pramen job. But it can have multiple writes to the same Iceberg table.pramen_info_date- the information date. Together withpramen_batchiddetermines a single write to an Iceberg table.pramen_snapshotid- uniquely defines a write to an Iceberg table. Corresponds 1:1 to Icebergsnapshot_id.Feature
Add
pramen_batchid,pramen_info_date,pramen_snapshotidproperty to Iceberg tables when writing to it.Example [Optional]
--
Proposed Solution [Optional]
--