Dataflow Templates 2024-07-01-00_RC00

Release Week of 2024-07-01

Note: This release is being rolled out. It may not be in your region yet.

Improvements

• Updated all templates to Beam 2.57.0
• Added DLQDirectory parameter to Sourcedb To Spanner template
• PubSub to Datadog template now includes full PubSub message by default
• Add support for using pre-existing private connectivity configuration for Datastream to Spanner terraform module
• Various IT's added

Contributors

@AdibH
@aksharauke
@AnandInguva
@dtru-ddog
@Dippatel98
@Polber
@justinuang
@manitgupta
@shreyakhajanchi
@Abacn
@darshan-sj

Dataflow Templates 2024-06-18-00_RC01

Release week of 2024-06-17

Dataflow Templates 2024-06-18-00_RC00

Release week of 2024-06-17

Dataflow Templates 2024-06-11-00_RC00

Release Week of 2024-06-11

New Templates

N/A

Improvements

• Add terraform module to Any SourceDB to Cloud Spanner template
• Update bigtable-beam-import to 2.14.2 to support Data Boost
• Update Template groupName details
• Update the skipOptions in Template annotation for kafka template
• Allow flags to be used as jinja template variables

Bug Fixes

• Add TLS authentication support for Kafka Templates

Contributors

• bharadwaj-aditya
• justinuang
• shreyakhajanchi
• akashorabek
• AnandInguva
• Dippatel98
• robertwb

Dataflow Templates 2024-06-04-00_RC01

Release week of 2024-06-03

Dataflow Templates 2024-06-04-00_RC00

Release week of 2024-06-03

Dataflow Templates 2024-05-28-00_RC00

Release Week of 2024-05-28

New Templates

N/A

Improvements

[datastream-to-spanner] Tests for additional CUJs (#1556)
[sourcedb-to-spanner] Add new feature (#1577, #1600, #1604)
[googlecloud-to-neo4j] Use built-in batch grouping transform (#1418)
[All Kafka] Add commit offsets (#1597)
[gcs-to-sourcedb] Add Integration test (#1562)
[kafka-to-kafka] Add authentication with SSL (#1555)
[spanner-change-streams-to-sharded-file-sink] Add Reverse replication reader IT (#1575)

Bug Fixes

[sourcedb-to-spanner] Fixed value mapping for blob and year (1578)

Contributors

Aditya Bharadwaj
Akarys Shorabek
Akshara Uke
Anand Inguva
Cloud Teleport
Deep1998
Dippatel98
djagaluru
Florent Biville
Jeffrey Kinard
jian jin
Manit Gupta
sharan-malyala
Sharan Teja M
Shreya Khajanchi
surabhiad
Svetak Sundhar
Tamanna Kakkar
Tanu Sharma
Vardhan Thigle
Vitaly Terentyev

Dataflow Templates 2024-05-21-00_RC00

New Templates

N/A

Improvements

• [BULK] Misc changes across types, logging and options
• [DataStreamToSpanner] - Convert sharded migration ITs to use Avro
• [Avro] - Add support for interval type
• [Plugins] - Add GcsReadBucket type
• [KafkaToBigQuery] - Add support for writing Kafka keys
• [Plugins] - Add support for KAFKA_TOPIC template parameters.
• Add Spanner PR workflow and exclude Spanner from global workflow
• [GcsToSourceDB] - Spanner Reverse replication writer IT
• [Plugins] - Template parameters: update old types to new ones
• [Documentation ] - Add more docs on adding ITs, flex templates, and deprecating parameters
• optimized the processing for lean Spanner write intervals

Bug Fixes

• [BULK MISC FIXES] - Add error counters; Fix null string handlings

Contributors

@ahmedabu98
@aksharauke
@akvelon
@an2x
@damccorm
@darshan-sj
@Deep1998
@Dippatel98
@manitgupta
@RajKumarGupta
@shreyakhajanchi

Dataflow Templates 2024-05-14-00_RC00

Release week of 2024-05-13

Dataflow Templates 2024-05-13-00_RC00

Release Week of 2024-05-13

New Templates

• Kafka to GCS template (#1448)
• Kafka To BigQuery Template (#1464)
• Python UDFs to MongoDB flex templates (#1497)
• Add Python UDF support to GCS to Splunk template (#1494)

Improvements

• [sourcedb-to-spanner] Unified Schema Mappings For Bulk Reader. (#1460)
• [jdbc] JDBC Table Schema Discovery (#1481)
• Upgrade beam version to 2.56.0 (#1504)
• [jdbc] Implementing JDBCIO.RowMapper to map ResultSet to SourceRow. (#1508)
• [bigtable-to-vectorembedding] Add custom App Profile support for Bigtable to Vector Embedding export (#1505)
• [spanner-migration] Add support for the unified custom logical types: JSON, NUMBER and VARCHAR for Spanner Migration Templates (#1509)
• Xlang Dockerfile speed improvements (#1499)
• [spanner-migration] Support UNION field shema types and improve debugging (#1524)
• [UI] Add parentName and parentTriggerValues attributes for templates (#1456)

Bug Fixes

• Parse secret name and password only when using Auth (#1512)

Contributors

Anand Inguva

Archana Gupta

Danny McCormick

Deep1998

Dippatel98

Jeffrey Kinard

Manit Gupta

Ron Gal

Sharan Teja M

Shreya Khajanchi

Supriya Koppa

Vardhan Thigle

Vardhan Vinay Thigle

Yi Hu

anikin

archanagupta03

fozzie15

gauravjainn

hugo-syn

sharan-malyala

surabhiad

vitaly.terentyev