Dataflow Templates 2025-03-11-00_RC00

Release week of 2025-03-10

Dataflow Templates 2025-03-04-00_RC00

Note: This release is being rolled out. It may not be in your region yet.

What's Changed

Improvements

• Support partitioned reads for DateTime column type in JDBC to BigQuery template by @sharan-malyala in #2084
• Add warning about caching with plugin by @derrickaw in #2221
• [DatastreamToSpanner] Spanner Exception handling by @darshan-sj in #2185

Bug fixes

• bug-fix: Use jdbc connection properties for reverse migration by @asthamohta in #2198

New Contributors

• @Rudra-Gujarathi made their first contribution in #2219
• @derrickaw made their first contribution in #2221

Full Changelog: 2025-02-26-00_RC00...2025-03-04-00_RC00

Dataflow Templates 2025-02-26-00_RC00

What's Changed

Improvements

• Change default shadow table prefix of reverse replication template by @manitgupta in #2186
• Remove sdk_container_image example from terraform.tfvars by @manitgupta in #2187
• Upgrade Beam version to 2.63.0 by @jrmccluskey in #2206
• Add load test for cross db txn by @Deep1998 in #2199
• Report Lineage for CsvToBigQuery template by @Abacn in #2205
• Spanner Import/Export INTERLEAVE IN by @jjfox15 in #2128
• Added SCRAM-SHA-512 support to dataflow kafka-to-bigquery by @vgnanasekaran in #2181

Bug fixes

• Fix inf issues in Datastream reader by @dhercher in #2213

New Contributors

• @jjfox15 made their first contribution in #2128
• @vgnanasekaran made their first contribution in #2181

Full Changelog: 2025-02-18-00_RC00...2025-02-26-00_RC00

Dataflow Templates 2025-02-18-00_RC00

What's Changed

Improvements

• Switch to use beam-sdks-java-io-aws2 for KinesisToPubsub by @Abacn in #2182
• Add support for byte arrays and uuid arrays in datastream to sql template. by @claudevdm in #2174
• IT for cross database transaction by @Deep1998 in #2160

Bug fixes

• Support inf and -inf values by @dhercher in #2203
• Fix issues for sequences and identity columns by @hengfengli in #2171
• fix: clean shadow tables by @asthamohta in #2207

Full Changelog: 2025-02-12-00_RC00...2025-02-18-00_RC00

Dataflow Templates 2025-02-12-00_RC00

What's Changed

Improvements

• Added VIRTUAL column support to PG dataflow by @atask-g in #2167
• Add UUID support in import/export templates by @lagarwal1 in #2145

Bug fixes

• Fix handling of empty arrays, json arrays and jsonb arrays. by @claudevdm in #2169
• Remove warning message by @dhercher in #2173

New Contributors

• @lagarwal1 made their first contribution in #2145

Full Changelog: 2025-02-04-00_RC00...2025-02-12-00_RC00

Dataflow Templates 2025-02-04-00_RC00

What's Changed

Improvements

• Search indexes by @n-d-joshi in #2152

Full Changelog: 2025-01-29-00_RC01...2025-02-04-00_RC00

Dataflow Templates 2025-01-29-00_RC01

What's Changed

Improvements

• Switch to Java 17 for all Dataflow Templates by @liferoad in #2050
• Support shadow table in different database by @Deep1998 in #2138
• feat: bump to Neo4j driver v5 by @fbiville in #2136
• Bump Beam to 2.62.0 by @Abacn in #2143
• Added support for Tokenlist columns by @n-d-joshi in #2144
• Support avro arrays for postgres insertion. by @claudevdm in #2154

Bug fixes

• Small Fix for Cassandra bulk e2e test by @VardhanThigle in #2137
• Fix the additional sequence kind options by @hengfengli in #2146
• fix: revamp Neo4j kernel version sniffing by @fbiville in #2148
• fix: account for on-premise Neo4j CalVer version format by @fbiville in #2149

New Contributors

• @claudevdm made their first contribution in #2154

Full Changelog: 2025-01-10-00_RC00...2025-01-29-00_RC01

Dataflow Templates 2025-01-10-00_RC00

What's Changed

Improvements

• feat(spanner): add support for identity columns in import/export by @hengfengli in #1812
• Creating new annotation and new workflow for spanner staging tests by @darshan-sj in #2053
• Formatting changes to templates by @sharan-malyala in #1667
• Cassandra datatype handler by @taherkl in #2054
• tests: Adding Forward Migration Tests by @asthamohta in #2001
• [Sourcedb-to-spanner] Bulk migration Mysql to spanner 1tb Load test by @darshan-sj in #2063
• Fix Spanner Load tests and add display test report by @darshan-sj in #2092
• Favor built-in transform over custom one by @fbiville in #2013
• Simplify dependency manager: remove unused code by @fbiville in #2011
• feat: drop node target's key constraint requirement by @fbiville in #2068

Bug fixes

• Fix disabled algorithm default by @fbiville in #2012
• Correcting Partitioning Logic for VarBinary Primary Key Type. by @VardhanThigle in #2064
• Fix precision error for datetime by @Deep1998 in #2106
• Fix renamed template mongodb-to-bigquery-cdc by @Abacn in #2116
• fix: eager source row fetching logic by @fbiville in #2071
• Fixing TypeMismatch in Cassandra Driver Config Loader by @VardhanThigle in #2118

New Contributors

• @hengfengli made their first contribution in #1812
• @taherkl made their first contribution in #2054
• @akashthawaitcc made their first contribution in #2081

Full Changelog: 2024-12-10-00_RC00...2025-01-10-00_RC00

Dataflow Templates 2024-12-10-00_RC00

What's Changed

Improvements

• Add extra validation to SpannerToBigQuery Template by @jrmccluskey in #2035
• custom transformation implementation by @shreyakhajanchi in #2040
• feat: introduce new configuration properties for bigquery source by @Emrehzl94 in #2049

Bug fixes

• Adding dataflow timestamp field in datastream-common json flow by @shreyakhajanchi in #2042

New Contributors

• @emmanuel-ferdman made their first contribution in #2027

Full Changelog: 2024-12-03-00_RC00...2024-12-10-00_RC00

Dataflow Templates 2024-12-03-00_RC00

What's Changed

Improvements

• Allow SourceSchemaReference and DataSource to represent Cassandra (and other similar non-jdbc) values by @VardhanThigle in #1998
• fix: add yaml support for specification parser by @Emrehzl94 in #1999
• Add enable backfill flag to terraform template var files by @Deep1998 in #2018
• Avro format Import/Export support for Spanner NOT ENFORCED Foreign Key by @jennyjiang98 in #2031
• Upgrade to Beam 2.61.0 by @damccorm in #2033
• add KMS key support to BigQueryConverters by @Polber in #2024
• feat: Modularization: Terraform changes for schema overrides by @manitgupta in #1841
• Add support for PropertyGraphs in import/export DDL by @smukil in #2003
• Add DLQ support to JdbcToBigQuery using BQ Storage Write API by @Polber in #2016
• Parameterizing spanner Host in spanner tests by @darshan-sj in #2020

Bug fixes

• Removing unsued Cassandra dependency by @VardhanThigle in #2009
• Update requirements annotation for JdbcToBigQuery templates by @Abacn in #2007
• Use safe deserialization option for Yaml specs by @fbiville in #2030
• Fix node target reference lookup and node match mode by @fbiville in #1961
• Allowing Jdbc FetchSize Configuration for handling large rows by @VardhanThigle in #2028
• Fixing Custom Transforms for BigInt Unsigned and Boolean types. by @VardhanThigle in #2039
• Setting useCursorFetch for MySql if FetchSize is configured. by @VardhanThigle in #2041
• [KafkaToBigQueryFlex Template]: Add support for Avro ENUM type and fix FLOAT type. by @an2x in #2029

New Contributors

• @jennyjiang98 made their first contribution in #2031
• @smukil made their first contribution in #2003

Full Changelog: 2024-11-12-00_RC00...2024-12-03-00_RC00