Skip to content
This repository was archived by the owner on Nov 11, 2022. It is now read-only.

Commit cd845a9

Browse files
reuvenlaxdavorbonaci
authored andcommitted
Reduce the number of calls to BigQuery table creation to reduce quota issues. Previously every thread tried to create the table at startup. Synchronizing this call prevents this.
[] ------------- Created by MOE: http://code.google.com/p/moe-java MOE_MIGRATED_REVID=87323976
1 parent 0a97da5 commit cd845a9

File tree

1 file changed

+12
-5
lines changed

1 file changed

+12
-5
lines changed

sdk/src/main/java/com/google/cloud/dataflow/sdk/io/BigQueryIO.java

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -682,11 +682,18 @@ public void startBundle(Context context) {
682682
JSON_FACTORY.fromString(jsonTableReference, TableReference.class);
683683

684684
if (!createdTables.contains(jsonTableSchema)) {
685-
TableSchema tableSchema = JSON_FACTORY.fromString(jsonTableSchema, TableSchema.class);
686-
Bigquery client = Transport.newBigQueryClient(options).build();
687-
BigQueryTableInserter inserter = new BigQueryTableInserter(client, tableReference);
688-
inserter.tryCreateTable(tableSchema);
689-
createdTables.add(jsonTableSchema);
685+
synchronized (createdTables) {
686+
// Another thread may have succeeded in creating the table in the meanwhile, so
687+
// check again. This check isn't needed for correctness, but we add it to prevent
688+
// every thread from attempting a create and overwhelming our BigQuery quota.
689+
if (!createdTables.contains(jsonTableSchema)) {
690+
TableSchema tableSchema = JSON_FACTORY.fromString(jsonTableSchema, TableSchema.class);
691+
Bigquery client = Transport.newBigQueryClient(options).build();
692+
BigQueryTableInserter inserter = new BigQueryTableInserter(client, tableReference);
693+
inserter.tryCreateTable(tableSchema);
694+
createdTables.add(jsonTableSchema);
695+
}
696+
}
690697
}
691698
} catch (IOException e) {
692699
throw new RuntimeException(e);

0 commit comments

Comments
 (0)