Conversation
New enterprise-only output that streams data into BigQuery using the Storage Write API. Supports JSON and Protobuf message formats, default stream type, connection multiplexing via managed stream cache, and IAM/credential-based authentication.
Add basic operational metrics (rows sent/failed, batches, latency, retries), classify gRPC errors as transient or permanent for smarter retry behavior, and support service account impersonation via target_principal and delegates config fields.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| const ( | ||
| // streamIdleTimeout is how long a cached stream can remain unused before | ||
| // being eligible for eviction by the idle sweep. | ||
| streamIdleTimeout = 5 * time.Minute | ||
|
|
||
| // streamSweepInterval is how often the background goroutine checks for | ||
| // idle streams. | ||
| streamSweepInterval = 1 * time.Minute | ||
| ) |
There was a problem hiding this comment.
Hardcoded time durations must be YAML-configurable. Per project Go patterns: "Every time-related value (timeouts, backoffs, intervals, retry delays) must be exposed as a YAML-configurable field. Do not hardcode durations."
streamIdleTimeout and streamSweepInterval should be added as config fields (e.g., stream_idle_timeout and stream_sweep_interval) with these values as defaults, following the same pattern as other duration fields in the codebase.
Add basic operational metrics (rows sent/failed, batches, latency, retries). Classify gRPC errors as transient or permanent for smarter retry behavior. Support service account impersonation via target_principal and delegates config fields. Make stream idle timeout and sweep interval YAML-configurable. Regenerate docs.
|
Commits
Review LGTM |
|
Top 2 commits can be squashed I think |
|
Would be nice to redo test comments into given/when/then logs |
|
Can we have a constructor for tests instead of |
| if err = license.CheckRunningEnterprise(mgr); err != nil { | ||
| return | ||
| } |
There was a problem hiding this comment.
Check license in constructor not here.
| if maxInFlight, err = conf.FieldMaxInFlight(); err != nil { | ||
| return | ||
| } | ||
| if batchPolicy, err = conf.FieldBatchPolicy(bqwaFieldBatching); err != nil { | ||
| return | ||
| } |
There was a problem hiding this comment.
Try preserving order where it makes sense
|
|
||
| When batching is enabled the table name is resolved from the first message in | ||
| each batch; all messages in the same batch are written to that table. | ||
| `). |
There was a problem hiding this comment.
nit: it's the best for maintenance to use sentence per line
|
It's handy to support |
Move license check from init() to constructor. Reorder config fields logically (core fields first, advanced last). Use sentence-per-line in config descriptions for easier diffs. Add newTestOutput helper for tests. Rewrite test comments as given/when/then. Inject test license via license.InjectTestService. Regenerate docs.
|
|
||
| // When we write a batch. | ||
| batch := service.MessageBatch{service.NewMessage([]byte(`{"foo":"bar"}`))} | ||
| err := out.WriteBatch(context.Background(), batch) |
There was a problem hiding this comment.
Test pattern violation: use t.Context() instead of context.Background() in test functions.
Per the project test patterns:
Use
t.Context()for test contexts. Exception: int.Cleanup()functions, usecontext.Background()becauset.Context()is already canceled during cleanup.
These are non-cleanup test functions, so t.Context() should be used. This also applies to TestWriteBatchEmptyBatch (line 187) and TestCloseNilClients (line 203).
|
Commits
Review
|
2 participants
No description provided.