feat(bulk): attach auto-create table metadata to Arrow schema - #59
Open
v0y4g3r wants to merge 3 commits into
Open
feat(bulk): attach auto-create table metadata to Arrow schema#59v0y4g3r wants to merge 3 commits into
v0y4g3r wants to merge 3 commits into
Conversation
- Attach `greptime:semantic_type` (`timestamp`/`tag`/`field`) metadata derived from each column's semantic type to every Arrow field in the bulk insert schema, enabling the server to auto-create tables for Flight bulk inserts - Add `greptime:type: Json` metadata for JSON columns, which are encoded as Arrow `Binary` and would otherwise be created as `Binary` columns - Unify Arrow field construction between `BulkStreamWriter` and `RowBatchBuilder` via `column_to_arrow_field` so the schema sent on the wire matches the writer's schema Modified: `src/bulk.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
- Never emit `greptime:type` on the timestamp column, which the auto-create contract forbids - Reject schemas with more than one timestamp column via new `MultipleTimestampColumns` error, validated in `BulkStreamWriter::new` and `find_timestamp_index_and_window` - Strengthen metadata tests with key-cardinality assertions and Decimal128 coverage Modified: `src/bulk.rs`, `src/error.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
There was a problem hiding this comment.
Pull request overview
This PR extends the bulk insert Arrow schema to include GreptimeDB Enterprise auto-create-table contract metadata on every Arrow Field, ensuring the server can infer each column’s semantic role (timestamp/tag/field) and extended type information (e.g., JSON) during the first Flight bulk insert.
Changes:
- Introduces
greptime:semantic_typefield metadata (timestamp/tag/field) on all bulk-insert Arrow fields via a sharedcolumn_to_arrow_fieldhelper. - Adds
greptime:type=Jsonmetadata for JSON columns (encoded as ArrowBinary) while ensuring it is never set on timestamp columns. - Adds early rejection for schemas containing multiple timestamp columns via a new
MultipleTimestampColumnserror, plus unit tests covering the new contract behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/error.rs | Adds a new MultipleTimestampColumns error variant used to reject invalid schemas. |
| src/bulk.rs | Attaches GreptimeDB contract metadata to Arrow fields, adds multi-timestamp validation, and adds unit tests for the metadata contract and schema validation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+201
to
+210
| // The server auto-create contract allows exactly one timestamp column | ||
| ensure!( | ||
| table_schema | ||
| .columns() | ||
| .iter() | ||
| .filter(|col| col.semantic_type == SemanticType::Timestamp) | ||
| .count() | ||
| <= 1, | ||
| error::MultipleTimestampColumnsSnafu | ||
| ); |
- Add explicit bulk writer constructors for controlling automatic table creation while preserving existing `BulkWriteOptions` compatibility and defaulting legacy constructors to disabled - Forward `auto_create_table=true|false` through the shared `x-greptime-hints` Flight request metadata in `src/database.rs` - Cover exact hint values and request metadata alongside database and authentication headers Modified: `src/bulk.rs`, `src/database.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Aligns the Rust bulk writer with the latest GreptimeDB Enterprise Flight bulk auto-create contract from GreptimeTeam/greptimedb-enterprise#845 (HEAD
9b277b5b).The client now provides both parts required by the server:
API
Existing constructors remain source-compatible and default automatic table creation to disabled:
Use the new constructor to enable it for a writer:
The client sends
x-greptime-hints: auto_create_table=true|false. A true hint only takes effect when automatic table creation is also enabled globally on the server.Changes
greptime:semantic_type(timestamp/tag/field) to every Arrow fieldgreptime:type: Jsonfor JSON fields encoded as ArrowBinary, never on the timestamp fieldBulkInserterandBulkStreamWriter; existing constructors send falseauto_create_tablethrough the sharedx-greptime-hintsmetadata while preserving database and authentication headersTesting
cargo check --all-targetscargo clippy --all-targets -- -D warningscargo fmt --checkcargo test -- --skip integration(47 tests passed)