Skip to content

feat(bulk): attach auto-create table metadata to Arrow schema - #59

Open
v0y4g3r wants to merge 3 commits into
mainfrom
feat/bulk-auto-create-table-metadata
Open

feat(bulk): attach auto-create table metadata to Arrow schema#59
v0y4g3r wants to merge 3 commits into
mainfrom
feat/bulk-auto-create-table-metadata

Conversation

@v0y4g3r

@v0y4g3r v0y4g3r commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Aligns the Rust bulk writer with the latest GreptimeDB Enterprise Flight bulk auto-create contract from GreptimeTeam/greptimedb-enterprise#845 (HEAD 9b277b5b).

The client now provides both parts required by the server:

  • Strict Arrow field metadata describing the schema to create
  • The Flight request hint that explicitly enables or disables automatic creation

API

Existing constructors remain source-compatible and default automatic table creation to disabled:

let writer = inserter
    .create_bulk_stream_writer(&schema, options)
    .await?;

Use the new constructor to enable it for a writer:

let writer = inserter
    .create_bulk_stream_writer_with_auto_create_table(&schema, options, true)
    .await?;

The client sends x-greptime-hints: auto_create_table=true|false. A true hint only takes effect when automatic table creation is also enabled globally on the server.

Changes

  • Attach greptime:semantic_type (timestamp/tag/field) to every Arrow field
  • Emit greptime:type: Json for JSON fields encoded as Arrow Binary, never on the timestamp field
  • Reject schemas containing multiple timestamp columns before opening the stream
  • Add source-compatible explicit auto-create constructors to BulkInserter and BulkStreamWriter; existing constructors send false
  • Forward auto_create_table through the shared x-greptime-hints metadata while preserving database and authentication headers

Testing

  • Verify semantic metadata values, allowed key cardinality, JSON/Binary/Decimal128 behavior, and multiple-timestamp rejection
  • Verify exact true/false hint pairs
  • Verify the constructed Flight request contains hint, database, and Basic authentication metadata
  • cargo check --all-targets
  • cargo clippy --all-targets -- -D warnings
  • cargo fmt --check
  • cargo test -- --skip integration (47 tests passed)

v0y4g3r added 2 commits August 2, 2026 21:55
- Attach `greptime:semantic_type` (`timestamp`/`tag`/`field`) metadata derived from each column's semantic type to every Arrow field in the bulk insert schema, enabling the server to auto-create tables for Flight bulk inserts
- Add `greptime:type: Json` metadata for JSON columns, which are encoded as Arrow `Binary` and would otherwise be created as `Binary` columns
- Unify Arrow field construction between `BulkStreamWriter` and `RowBatchBuilder` via `column_to_arrow_field` so the schema sent on the wire matches the writer's schema

Modified: `src/bulk.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
- Never emit `greptime:type` on the timestamp column, which the auto-create contract forbids
- Reject schemas with more than one timestamp column via new `MultipleTimestampColumns` error, validated in `BulkStreamWriter::new` and `find_timestamp_index_and_window`
- Strengthen metadata tests with key-cardinality assertions and Decimal128 coverage

Modified: `src/bulk.rs`, `src/error.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Copilot AI review requested due to automatic review settings August 2, 2026 14:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the bulk insert Arrow schema to include GreptimeDB Enterprise auto-create-table contract metadata on every Arrow Field, ensuring the server can infer each column’s semantic role (timestamp/tag/field) and extended type information (e.g., JSON) during the first Flight bulk insert.

Changes:

  • Introduces greptime:semantic_type field metadata (timestamp/tag/field) on all bulk-insert Arrow fields via a shared column_to_arrow_field helper.
  • Adds greptime:type=Json metadata for JSON columns (encoded as Arrow Binary) while ensuring it is never set on timestamp columns.
  • Adds early rejection for schemas containing multiple timestamp columns via a new MultipleTimestampColumns error, plus unit tests covering the new contract behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
src/error.rs Adds a new MultipleTimestampColumns error variant used to reject invalid schemas.
src/bulk.rs Attaches GreptimeDB contract metadata to Arrow fields, adds multi-timestamp validation, and adds unit tests for the metadata contract and schema validation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/bulk.rs
Comment on lines +201 to +210
// The server auto-create contract allows exactly one timestamp column
ensure!(
table_schema
.columns()
.iter()
.filter(|col| col.semantic_type == SemanticType::Timestamp)
.count()
<= 1,
error::MultipleTimestampColumnsSnafu
);
- Add explicit bulk writer constructors for controlling automatic table creation while preserving existing `BulkWriteOptions` compatibility and defaulting legacy constructors to disabled
- Forward `auto_create_table=true|false` through the shared `x-greptime-hints` Flight request metadata in `src/database.rs`
- Cover exact hint values and request metadata alongside database and authentication headers

Modified: `src/bulk.rs`, `src/database.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants