perf: Optimize wide table write performance #16699

luoluoyuyu · 2025-11-04T08:21:27Z

Description

This PR primarily addresses two performance bottlenecks in wide-table scenarios:

Metadata Analysis Phase - StatementAnalyze takes too long (even longer than writing to MemTable time)
TSFile Table Registration - Unnecessary schema conversion overhead

I. Metadata Phase Optimization

Problem Background

Through flame graph analysis, it was found that in wide-table scenarios, the CPU time consumption of the StatementAnalyze stage is too high, becoming the main bottleneck for write performance.

Optimization Measures

1. Reduce Redundant `TsTable` → `TableSchema` Conversion

Problem Positioning: Frequent schema conversion operations occupy a large amount of CPU

2. Reduce Unnecessary `semanticCheck` Execution

Problem Positioning: In wide-table scenarios, complete semantic checks are executed on every write, but most check items are repeated

Optimization Solution: Introduce a check item caching mechanism to skip repeated checks

3. Optimize `TsTable.getColumnSchema` Read Lock Time - Introduce Optimistic Locking

Problem Positioning: Severe read lock contention under high concurrency

Optimization Solution:

Read operations prefer optimistic read lock, downgrade to read lock on failure

II. TSFile Table Registration Optimization

Problem Background

The TSFile registration stage has unnecessary TsTableSchema conversion, causing additional overhead.

Optimization Measures

1. Eliminate Redundant `TsTableSchema` Conversion

Implementation Points:

Remove intermediate TsTableSchema conversion in TsFileRegister

This PR has:

Key changed/added classes (or packages if there are too many classes) in this PR

jt2594838 · 2025-11-11T08:53:09Z

.../node-commons/src/main/java/org/apache/iotdb/commons/schema/table/TsFileTableSchemaUtil.java

+    // Skip column name
+    ReadWriteIOUtils.readString(buffer);
+    // Skip data type
+    ReadWriteIOUtils.readDataType(buffer);
+    // Skip encoding and compression for FIELD columns
+    if (category == TsTableColumnCategory.FIELD) {
+      ReadWriteIOUtils.readEncoding(buffer);
+      ReadWriteIOUtils.readCompressionType(buffer);
+    }
+    // Skip column props
+    ReadWriteIOUtils.readMap(buffer);


May add skipString and skipMap, which only change the buffer position instead of creating temporary objects.

Copilot

Pull Request Overview

This PR optimizes wide-table write performance by addressing two main bottlenecks: metadata analysis phase and TSFile table registration. Key optimizations include:

Optimistic locking in TsTable - Introduces lock-free fast paths for read operations using version tracking and write flags
Semantic check caching - Adds flags to skip redundant validation of InsertNode measurements
Direct schema conversion - Eliminates intermediate schema conversions during TSFile registration
Lower-case transformation optimization - Adds caching to prevent redundant toLowerCase operations
Test utilities - Introduces TSDataTypeTestUtils for consistent handling of supported data types in tests

Reviewed Changes

Copilot reviewed 35 out of 35 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
pom.xml	Updates tsfile dependency version to 2.2.0-251111-SNAPSHOT
TsTable.java	Adds optimistic locking mechanism with version tracking for improved read performance
TsTableColumnSchema.java, FieldColumnSchema.java	Adds getMeasurementSchema() method for schema conversion
TsFileTableSchemaUtil.java	New utility class for optimized TsTable↔TableSchema conversion without intermediate serialization
InsertNodeMeasurementInfo.java	New class encapsulating insert node measurements with lazy evaluation support
InsertBaseStatement.java	Adds caching flags for semantic checks, toLowerCase, and attribute columns
InsertTabletStatement.java, InsertRowStatement.java	Implements rebuildArraysAfterExpansion for TAG column reordering
WrappedInsertStatement.java	Refactors validation to use new InsertNodeMeasurementInfo and optimized TAG column handling
TableHeaderSchemaValidator.java	Adds validateInsertNodeMeasurements with custom handlers for optimized validation
DataRegion.java	Replaces schema cache with version-aware TableSchema cache
LoadTsFileManager.java, UnsealedTsFileRecoverPerformer.java	Uses TsFileTableSchemaUtil instead of intermediate conversions
DataNodeTableCache.java	Renames version to instanceVersion for clarity
TSDataTypeTestUtils.java	New test utility for filtering unsupported TSDataType values
AlignedTVList.java	Optimizes bitmap initialization using markRange
TVList.java	Changes hasLimit() to hasSetLimit() for pagination controller
IoTDBConfig.java, IoTDBDescriptor.java	Removes deprecated loadTableSchemaCacheSizeInBytes configuration

Comments suppressed due to low confidence (1)

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/ast/WrappedInsertStatement.java:1

This duplicates the same incorrect logic from InsertBaseStatement.semanticCheck() (see previous comment). If a measurement has failed and is in failedMeasurementIndex2Info, it could be null but would skip the null check, then get added to deduplicatedMeasurements, causing incorrect behavior. Failed measurements should be completely skipped from duplicate detection.

/*

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/table/TsTable.java

...e/src/main/java/org/apache/iotdb/db/queryengine/plan/statement/crud/InsertBaseStatement.java

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/DataRegion.java

...ain/java/org/apache/iotdb/db/queryengine/plan/relational/sql/ast/WrappedInsertStatement.java

Copilot · 2025-11-20T10:47:43Z

...apache/iotdb/db/queryengine/plan/relational/metadata/fetcher/TableHeaderSchemaValidator.java

+      measurementValidator.validate(
+          i,
+          measurementInfo.getMeasurementName(i),
+          measurementInfo.getType(i),
+          columnCategories[i],
+          table.getColumnSchema(measurementInfo.getMeasurementName(i)));
+    }


Variable measurementValidator may be null at this access as suggested by this null guard.

Suggested change

measurementValidator.validate(

i,

measurementInfo.getMeasurementName(i),

measurementInfo.getType(i),

columnCategories[i],

table.getColumnSchema(measurementInfo.getMeasurementName(i)));

}

if (measurementValidator != null) {

measurementValidator.validate(

i,

measurementInfo.getMeasurementName(i),

measurementInfo.getType(i),

columnCategories[i],

table.getColumnSchema(measurementInfo.getMeasurementName(i)));

}

Copilot · 2025-11-20T10:47:44Z

...apache/iotdb/db/queryengine/plan/relational/metadata/fetcher/TableHeaderSchemaValidator.java

+          i,
+          measurementInfo.getMeasurementName(i),
+          measurementInfo.getType(i),
+          columnCategories[i],


Variable columnCategories may be null at this access as suggested by this null guard.

Suggested change

columnCategories[i],

columnCategories != null ? columnCategories[i] : null,

Copilot · 2025-11-20T10:47:44Z

...rc/main/java/org/apache/iotdb/db/queryengine/plan/relational/metadata/TableMetadataImpl.java

        .validateTableHeaderSchema(
            database, tableSchema, context, allowCreateTable, isStrictTagColumn);
  }



This method overrides Metadata.validateInsertNodeMeasurements; it is advisable to add an Override annotation.

Suggested change

@Override

Copilot · 2025-11-20T10:47:44Z

...ode/src/test/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/AnalyzerTest.java

        assertEquals(tableSchema, schema);
        return Optional.of(tableSchema);
      }



This method overrides TestMetadata.validateInsertNodeMeasurements; it is advisable to add an Override annotation.

Suggested change

@Override

Copilot · 2025-11-20T10:47:45Z

...src/main/java/org/apache/iotdb/db/queryengine/plan/statement/crud/InsertTabletStatement.java

+    for (int oldIdx = 0; oldIdx < oldLength; oldIdx++) {
+      final int newIdx = oldToNewMapping[oldIdx];
+      columns[newIdx] = oldColumns[oldIdx];
+      if (nullBitMaps != null && oldNullBitMaps != null) {


This check is useless. oldNullBitMaps cannot be null at this check, since it is guarded by ... != ....

Suggested change

if (nullBitMaps != null && oldNullBitMaps != null) {

if (nullBitMaps != null) {

jt2594838 · 2025-11-24T02:32:09Z

integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/IoTDBInsertTableIT.java

  public void testInsertMultiRowWithNull() throws SQLException {
    try (Connection connection = EnvFactory.getEnv().getConnection(BaseEnv.TABLE_SQL_DIALECT);
        Statement st1 = connection.createStatement()) {
+      st1.execute("SET CONFIGURATION enable_auto_create_schema='false'");


Reset configurations when the test is done.

jt2594838 · 2025-11-25T03:09:51Z

...src/main/java/org/apache/iotdb/db/queryengine/plan/statement/crud/InsertTabletStatement.java

+        newNullBitMaps[newIdx] = new BitMap(rowCount);
+        newNullBitMaps[newIdx].markAll();


Not relevant to this PR, but we may add an extension of BitMap like AllMarkedBitMap, which cannot be marked/unmarked and always returns true when a position is tested.
This could somehow reduce the memory footprint, because it will not store any underlying array.

...tion-test/src/test/java/org/apache/iotdb/relational/it/session/IoTDBSessionRelationalIT.java

luoluoyuyu added 8 commits November 4, 2025 16:20

Refactor TableSchema handling to reduce conversion overhead

b5ac4a5

update

a0a5051

update

5a91fcb

update

022d92e

update

c4cdae2

update

aa406fd

update

514e9e7

update

a5a51e3

luoluoyuyu closed this Nov 6, 2025

luoluoyuyu deleted the impl-table-util branch November 6, 2025 10:11

update

195b767

luoluoyuyu restored the impl-table-util branch November 7, 2025 02:13

luoluoyuyu reopened this Nov 7, 2025

luoluoyuyu changed the title ~~Optimize TableSchema conversion for write performance~~ perf: Optimize wide table write performance Nov 7, 2025

luoluoyuyu added 4 commits November 7, 2025 11:59

fix

6392beb

update semanticCheck

995b3c5

fix: cache table schema by database and table

626ceb3

spotless

ceb4609

jt2594838 reviewed Nov 11, 2025

View reviewed changes

update

bcab1cf

jt2594838 approved these changes Nov 12, 2025

View reviewed changes

luoluoyuyu added 9 commits November 18, 2025 12:05

update

4a796fb

update

81ed8ba

update

21be63e

update

f512f0f

fix InsertStatementTest

5d268cd

update

f604a7f

update

8f8a4e4

update

d987788

update

7eb3b3a

Copilot started reviewing on behalf of HTHou November 20, 2025 10:37 View session

Copilot finished reviewing on behalf of HTHou November 20, 2025 10:41

update

ea6e650

Copilot AI reviewed Nov 20, 2025

View reviewed changes

jt2594838 approved these changes Nov 20, 2025

View reviewed changes

luoluoyuyu added 3 commits November 21, 2025 12:24

fix it

ad1d0d5

Merge branch 'master' into impl-table-util

72357e8

fix it

0f84272

jt2594838 reviewed Nov 24, 2025

View reviewed changes

luoluoyuyu added 5 commits November 24, 2025 13:15

fix

84aa436

fix

16085a2

fix

1d43168

fix

4e2a960

fix

a46a6b9

jt2594838 approved these changes Nov 25, 2025

View reviewed changes

luoluoyuyu added 8 commits November 25, 2025 11:59

fix

97a6418

update

9a08ca8

fix

e6a5abf

fix

1f05cff

fix

1633337

fix

c22f6ab

fix

e611862

fix

30af377

jt2594838 approved these changes Nov 27, 2025

View reviewed changes

...tion-test/src/test/java/org/apache/iotdb/relational/it/session/IoTDBSessionRelationalIT.java Show resolved Hide resolved

jt2594838 approved these changes Nov 27, 2025

View reviewed changes

fix

ece8fc3

jt2594838 approved these changes Nov 27, 2025

View reviewed changes

fix

f229c30

jt2594838 approved these changes Nov 28, 2025

View reviewed changes

jt2594838 merged commit 23be220 into apache:master Nov 28, 2025
27 of 28 checks passed

	columnCategories[i],
	columnCategories != null ? columnCategories[i] : null,

	if (nullBitMaps != null && oldNullBitMaps != null) {
	if (nullBitMaps != null) {

		newNullBitMaps[newIdx] = new BitMap(rowCount);
		newNullBitMaps[newIdx].markAll();

perf: Optimize wide table write performance #16699

perf: Optimize wide table write performance #16699

Conversation

luoluoyuyu commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

I. Metadata Phase Optimization

Problem Background

Optimization Measures

1. Reduce Redundant TsTable → TableSchema Conversion

2. Reduce Unnecessary semanticCheck Execution

3. Optimize TsTable.getColumnSchema Read Lock Time - Introduce Optimistic Locking

II. TSFile Table Registration Optimization

Problem Background

Optimization Measures

1. Eliminate Redundant TsTableSchema Conversion

Key changed/added classes (or packages if there are too many classes) in this PR

Uh oh!

jt2594838 Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

jt2594838 Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

jt2594838 Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

luoluoyuyu commented Nov 4, 2025 •

edited

Loading

1. Reduce Redundant `TsTable` → `TableSchema` Conversion

2. Reduce Unnecessary `semanticCheck` Execution

3. Optimize `TsTable.getColumnSchema` Read Lock Time - Introduce Optimistic Locking

1. Eliminate Redundant `TsTableSchema` Conversion