Fix slowQueryThreshold & optimize encodeBatch #16765
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #16765 +/- ##
============================================
+ Coverage 38.72% 38.75% +0.03%
Complexity 207 207
============================================
Files 5005 5006 +1
Lines 331856 332051 +195
Branches 42201 42227 +26
============================================
+ Hits 128498 128675 +177
- Misses 203358 203376 +18 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
shuwenwei
changed the title
There was a problem hiding this comment.
Pull request overview
This PR introduces two key improvements: changing the default slowQueryThreshold from 30 seconds to 10 seconds, and optimizing memory usage in encodeBatch by replacing BitMap with LazyBitMap for tracking duplicate timestamps in sparse scenarios.
- Changed default slow query threshold from 30s to 10s to align with documented defaults
- Introduced
LazyBitMapoptimization to reduce memory allocation by up to 80% when tracking sparse duplicate timestamps - Added test coverage for the
encodeBatchfunctionality with multiple TVList scenarios
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| IoTDBConfig.java | Updated default slowQueryThreshold from 30000ms to 10000ms |
| AlignedTVList.java | Replaced BitMap with LazyBitMap in encodeBatch method; added @SuppressWarnings annotations for cognitive complexity |
| AlignedTVListIteratorTest.java | Added new testEncodeBatch test method to verify encodeBatch functionality with different TVList configurations |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| } else { | ||
| if (Objects.isNull(timeDuplicateInfo)) { | ||
| timeDuplicateInfo = new BitMap(rows); | ||
| timeDuplicateInfo = new LazyBitMap(index, maxRowCountOfCurrentBatch, rows - 1); |
There was a problem hiding this comment.
The LazyBitMap is initialized with index as the startPosition, but later at line 2206, the code iterates from startIndex (saved at line 2160) to check isMarked(sortedRowIndex) at line 2220. If the first duplicate is found after some iterations, sortedRowIndex could be less than index, causing incorrect behavior. The first parameter should be startIndex instead of index to align with the iteration range at line 2206.
Example: If startIndex=0 and the first duplicate is at index=50, LazyBitMap(50,...) is created, but isMarked(0) through isMarked(49) will be called, which are below the startPosition.
| timeDuplicateInfo = new LazyBitMap(index, maxRowCountOfCurrentBatch, rows - 1); | |
| timeDuplicateInfo = new LazyBitMap(startIndex, maxRowCountOfCurrentBatch, rows - 1); |
...st/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AlignedTVListIteratorTest.java
Show resolved
Hide resolved
…ngine/dataregion/memtable/AlignedTVListIteratorTest.java Co-authored-by: Copilot <[email protected]>
|
(cherry picked from commit 03b60d1)
2 participants
Description
Default slowQueryThreshold should be 10s
optimize encodeBatch for multi tvlist scene
Scenario
Rows: 0–99999 (100,000 rows)
Duplicate timestamps occur at only two positions:
index 123
index 90732
Using regular BitMap
Must allocate 100,000 bits immediately.
Even though only 2 bits will be marked.
Using LazyBitMap (blockSize = 10,000)
LazyBitMap creates blocks only when needed:
Block for index 123 → block index = 0
Block for index 90732 → block index = 90
Total allocated blocks: 2 blocks
Each block has only 10,000 bits, so memory is:
2 blocks × 10,000 bits = 20,000 bits total
Savings
Regular BitMap: 100,000 bits
LazyBitMap: 20,000 bits
➡ 80% memory reduction
➡ Matches the sparse nature of duplicated timestamps