Add binary doc value compression with variable doc count blocks #137139

parkertimmins · 2025-10-24T23:15:50Z

No description provided.

Reintroduce the LZ4 binary doc values compression originally added to Lucene in LUCENE-9211. Modify so that works in ES819TSDBDocValuesFormat

parkertimmins · 2025-10-24T23:17:24Z

server/src/main/java/org/elasticsearch/index/codec/tsdb/es819/DelayedOffsetAccumulator.java

+        boolean success = false;
+        try {
+            tempOutput = dir.createTempOutput(data.getName(), suffix, context);
+            CodecUtil.writeHeader(


Does it make sense to add the header/footer and then check checksum, given that we are immediately using and deleting the temp file?

parkertimmins · 2025-10-27T00:50:56Z

server/src/main/java/org/elasticsearch/index/codec/PerFieldFormatSupplier.java

+    boolean isBinaryDocValueField(final String field) {
+        if (mapperService != null) {
+            Mapper mapper = mapperService.mappingLookup().getMapper(field);
+            if (mapper != null && "wildcard".equals(mapper.typeName())) {


I believe this is not actually how we want to enable this feature, given that we want it for all BinaryDocValues. I'm thinking we need some sort of PerTypeDelegatingDocValueFormat, which would delegate all types to Lucene90DocValuesFormat except for binary would be delegate to ES819TSDBDocValuesFormat. Thoughts?

I see this is for using compressed binary doc values out side of logsdb / tsdb use cases. Let's wait with doing this for now? Let's just always enable compressed binary doc values if tsdb doc values codec is used.

I think we should add the compressed binary doc. values compression to Lucene90DocValuesFormat in Lucene. In order to do this, I think we need to make compression more pluggable (similar to stored fields). Because zstds isn't available in Lucene and I think we need to default to lz4.

Ahh gotcha. I had thought we want to enable outside logsdb /tsdb, but yeah that makes sense that the way to add this outside logsdb is to eventually add it to Lucene.

martijnvg

Good stuff Parker! I did a first review round.

Additionally I also think we should get real bwc test coverage, I think we can get that by adding a bwc java integration test for wildcard field type. Similar to MatchOnlyTextRollingUpgradeIT or TextRollingUpgradeIT.

martijnvg · 2025-10-27T09:48:48Z

server/src/main/java/org/elasticsearch/index/codec/tsdb/BinaryDVCompressionMode.java

+
+package org.elasticsearch.index.codec.tsdb;
+
+public enum BinaryDVCompressionMode {


Maybe make more use of this abstraction here? For example I think we can add methods:

To get the Compressor instance. For NO_COMPRESS this would return null and for the other this would return ZstdCompressor.

Add minBlockBytes() that returns CompressedBinaryBlockWriter#MIN_BLOCK_BYTES or -1 (for uncompressed).

And perhaps add a level() method that returns -1 or CompressedBinaryBlockWriter#ZSTD_LEVEL? But maybe this isn't necessary.

martijnvg · 2025-10-27T09:49:05Z

server/src/main/java/org/elasticsearch/index/codec/tsdb/BinaryDVCompressionMode.java

+public enum BinaryDVCompressionMode {
+
+    NO_COMPRESS((byte) 0),
+    COMPRESSED_WITH_ZSTD((byte) 1);


Maybe rename to COMPRESSED_WITH_ZSTD_LEVEL)1?

martijnvg · 2025-10-27T09:49:56Z

server/src/main/java/org/elasticsearch/index/codec/zstd/Zstd814StoredFieldsFormat.java

    }

-    private static final class ZstdDecompressor extends Decompressor {
+    public static final class ZstdDecompressor extends Decompressor {


Maybe move to be a top level class in this package?

martijnvg · 2025-10-27T09:55:09Z

server/src/main/java/org/elasticsearch/index/codec/PerFieldFormatSupplier.java

+    boolean isBinaryDocValueField(final String field) {
+        if (mapperService != null) {
+            Mapper mapper = mapperService.mappingLookup().getMapper(field);
+            if (mapper != null && "wildcard".equals(mapper.typeName())) {


I see this is for using compressed binary doc values out side of logsdb / tsdb use cases. Let's wait with doing this for now? Let's just always enable compressed binary doc values if tsdb doc values codec is used.

I think we should add the compressed binary doc. values compression to Lucene90DocValuesFormat in Lucene. In order to do this, I think we need to make compression more pluggable (similar to stored fields). Because zstds isn't available in Lucene and I think we need to default to lz4.

martijnvg · 2025-10-27T09:55:45Z

...er/src/test/java/org/elasticsearch/index/codec/tsdb/es819/ES819TSDBDocValuesFormatTests.java

+    public static BinaryDVCompressionMode randomBinaryCompressionMode() {
+        // BinaryDVCompressionMode[] modes = BinaryDVCompressionMode.values();
+        // return modes[random().nextInt(modes.length)];
+        return BinaryDVCompressionMode.COMPRESSED_WITH_ZSTD;


We need to randomize between uncompressed and this one here?

martijnvg · 2025-10-27T10:11:17Z

server/src/main/java/org/elasticsearch/index/codec/tsdb/es819/ES819TSDBDocValuesFormat.java

+import org.elasticsearch.index.codec.tsdb.BinaryDVCompressionMode;

 import java.io.IOException;



Maybe also update the class level java doc to include that version 1 also includes compressed binary doc values?

martijnvg · 2025-10-27T10:12:41Z

server/src/main/java/org/elasticsearch/index/codec/tsdb/es819/ES819TSDBDocValuesConsumer.java

+         * _docId is unused. This is because docId may not be dense.
+         * But we can guarantee that the lookup value is dense on the range of inserted values.
+         */
+        void addDoc(int _docId, BytesRef v) throws IOException {


Remove _docId parameter? Given that it is unused.

martijnvg · 2025-10-27T10:20:31Z

server/src/main/java/org/elasticsearch/index/codec/tsdb/es819/ES819TSDBDocValuesConsumer.java

        }
    }

+    public void doAddCompressedBinary(FieldInfo field, DocValuesProducer valuesProducer) throws IOException {


This method is missing the logic for optimized merging? (that does exist in doAddUncompressedBinary(...))

I wonder whether doAddCompressedBinary(...) and doAddUncompressedBinary(...) can be unified here. Maybe add an interface for CompressedBinaryBlockWriter, and name it something like BinaryWriter with addDoc(...), flushData() and writeMetadata() methods? For uncompressed the addDoc(...) impl would delegate to data.writeBytes(....) and the other methods would be no-op.

martijnvg · 2025-10-27T10:26:51Z

server/src/main/java/org/elasticsearch/index/codec/tsdb/es819/ES819TSDBDocValuesFormat.java

    static final int VERSION_START = 0;
-    static final int VERSION_CURRENT = VERSION_START;
+    static final int VERSION_BINARY_DV_COMPRESSION = 1;
+    static final int VERSION_CURRENT = VERSION_BINARY_DV_COMPRESSION;


+1 to handle bwc via codec version. The alternative is forking the entire codec, but given that the change in the format isn't all over the place, I think this approach is good.

martijnvg · 2025-10-27T10:28:04Z

server/src/main/java/org/elasticsearch/index/codec/tsdb/es819/ES819TSDBDocValuesProducer.java

        }
    }

+    static boolean isDense(int firstDocId, int lastDocId, int length) {


I think this change is unrelated and was for bulk loading? If so let's undo this for now.

parkertimmins added 10 commits October 23, 2025 13:02

Copy binary compression from LUCENE-9211

74880a0

Reintroduce the LZ4 binary doc values compression originally added to Lucene in LUCENE-9211. Modify so that works in ES819TSDBDocValuesFormat

Initial version of block withs variable number values

a973713

Fix issue with index output unclosed

3fc95dc

Changes docRanges to single limit per block, plus start of 0

c302cc2

Factor block address and block doc offset to accumulator class

99748c8

Rename offset accumulator

fa2ea11

Change lz4 to zstd

b67dd58

Fix direct monotonic reader size

638dbbc

Fix docRangeLen bug, use for non-logsdb wildcards

fdf3428

Change offset encoding from zstd to numeric

36b3e10

elasticsearchmachine added the v9.3.0 label Oct 24, 2025

parkertimmins commented Oct 24, 2025

View reviewed changes

elasticsearchmachine and others added 3 commits October 24, 2025 23:22

[CI] Auto commit changes from spotless

eeded36

Fix missing compression in es819 format

2d8e6dc

Merge branch 'main' into parker/compressed-binary-doc-values

efa270f

parkertimmins changed the title ~~Add binary doc value compression with variably doc count blocks~~ Add binary doc value compression with variable doc count blocks Oct 25, 2025

parkertimmins and others added 5 commits October 24, 2025 21:36

Store offsets rather than lengths

c4d67e5

[CI] Auto commit changes from spotless

06a2035

Remove forbidden APIs

7ccb18d

[CI] Auto commit changes from spotless

a57e0d4

Binary search to find block containing docNum

f156e55

parkertimmins commented Oct 27, 2025

View reviewed changes

[CI] Auto commit changes from spotless

91e5842

martijnvg reviewed Oct 27, 2025

View reviewed changes


		package org.elasticsearch.index.codec.tsdb;

		public enum BinaryDVCompressionMode {

		import org.elasticsearch.index.codec.tsdb.BinaryDVCompressionMode;

		import java.io.IOException;

Uh oh!

Add binary doc value compression with variable doc count blocks #137139

Are you sure you want to change the base?

Add binary doc value compression with variable doc count blocks #137139

Conversation

parkertimmins commented Oct 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

martijnvg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants