Make the perThreadHardLimitMB to be configurable above 2GB

[punAhuja]

Description

The parameter perThreadHardLimitMB cannot be larger than 2GB, which means a single thread cannot write segments larger than 2GB.
Refer: https://lucene.apache.org/core/9_9_0/core/org/apache/lucene/index/IndexWriterConfig.html#setRAMPerThreadHardLimitMB(int)

This issue proposes to make this parameter configurable above the 2GB limit, so that each thread can write a bigger segment.

When indexing high dimensional vector data, each segment has its own HNSW graph. So more segments mean more graphs to search per shard and more graph rebuild work during merges. With this change, a single indexing thread can flush fewer, and larger segments, which is generally more resource-efficient for vector-heavy workloads.

[dsmiley]

That limit prevents new (aka "flushed") segments from being larger than 2GB. It doesn't prevent larger segments (via merges). That limit is governed by setMaxMergedSegmentMB (default is 5GB; consider raising this).

[chatman]

That limit prevents new (aka "flushed") segments from being larger than 2GB. It doesn't prevent larger segments (via merges). That limit is governed by setMaxMergedSegmentMB (default is 5GB; consider raising this).

During bulk ingestion of vectors, if someone is not in a memory constrained environment, it is better to flush larger segments (>2GB) than creating several small segments and merging them up to become large segments.

As an example, 1M documents with a 2048 dimension vector, would take up 8GB. It is arguably more efficient to write this segment at once than creating several smaller segments and merging them up to become a 8GB segment.

[chatman]

setMaxMergedSegmentMB (default is 5GB; consider raising this).

+1

[bharath-techie]

#15152 - byteBlockPool can hold only upto 2 GB for reference, will that not be a problem if hard limit breaches 2 GB ?

[punAhuja]

#15152 - byteBlockPool can hold only upto 2 GB for reference, will that not be a problem if hard limit breaches 2 GB ?

The byteBlockPool is only relevant to the postings layer. I had done an experiment where I generated many unique tokens to index on a single thread, which caused an Arithmetic Exception. But that is an unrealistic scenario. Especially in our case of vector heavy workload. Vectors dont use the byteBlockPool. Postings data (from text fields) use byteBlockPool that remain small.
In vector heavy payloads (like high dimensional vectors), most of the in-memory data comes from vector graphs, not text postings, so the ByteBlockPool usage remains minimal.

Maybe we can add a note saying raising the per thread ram limit is beneficial for vector-heavy data, but could be unsafe for extreme text-heavy payloads.