Indirect loading Faiss index in warmup API when memory optimized search is enabled.

[0ctopus13prime]

Description

When memory-optimized search is enabled, the warm-up operation performs a partial index load and determines the starting offset of each logical section within the index. This does not load anything from an underlying file or stream.
Because of this, users still incur the startup cost regardless of using the warm-up API.

Code reference: KNNIndexShard.java#L114-L118

In this proposal, we start with a simple approach that touches the entire file, indirectly delegating page loading to the kernel. With this mechanism, once an index has been warmed up, the cold start overhead will be reduced compared to when this optimization is absent.

Pseudocode:

if (directory is not FSDirectory) {
    // Do partial loading
    ...
    return
}

IndexInput input = ...;  // Get random accessor from FSDirectory
long fileSize = input.length();
for (long i = 0 ; i < fileSize ; i += 4096) {
    // Touch 1 byte from each page to trigger page fault, then kernel will load 4KB page from the file.
    input.readByte();
}

[0ctopus13prime]

Or, we can just extract the file path like this, and do fadvise directly.

https://man7.org/linux/man-pages/man2/posix_fadvise.2.html

       POSIX_FADV_WILLNEED
              The specified data will be accessed in the near future.

              POSIX_FADV_WILLNEED initiates a nonblocking read of the
              specified region into the page cache.  The amount of data
              read may be decreased by the kernel depending on virtual
              memory load.  (A few megabytes will usually be fully
              satisfied, and more is rarely useful.)

[navneet1v]

@0ctopus13prime lets try to move towards IndexInput based approach rather than building anything related to directory specific. If we want to use POSIX_FADV_WILLNEED can we not add this feature in Lucene and use this madvise?

[0ctopus13prime]

The tricky part is that IndexInput is located deep inside VectorReader, so we need a hacky way to obtain it 😓. We can reuse the approach we’re using today (see, for example, the hack in NativeEngines990KnnVectorsReader).

I think it would be better to start with FSDirectory first, as in the worst case, large index stored remotely (ex: S3) and a small instance serving light traffic then one warm up API could trigger downloading the entire index onto the small instance, which may crash the server or cause resource exhaustion.

I think I know the reason you're reluctant to go with fadvise, and alternatively, we can use File API in Java to load it once acquired file path.

[navneet1v]

The tricky part is that IndexInput is located deep inside VectorReader, so we need a hacky way to obtain it 😓. We can reuse the approach we’re using today (see, for example, the hack in NativeEngines990KnnVectorsReader).

I think it would be better to start with FSDirectory first, as in the worst case, large index stored remotely (ex: S3) and a small instance serving light traffic then one warm up API could trigger downloading the entire index onto the small instance, which may crash the server or cause resource exhaustion.

I think I know the reason you're reluctant to go with fadvise, and alternatively, we can use File API in Java to load it once acquired file path.

I understand all the benefits here, but you already know we did a lot of effort in 2.18 and 2.19 version of OpenSearch to move away from FSDirectory and direct file paths. Lets not go backwards now. Lets come up with some creative ways here to leverage this optimization. May be this POSIX_FADV_WILLNEED might already be present in Lucene. Let check and confirm.

[0ctopus13prime]

Sounds good!

[navneet1v]

@0ctopus13prime I see lucene has this madvise present: https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/store/PosixNativeAccess.java#L45-L46 lets see how we can use this madvise. Because what I can see is Lucene is not using this mdvise: https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/store/PosixNativeAccess.java#L154-L166

What I can see if Lucene is using this WILL NEED madvise during prefetch of an IndexInput.

Ref: https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/store/PosixNativeAccess.java#L127-L130 which is called from here: https://github.com/apache/lucene/blob/1d370a9fd84de2896c7c0dc5567f729dde79e526/lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInput.java#L358 which is called from here: https://github.com/apache/lucene/blob/b7957ff090ab2a48e7143e671c688e477ed9094a/lucene/core/src/java/org/apache/lucene/store/IndexInput.java#L206-L209

I think we can build a prefetch function on our NativeEngines990KnnVectorsReader which can be used to this warmup operations for us by doing prefetch.

[0ctopus13prime]

That's only working for MMapDirectory, and not for NIOFSDirectory.
I believe we should make the behavior general as possible to minimize unncecessary confusions for user.
Agree on adding warmup API in vector reader, and iterating through IndexInput, so that warmup behavior can be consistent regardless of Directory configured.

What do you think? @navneet1v