[LuceneOnFaiss Part-8] Added index.knn.memory_optimized_search index setting.

[0ctopus13prime]

Description

This PR adds index.knn.memory_optimized_search index settings.
Overall, this brings two major changes listed below along with the new setting definition:

1. In codec level, it relies on MapperService to retrieve index.knn.memory_optimized_search bool flag. Which then be used to determine whether to initialize partial loading on KNN fields.
   But this is not always working, during recovery MapperService can be null, therefore adding lazy-loading logic in both constructor and search method. For 99%, it will be loaded in the constructor, so it won't affect search p99 latency as by the time the search method is called, MemoryOptimizedSearch should be loaded already.

2. Branching in KNNQueryBuilder
   This PR will make each knn field type decide whether memory-optimized-search is supported or not.
   Currently only Float HNSW is supported, therefore any binary HNSW graphs will return false to indicate the feature is not supported.
   If target field in search is supported for memory-optimized-search, control will fallback to Lucene query to proceed vector search. In which, Lucene's HNSW graph searcher will perform KNN search + Radius search on FAISS index.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

RFC: #2401

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[0ctopus13prime]

Part-7 #2608 was merged.
Will rebase and re-raise a new revision.
In the new revision, it will have unit tests.

[0ctopus13prime]

Hi @jmazanec15 ,

In this revision, I've included a hybrid approach inspired by @shatejas 's suggestion.

Previously, I added lazy loading in search because accessing the index.knn.memory_optimized_search setting in the constructor seemed tricky. Retrieving this value requires knowing the index name, and more importantly, during recovery, the codec is loaded via Lucene's SPI, where OpenSearch's internal framework isn't available. That was the main reason for switching to lazy loading within search.

However, this scenario is rare. In most cases, OpenSearch's internal framework (e.g., IndexSettings) is available. Additionally, since we're trying to compress long[] on the fly during loading in #2609 , relying solely on lazy loading in search could negatively impact p99, as it might take several seconds for larger datasets.

With this hybrid approach, we attempt to initialize in the constructor. If IndexSettings is available, we fetch the boolean value; otherwise, we do nothing. This way, most searches will find the table already initialized, no need for lazy loading. Only searches on a just-recovered index will require it.

Let me know if you have any major concerns. I think this approach not only mitigates potential p99 issues but also avoids the Lucene SPI limitation.

[shatejas]

Looks good overall - couple of comments

Will update in the next PR.
It's being tracked in #2401 (comment)