[LuceneOnFaiss Part-8] Added index.knn.memory_optimized_search index setting.

[0ctopus13prime]

Description

This PR adds index.knn.memory_optimized_search index settings.
Overall, this brings two major changes listed below along with the new setting definition:

1. Add 'index name' in field attributes.
   In VectorReader, we check whether memory optimized search is enabled. If it is enabled, then use IndexInput to partial load FAISS index for searching. Partial loaded index hierarchy will be used for serving a query via Lucene's HNSW graph searcher.
   Since this setting sits on index level, we need an index name to query whether if it's enabled or not.
   To this, we inject index name into field attributes.

2. Branching in KNNQueryBuilder
   This PR will make each knn field type decide whether memory-optimized-search is supported or not.
   Currently only Float HNSW is supported, therefore any binary HNSW graphs will return false to indicate the feature is not supported.
   If target field in search is supported for memory-optimized-search, control will fallback to Lucene query to proceed vector search. In which, Lucene's HNSW graph searcher will perform KNN search + Radius search on FAISS index.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[0ctopus13prime]

Part-7 #2608 was merged.
Will rebase and re-raise a new revision.
In the new revision, it will have unit tests.

[0ctopus13prime]

Hi @jmazanec15 ,

In this revision, I've included a hybrid approach inspired by @shatejas 's suggestion.

Previously, I added lazy loading in search because accessing the index.knn.memory_optimized_search setting in the constructor seemed tricky. Retrieving this value requires knowing the index name, and more importantly, during recovery, the codec is loaded via Lucene's SPI, where OpenSearch's internal framework isn't available. That was the main reason for switching to lazy loading within search.

However, this scenario is rare. In most cases, OpenSearch's internal framework (e.g., IndexSettings) is available. Additionally, since we're trying to compress long[] on the fly during loading in #2609 , relying solely on lazy loading in search could negatively impact p99, as it might take several seconds for larger datasets.

With this hybrid approach, we attempt to initialize in the constructor. If IndexSettings is available, we fetch the boolean value; otherwise, we do nothing. This way, most searches will find the table already initialized, no need for lazy loading. Only searches on a just-recovered index will require it.

Let me know if you have any major concerns. I think this approach not only mitigates potential p99 issues but also avoids the Lucene SPI limitation.

[shatejas]

Looks good overall - couple of comments