[RFC] Native Maximal Marginal Relevance (MMR) Support

[bzhangam]

Background

Maximal Marginal Relevance (MMR) is a greedy re-ranking algorithm designed to select documents that are both highly relevant to a query and diverse with respect to each other. It works by iteratively choosing the next document that maximizes:

MMR ( 𝐷 𝑖 ) = 𝜆 ⋅ Similarity ( 𝐷 𝑖 , 𝑄 ) − ( 1 − 𝜆 ) ⋅ max ⁡ 𝐷 𝑗 ∈ 𝑆 Similarity ( 𝐷 𝑖 , 𝐷 𝑗 )

Where:
𝜆 (diversity parameter) controls the trade-off between relevance and novelty.
Similarity ( 𝐷 𝑖 , 𝑄 ) measures the relevance of document 𝐷 𝑖 to the query.
Similarity ( 𝐷 𝑖 , 𝐷 𝑗 ) measures similarity between candidate documents to avoid redundancy.
𝑆 is the set of documents already selected.

In vector search contexts such as knn or neural queries, MMR can improve retrieval results by preventing the top‑k from being dominated by near-duplicate embeddings. Without MMR, results tend to cluster closely in the embedding space, reducing information diversity.

Currently, MMR can only be implemented externally or via custom pipelines, which adds complexity, increases latency, and requires manual request/response manipulation.

The goal is to provide native MMR support in OpenSearch for single knn and neural queries using knn_vector.

Related Github requests:
#2804
opensearch-project/neural-search#1481

Requirement

In Scope

1. Handle a single knn query.
2. Handle a single neural query using knn_vector.

Out of Scope

1. Embeddings in nested objects (unclear similarity calculation for multiple embeddings per doc).
2. MMR for queries nested inside hybrid queries. (not sure if we want to simply rerank the response)
3. MMR for queries nested inside bool queries. (not sure if we want to simply rerank the response)

Solutions

Flow:

1. Search Request Processor: modifies the original query k and size to oversample candidates. Also collect the info needed by the response processor and set it in the PipelineContext so that later the response processor can use it.
2. Search Response Processor: reranks the candidates using MMR and selects the top results based on the original query size.

Note: We will rely on the System Generated Search Pipeline to support this feature so that we don't need to manual set up the search pipeline to use the search processors.

How to take mmr parameters?

Option 1: New Query Type (mmr)

Example Request:

{
  "query": {
    "mmr": {
      "query": {
         "knn": {
           "vector_field": [...],
           "k": 10
         }
      },
      "candidates": 100,
      "diversity": 0.5 // 0 - 1, higher value means higher weight for diversity in rerank
    }
  },
  "_source": {
    "excludes": [
      "vector_field"
    ]
  }
}

Processor Transformation:

{
  "query": {
    "knn": {
        "vector_field": [...],
        "k": 100
    }
  },
  "size": 100
}

Pros:

• Self-contained — all MMR parameters and the inner query are together.
• Clear discoverability — users see mmr as a distinct query type.

Cons:

• Increases API surface — more parsing, serialization, and query builder code.
• Potential user confusion — may be mistaken as a normal query rather than a transformation wrapper.

Implementation Note:

• Must be top-level only. doRewrite and doToQuery functions should never be invoked.

Option 2: Query Extension (ext.mmr) (recommended)

Example Request:

{
  "query": {
    "knn": {
        "vector_field": [...],
        "k":10
    }
  },
  "ext": {
    "mmr": {
      "candidates": 100,
      "diversity": 0.5
    }
  }
}

Processor Transformation:

{
  "query": {
    "knn": {
        "vector_field": [...],
        "k":100
    }
  },
  "size": 100,
  "ext": {
    "mmr": {
      "candidates": 100,
      "diversity": 0.5
    }
  }
}

Pros:

• Minimal API change — fits naturally into existing ext mechanism.
• Easier for users to toggle MMR on/off without changing the query structure.

Cons:

• Less discoverable — hidden under ext.

Recommendation on Query Type vs Extension

• If API clarity and discoverability are priorities → Option 1.
• If minimal disruption and composability are priorities → Option 2.

The implementation effort is similar and theoritically this is more like an extension to enhance the query function so we would recommend the option 2.

Where to introduce the change?

Where to Introduce the New Query/Extension

• Add to the KNN plugin, since both Neural Search and KNN plugins rely on vectors.
• Do not add to core — MMR is vector-specific and not generic enough.

Where to Introduce the processors

Search Request Processor

The search request processor needs access to the original query to modify the k parameter:

• KNN query: belongs to the KNN plugin.
• Neural query: belongs to the Neural plugin, which depends on the KNN plugin.

Because the Neural plugin depends on the KNN plugin:

• The Neural plugin can access the KNN query, but the KNN plugin cannot access the Neural query.
• We also want MMR to work when only the KNN plugin is installed.

Proposed solution:

• Implement the search request processor in the KNN plugin.
• Introduce a transformer registry that allows other plugins (e.g., Neural plugin) to register a transformer for their query type.

public interface MMRQueryTransformer<T extends QueryBuilder> {
    /**
     * Transform the queryBuilder to oversample for MMR.
     * Also need to figure out the vector field path and the space type and set them in the MMRProcessingContext for
     * response processor to consume.
     * @param queryBuilder
     * @param listener
     * @param mmrTransformContext
     */
    void transform(T queryBuilder, ActionListener<Void> listener, MMRTransformContext mmrTransformContext);
}

This allows:

• KNN queries to be handled natively in the KNN plugin.
• Neural queries to have dedicated logic in the Neural plugin.

Search Response Processor

The search response processor only needs to perform MMR reranking, which depends on:

• The vector in each hit.
• The vector field path, space type, and data type to compute similarity.

It does not depend on the original query type.

• Collecting the vector field info and space type for different query types requires dedicated logic, but this can be done during the search request processing using the transformers.

Advantages:

• Only one search response processor is needed in the KNN plugin.
• Transformers populate the MMRRerankContext with all necessary info for reranking.
• Neural plugin can inject custom logic without duplicating the response processing logic.

Processor Implementation Detail

Search Request Processor

• Modify the query size
  • Should set the query size as the candidates to oversample.
• Modify k
  • Should set the k as the candidates to oversample.
  • If we are using min_score or max_distance then we don’t need to change it.
• Ensure embedding in the source
  • Should fetch all the fields from the source to ensure we have the vector for MMR. It’s tricky to modify the _source to ensure it doesn’t exclude the embedding so we simply include all fields and do filter after the MMR rerank.
• Collect info
  • We should collect necessary info needed by the search response processor and set it in the pipeline context. We need to collect the info from the index mapping or the model metadata.

{
          "mmr_rerank_context": {
            "original_query_size": 10, // used to decide how many hits should return
            "diversity":0.5,
            "vectorFieldPath": "vector_field", // used to access the vector in the source
            "spaceType": "l2" // used to decide the similarity function
            "vectorDataType": "float", // used to vector data type when parse it from the source
            "originalFetchSourceContext": {"excludes": ["vector_field"]} // used to filter the source after the rerank
           }
        }

• Validation:
  • Ensure the vector field in all the target indices should have the same space type and data type so that we can use the same similarity function.
• Order
  • This search request processor modifies the query size and k if we are using it. Which just needs to be done before the query phase. Considering it is possible there can be other processors modify the search request to inject knn query so would recommend to execute this one after the user defined query.

Search Response Processor

• Extract vectors from hits
  • Use vector field path to extract the vector from the hits and use the vector data type to decide it should be float or byte. This can impact how we calculate the similarity later.
• Similarity computation
  • Option 1: reuse existing doc score (recommended)
    • Pros:
      • Fast, leverages existing KNN/neural scores.
    • Cons:
      • Slight bias possible if score transformations exist. e.g. boost
  • Option 2: recalculate similarity
    • Pros:
      • Accurate, independent of score transformations.
    • Cons:
      • Slower, requires raw vectors. For neural query a phase result processor will be needed to access the vector in the rewritten query.
  • Recommendation: Most KNN or neural searches already produce high-quality similarity scores. So would recommend to reuse doc score for now; optional enhancement to recompute similarity if needed in future.
• Similarity Function
  • Determined by vector field’s space type (e.g., L2, cosine). For now we only support the knn_vector field and it should always have a space type value.
  • Ensures MMR reranking uses correct similarity function. This mapping is already defined in the knn pluign.
• Order
  • Since we will do MMR re-rank and reduce the results to the original query size so we would do this before the user defined processors.
• Validation
  • Nested vectors are not supported.

Extendability

To support the bool and hybrid queries with MMR rerank we can allow users to specify:

• Vector field for MMR.
• Query embedding vector if don’t want to use the doc score directly
• Model ID if generating embedding from plain text.

Then we should be able to do MMR rerank.

[bzhangam]

#2868 is raised for this feature.

[navneet1v]

@bzhangam is this feature planned for 3.3 release?

[bzhangam]

@navneet1v yes, it's for 3.3.

[bzhangam]

Benchmark

Local test cluster on MacBook Pro with Apple M3 Pro chip and 36G memory.
Using workload: https://github.com/opensearch-project/opensearch-benchmark-workloads/blob/main/vectorsearch/params/corpus/1million/faiss-cohere-768-dp.json

k	query size	mmr candidates	knn query without mmr (p50 ms)	knn query without mmr (p90 ms)	knn query with mmr (p50 ms)	knn query with mmr (p90 ms)	p50 delta (%)	p90 delta (%)	p50 delta (ms)	p90 delta (ms)
1	1	1	2.11548	2.42109	2.25152	2.54534	6.43069	5.13199	0.13604	0.12425
10	10	10	2.53146	2.42109	3.92	4.35287	54.85135	79.78968	1.38854	1.93178
30	10	30	2.3281	2.63308	7.28085	7.80728	212.73785	196.50751	4.95275	5.1742
50	10	50	2.32081	2.63547	10.6428	11.5742	358.58127	339.17024	8.32199	8.93873
50	20	50	2.31448	2.59355	11.2581	12.3453	386.42028	376.00008	8.94362	9.75175
50	50	50	2.27188	2.58752	12.6204	13.7177	455.50469	430.14856	10.34852	11.13018
100	10	100	2.34744	2.66638	18.6883	20.1658	696.11406	656.2988	16.34086	17.49942

Latency Summary

MMR reranking can add noticeable latency to a KNN query, and the extra latency increases rapidly as the number of candidates grows. The latency primarily comes from two sources:

1. Vector extraction from the hit source: currently, vectors are retrieved from _source, parsed into a Map, and numeric values are unboxed (e.g., Double → float).
2. Pairwise similarity calculation for MMR reranking: this is inherent to the MMR algorithm and cannot be avoided.

The latency of a pure KNN query grows only slightly with larger k. The dominant cost is graph traversal, controlled by ef_search, while selecting the top-k candidates is relatively cheap. As long as k is reasonable compared to ef_search, query latency remains largely stable.

The latency of extracting the vectors grows linearly when we have more candidates. We may be able to improve the first source of latency if we can fetch the vector as primitive arrays (float[] or byte[]). But more investigation is needed.

This benchmark is using a local cluster because there is some build issue to set up a remote cluster. But it's ok to use it to give us a rough idea how much latency the MMR can add to the knn query. Later will do the benchmark with remote cluster to provide more accurate data.