Support native Maximal Marginal Relevance

[bzhangam]

Description

Support native Maximal Marginal Relevance. It relies on the system generated search pipeline to auto generate:

• MMR oversample search request processor. It will modify the search request to oversample for MMR and also resolve the space type for response processor to use.
• MMR rerank search response processor. It will rerank the hits based on Maximal Marginal Relevance.

We also introduce the MMRQueryTransformer to support different transform logic for different query builder. Here we just implement the transformer for the knn query for now. And we will implement the transformer in neural plugin to handle the neural query properly.

Checks will fail until this PR is merged in core.

example query with mmr:

{
	"_source": {
		"excludes": ["passage_embedding"]
	},
	"query": {
		"knn": {
				"passage_embedding": {
					"vector": [1.0, 2.0, 3.0],
					"k": 10
				}
		}
  },
	"ext":{
		"mmr":{
			"diversity": 0.5,
			"candidates": 20
		}
	}
}

Check benchmark

Related Issues

Resolves #2804

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[bzhangam]

We cleaned up this logic in this PR but didn't clean up the comments which can cause confusion. So cleaned it up here since this PR also needs to resolve the space type.

[bzhangam]

@navneet1v @Vikasht34 Could you help review this PR?

[Vikasht34]

@navneet1v @Vikasht34 Could you help review this PR?

I will start ...Please Make Sure all Integs tests are passing

[bzhangam]

@Vikasht34 I have added some benchmark data.

[Vikasht34]

Looks good me , though one test is failing !! Please make sure to fix that.