[FEATURE][RFC] Introduce MultiVector Field Type For Late-interaction Score

[luyuncheng]

Statement

In ColPali paper we found multi vector would optimize large document search ndcg result with Late-interaction like paper shows.

we have been tried using nested document implement multiVector, and do multi KNN query responses calculating ColPali score. but it is not ok, because we found that when KNN queries results with different TopK,

[Q1->Res Document:1,2,3]
[Q2->Res Document:4,5,6]
....

Score function is max(Q1, D)+max(Q2,D) + ... which should include Document [1,6] socres

we can not get the complete score for a paper(parent document):

📌 so i want to introduce Late-interaction capability with new field: multiVector

Proposal

1st introduce new field type: multi_knn_vector

"mappings": {
  "properties": {
    "my_multi_knn_vector_field": {
      "type": "multi_knn_vector"
      "dimension": 3
    }
  }
}
 
PUT xx_index/_doc/1
{ "my_multi_knn_vector_field": [ [1,2,3], [4,5,6], ...]}

for multi_knn_vector we only use binary docvalues, and can combined with derived source.

2nd introduce multi knn query

we can reuse dot product calculation and implement: maximum dot product score like paper says

POST xx_index/_search
{
   "query": {
         "multi_knn_query" : {
             "my_multi_knn_vector_field": {
                 "score_mode": "max_sim_dot",
                 "vectors": [ [1,2,3], [5,6,7] ...]
             }
         }
   }
}

3rd introduce rescore for query which is Late-interaction Score

POST xx_index/_search
{
   "query": {
       "match_all": { }
   },
   "rescore" : {
      "window_size" : 50,
      "query" : {
         "multi_knn_query" : {
             "my_multi_knn_vector_field": {
                "score_mode": "max_sim_dot",
                "vectors": [ [1,2,3], [5,6,7] ...]
             }
         }
      }
   }
}

Future Plan

we can use binary quantize for the multi vector optimize performance

WIP PR： #2707

[jmazanec15]

Thanks @luyuncheng - seems like an awesome feature. One question: why create a new field type as opposed to extending existing field type and just adding array support? See https://docs.opensearch.org/docs/latest/field-types/supported-field-types/index/#arrays.

[q-andy]

@luyuncheng thanks for this, this looks awesome! We've also been exploring how to provide this solution in neural search, some discussion in opensearch-project/OpenSearch#18091. With this RFC are you planning to run late-interaction models outside of OpenSearch and ingest the token level embeddings into the proposed multi-vector field directly?

[luyuncheng]

run late-interaction models outside of OpenSearch

NO

With this RFC are you planning to run late-interaction models outside of OpenSearch and ingest the token level embeddings into the proposed multi-vector field directly?

@q-andy in this rfc, i just want to introduce multi-vector type and do maximum dot product score in query phase AND rescore phase. these are the fundamental ability in ColPali model's search schema. meanwhile, vector search, vector distance compute, vector quantize can reuse k-NN ability.

also, i think ColPali models process can be introduced in OpenSearch ml-commons repo, ingest process token can handled in neural search

[luyuncheng]

One question: why create a new field type as opposed to extending existing field type and just adding array support?

@jmazanec15 because i also want to introduce binary vector, quantized vector etc. after this features introduces, i would like to add more types. so i think a new field type would be better, that we can set some dimension and data_type like knn_vector#byte-vectors