Adds ZScore Normalization Technique (#1224)

* Initial implementation of zscore

Signed-off-by: Owais <[email protected]>

* Added negative technique for GP and HP

Signed-off-by: Owais <[email protected]>

* Minor refactoring and removed negative HP and GP

Signed-off-by: Owais <[email protected]>

* Added UTs and ITs and few edge cases

Signed-off-by: Owais <[email protected]>

* Used DescriptiveStatistics and added validations for combination technique

Signed-off-by: Owais <[email protected]>

* Created DTO for validation

Signed-off-by: Owais <[email protected]>

* Addressed PR comments

Signed-off-by: Owais <[email protected]>

---------

Signed-off-by: Owais <[email protected]>

Author: owaiskazi19

CHANGELOG.md

@@ -4,12 +4,20 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). See the [CONTRIBUTING guide](./CONTRIBUTING.md#Changelog) for instructions on how to add changelog entries.
 
 ## [Unreleased 3.x](https://github.com/opensearch-project/neural-search/compare/main...HEAD)
+
 ### Features
 - Lower bound for min-max normalization technique in hybrid query ([#1195](https://github.com/opensearch-project/neural-search/pull/1195))
 - Support filter function for HybridQueryBuilder and NeuralQueryBuilder ([#1206](https://github.com/opensearch-project/neural-search/pull/1206))
+- Add Z Score normalization technique ([#1224](https://github.com/opensearch-project/neural-search/pull/1224))
+
 ### Enhancements
+
 ### Bug Fixes
+
 ### Infrastructure
+
 ### Documentation
+
 ### Maintenance
+
 ### Refactoring

DEVELOPER_GUIDE.md

@@ -351,9 +351,9 @@ through the same build issue.
 
 ### Class and package names
 
-Class names should use `CamelCase`. 
+Class names should use `CamelCase`.
 
-Try to put new classes into existing packages if package name abstracts the purpose of the class. 
+Try to put new classes into existing packages if package name abstracts the purpose of the class.
 
 Example of good class file name and package utilization:
 
@@ -371,7 +371,7 @@ methods rather than a long single one and does everything.
 ### Documentation
 
 Document you code. That includes purpose of new classes, every public method and code sections that have critical or non-trivial
-logic (check this example https://github.com/opensearch-project/neural-search/blob/main/src/main/java/org/opensearch/neuralsearch/query/NeuralQueryBuilder.java#L238). 
+logic (check this example https://github.com/opensearch-project/neural-search/blob/main/src/main/java/org/opensearch/neuralsearch/query/NeuralQueryBuilder.java#L238).
 
 When you submit a feature PR, please submit a new
 [documentation issue](https://github.com/opensearch-project/documentation-website/issues/new/choose). This is a path for the documentation to be published as part of https://opensearch.org/docs/latest/ documentation site.
@@ -384,17 +384,17 @@ For the most part, we're using common conventions for Java projects. Here are a
 
 1. Use descriptive names for classes, methods, fields, and variables.
 2. Avoid abbreviations unless they are widely accepted
-3. Use `final` on all method arguments unless it's absolutely necessary 
+3. Use `final` on all method arguments unless it's absolutely necessary
 4. Wildcard imports are not allowed.
 5. Static imports are preferred over qualified imports when using static methods
 6. Prefer creating non-static public methods whenever possible. Avoid static methods in general, as they can often serve as shortcuts.
 Static methods are acceptable if they are private and do not access class state.
-7. Use functional programming style inside methods unless it's a performance critical section. 
+7. Use functional programming style inside methods unless it's a performance critical section.
 8. For parameters of lambda expression please use meaningful names instead of shorten cryptic ones.
 9. Use Optional for return values if the value may not be present. This should be preferred to returning null.
 10. Do not create checked exceptions, and do not throw checked exceptions from public methods whenever possible. In general, if you call a method with a checked exception, you should wrap that exception into an unchecked exception.
 11. Throwing checked exceptions from private methods is acceptable.
-12. Use String.format when a string includes parameters, and prefer this over direct string concatenation. Always specify a Locale with String.format; 
+12. Use String.format when a string includes parameters, and prefer this over direct string concatenation. Always specify a Locale with String.format;
 as a rule of thumb, use Locale.ROOT.
 13. Prefer Lombok annotations to the manually written boilerplate code
 14. When throwing an exception, avoid including user-provided content in the exception message. For secure coding practices,
@@ -440,17 +440,17 @@ Fix any new warnings before submitting your PR to ensure proper code documentati
 
 ### Tests
 
-Write unit and integration tests for your new functionality. 
+Write unit and integration tests for your new functionality.
 
 Unit tests are preferred as they are cheap and fast, try to use them to cover all possible
-combinations of parameters. Utilize mocks to mimic dependencies. 
+combinations of parameters. Utilize mocks to mimic dependencies.
 
-Integration tests should be used sparingly, focusing primarily on the main (happy path) scenario or cases where extensive 
-mocking is impractical. Include one or two unhappy paths to confirm that correct response codes are returned to the user. 
-Whenever possible, favor scenarios that do not require model deployment. If model deployment is necessary, use an existing 
+Integration tests should be used sparingly, focusing primarily on the main (happy path) scenario or cases where extensive
+mocking is impractical. Include one or two unhappy paths to confirm that correct response codes are returned to the user.
+Whenever possible, favor scenarios that do not require model deployment. If model deployment is necessary, use an existing
 model, as tests involving new model deployments are the most resource-intensive.
 
-If your changes could affect backward compatibility, please include relevant backward compatibility tests along with your 
+If your changes could affect backward compatibility, please include relevant backward compatibility tests along with your
 PR. For guidance on adding these tests, refer to the [Backwards Compatibility Testing](#backwards-compatibility-testing) section in this guide.
 
 ### Outdated or irrelevant code

build.gradle

@@ -260,6 +260,7 @@ dependencies {
     api group: 'org.opensearch', name:'opensearch-ml-client', version: "${opensearch_build}"
     testFixturesImplementation "org.opensearch.test:framework:${opensearch_version}"
     implementation group: 'org.apache.commons', name: 'commons-lang3', version: '3.14.0'
+    implementation group: 'org.apache.commons', name: 'commons-math3', version: '3.6.1'
     // ml-common excluded reflection for runtime so we need to add it by ourselves.
     // https://github.com/opensearch-project/ml-commons/commit/464bfe34c66d7a729a00dd457f03587ea4e504d9
     // TODO: Remove following three lines of dependencies if ml-common include them in their jar

src/main/java/org/opensearch/neuralsearch/processor/TechniqueCompatibilityCheckDTO.java

@@ -0,0 +1,25 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.opensearch.neuralsearch.processor;
+
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Getter;
+import lombok.NonNull;
+import org.opensearch.neuralsearch.processor.combination.ScoreCombinationTechnique;
+import org.opensearch.neuralsearch.processor.normalization.ScoreNormalizationTechnique;
+
+/**
+ * DTO object to hold data required for validation.
+ */
+@AllArgsConstructor
+@Builder
+@Getter
+public class TechniqueCompatibilityCheckDTO {
+    @NonNull
+    private ScoreCombinationTechnique scoreCombinationTechnique;
+    @NonNull
+    private ScoreNormalizationTechnique scoreNormalizationTechnique;
+}

src/main/java/org/opensearch/neuralsearch/processor/combination/ArithmeticMeanScoreCombinationTechnique.java

@@ -58,6 +58,11 @@ public float combine(final float[] scores) {
         return combinedScore / sumOfWeights;
     }
 
+    @Override
+    public String techniqueName() {
+        return TECHNIQUE_NAME;
+    }
+
     @Override
     public String describe() {
         return describeCombinationTechnique(TECHNIQUE_NAME, weights);

src/main/java/org/opensearch/neuralsearch/processor/combination/GeometricMeanScoreCombinationTechnique.java

@@ -58,6 +58,11 @@ public float combine(final float[] scores) {
         return sumOfWeights == 0 ? ZERO_SCORE : (float) Math.exp(weightedLnSum / sumOfWeights);
     }
 
+    @Override
+    public String techniqueName() {
+        return TECHNIQUE_NAME;
+    }
+
     @Override
     public String describe() {
         return describeCombinationTechnique(TECHNIQUE_NAME, weights);

src/main/java/org/opensearch/neuralsearch/processor/combination/HarmonicMeanScoreCombinationTechnique.java

@@ -55,6 +55,11 @@ public float combine(final float[] scores) {
         return sumOfHarmonics > 0 ? sumOfWeights / sumOfHarmonics : ZERO_SCORE;
     }
 
+    @Override
+    public String techniqueName() {
+        return TECHNIQUE_NAME;
+    }
+
     @Override
     public String describe() {
         return describeCombinationTechnique(TECHNIQUE_NAME, weights);

src/main/java/org/opensearch/neuralsearch/processor/combination/RRFScoreCombinationTechnique.java

@@ -37,6 +37,11 @@ public float combine(final float[] scores) {
         return sumScores;
     }
 
+    @Override
+    public String techniqueName() {
+        return TECHNIQUE_NAME;
+    }
+
     @Override
     public String describe() {
         return describeCombinationTechnique(TECHNIQUE_NAME, List.of());

src/main/java/org/opensearch/neuralsearch/processor/combination/ScoreCombinationTechnique.java

@@ -12,4 +12,9 @@ public interface ScoreCombinationTechnique {
      * @return combined score
      */
     float combine(final float[] scores);
+
+    /**
+     * Returns the name of the combination technique.
+     */
+    String techniqueName();
 }

src/main/java/org/opensearch/neuralsearch/processor/factory/NormalizationProcessorFactory.java

@@ -12,6 +12,7 @@
 
 import org.opensearch.neuralsearch.processor.NormalizationProcessor;
 import org.opensearch.neuralsearch.processor.NormalizationProcessorWorkflow;
+import org.opensearch.neuralsearch.processor.TechniqueCompatibilityCheckDTO;
 import org.opensearch.neuralsearch.processor.combination.ArithmeticMeanScoreCombinationTechnique;
 import org.opensearch.neuralsearch.processor.combination.ScoreCombinationFactory;
 import org.opensearch.neuralsearch.processor.combination.ScoreCombinationTechnique;
@@ -77,6 +78,13 @@ public SearchPhaseResultsProcessor create(
             Map<String, Object> combinationParams = readOptionalMap(NormalizationProcessor.TYPE, tag, combinationClause, PARAMETERS);
             scoreCombinationTechnique = scoreCombinationFactory.createCombination(combinationTechnique, combinationParams);
         }
+
+        TechniqueCompatibilityCheckDTO techniqueCompatibilityCheckDTO = TechniqueCompatibilityCheckDTO.builder()
+            .scoreNormalizationTechnique(normalizationTechnique)
+            .scoreCombinationTechnique(scoreCombinationTechnique)
+            .build();
+        scoreNormalizationFactory.isTechniquesCompatible(techniqueCompatibilityCheckDTO);
+
         log.info(
             "Creating search phase results processor of type [{}] with normalization [{}] and combination [{}]",
             NormalizationProcessor.TYPE,

src/main/java/org/opensearch/neuralsearch/processor/normalization/L2ScoreNormalizationTechnique.java

@@ -23,6 +23,7 @@
 import org.opensearch.neuralsearch.processor.explain.ExplainableTechnique;
 
 import static org.opensearch.neuralsearch.processor.explain.ExplanationUtils.getDocIdAtQueryForNormalization;
+import static org.opensearch.neuralsearch.processor.util.ProcessorUtils.getNumOfSubqueries;
 
 /**
  * Abstracts normalization of scores based on L2 method
@@ -69,6 +70,11 @@ public void normalize(final NormalizeScoresDTO normalizeScoresDTO) {
         }
     }
 
+    @Override
+    public String techniqueName() {
+        return TECHNIQUE_NAME;
+    }
+
     @Override
     public String describe() {
         return String.format(Locale.ROOT, "%s", TECHNIQUE_NAME);
@@ -108,13 +114,7 @@ private List<Float> getL2Norm(final List<CompoundTopDocs> queryTopDocs) {
         // find any non-empty compound top docs, it's either empty if shard does not have any results for all of sub-queries,
         // or it has results for all the sub-queries. In edge case of shard having results only for one sub-query, there will be TopDocs for
         // rest of sub-queries with zero total hits
-        int numOfSubqueries = queryTopDocs.stream()
-            .filter(Objects::nonNull)
-            .filter(topDocs -> topDocs.getTopDocs().size() > 0)
-            .findAny()
-            .get()
-            .getTopDocs()
-            .size();
+        int numOfSubqueries = getNumOfSubqueries(queryTopDocs);
         float[] l2Norms = new float[numOfSubqueries];
         for (CompoundTopDocs compoundQueryTopDocs : queryTopDocs) {
             if (Objects.isNull(compoundQueryTopDocs)) {

src/main/java/org/opensearch/neuralsearch/processor/normalization/MinMaxScoreNormalizationTechnique.java

@@ -32,6 +32,7 @@
 import org.opensearch.neuralsearch.processor.explain.ExplainableTechnique;
 
 import static org.opensearch.neuralsearch.processor.explain.ExplanationUtils.getDocIdAtQueryForNormalization;
+import static org.opensearch.neuralsearch.processor.util.ProcessorUtils.getNumOfSubqueries;
 import static org.opensearch.neuralsearch.query.HybridQueryBuilder.MAX_NUMBER_OF_SUB_QUERIES;
 
 /**
@@ -126,6 +127,11 @@ private MinMaxScores getMinMaxScoresResult(final List<CompoundTopDocs> queryTopD
         return new MinMaxScores(minScoresPerSubquery, maxScoresPerSubquery);
     }
 
+    @Override
+    public String techniqueName() {
+        return TECHNIQUE_NAME;
+    }
+
     @Override
     public String describe() {
         return lowerBoundsOptional.map(lb -> {
@@ -172,16 +178,6 @@ public Map<DocIdAtSearchShard, ExplanationDetails> explain(final List<CompoundTo
         return getDocIdAtQueryForNormalization(normalizedScores, this);
     }
 
-    private int getNumOfSubqueries(final List<CompoundTopDocs> queryTopDocs) {
-        return queryTopDocs.stream()
-            .filter(Objects::nonNull)
-            .filter(topDocs -> topDocs.getTopDocs().isEmpty() == false)
-            .findAny()
-            .get()
-            .getTopDocs()
-            .size();
-    }
-
     private float[] getMaxScores(final List<CompoundTopDocs> queryTopDocs, final int numOfSubqueries) {
         float[] maxScores = new float[numOfSubqueries];
         Arrays.fill(maxScores, Float.MIN_VALUE);

src/main/java/org/opensearch/neuralsearch/processor/normalization/RRFNormalizationTechnique.java

@@ -76,6 +76,11 @@ public String describe() {
         return String.format(Locale.ROOT, "%s, rank_constant [%s]", TECHNIQUE_NAME, rankConstant);
     }
 
+    @Override
+    public String techniqueName() {
+        return TECHNIQUE_NAME;
+    }
+
     @Override
     public Map<DocIdAtSearchShard, ExplanationDetails> explain(List<CompoundTopDocs> queryTopDocs) {
         Map<DocIdAtSearchShard, List<Float>> normalizedScores = new HashMap<>();

src/main/java/org/opensearch/neuralsearch/processor/normalization/ScoreNormalizationFactory.java

@@ -4,7 +4,14 @@
  */
 package org.opensearch.neuralsearch.processor.normalization;
 
+import org.opensearch.neuralsearch.processor.TechniqueCompatibilityCheckDTO;
+import org.opensearch.neuralsearch.processor.combination.ArithmeticMeanScoreCombinationTechnique;
+import org.opensearch.neuralsearch.processor.combination.GeometricMeanScoreCombinationTechnique;
+import org.opensearch.neuralsearch.processor.combination.HarmonicMeanScoreCombinationTechnique;
+
+import java.util.Locale;
 import java.util.Map;
+import java.util.Set;
 import java.util.Optional;
 import java.util.function.Function;
 
@@ -17,13 +24,38 @@ public class ScoreNormalizationFactory {
 
     public static final ScoreNormalizationTechnique DEFAULT_METHOD = new MinMaxScoreNormalizationTechnique();
 
-    private final Map<String, Function<Map<String, Object>, ScoreNormalizationTechnique>> scoreNormalizationMethodsMap = Map.of(
+    private static final Map<String, Function<Map<String, Object>, ScoreNormalizationTechnique>> SCORE_NORMALIZATION_METHODS = Map.of(
         MinMaxScoreNormalizationTechnique.TECHNIQUE_NAME,
         params -> new MinMaxScoreNormalizationTechnique(params, scoreNormalizationUtil),
         L2ScoreNormalizationTechnique.TECHNIQUE_NAME,
         params -> new L2ScoreNormalizationTechnique(params, scoreNormalizationUtil),
         RRFNormalizationTechnique.TECHNIQUE_NAME,
-        params -> new RRFNormalizationTechnique(params, scoreNormalizationUtil)
+        params -> new RRFNormalizationTechnique(params, scoreNormalizationUtil),
+        ZScoreNormalizationTechnique.TECHNIQUE_NAME,
+        params -> new ZScoreNormalizationTechnique()
+    );
+
+    private static final Map<String, Set<String>> COMBINATION_TECHNIQUE_FOR_NORMALIZATION_METHODS = Map.of(
+        MinMaxScoreNormalizationTechnique.TECHNIQUE_NAME,
+        Set.of(
+            ArithmeticMeanScoreCombinationTechnique.TECHNIQUE_NAME,
+            GeometricMeanScoreCombinationTechnique.TECHNIQUE_NAME,
+            HarmonicMeanScoreCombinationTechnique.TECHNIQUE_NAME
+        ),
+        L2ScoreNormalizationTechnique.TECHNIQUE_NAME,
+        Set.of(
+            ArithmeticMeanScoreCombinationTechnique.TECHNIQUE_NAME,
+            GeometricMeanScoreCombinationTechnique.TECHNIQUE_NAME,
+            HarmonicMeanScoreCombinationTechnique.TECHNIQUE_NAME
+        ),
+        RRFNormalizationTechnique.TECHNIQUE_NAME,
+        Set.of(
+            ArithmeticMeanScoreCombinationTechnique.TECHNIQUE_NAME,
+            GeometricMeanScoreCombinationTechnique.TECHNIQUE_NAME,
+            HarmonicMeanScoreCombinationTechnique.TECHNIQUE_NAME
+        ),
+        ZScoreNormalizationTechnique.TECHNIQUE_NAME,
+        Set.of(ArithmeticMeanScoreCombinationTechnique.TECHNIQUE_NAME)
     );
 
     /**
@@ -36,8 +68,30 @@ public ScoreNormalizationTechnique createNormalization(final String technique) {
     }
 
     public ScoreNormalizationTechnique createNormalization(final String technique, final Map<String, Object> params) {
-        return Optional.ofNullable(scoreNormalizationMethodsMap.get(technique))
+        return Optional.ofNullable(SCORE_NORMALIZATION_METHODS.get(technique))
             .orElseThrow(() -> new IllegalArgumentException("provided normalization technique is not supported"))
             .apply(params);
     }
+
+    /**
+     * Validate normalization technique based on combination technique and other params that needs to be validated
+     * @param techniqueCompatibilityCheckDTO data transfer object that contains combination technique and other params that needs to be validated
+     */
+    public void isTechniquesCompatible(TechniqueCompatibilityCheckDTO techniqueCompatibilityCheckDTO) {
+        ScoreNormalizationTechnique normalizationTechnique = techniqueCompatibilityCheckDTO.getScoreNormalizationTechnique();
+        Set<String> supportedTechniques = COMBINATION_TECHNIQUE_FOR_NORMALIZATION_METHODS.get(normalizationTechnique.techniqueName());
+
+        if (supportedTechniques.contains(techniqueCompatibilityCheckDTO.getScoreCombinationTechnique().techniqueName()) == false) {
+            throw new IllegalArgumentException(
+                String.format(
+                    Locale.ROOT,
+                    "provided combination technique %s is not supported for normalization technique %s. Supported techniques are: %s",
+                    techniqueCompatibilityCheckDTO.getScoreCombinationTechnique().techniqueName(),
+                    normalizationTechnique.techniqueName(),
+                    String.join(", ", supportedTechniques)
+                )
+            );
+        }
+    }
+
 }

src/main/java/org/opensearch/neuralsearch/processor/normalization/ScoreNormalizationTechnique.java

@@ -20,4 +20,8 @@ public interface ScoreNormalizationTechnique {
      */
     void normalize(final NormalizeScoresDTO normalizeScoresDTO);
 
+    /**
+     * Returns the name of the normalization technique.
+     */
+    String techniqueName();
 }