[BUG]Unable to call aggregation in ml-commons API /_plugins/_ml/models/_search API

[xinyual]

What is the bug?

When we use aggregation in /_plugins/_ml/models/_search API, it returns error like

{
    "error": {
        "root_cause": [
            {
                "type": "status_exception",
                "reason": "Fail to search model version"
            }
        ],
        "type": "status_exception",
        "reason": "Fail to search model version"
    },
    "status": 500
}

How can one reproduce the bug?

In Opensearch 2.19 or 3.0.0 alpha1, we create any connector and corresponding remote ml model. Then we call

POST /_plugins/_ml/models/_search
{
  "size": 0,
  "aggs": {
    "unique2_connector_names": {
      "terms": {
        "field": "connector.name.keyword",
        "size": 10000
      }
    }
  }
}

Then we will get error.

What is the expected behavior?

A clear and concise description of what you expected to happen.

What is your host/environment?

Operating system, version.

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

I think it's related to current repo because In ml commons, In line 142 in MLSearchHandler we use sdkClient.searchDataObjectAsync from library remote-metadata-sdk. After changing it to cient.search , it works. Also, I fetch the search request body in the debug mod and call _search directly, it also works. So I think the problem is from the remote-metadata-sdk library.

[dbwiddis]

In Opensearch 2.19 or 3.0.0 alpha1

Can you be more clear on what you're testing on? Is this the released versions? Or the current snapshot (2.19.2 / 3.0.0 beta1) versions? There were recent changes in ML Commons to the searching.

Also, I fetch the search request body in the debug mod and call _search directly, it also works.

Generally speaking the remote sdk client just copies the search source verbatim. Do you have more debug information on what the search source is?

[dhrubo-os]

error log:

org.opensearch.core.common.ParsingException: Could not parse aggregation keyed as [unique_connector_names]
        at org.opensearch.search.aggregations.Aggregations.fromXContent(Aggregations.java:165) ~[opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.action.search.SearchResponse.innerFromXContent(SearchResponse.java:423) ~[opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.action.search.SearchResponse.fromXContent(SearchResponse.java:376) ~[opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.ml.action.handler.MLSearchHandler.lambda$search$2(MLSearchHandler.java:146) [opensearch-ml-3.0.0.0-beta1-SNAPSHOT.jar:3.0.0.0-beta1-SNAPSHOT]
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) [?:?]
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) [?:?]
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) [?:?]
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) [?:?]
        at org.opensearch.remote.metadata.client.impl.LocalClusterIndicesClient.lambda$searchDataObjectAsync$15(LocalClusterIndicesClient.java:462) [opensearch-remote-metadata-sdk-3.0.0.0-beta1-SNAPSHOT.jar:?]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115) [opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109) [opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.search.pipeline.PipelinedRequest.lambda$transformResponseListener$0(PipelinedRequest.java:47) [opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.core.action.ActionListener$5.onResponse(ActionListener.java:268) [opensearch-core-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:764) [opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132) [opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.action.search.SearchPhase.recordAndRun(SearchPhase.java:62) [opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:481) [opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:448) [opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:300) [opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:138) [opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:150) [opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:122) [opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:994) [opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-3.0.0-beta1-SNAPSHOT.jar:3.0.0-beta1-SNAPSHOT]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

It's failing here and then coming to the exception block

[dbwiddis]

Stack trace points to "typed keys":

parseTypedKeysObject(parser, Aggregation.TYPED_KEYS_DELIMITER, Aggregation.class, typedAgg::set);
if (typedAgg.get() != null) {
    aggregations.add(typedAgg.get());
} else {
    throw new ParsingException(
        parser.getTokenLocation(),
        String.format(Locale.ROOT, "Could not parse aggregation keyed as [%s]", currentField)
    );
}

where parseTypedKeysObject() javadoc says:

This method expects that the current field name is the concatenation of a type, a delimiter and a name (ex: terms#foo where "terms" refers to the type of a registered {@link NamedXContentRegistry.Entry}, "#" is the delimiter and "foo" the name of the object to parse).

But unique_connector_names does not match that pattern.

[dbwiddis]

I have a suspicion that using "aggs" in a query with an arbitrary name is probably supposed to temporarily add it to the registry context for long enough to be parsed, and that might be being lost in the query wrapping. But I don't have not that much experience with custom aggregations like this.

[xinyual]

Stack trace points to "typed keys":

parseTypedKeysObject(parser, Aggregation.TYPED_KEYS_DELIMITER, Aggregation.class, typedAgg::set);
if (typedAgg.get() != null) {
aggregations.add(typedAgg.get());
} else {
throw new ParsingException(
parser.getTokenLocation(),
String.format(Locale.ROOT, "Could not parse aggregation keyed as [%s]", currentField)
);
}

where parseTypedKeysObject() javadoc says:

This method expects that the current field name is the concatenation of a type, a delimiter and a name (ex: terms#foo where "terms" refers to the type of a registered {@link NamedXContentRegistry.Entry}, "#" is the delimiter and "foo" the name of the object to parse).

But unique_connector_names does not match that pattern.

But if we use

POST /_plugins/_ml/models/_search
{
  "size": 0,
  "aggs": {
    "terms#foo": {
      "terms": {
        "field": "connector.name.keyword",
        "size": 10000
      }
    }
  }
}

We get

org.opensearch.core.xcontent.XContentParseException: unknown named object category [org.opensearch.search.aggregations.Aggregation]
        at org.opensearch.core.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:160) ~[opensearch-core-2.19.1-SNAPSHOT.jar:2.19.1-SNAPSHOT]
        at org.opensearch.core.xcontent.AbstractXContentParser.namedObject(AbstractXContentParser.java:429) ~[opensearch-core-2.19.1-SNAPSHOT.jar:2.19.1-SNAPSHOT]
        at org.opensearch.core.xcontent.XContentParserUtils.parseTypedKeysObject(XContentParserUtils.java:172) ~[opensearch-core-2.19.1-SNAPSHOT.jar:2.19.1-SNAPSHOT]
        at org.opensearch.search.aggregations.Aggregations.fromXContent(Aggregations.java:159) ~[opensearch-2.19.1-SNAPSHOT.jar:2.19.1-SNAPSHOT]
        at org.opensearch.action.search.SearchResponse.innerFromXContent(SearchResponse.java:431) ~[opensearch-2.19.1-SNAPSHOT.jar:2.19.1-SNAPSHOT]
        at org.opensearch.action.search.SearchResponse.fromXContent(SearchResponse.java:384) ~[opensearch-2.19.1-SNAPSHOT.jar:2.19.1-SNAPSHOT]
        at org.opensearch.ml.action.handler.MLSearchHandler.lambda$search$2(MLSearchHandler.java:145) [opensearch-ml-2.19.1.0-SNAPSHOT.jar:2.19.1.0-SNAPSHOT]
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) [?:?]
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) [?:?]
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) [?:?]
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) [?:?]
        at org.opensearch.remote.metadata.client.impl.LocalClusterIndicesClient.lambda$searchDataObjectAsync$15(LocalClusterIndicesClient.java:459) [opensearch-remote-metadata-sdk-2.19.1.0-SNAPSHOT.jar:?]

[xinyual]

I did more debug. In Line 362 LocalClusterIndicesClient, we get searchResponse and it looks good. But we construct a parser and when we parse the parser, it throws the error. Could you please take a check?

[xinyual]

Also, this will influence some front end feature opensearch-project/ml-commons-dashboards#410.

[dbwiddis]

I did more debug. In Line 362 LocalClusterIndicesClient, we get searchResponse and it looks good. But we construct a parser and when we parse the parser, it throws the error. Could you please take a check?

Can you post what this response is?

Can we fix this in AOS 2.19?

Unlikely. First we need to identify the bug. Then we can fix it. Given the imminent release of 2.19, I don't think that will happen.

Is this really the only way to query connectors by name? The SDK wasn't meant to replace a lot of complex searching in arbitrary documents, but for plugins with generally well defined and well known schema. Can't we filter on names without using named aggregations?

My current primary suspicion is that OpenSearch isn't set up to parse aggregations from JSON. This was a major blocker when we were developing the Extensions SDK and I've spent dozens of hours trying to solve it. The SDK assumes you can turn a search response into a JSON String and back into a parser. This may not be the case. I am not sure of any workaround.

I may be wrong. I welcome any and all help trying to identify what's wrong here. I don't know if there is a quick fix and I don't know if there's a way to implement the SDK while allowing any arbitray search queries to be used against it.

It would be nice to confirm that suspicion and identify any possible way to work around it.

[dhrubo-os]

This is affecting ml-commons dashboard: opensearch-project/ml-commons-dashboards#410

[dbwiddis]

This is affecting ml-commons dashboard: opensearch-project/ml-commons-dashboards#410

Apparently already worked around using hits: opensearch-project/ml-commons-dashboards#413

Root cause seems to be that SearchResponse.fromXContent() can't handle the JSON generated by a search response for the type of aggregations in this query.

We could, for the local client case, try to pass through the response without parsing but it would still break on remote client.

[dbwiddis]

But if we use

POST /_plugins/_ml/models/_search
{
  "size": 0,
  "aggs": {
    "terms#foo": {
      "terms": {
        "field": "connector.name.keyword",
        "size": 10000
      }
    }
  }
}

Looking at this StackOverflow question (for ElasticSearch, so pre-fork Opensearch) I see a few things:

• the user does custom namedXContent
• there appears to be built in namedXContent for things like filter#matched that I presume would not fail the parsing

[dbwiddis]

It looks like the SearchModule has most of the built-in aggregations here.

So for parsing we could possibly add all those to our parser?

NamedXContentRegistry registryWithSearchAggregations = new NamedXContentRegistry(
        new SearchModule(Settings.EMPTY, Collections.emptyList()).getNamedXContents()
);

(If that works at org.opensearch.ml.action.handler.MLSearchHandler.lambda$search$2(MLSearchHandler.java:146) we can add it to the util here as well.)