Introduce Insights API #1610

jackiehanyang · 2025-11-11T06:12:29Z

Description

Introducing a new Insights API
- POST /_plugins/_anomaly_detection/insights/_start - Start insights job
- GET /_plugins/_anomaly_detection/insights/_status - Get insights job status
- GET /_plugins/_anomaly_detection/insights/_results - Get latest insights results
- POST /_plugins/_anomaly_detection/insights/_stop - Stop insights job
Introducing ml-commons metrics correlation runtime dependency
- sending anomaly results to ml-commons metrics correlation algorithm to analyze
- write analyze results into insights-results index
- frontend will read from this index to display insights on dashboard

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Jackie <[email protected]>

…nsights

Signed-off-by: Jackie <[email protected]>

kaituo · 2025-11-13T01:37:43Z

CI failed due to jacoco changes in build.gradle. Not sure how to fix. One naive way is to add correlation request, response, and Action in AD to avoid ml-commons dependency.

* What went wrong:
Execution failed for task ':jacocoTestCoverageVerification'.
> A failure occurred while executing org.gradle.internal.jacoco.JacocoCoverageAction
   > Rule violated for class org.opensearch.ad.AnomalyDetectorRunner: branches covered ratio is 0.35, but expected minimum is 0.60
     Rule violated for class org.opensearch.ad.AnomalyDetectorRunner: lines covered ratio is 0.47, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.util.ModelUtil: branches covered ratio is 0.32, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.util.ModelUtil: lines covered ratio is 0.48, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.util.DataUtil: lines covered ratio is 0.72, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.feature.AbstractRetriever: branches covered ratio is 0.55, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.feature.AbstractRetriever: lines covered ratio is 0.63, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.feature.SearchFeatureDao: branches covered ratio is 0.28, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.feature.SearchFeatureDao: lines covered ratio is 0.59, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.handler.ModelValidationActionHandler: branches covered ratio is 0.00, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.handler.ModelValidationActionHandler: lines covered ratio is 0.00, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.handler.ConfigUpdateConfirmer: branches covered ratio is 0.06, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.handler.ConfigUpdateConfirmer: lines covered ratio is 0.19, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.handler.AggregationPrep: branches covered ratio is 0.36, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.handler.AggregationPrep: lines covered ratio is 0.40, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.handler.IntervalCalculation: branches covered ratio is 0.06, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.handler.IntervalCalculation: lines covered ratio is 0.18, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.handler.LatestTimeRetriever: branches covered ratio is 0.00, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.handler.LatestTimeRetriever: lines covered ratio is 0.00, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.handler.IntervalCalculation.IntervalRecommendationListener: branches covered ratio is 0.37, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.handler.IntervalCalculation.IntervalRecommendationListener: lines covered ratio is 0.55, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.ratelimit.ColdStartWorker: branches covered ratio is 0.45, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.ratelimit.ColdStartWorker: lines covered ratio is 0.73, but expected minimum is 0.75
     Rule violated for class org.opensearch.ad.transport.SuggestAnomalyDetectorParamTransportAction: branches covered ratio is 0.00, but expected minimum is 0.60
     Rule violated for class org.opensearch.ad.transport.SuggestAnomalyDetectorParamTransportAction: lines covered ratio is 0.11, but expected minimum is 0.75
     Rule violated for class org.opensearch.ad.transport.ADSuggestName: branches covered ratio is 0.00, but expected minimum is 0.60
     Rule violated for class org.opensearch.ad.transport.ADSuggestName: lines covered ratio is 0.57, but expected minimum is 0.75
     Rule violated for class org.opensearch.ad.transport.ADResultProcessor: branches covered ratio is 0.00, but expected minimum is 0.60
     Rule violated for class org.opensearch.ad.transport.ADResultProcessor: lines covered ratio is 0.61, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.model.InitProgressProfile: branches covered ratio is 0.50, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.model.IntervalTimeConfiguration: branches covered ratio is 0.50, but expected minimum is 0.60
     Rule violated for class org.opensearch.ad.ml.MLCommonsClient: lines covered ratio is 0.62, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.RestValidateAction: branches covered ratio is 0.00, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.RestValidateAction: lines covered ratio is 0.26, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.RestJobAction: branches covered ratio is 0.00, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.RestJobAction: lines covered ratio is 0.25, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.AbstractSearchAction: lines covered ratio is 0.60, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.transport.SuggestConfigParamResponse: branches covered ratio is 0.59, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.transport.SuggestConfigParamRequest: branches covered ratio is 0.50, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.transport.SuggestConfigParamRequest: lines covered ratio is 0.68, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.transport.SuggestConfigParamResponse.Builder: lines covered ratio is 0.57, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.transport.handler.IndexMemoryPressureAwareResultHandler: branches covered ratio is 0.54, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.transport.handler.IndexMemoryPressureAwareResultHandler: lines covered ratio is 0.68, but expected minimum is 0.75
     Rule violated for class org.opensearch.ad.ratelimit.ADSaveResultStrategy: branches covered ratio is 0.43, but expected minimum is 0.60
     Rule violated for class org.opensearch.ad.ratelimit.ADSaveResultStrategy: lines covered ratio is 0.53, but expected minimum is 0.75

kaituo

partial review

src/main/java/org/opensearch/ad/InsightsJobProcessor.java

kaituo · 2025-11-13T02:50:01Z

src/main/java/org/opensearch/ad/InsightsJobProcessor.java

+
+        InjectSecurity injectSecurity = new InjectSecurity(jobParameter.getName(), settings, localClient.threadPool().getThreadContext());
+        try {
+            injectSecurity.inject(user, roles);


A normal user cannot query system index. Please add security tests.

updated in the new revision

src/main/java/org/opensearch/ad/InsightsJobProcessor.java

kaituo · 2025-11-13T02:56:55Z

src/main/java/org/opensearch/ad/constant/ADCommonName.java

+    // Insights job
+    // ======================================
+    // The Insights job name
+    public static final String INSIGHTS_JOB_NAME = "insights_job";


how about changing to ad_insights_job in case we need forecasting job later?

updated in the new revision

kaituo

partial review

kaituo · 2025-11-14T19:17:33Z

src/main/java/org/opensearch/ad/rest/RestInsightsJobAction.java

+        return ImmutableList
+            .of(
+                // Start insights job
+                new ReplacedRoute(


You don't need ReplaceRoute as this is a new API. TimeSeriesAnalyticsPlugin.AD_BASE_URI alone is enough.

updated in the new revision

kaituo · 2025-11-14T19:51:31Z

src/main/java/org/opensearch/ad/ml/InsightsGenerator.java

+        builder.startObject();
+
+        // Task metadata
+        builder.field("task_id", "task_" + ADCommonName.INSIGHTS_JOB_NAME + "_" + UUID.randomUUID().toString());


Why do you need task id? AD task id is the doc id of state index.

kaituo · 2025-11-14T20:08:22Z

src/main/java/org/opensearch/ad/ml/InsightsGenerator.java

+
+                if (parts.length > 1) {
+                    String seriesKey = parts[1];
+                    seriesKeys.add(seriesKey);


Is the entities set redundant with seriesKeys set?

We don't necessarily need it. Just followed the current practice to have this logical run identifier for the insights generation. Maybe it's useful in the future when integrate with Investigation so we can refer to a specific insights run this this id.

src/main/java/org/opensearch/ad/ml/MLMetricsCorrelationInputBuilder.java

kaituo · 2025-11-14T20:54:28Z

src/main/resources/mappings/insights-results.json

+            "fields": {
+              "raw": {
+                "type": "keyword",
+                "ignore_above": 32766
+              }


do you need keyword?

Not strictly, was following other AD index mapping to see how is it storing text field. Removed in the new revision

Signed-off-by: Jackie <[email protected]>

kaituo

partial review

kaituo · 2025-11-21T20:25:06Z

src/main/java/org/opensearch/ad/transport/InsightsJobTransportAction.java

+            handleStartOperation(request, listener);
+        } else if (request.isStatusOperation()) {
+            handleStatusOperation(request, listener);
+        } else if (request.isStopOperation()) {
+            handleStopOperation(request, listener);


Do you need to stash context before accessing job index (system index)? Please add security tests.

yes, I'm using stashed context when accessing job index, will add security tests

kaituo · 2025-11-21T20:25:43Z

src/main/java/org/opensearch/ad/transport/InsightsJobTransportAction.java

+                        .sort("generated_at", SortOrder.DESC)
+                );
+
+            client.search(searchRequest, ActionListener.wrap(searchResponse -> {


Do you need to add backend role filtering before search? Please add security tests with backend role filtering on.

We need to do tenant-isolated search, but not necessarily backend role filtering here. For insights generation, it's a background job, so followed existing pattern to use InjectSecurity directly for background work, just impersonate the stored user via InjectSecurity, then execute search directly. For user-facing search APIs like search anomaly result transport action, AD reads the current user from thread context and then adds backend role filtering.

Adding security tests in the next revision

kaituo · 2025-11-21T22:51:49Z

src/main/java/org/opensearch/ad/InsightsJobProcessor.java

+
+    private static final Logger log = LogManager.getLogger(InsightsJobProcessor.class);
+
+    private static InsightsJobProcessor INSTANCE;


add volatile. read https://en.wikipedia.org/wiki/Double-checked_locking

good catch, we should do the same for ADJobProcessor and ForecastJobProcessor too

kaituo · 2025-11-21T23:18:08Z

src/main/java/org/opensearch/ad/InsightsJobProcessor.java

+        try {
+            injectSecurity.inject(user, roles);
+
+            localClient


We should verify if mapping is changed by customer before writing. If yes, report error/stop job and stop writing.

good catch, updated in the new revision

kaituo · 2025-11-21T23:21:10Z

src/main/java/org/opensearch/ad/rest/handler/InsightsJobActionHandler.java

+                Instant.now(),
+                lockDurationSeconds,
+                user,
+                ADCommonName.INSIGHTS_RESULT_INDEX_ALIAS,


Can you add this index in ADIndex? This would be consistent with other indexes.

already added, check file in diff - src/main/java/org/opensearch/ad/indices/ADIndex.java

Signed-off-by: Jackie <[email protected]>

kaituo

partial review

src/main/java/org/opensearch/ad/rest/handler/InsightsJobActionHandler.java

kaituo · 2025-11-24T23:01:21Z

src/main/java/org/opensearch/ad/rest/handler/InsightsJobActionHandler.java

+    private void createNewJob(String frequency, User user, ActionListener<InsightsJobResponse> listener) {
+        try {
+            IntervalSchedule schedule = createSchedule(frequency);
+            long lockDurationSeconds = java.time.Duration.of(schedule.getInterval(), schedule.getUnit()).getSeconds() * 2;


lockDurationSeconds should be less than or equal to frequency, right?

yes good catch, updated it to

long lockDurationSeconds = java.time.Duration.of(schedule.getInterval(), schedule.getUnit()).getSeconds();

kaituo · 2025-11-24T23:04:10Z

src/main/java/org/opensearch/ad/rest/handler/InsightsJobActionHandler.java

+                            Instant.now(),
+                            null,
+                            Instant.now(),
+                            existingJob.getLockDurationSeconds(),


When you update frequency, you need to update lock duration as well.

good catch, updated in the new revision

kaituo · 2025-11-24T23:06:16Z

src/main/java/org/opensearch/ad/rest/handler/InsightsJobActionHandler.java

+                            null,
+                            Instant.now(),
+                            existingJob.getLockDurationSeconds(),
+                            user != null ? user : existingJob.getUser(),


If we allow overwrite user, it would enable malicious user to update a job with. new user and thus cause original user to lose access.

good catch, updated it to preserve the original job user and only fall back to the current request user if the job has no user

kaituo · 2025-11-24T23:14:32Z

src/main/java/org/opensearch/ad/InsightsJobProcessor.java

+
+        log.info("Running Insights job for time window: {} to {}", executionStartTime, executionEndTime);
+
+        querySystemResultIndex(jobParameter, lockService, lock, executionStartTime, executionEndTime);


Only querying system result index would hinder your ability to go GA alone. You have to tie insights with Auto AD creation. One route to go to GA is to add a text box in AD overview page. That would add a summary on top of existing detectors' results.

jackiehanyang added 3 commits November 6, 2025 10:48

initial commit

64021b5

Signed-off-by: Jackie <[email protected]>

Merge remote-tracking branch 'origin/main' into tmp/merge-main-into-i…

45e795f

…nsights

Introduce Insights API

b14da81

Signed-off-by: Jackie <[email protected]>

jackiehanyang requested review from VijayanB, amitgalitz, dbwiddis, jmazanec15, jngz-es, joshpalis, kaituo, ohltyler, owaiskazi19, saratvemulapalli, sean-zheng-amazon, vamshin and ylwu-amzn as code owners November 11, 2025 06:12

opensearch-trigger-bot bot added infra Changes to infrastructure, testing, CI/CD, pipelines, etc. backport 2.x labels Nov 11, 2025

jackiehanyang added 4 commits November 10, 2025 22:28

spotless apply

ed68a6d

Signed-off-by: Jackie <[email protected]>

add change log

200d461

Signed-off-by: Jackie <[email protected]>

fix

671e103

Signed-off-by: Jackie <[email protected]>

make timestamp more user friendly when reading

77fcda9

Signed-off-by: Jackie <[email protected]>

jackiehanyang force-pushed the insights branch from 4883d42 to 77fcda9 Compare November 11, 2025 20:00

jackiehanyang added 2 commits November 11, 2025 12:11

fix forbidden api violation by avoiding default-locale usage

3d50179

Signed-off-by: Jackie <[email protected]>

add more tests

49018e0

Signed-off-by: Jackie <[email protected]>

jackiehanyang force-pushed the insights branch from 0eba385 to 49018e0 Compare November 12, 2025 17:31

kaituo reviewed Nov 13, 2025

View reviewed changes

kaituo reviewed Nov 14, 2025

View reviewed changes

use stashed context when starting insights job

816d69b

Signed-off-by: Jackie <[email protected]>

kaituo reviewed Nov 22, 2025

View reviewed changes

add one time insights job run

1356920

Signed-off-by: Jackie <[email protected]>

jackiehanyang force-pushed the insights branch from 3076f15 to 1356920 Compare November 22, 2025 09:20

do immediate one time run job right after starting insights job

b731aef

Signed-off-by: Jackie <[email protected]>

kaituo reviewed Nov 25, 2025

View reviewed changes


		private static final Logger log = LogManager.getLogger(InsightsJobProcessor.class);

		private static InsightsJobProcessor INSTANCE;


		log.info("Running Insights job for time window: {} to {}", executionStartTime, executionEndTime);

		querySystemResultIndex(jobParameter, lockService, lock, executionStartTime, executionEndTime);

Introduce Insights API #1610

Are you sure you want to change the base?

Introduce Insights API #1610

Conversation

jackiehanyang commented Nov 11, 2025

Description

Related Issues

Check List

Uh oh!

kaituo commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kaituo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaituo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaituo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaituo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

kaituo commented Nov 13, 2025 •

edited

Loading