[Pull-based Ingestion] Introduce ingestion management APIs #17631

varunbharadwaj · 2025-03-19T06:26:09Z

Description

This PR adds ingestion management APIs - mainly Pause, Resume and GetIngestionState APIs. This PR is a first version to introduce the APIs and subsequent PRs will build on this to support pagination and consumer reset options.

Related Issues

Part one of #17442

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-03-19T07:20:13Z

❌ Gradle check result for dcccf22: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-21T16:45:39Z

❌ Gradle check result for 77aa63a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Varun Bharadwaj <[email protected]>

github-actions · 2025-03-21T23:12:47Z

❌ Gradle check result for 7402d31: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Varun Bharadwaj <[email protected]>

github-actions · 2025-03-22T00:27:29Z

✅ Gradle check result for 5756fd0: SUCCESS

codecov · 2025-03-22T00:28:21Z

Codecov Report

Attention: Patch coverage is 54.02098% with 263 lines in your changes missing coverage. Please review.

Project coverage is 72.33%. Comparing base (6d53f9d) to head (060d76c).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...tadata/MetadataStreamingIngestionStateService.java	11.62%	38 Missing ⚠️
...ingestion/pause/TransportPauseIngestionAction.java	17.14%	29 Missing ⚠️
...gestion/resume/TransportResumeIngestionAction.java	17.14%	29 Missing ⚠️
...in/java/org/opensearch/index/shard/IndexShard.java	0.00%	18 Missing ⚠️
...reamingingestion/IngestionUpdateStateResponse.java	52.94%	16 Missing ⚠️
...streamingingestion/IngestionStateShardFailure.java	61.76%	13 Missing ⚠️
...mingingestion/state/GetIngestionStateResponse.java	43.47%	13 Missing ⚠️
.../streamingingestion/state/ShardIngestionState.java	72.50%	11 Missing ⚠️
...a/org/opensearch/index/engine/IngestionEngine.java	0.00%	10 Missing and 1 partial ⚠️
...ion/admin/indices/RestGetIngestionStateAction.java	23.07%	10 Missing ⚠️
... and 17 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17631      +/-   ##
============================================
- Coverage     72.40%   72.33%   -0.07%     
- Complexity    65828    65851      +23     
============================================
  Files          5316     5339      +23     
  Lines        305294   305905     +611     
  Branches      44289    44338      +49     
============================================
+ Hits         221033   221266     +233     
- Misses        66187    66495     +308     
- Partials      18074    18144      +70

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2025-03-22T06:14:25Z

❌ Gradle check result for 371e634: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-22T06:33:16Z

❌ Gradle check result for 7eccc3c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Varun Bharadwaj <[email protected]>

github-actions · 2025-03-22T07:45:57Z

❕ Gradle check result for 060d76c: UNSTABLE

TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testSnapshotWithStuckNode

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

yupeng9

overall lgtm

yupeng9 · 2025-03-22T15:42:16Z

.../java/org/opensearch/action/admin/indices/streamingingestion/IngestionStateShardFailure.java

+    @Override
+    public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
+        builder.startObject();
+        builder.field(SHARD, shard);


We skip index from XContent since shard level failures generally follow the format of indexName: {[<shard failures>]}. So the indexName will be used as a key from the container class holding this failure.

yupeng9 · 2025-03-22T16:22:17Z

.../opensearch/action/admin/indices/streamingingestion/pause/TransportPauseIngestionAction.java

+                public void onResponse(UpdateIngestionStateResponse updateIngestionStateResponse) {
+                    boolean shardsAcked = updateIngestionStateResponse.isAcknowledged()
+                        && updateIngestionStateResponse.getTotalShards() > 0
+                        && updateIngestionStateResponse.getFailedShards() == 0;


do we need this condition?

We are checking if the request was processed successfully on all individual shards in order to set shard level ack to true. If atleast 1 shard failed to explicitly update the state, we set this to false and add the shard in the failure list.

yupeng9 · 2025-03-22T16:28:46Z

...rg/opensearch/action/admin/indices/streamingingestion/state/UpdateIngestionStateRequest.java

+/**
+ * Holds metadata required for updating ingestion state.
+ *
+ * <p> This is for internal use only and will not be exposed to the user. </p>


how do we ensure user cannot access this?

Good question. Right now, I just create the transport action but do not register it as a REST action in the action module so users cannot call this externally.

yupeng9 · 2025-03-22T16:30:44Z

server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java

+            if (in.getVersion().onOrAfter(Version.V_3_0_0)) {
+                ingestionPaused = in.readBoolean();
+            } else {
+                ingestionPaused = false;


do we need this one? the default value of ingestionPaused is false?

It is a final variable and hence we have to initialize it explicitly to false. If we don't want to, we can make it non-final.

This was referenced Mar 19, 2025

[AUTOCUT] Gradle Check Flaky Test Report for MinimumClusterManagerNodesIT #14289

Open

[AUTOCUT] Gradle Check Flaky Test Report for SpecificClusterManagerNodesIT #15944

Open

varunbharadwaj force-pushed the vb/ingestionmgmt branch from 77aa63a to 7402d31 Compare March 21, 2025 23:05

Add ingestion management APIs

1b6951d

Signed-off-by: Varun Bharadwaj <[email protected]>

varunbharadwaj force-pushed the vb/ingestionmgmt branch from 7402d31 to b7617bf Compare March 21, 2025 23:18

Refactor and support shard level acks

5756fd0

Signed-off-by: Varun Bharadwaj <[email protected]>

varunbharadwaj force-pushed the vb/ingestionmgmt branch from b7617bf to 5756fd0 Compare March 21, 2025 23:24

varunbharadwaj changed the title ~~[WIP] [Pull-based Ingestion] Introduce ingestion management APIs~~ [Pull-based Ingestion] Introduce ingestion management APIs Mar 22, 2025

varunbharadwaj force-pushed the vb/ingestionmgmt branch from 5e33b66 to 371e634 Compare March 22, 2025 05:16

varunbharadwaj force-pushed the vb/ingestionmgmt branch from 371e634 to 7eccc3c Compare March 22, 2025 06:26

Add unit tests for ingestion management APIs

060d76c

Signed-off-by: Varun Bharadwaj <[email protected]>

varunbharadwaj force-pushed the vb/ingestionmgmt branch from 7eccc3c to 060d76c Compare March 22, 2025 06:47

yupeng9 reviewed Mar 22, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Pull-based Ingestion] Introduce ingestion management APIs #17631

[Pull-based Ingestion] Introduce ingestion management APIs #17631

varunbharadwaj commented Mar 19, 2025 •

edited

Loading

github-actions bot commented Mar 19, 2025

github-actions bot commented Mar 21, 2025

github-actions bot commented Mar 21, 2025

github-actions bot commented Mar 22, 2025

codecov bot commented Mar 22, 2025 •

edited

Loading

github-actions bot commented Mar 22, 2025

github-actions bot commented Mar 22, 2025

github-actions bot commented Mar 22, 2025

yupeng9 left a comment

yupeng9 Mar 22, 2025

varunbharadwaj Mar 22, 2025

yupeng9 Mar 22, 2025

varunbharadwaj Mar 22, 2025

yupeng9 Mar 22, 2025

varunbharadwaj Mar 22, 2025

yupeng9 Mar 22, 2025

varunbharadwaj Mar 22, 2025

[Pull-based Ingestion] Introduce ingestion management APIs #17631

Are you sure you want to change the base?

[Pull-based Ingestion] Introduce ingestion management APIs #17631

Conversation

varunbharadwaj commented Mar 19, 2025 • edited Loading

Description

Related Issues

github-actions bot commented Mar 19, 2025

github-actions bot commented Mar 21, 2025

github-actions bot commented Mar 21, 2025

github-actions bot commented Mar 22, 2025

codecov bot commented Mar 22, 2025 • edited Loading

Codecov Report

github-actions bot commented Mar 22, 2025

github-actions bot commented Mar 22, 2025

github-actions bot commented Mar 22, 2025

yupeng9 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

varunbharadwaj commented Mar 19, 2025 •

edited

Loading

codecov bot commented Mar 22, 2025 •

edited

Loading