Skip to content

Conversation

varunbharadwaj
Copy link
Contributor

Description

Add documentation for all-active ingestion mode and related APIs in pull-based ingestion. This mode has been introduced as a document replication equivalent with additional fields added in the GetIngestionState API.

Issues Resolved

Closes #11121

Version

3.3 onwards

Checklist

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
    For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Varun Bharadwaj <[email protected]>
Copy link

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

Copy link
Collaborator

@kolchfa-aws kolchfa-aws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @varunbharadwaj! I provided suggestions based on my assumption that the ingestion mode depends entirely on the index’s replication type (segment vs. document replication). If that’s the case, what does the all_active parameter do? If an index is in segment replication, then you can't set all_active but if it's in document replication, what's the difference between all_active true and false?

@kolchfa-aws kolchfa-aws added Doc review PR: Doc review in progress release-notes PR: Include this PR in the automated release notes v3.3.0 labels Sep 29, 2025
@varunbharadwaj
Copy link
Contributor Author

Thank you, @varunbharadwaj! I provided suggestions based on my assumption that the ingestion mode depends entirely on the index’s replication type (segment vs. document replication). If that’s the case, what does the all_active parameter do? If an index is in segment replication, then you can't set all_active but if it's in document replication, what's the difference between all_active true and false?

Thanks for the quick review @kolchfa-aws . The all-active mode is only supported when the replication type is DOCUMENT (the default replication type today). We do not want to support this mode in segment replication mode. However, the all-active mode does not really do any replication at all, and therefore a different concept from replication type.

In all-active mode, the replicas also process the indexing updates similar to document replication mode, and hence i was thinking it can be portrayed as a doc-rep equivalent in pull-based ingestion.

@varunbharadwaj
Copy link
Contributor Author

If an index is in segment replication, then you can't set all_active but if it's in document replication, what's the difference between all_active true and false?

If all_active = true, then replication type must be document. If all_active = false, replication type must be segment.

@varunbharadwaj
Copy link
Contributor Author

Updated the ingestion mode descriptions and added an image for each mode for clarity.

Signed-off-by: Fanit Kolchina <[email protected]>
@kolchfa-aws kolchfa-aws added Editorial review PR: Editorial review in progress and removed Doc review PR: Doc review in progress labels Oct 1, 2025
Copy link
Collaborator

@natebower natebower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@natebower natebower removed the Editorial review PR: Editorial review in progress label Oct 1, 2025
@natebower natebower merged commit 85b3ade into opensearch-project:main Oct 1, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-notes PR: Include this PR in the automated release notes v3.3.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[DOC] Documentation for pull-based ingestion all-active mode

3 participants