From b23983c002cbd1793f6c5bf0a9ce576fa96bc077 Mon Sep 17 00:00:00 2001 From: Varun Bharadwaj Date: Sun, 28 Sep 2025 17:35:42 -0700 Subject: [PATCH 1/9] add all-active ingestion mode Signed-off-by: Varun Bharadwaj --- .../pull-based-ingestion-management.md | 6 +++++- .../document-apis/pull-based-ingestion.md | 20 +++++++++++++++++-- 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/_api-reference/document-apis/pull-based-ingestion-management.md b/_api-reference/document-apis/pull-based-ingestion-management.md index c2608eb0923..39e30f86def 100644 --- a/_api-reference/document-apis/pull-based-ingestion-management.md +++ b/_api-reference/document-apis/pull-based-ingestion-management.md @@ -315,7 +315,11 @@ response = client.ingestion.get_state( "shard": 0, "poller_state": "POLLING", "error_policy": "DROP", - "poller_paused": false + "poller_paused": false, + "write_block_enabled" : false, + "batch_start_pointer" : "KafkaOffset{offset=2}", + "is_primary" : true, + "node" : "node_name" } ] } diff --git a/_api-reference/document-apis/pull-based-ingestion.md b/_api-reference/document-apis/pull-based-ingestion.md index 1153b7646ba..d0c7a23820b 100644 --- a/_api-reference/document-apis/pull-based-ingestion.md +++ b/_api-reference/document-apis/pull-based-ingestion.md @@ -22,12 +22,11 @@ Before using pull-based ingestion, ensure that the following prerequisites are m * Install an ingestion plugin for your streaming source using the command `bin/opensearch-plugin install `. For more information, see [Additional plugins]({{site.url}}{{site.baseurl}}/install-and-configure/additional-plugins/index/). OpenSearch supports the following ingestion plugins: - `ingestion-kafka` - `ingestion-kinesis` -* Enable [segment replication]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/segment-replication/index/) with [remote-backed storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/). Pull-based ingestion is not compatible with document replication. * Configure pull-based ingestion during [index creation](#creating-an-index-for-pull-based-ingestion). You cannot convert an existing push-based index to a pull-based one. ## Creating an index for pull-based ingestion -To ingest data from a streaming source, first create an index with pull-based ingestion settings. The following request creates an index that pulls data from a Kafka topic: +To ingest data from a streaming source, first create an index with pull-based ingestion settings. The following request creates an index that pulls data from a Kafka topic in segment replication mode. See [ingestion modes](#ingestion-modes) for other available modes. ```json PUT /my-index @@ -75,8 +74,25 @@ The `ingestion_source` parameters control how OpenSearch pulls data from the str | `poll.timeout` | The maximum time to wait for data in each poll operation. Optional. | | `num_processor_threads` | The number of threads for processing ingested data. Optional. Default is 1. | | `internal_queue_size` | The size of the internal blocking queue for advanced tuning. Valid values are from 1 to 100,000, inclusive. Optional. Default is 100. | +| `all_active` | Boolean value determining if all-active ingestion mode should be enabled. This is not supported in segment replication mode. Default value is false. See [ingestion modes](#ingestion-modes) | | `param` | Source-specific configuration parameters. Required.
 • The `ingest-kafka` plugin requires:
  - `topic`: The Kafka topic to consume from
  - `bootstrap_servers`: The Kafka server addresses
  Optionally, you can provide additional standard Kafka consumer parameters (such as `fetch.min.bytes`). These parameters are passed directly to the Kafka consumer.
 • The `ingest-kinesis` plugin requires:
  - `stream`: The Kinesis stream name
  - `region`: The AWS Region
  - `access_key`: The AWS access key
  - `secret_key`: The AWS secret key
  Optionally, you can provide an `endpoint_override`. | + +### Ingestion modes + +Pull-based ingestion supports the following modes: + +#### Segment replication mode + +Pull-based ingestion can be used in the [segment replication]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/segment-replication/index/) mode. +In this mode, the primary shards ingest events from a streaming source and index the documents. Segments are then copied over to the replica shards. +It is recommended to use this mode with a [remote-backed storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/). + +#### All-active mode + +All-active mode is the document replication equivalent in pull-based ingestion. Primary and replica shards ingest events from the streaming source and index documents independently. +There is no replication or coordination between the primary and replica shards. Replica shards however may depend on the primary shard during bootstrapping to copy the segment files if a local copy is unavailable. + ### Stream position When creating an index, you can specify where OpenSearch should start reading from the stream by configuring the `pointer.init.reset` and `pointer.init.reset.value` settings in the `ingestion_source` parameter. OpenSearch will resume reading from the last commited position for existing indexes. From e1d4e51634f86e103ff9753540daff3e95f83e5f Mon Sep 17 00:00:00 2001 From: Varun Bharadwaj Date: Mon, 29 Sep 2025 14:35:02 -0700 Subject: [PATCH 2/9] Update _api-reference/document-apis/pull-based-ingestion.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Varun Bharadwaj --- _api-reference/document-apis/pull-based-ingestion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_api-reference/document-apis/pull-based-ingestion.md b/_api-reference/document-apis/pull-based-ingestion.md index d0c7a23820b..da9dc940554 100644 --- a/_api-reference/document-apis/pull-based-ingestion.md +++ b/_api-reference/document-apis/pull-based-ingestion.md @@ -26,7 +26,7 @@ Before using pull-based ingestion, ensure that the following prerequisites are m ## Creating an index for pull-based ingestion -To ingest data from a streaming source, first create an index with pull-based ingestion settings. The following request creates an index that pulls data from a Kafka topic in segment replication mode. See [ingestion modes](#ingestion-modes) for other available modes. +To ingest data from a streaming source, first create an index with pull-based ingestion settings. The following request creates an index that pulls data from a Kafka topic in segment replication mode. For other available modes, see [Ingestion modes](#ingestion-modes). ```json PUT /my-index From b2c599423ecd7a4fbaecb000eca2a943f6481b7d Mon Sep 17 00:00:00 2001 From: Varun Bharadwaj Date: Mon, 29 Sep 2025 14:35:54 -0700 Subject: [PATCH 3/9] Update _api-reference/document-apis/pull-based-ingestion.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Varun Bharadwaj --- _api-reference/document-apis/pull-based-ingestion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_api-reference/document-apis/pull-based-ingestion.md b/_api-reference/document-apis/pull-based-ingestion.md index da9dc940554..52d5b6ed349 100644 --- a/_api-reference/document-apis/pull-based-ingestion.md +++ b/_api-reference/document-apis/pull-based-ingestion.md @@ -74,7 +74,7 @@ The `ingestion_source` parameters control how OpenSearch pulls data from the str | `poll.timeout` | The maximum time to wait for data in each poll operation. Optional. | | `num_processor_threads` | The number of threads for processing ingested data. Optional. Default is 1. | | `internal_queue_size` | The size of the internal blocking queue for advanced tuning. Valid values are from 1 to 100,000, inclusive. Optional. Default is 100. | -| `all_active` | Boolean value determining if all-active ingestion mode should be enabled. This is not supported in segment replication mode. Default value is false. See [ingestion modes](#ingestion-modes) | +| `all_active` | Whether to enable the all-active ingestion mode. Cannot be enabled for indexes that use segment replication mode. Default is `false`. See [Ingestion modes](#ingestion-modes) | | `param` | Source-specific configuration parameters. Required.
 • The `ingest-kafka` plugin requires:
  - `topic`: The Kafka topic to consume from
  - `bootstrap_servers`: The Kafka server addresses
  Optionally, you can provide additional standard Kafka consumer parameters (such as `fetch.min.bytes`). These parameters are passed directly to the Kafka consumer.
 • The `ingest-kinesis` plugin requires:
  - `stream`: The Kinesis stream name
  - `region`: The AWS Region
  - `access_key`: The AWS access key
  - `secret_key`: The AWS secret key
  Optionally, you can provide an `endpoint_override`. | From 3572f2728d8c0ada021c193be1510ef6079c7097 Mon Sep 17 00:00:00 2001 From: Varun Bharadwaj Date: Mon, 29 Sep 2025 14:36:07 -0700 Subject: [PATCH 4/9] Update _api-reference/document-apis/pull-based-ingestion.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Varun Bharadwaj --- _api-reference/document-apis/pull-based-ingestion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_api-reference/document-apis/pull-based-ingestion.md b/_api-reference/document-apis/pull-based-ingestion.md index 52d5b6ed349..88dd79e8f9e 100644 --- a/_api-reference/document-apis/pull-based-ingestion.md +++ b/_api-reference/document-apis/pull-based-ingestion.md @@ -80,7 +80,7 @@ The `ingestion_source` parameters control how OpenSearch pulls data from the str ### Ingestion modes -Pull-based ingestion supports the following modes: +Pull-based ingestion supports the following modes. #### Segment replication mode From cf767776094d47a8dcf272d59b51d242a0e7baa5 Mon Sep 17 00:00:00 2001 From: Varun Bharadwaj Date: Mon, 29 Sep 2025 15:22:21 -0700 Subject: [PATCH 5/9] update ingestion mode names Signed-off-by: Varun Bharadwaj --- .../document-apis/pull-based-ingestion.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/_api-reference/document-apis/pull-based-ingestion.md b/_api-reference/document-apis/pull-based-ingestion.md index 88dd79e8f9e..c29eabafc85 100644 --- a/_api-reference/document-apis/pull-based-ingestion.md +++ b/_api-reference/document-apis/pull-based-ingestion.md @@ -26,7 +26,7 @@ Before using pull-based ingestion, ensure that the following prerequisites are m ## Creating an index for pull-based ingestion -To ingest data from a streaming source, first create an index with pull-based ingestion settings. The following request creates an index that pulls data from a Kafka topic in segment replication mode. For other available modes, see [Ingestion modes](#ingestion-modes). +To ingest data from a streaming source, first create an index with pull-based ingestion settings. The following request creates an index that pulls data from a Kafka topic in the default ingestion mode. For other available modes, see [Ingestion modes](#ingestion-modes). ```json PUT /my-index @@ -82,16 +82,16 @@ The `ingestion_source` parameters control how OpenSearch pulls data from the str Pull-based ingestion supports the following modes. -#### Segment replication mode +#### Default mode -Pull-based ingestion can be used in the [segment replication]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/segment-replication/index/) mode. -In this mode, the primary shards ingest events from a streaming source and index the documents. Segments are then copied over to the replica shards. -It is recommended to use this mode with a [remote-backed storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/). +In the default mode, the primary shards ingest events from a streaming source and index the documents. The pull-based index is configured to use [segment replication]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/segment-replication/index/) to copy over the segment files from primary to replica shards. +We recommend using this mode with a [remote-backed storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/). #### All-active mode -All-active mode is the document replication equivalent in pull-based ingestion. Primary and replica shards ingest events from the streaming source and index documents independently. -There is no replication or coordination between the primary and replica shards. Replica shards however may depend on the primary shard during bootstrapping to copy the segment files if a local copy is unavailable. +Enabling all-active mode allows both primary and replica shards to independently ingest and index events from the streaming source. +There is no replication or coordination between the shards, although replica shards may fetch segment files from the primary shard during bootstrapping if a local copy is unavailable. +This mode is currently not supported along with segment replication. ### Stream position From cd17c2728bf52dc4e0a4635993599e1fe23ccf68 Mon Sep 17 00:00:00 2001 From: Varun Bharadwaj Date: Tue, 30 Sep 2025 11:17:55 -0700 Subject: [PATCH 6/9] update the ingestion mode names Signed-off-by: Varun Bharadwaj --- _api-reference/document-apis/pull-based-ingestion.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_api-reference/document-apis/pull-based-ingestion.md b/_api-reference/document-apis/pull-based-ingestion.md index c29eabafc85..33f1e0a4e10 100644 --- a/_api-reference/document-apis/pull-based-ingestion.md +++ b/_api-reference/document-apis/pull-based-ingestion.md @@ -26,7 +26,7 @@ Before using pull-based ingestion, ensure that the following prerequisites are m ## Creating an index for pull-based ingestion -To ingest data from a streaming source, first create an index with pull-based ingestion settings. The following request creates an index that pulls data from a Kafka topic in the default ingestion mode. For other available modes, see [Ingestion modes](#ingestion-modes). +To ingest data from a streaming source, first create an index with pull-based ingestion settings. The following request creates an index that pulls data from a Kafka topic in the segment replication mode. For other available modes, see [Ingestion modes](#ingestion-modes). ```json PUT /my-index @@ -82,9 +82,9 @@ The `ingestion_source` parameters control how OpenSearch pulls data from the str Pull-based ingestion supports the following modes. -#### Default mode +#### Segment replication mode -In the default mode, the primary shards ingest events from a streaming source and index the documents. The pull-based index is configured to use [segment replication]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/segment-replication/index/) to copy over the segment files from primary to replica shards. +The primary shards ingest events from a streaming source and index the documents. The pull-based index is configured to use [segment replication]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/segment-replication/index/) to copy over the segment files from primary to replica shards. We recommend using this mode with a [remote-backed storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/). #### All-active mode From b8fcb48c713b8c219d02ebeb95a630f6465716d1 Mon Sep 17 00:00:00 2001 From: Varun Bharadwaj Date: Tue, 30 Sep 2025 13:22:40 -0700 Subject: [PATCH 7/9] include images for ingestion modes Signed-off-by: Varun Bharadwaj --- .../document-apis/pull-based-ingestion.md | 4 ++++ .../pull-based-all-active-mode.png | Bin 0 -> 10188 bytes .../pull-based-segrep-mode.png | Bin 0 -> 10786 bytes 3 files changed, 4 insertions(+) create mode 100644 images/pull-based-ingestion/pull-based-all-active-mode.png create mode 100644 images/pull-based-ingestion/pull-based-segrep-mode.png diff --git a/_api-reference/document-apis/pull-based-ingestion.md b/_api-reference/document-apis/pull-based-ingestion.md index 33f1e0a4e10..668091b24c8 100644 --- a/_api-reference/document-apis/pull-based-ingestion.md +++ b/_api-reference/document-apis/pull-based-ingestion.md @@ -84,11 +84,15 @@ Pull-based ingestion supports the following modes. #### Segment replication mode +![Pull-based ingestion segment replication mode]({{site.url}}{{site.baseurl}}/images/pull-based-ingestion/pull-based-segrep-mode.png){: width="50%" } + The primary shards ingest events from a streaming source and index the documents. The pull-based index is configured to use [segment replication]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/segment-replication/index/) to copy over the segment files from primary to replica shards. We recommend using this mode with a [remote-backed storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/). #### All-active mode +![Pull-based ingestion all active mode]({{site.url}}{{site.baseurl}}/images/pull-based-ingestion/pull-based-all-active-mode.png){: width="50%" } + Enabling all-active mode allows both primary and replica shards to independently ingest and index events from the streaming source. There is no replication or coordination between the shards, although replica shards may fetch segment files from the primary shard during bootstrapping if a local copy is unavailable. This mode is currently not supported along with segment replication. diff --git a/images/pull-based-ingestion/pull-based-all-active-mode.png b/images/pull-based-ingestion/pull-based-all-active-mode.png new file mode 100644 index 0000000000000000000000000000000000000000..235c7001a501b3c3278752d9b1e8ce32de3f56e0 GIT binary patch literal 10188 zcmd^lXEa>V+wZ8+YoeDZQ4=*nh!LV^5QFF?YV=-WFa$vmz4tPi=tgJMAbKYVL-gKz z8Q#hN-gVdg@_xCW?pp6&=UHdY@7d4ZXYX^KIs5lKJK~+X5-AY_5dZ)nRZ*4)0{}SG z0034#0R{jxN_2T*PLC~=!D;}2FBbq17y=8DioS>GK1pv(9tSSp^ui5T#Khl-lQgDpaqheSlOMnA{_0Im!b zd0B0rKYJ)>ILAiT#*sIx2oro(x*Eb@r^i@ATitc!RU14_vbAqzZnPJh9i+b^lxGK$ zsFrfE82K?G4ow~WVp}rXwae=M!5~XQDC9MrN@0X6$$Lgfjl`Qmef*wMc+DM_{d>tw zA-N`$;fpe<&$uIKO>lj7_P%YA4p_$vB_1AX~BRgPe6O$HYQV3Si6 zVGB|JA1ne?9yndS&z+jJ$eT##Ao~$|qiP=+wPfpGNA+$W?ed0_sILnP;mfqM_wF8C?$^lhji#UOCa^(q*_RIJG8cItbz7Fph!r1! z!!b~ln^Boy8`(zOMEW>Qf=BgGZ`5&S?4+5+gZj0I>$}9teA)UBN?HxF;#i#250rRb zZ0!=p6S8^ye<=}~F26)%#br-HB9wNXJ%mk!4`M9YCmCiPM%5&IIH~i?C_Qj~%83X| zT3D1yTBsxZtK(jZWZ2e(F_d-B3d3iPS$Lmmu;@Rn`wapmZbr})26Y9`w=ADO*@eem{9{Jas#BQ@=k$#p@xM?U@;s32RspMdyu74 zRCS>=8%9%zn7%iG9!#LL3MhFhA$^<4%7|CftrAX>8ZLMvZWg{7H5w_mZ|!F<8!heinz^^^rmIY= z(N9Hk80K|Sdhn2ksLOodhb5}07S*2DdoYZvJ~A7E@Z@QeY4J`CuKo7AU7x&(r~<2s zJp%`{_ST;J4ZKSZkUQN}2o#+UksOq}ftJRG`;z;UkT27)iHOrOJ6W&`x;}DY`g)?? zg05b0-G+7dt>KqVUG;&Gth`oC+*(1=MQ^hlyI|4L-OlFSSj(ZO&MlDlv{_T<6h#Qh zgQql%fB|(x)n+ss-^odquy!ggb3abrgvRqEJ81BC;pQ-san;$>I>)&>JXK!mDEV^f zy~FE>b6Uy5KsN}{0gd7We!GrpQR0P)itmUT&YBnx4=hnNc5|>UH`=d%W+9vxK6&la z=fDSZpu(R~edGY(cvKg_{xEvxLNe{0YMIT?W0opqeyfj*nnN{MbhcuvpjDwnaA2B1 zMVZa#kPJ~_m8O@Gh2Ua9-{*`|Z0yvcP;cwr=^(TrA)HI*|`Rg2PtlK9~y zRI0p;QH49jBC$_XF-{Y{KP+#9yA;j8JJhy2u;6{+BDxgKEy>!a&h-?gjYXwegsrGm zl$o@h{b#_fW9QT@YhRrgMubln;q|mX_SnQ!t;k^f{QHX}u-6T2F4bPO$YWE4Z6(6& zmeP-4@BRp{+)HMLJ)qp|Fd^?5dPNmh39(c}UgW_+0o{9av=#F8%JO}5JU3{7~ ztN_10@W9n;C2b-H9=$z2X!%`Q{t};}wt)=smfkrdnK+OculLyPp|k7~dm4e_{Bo#y zEb1PfyQKC|?mBVB(Jjv=EK+x};`zP{1}uY516E z@&uLkc-hZHnQnjU0Nbn^knJq0npN(9SBHb`mbSKj}MY0cLyBffQDxOA96SFk`g z^s?dN`>7Q&;twc3{q{%41NRhmN#1R@3X&a+V3k#RQZLbMvMQH@|9P)s>OvvHp78P1 zMVyMEqxbgwc6L5mZ(3%L=EYXPy#1#?b2f-pf`V$^vPuN2lv-2EjtMWLwhc!n_`7$Y zj%=M2VT^^qi(FSk3*~nk1b=zQRS!Rh{m0355DamBX?(#gF(hj2oheeRm8NRzO62Z4 zQ6uwL3a034B15~Xc1&)qIy~5GqB8QCY)w`N0E)c(=Bow?lY<4yLyf!lZwRMY3=K1` z3D@dk4vkZG{Js;}X~~S8dZ}{N2FR^}037sn0dl{nvC_OuhnnbYa=u<174jNAcH8et zyB+V3S@&xEnDN7F;3QVM_DJtShPpP#WNp^@j_UaUS&N4PAcFf1o!7AjMHd@xEa+UVH>!z^wjt%_koX88`+1)pDS4@* zDT}#VpBSsw%Ex(!NtWz2s^aBMfb}#B>WlP!Dq`Xh(CG_}F`Pbmp0iO|hla~0!+6l8 zk_aU*i|nW5C-#Mb(>NPcec^2bl#_Z1481p?X_xTaMr4cM%M3zf>mf36pb({<1aXst zlHi#`%TdQg4Nht{L}?zFqOb(e17P5EcETmErX545o%`iRry7OS{C) z_6V!$fk4goA*^~B82w+VQM)mxX$)pkL#GeSJlBlDqM;sM@#)*HJC%-o7itA!=i%@`d_@sqhG}}oTc2nW!2`vQk;`H0Vg$w>vy}|CWplqWpQEL_y|qCp=Di82G4NMCj2d~FS6TS zQ9s2TOxZQ{v!$N{fdRcmKJ~B+I{dyao=Z*68UBwe#_gO^*;tsm@*=`SP@zo{zJcm6 z>*V58-ufc#j#W9Ff2v>9dEVYFpSLo|bp=f@9V>HE7a-=Szt;-C6?{EBwzbC}$dP!m+i; z&xqDb{gT>JoXqJk13H?NJLhgG+cqKob9?)%mUcyYNb^)hM3_oxDS_qNo#DabVPLJ6 zY{uri(6>jF%3O+^=4*>HY}Boj{1iYsSuQ?tnVVA%@l9Gu+!x&Y(59Y8mwtC=kg3j# z5|g0Aj>^Tr-^j&3F|L5eZ)xINk{WZmQBL-|z7Ia+#FDX-e94HoUVMtRuKR7_zB||S zhoHe?+j-kko8J{L%SO;*^4*zrawy)(8p{jb?x^h>LsGyOSIJ_!Scd%Pud7}SDnupP z1HAHfvaN@)OaBaysF^#q+Ly%&0V>_42hD_RcjL;7jd{s0R%S}rUgTQ`X@0~<6Zubo z07bP(6Kg8vgUem)M5J?y77f69oVlAX9(3yD6WxmyfleUM5v|{$Fe*qX|Jk)$8eMCI z3|3zK?d6-UL9}$)fHGtd-Q?mC>jFJ&cL@&g1QyfJUEmMl(n*Ep&zrD0^`AL_`R~P9v8Sw62UK9e`-QQ``M|3 ztP~{SrBB8|UDji~a^nAxkcO70$2-&ROrr4}Yf61<=REg%K|q~T{{dA5@@=tq1LW%-df0cj-yU%Dtt`#-@230?o?3;mz_W%j>Vuss=o0VIT?)JChY_tr zk;dIg-^@o&q^Xmh3gV;lZzpv>AR2~BWX1JJ`Y%AYxbBjhv}h^UfRkvO~_%;zit}er*Eb9Nazl{9qy9iJfC23a56hR z(>i3-#!QPo$*&DiP4e}P0cnf(wV$+bLj7l_YPzVAv`btT;_D|@#RpM2OGCMHPw`jW zxm5Muyj+G0fAvI!sxH#YC<1fqh;lA7BkC_t_A|@lDmv1Y)8KQx2gG>T`1=jyUWSxg}3yaZ9QBo@f7cRNlX3oCXxF}JcmZ$^OCcS zX3&EdPG#CdI}{n#OQEtcQ9FACe5|xH=PVZ&H;F|KGAZd4ODi`BW@9$w=mN}N80}WD zfBWp->=%v!vq#>cW1x8y{Lh996{L+_I5YwJ3W z>_K|B&hU9fvoz5XyfgLU+R!y*bPu$G6JYpGQW`UkURF+~sbkDvLwXx8ebym+w23o* z>#rh$^ok_caRV%6WJ!q5tf7HE3;}8{VN-2*M6>@bURH^5{(U{xFbmASB`C=slR zSsaK?>3H`2wzqy175&6x`|oUF5}(CfAlTmI8IVYoS7$%H*+T26)uaKE*Oc1upiOkl zIA|085Nsk~Z_bpL`+mfe)6bpvr6I z6DqD{;0bgLda<4kE+z>@ipGH#-ZY-QFY8P4{_HJH!7ZOAYQ4y ze*gFYkY6vsh?W1M?eitf9RLN<1h(Pl)7Ugao9arkv3w#*-W46|QvSW#qfK2~AXZ4_+!h-#Em z*bTQ0{)O=l;*v98{3+}`oc6!!w>0Ryj2srGm<7REQl@S`P(C54*us3ONWd8NTo#`1 zr|a=LB}2civ9mV*URiqVL#W{{jZIp2fMQ)$cVUqmqc)B5!~Wv>V>$px^Q2;00atH)CQw4|h)&;(=gK%jzJRVcT%zm<{|b@`{?r$IT&52nmmYH7G?XaL{%U;U$^Tw!u16sE~oLST(;e7yIjJqI>c*(_BW_o}*>0inBxHGybXB8GU>ZV`;e~VgcKI4=*`L z+vYw*)r=E`s|YbHy})JM9lnbY!{N~Ja@^bI^CH(>`_pw^ck)$wV8d|>#dwjleH(r# zV2}hLVnho4xk2>t!qOLV`hX&Nzrpx)TOJqi_oikS0S*V(QJmujTs7#Kg@@N?epH;- z#(38E!c=+7CtloAH{`|-LQDnAC#$`76TPRC+I&|g7B|=+{Scw^j{Y*!SJ(5@!jH$B zYmXysa;_6Ju)a4t7HGYyxr@2roP#{J!=aG&F2B5?{>c4)lQ`20>U&a9;WvdWz3IwY z?q>yho+hJ5ubCeE`k!T)`oa8%awA4`7p^z?P5tw4;3Jv=V;c+~8$nk{_3rR1BS!bD z+umXVNQ#-WwB9okM7Fk7^Wxi$(|Zz15ukG-!3 z+{~QcaMxI?%p2s{6PjO{-5l^RL-S|mM~cu9p9v{|!mcsUCjWk9)JcsNR7I@AFE&qQ zKXT%0Iw5|{#5hDm(YfqhhMrpam8DtRa@N--^I~@gdy4$Rdk8kQLbA2pZ?1g{tXrYe zxi7ZfeJmm@efrj_8x$};a=OU=vN%QXOV*FXBtGt*p{6{Iy_$YJX$BQF5 zKCU}Bcju+>+3nNFJ)^Fu2mirEg&o~o4;w?2IQv4grM@aAv*M!6gn8~qs&6`gJ3IpK z_8JQ$sz(Hr-G#1lB5d8Ej(IU$jiVNgf4+K9-*iw6_p~Yoh;62rI2p67=W$qmAvy;reh`MEn3{pjoRAX(lf8&%msU0dRi zDEG2#+vD6L+E1k^*X5MbBI1kv;{(p`?@70ATR1s`fnlQ>pIIU8$RHz-vZO9iEm92s;vH2 zyi@%XNyf|3XmV?%m3KWxXKj3GjV79vI@%v$zTA;k3>8bBpiKl(2yv?X| z@%~RSp>d9b3}4*={T+ULR%+|y?w-0k${RPY(++GVT6qVnn8iA9ogCXFd z%ofu~B1W%`bY&@3JKbyKfPui$knk2+i;~#p65OORJ?k5LEQrJznehE=Cs(Eoc$0E} z@wz4lrIN6(-Xm8F{Iudp+aBPg2Wi@w4!VYJDBZh0=Pwe}Ix1&8c^knGNSoj&$(w$% zSrHPPw{D#3*3J zmX*mO2+g)Fx_o4~Py%PYK(->>(SNn-)K&Rg9up?k$X-&xVeCuq!QFU4U{#Hsx|XZk z*-{Fd-F%3>$Nj|@r8V}uA3~e8rFlEQS8IfQ3@2yRf9j~ume&Bd>ISYiem;|ZwRz?u zsL8E_S(fsf@odaSAm{UU?Bb;(hK+5TR1&s7f6A%$y}0^YVaBEZ;}qEW$b_heJYjDr zt$?r8*BQh8E~R$wWMbnY-qzdO5?A$*s|oBZ<|e(8X_(ZQ%Jog+{%k@lCD|z$?1YVB zO)FXl{R`;Gm(v-cg+Y)7rVK$R4_X@`_)Rf`2easH%B&R4w*ld|Tt5Xw{I}F!T6K&M z=FP52zV7UEFIzlzTSBeY>|fyLgtB6&;86vRv{b-;llUeOdKtbVL2jd>4v=|_%poca z3AHqT*ec``hv93F*0|J)XC3P7Rt2y?8GFQj#4;FWyo6oO8pZ=MYBC;xO>CXoxGykQ zRE=6h$c-T@opa_`^S#bbV=8|6r)oKr-~66XJUb5H!=og_gq*u{GHx&5-_YYzH$G(! zy~l7z%Atn!e}HHIU$@Z;$Hh2MRBzXV&=AZ3Ux}EW9=P?5Ymx+6nrpL9=;7~eGK)P{ ziSn4JK|y_h%i{Ran}mU+4r`jd4PQt;ZG41TSLTlWLN3IwUDjJ>dXBm!7KFA&_( z>N3|*lBi9E3tP-}|J-1BpQOoKJcd}?84oRa{-{rWG~Ye#d&YAY+ChXdX;XoU3Bk3( zE`NLkML1?|Cg?pz&Sf?O@O1-TT`7xDQf0|{-`!rub>HEmV%h`vd1-n2rkYu@ z_0_0cDbe65B^5>lN1uGJ+P}W^W2FOTX4xMl^-a*pq*QOCInxN*W-@h2gzRa=rN<4J zyjA>wg>M^}wBsYB+5Wv=9Y2OQrMtI-iBs43J1(eT{yM&DA_yB0)akDWs$8n4q%chT z^&&mvzmj1*|H073dN3&dlf*-XV}JIv87cOI_zDlFnq+bt#B}D87u8j07-SQh1yQQ> z+2kcK9<<9w%?M3qk%?Kd{vr{t%nzGS)I@I$CnzqE^(oY2!Y?zC4Bximul>vqU!vx^ z8gq`ahDB9-rfh2IM?}ki6;631nloHvZIFjkK_z!7c7t3BSfsB5SLiG?sGS=+%`OG=q|F$%dVNep za;grP_C$e4+#Cg{Jcr$|0Q}T35;>awx*3;v@o5r>oDArlRKC;o9$lOk{yozv!%hyy zZMPqpwhFLT_F5D=_b836Pgq!3tX~x;g z*wNl4yB*9JKhR8Ny@36&U)@j~Z&@J$7A)vH_x#Oq?fHc%Wc3Als_CBRM z)=yTVm?eSz->JEk88spyCPP|z#iId4~T|)yN+5jNi z#Zo_$)@Zl5yUqpPwU97%Ydg9IUSRnUJ5K062&`$vJpS8WzhKiC*07@^ z0P&R3a|xW7)LS)Ty;r+?xq7^E7A)UuY!m$81eU{4^!?%ETAITZ*%X;ay*jH&|0@Yc zMaM;lp^l;*LhYPoiF^Q>*u(F?L^{A?bJ%m`SEHYwp9>h2YBThkoKV>^5~(J2w1cqv zhW!R|6YUtxzCY{bg@H=7#{vOCA`uN$(C{WFfloGuBgvofd)+0hXduX4pQcy zdagXFElOGrw$jrYdbsL$3)+1_t%hUByT$-0Oaxo~pbs?G%W*~~iP`(J>j(kj&rJqH z9K0D|)e}N-dcGE)eUo$4I_IGnidS!E^o_zbAQLLqsW!+H=e~HUA)5{KUPCSbB$rw)e$B=eXK=tCK}aPe)p!lhixRY>Vs~DZ}r2J!q4an+aWpIByh39i`1Iqoj!zScIHMXvh;= zvRud9Hv(fCSk120*H#{|9xX&SHvS50TU@{raz!r0j;9nnNu$Gap4OR1F6}MZ8Qpr3 z@HD>KL!nyQ_$6De8qGa8F&PgxnSd=i)>b+2!spmHan8q|L=Twp{n<8?!__eA2!F4Y zEYrzhpr`giP-UDqC(l@;=F~?jbetnXZj+PRT;C1T9erm)uQi>wG+ zoiOCmhsE3W6;ABXxn6Bq9CNdPY&)jDeRp%5hdo}#1f+6x=e1?jv$}Ka6y|VdbZ?8F z?P25IlW~pkXF5g|PaQaO0%nl#=TJ+8zl1r5D733@66@=*o4>0YNp7n|<2=1*jcTOQ zy6(kCa?$H50(JoU_uf0TPX6{Wd=NaG^xX1Z`B36P^Mp!D6Ud z+$mAnwkTtIe-Z`2^*;AxI;mdKxwt3f{In{SyW2cTVz05%(LFZdRV?zWo~# zC+wR05&7@Qb$T-i77l$OGvl{25oWs_W;}SsgYqtr)rTcYKw$mJK?q1zMRhC>4NFi3-J^g9q3^S%Z@T)%4;fiD$ zZWMAD_mzrQBREs04E3qsY@SzgWg(&HP61JMEwlf1VnOJ7sEr+foRJx$mxCvPiTYJN z&uMGCZ4DEV)<)<1YQf-QV-j!Y{hnkCK=<$boVFWvbb%5)I|jS3<bBoD<>&E>!=J?96NmVF@vc`+Zbl(UG-xTR5jDg z!}u{CA_qTHlVF%%#qLga<1vLh+5L~|D6-y)!R}pu8viV{+*ON6=$cRTLK#G|{u=?Tg z&*ifo9Mqqhe{KEjGtYWO>A_!jHYaXU5}!fbM331!AYouS@BkIUOn2~Qg+btWZYww3 z$Xm;LG`isd|AoE;I~%&IV?2H@O6ouJxTyZ`-{F6T+{t8wzUYaYstd;KTzlf7pyy%v z!NW?@!p#bE0*DBSh&>Z}`Ak$uTSQ1w_}{@VBqS*$#PtP)*%kMH88|vy+FF15?+tY4 nPB}3KJpZQ$4_ikocaIN_F8{p^V0h{VV*^l8P?s;4GyD8sn&Nq$ literal 0 HcmV?d00001 diff --git a/images/pull-based-ingestion/pull-based-segrep-mode.png b/images/pull-based-ingestion/pull-based-segrep-mode.png new file mode 100644 index 0000000000000000000000000000000000000000..ad6f59adc4969ed51c391b9b5bc8c5d5042d0796 GIT binary patch literal 10786 zcmcI~bx@nZ*KQ~+4sC-LFAeUlg;Ja#MFNx{ZE&Z!L!m%}6fG_-?vUbMN`T_-F2UWM zOWWVQGv6QI%$+;m%}i#qyU#g$-m_=V?ml_pZ&c*)amaB1006#%ytFz1fDQuy(9*Hc z?^|$|qEzk+S|fRNWdOi~0RZs(1OQy#H~Fmt0M1+hz~(yuKqLVGAhl1aQxm)IXn&)m zDRXysH$Oig6BG0H?c0wZKO&LH%Ye0)SC6~9C$v~_gt?(L_3&t6?y9~>T4 zQis|)dR$#!>l;}Xmi#CzEPVIwU1VfrPft&6eY2^RLkXht^z>|Ic79@N`d3>A01b_h zkT5tn7#|;BLP8=VBLf0~_yoXpbae0^KQ*`h;N~5mrepY8PUXP^^uXYdGm13TdvnoX zU^!{P=SNiX_Z4h=`L~V$09o>%FPegLKid@|JvJ^c^65D2=%zTW0(4}CDup(B(8@xxtUe4SqQ4_n7+=cb#r(+4(PjQM!>T8o}no{|f!AxA)~QM237-`-hziyedo_VV!FJr(*8`2;-He7e8T+ z5QHK5RJHQ0r%2%YBw~hinhkJM#^B-B2fe?8espco=FK!|r})?y8xkbQc1f-?>Xfx{ zA-{Wu3w3=|zIzIfc3S;e&n2(4!4NaGC-$L~#cD>*>31DzV(nd+gy>%CEl&juP-GNI z#+ocotQXDog+|{8S*eEVas}yWlI+Sp zAptrqw1}j6gthU2nrd)LMT1>(5J7Fkrg(xlEjK`mD@pX6s+*w#dK;OLlHpvEXyhoT z5w=!R@0|fk72fC2i_%d>FVUEAk5us5Chi57lNjQ#6?dQvCYsbx*;rFsf2Po|pXt z6z8PA=kb~$&9^r+w-AmC_+UcmwS<0*PU2DZ;3atiW=nLMuI|axTaJxZ=W5khn?ynw zlQlE)Yi6OXdWhZTO0qUO%HlGxshQ}}mCfC+@I`*OTJjw6DUsznwfgCmrnXJd?BtpG za{GYFQkEVH1*NUULVoQMsC(gJ=&h-a0at7RbX`tSO%1mKhIC4$Q<4;2t)4xWk%WO^ zVC%2=n<&i^@M!jy>JdVHfPBtldAM~|JmvM#r*vE(^pKri7JP8o$osUEi0^F!_ixoU z@0KE;pcnIkab$a!C0z&R=SGNU4SEgv93a!c)SH*SL_=uQH-qff-Ob-p1F0zctN3v6 z&X{F(WO0_3#;(*s6e(+OvzbPy$X|>QQh&B%;~sxUHtrlrM~Nbx1M-FWP6myA`cfBZ z?5!(bl-uGkdo(|SJNRS&M_If6BqSwcFei69Hm?7)dKF*x*4n|MFTmtO(hk?=*2YSz z#fj*{#LEUE6>du-L$!oHv9{xrF`(mU!lxF)#%5y3w(0ttH+Pa4N32_|RQYB=$4)^f zwArfcSHZh9oWwg2p>eR1CF*wj8n+bZ$>MhVp}yr^5T0HMIvu@)!^IeO0%!M5C8uN) zi%g|qSzUD;8T%NUWAkJ46e%`$DXUI9kKE!fCPwWqVvugNmBS?YIp51_sun!oqU?(N zk?XwU8G)qg%F6h$&a*K+xRt1pf?*+&<@Xi!C%3EM@n21nTtlT`we`jfuf}TRIkbOr zzl$dSNtx2Yxc=eA&b2&$ypH+&deJXC29bj-ABm;|pvJKqo@iJD+ZuD|JKj@Pa4Ry$ zFx`kSZ@Q3#3Dh5=ZtNL&;b8^85Dxe6eI0s|GRu?|67;V8MXwSp)%Mb%CNsq$Hkdy{ zCQ?USG{7*ZSdKWG&42D1`#Z6teg_@=K=tHjmYPTri_A`XV1I70MCz=M4vc#T7NOX4 zhNme0#D(6i^L)b3WFCJt_&{EqCN%%UO?Zk{BV>80-+{~ZoR1OiJUj_wL%xRbRSciR zgFTlBi|4YnLs+!5qm<37?HZd%f+0PV>ZLM0YAafBeDA=p=~79U3kNra1C>X#;%b{R zJkS{t60a%P%CozvRIg2+lT%)*xNWRl4zo~o3W*KK60avRJ&#IPDz4X^PutTo4DG-? zqPul?4J?#8vtHln%$Lwkb0&H#A6UA0YAP4?&7wO<8mgLZ@*Q((J~{!LnG**QJF$Tb zN@(n*>fIG~d(T`H0u)VY@E)k>DQ;OHfuFqoTqNuXtLSth>L5-m_RUgSCD>b9;Y$!+ z7u;x1EjIPZN*TW_!0%vHk4e|di3k#$Wrq_lPo4%Lr70Y$>^O@PM8+{qotZD0ddbfG zUjzq z?gp0(>eG`qa)XVKsmFnh{R+CKgMr!OC2a+acCov9Ny1vVp&4-_mXGT{ZK}9iU^P2) zH`8m97-9n=#g47eR+Nrc1b*e{;WptURL(!ZN&`YFz>N4744SIO{_V#Y~^C4cg?jr z2pGOie`jO65EvE8#;8iO_wkfEL-L+x8w_E}KDnj?a-*`@SwOx6xS z>Jwp3+E)@>o}YFyEL*MWZau^?*YtD|Ut_XGj>iV{!sS;sQ=h8sR`6`V%2B;x{!TzZUWv*)*(NRH9Fe86;$VxsHMra?QaUr_bQKwe!FSXrj>xJyZ{PAOG7L!<1){;}P` zjxC!E|DGgq{~bmN+tFuw?m~>=8+BNqZP%z3IV~cgr-;95o#&`#qjk*c3SU)mzhZWk zkLFcJ!e#Y|dpWr>?AFdxfMgl_bdTB-dbprcSV;YeY`rPwXD zI$L25WNlO{mx$M?YR2a)1taNrd}@2Gt=WPar%JfD+$7wva5eXp6h@D57z}HyBbjDT20TZ64tts7QzhVH-G;}h|GSZflaSJJFerRxYfx`A<*WM@ygKiN%0{)U>Yj=x*cw;z^$4>!M)s$y%i zG_4B};I83>U276U$9M`z3IAe^iKS?1tIpqt*MDjSbeF=C#832GFdj7a>z)Mt<%d2irZmC z_{>=Rz`570fij+NZ=gTnizk!7F%c@nX7Ii&O{kZ(2#WWKCiDH(MAkL%j-4G4`Y`zP zb2T^(%=Y`YHoD{%TJTA+B_h}yy)-gm29aql7_NNp5jL0hgB(1PQHOboLXv{~fJH&VDKnlg-zu^-BzFG3s($kN5=}oE2(<-+ z>qzQ*yeU7?_*`4;*;&0%x*6BTIg+iE5qon-5SFB=5HO8UF7YH z`~oHJLXex2m7}rG-Lr{U-993Fe)V9(UxqhNUTsaa%b ztshShBvj>ST~jj{qC70-()H#;C6OV}5Z2}}t9S{#T7G&y{Mh&Q{QSe-o)uwcfDx~o z66!?S3<$N}!EJ2A2NJ$NXqyRZRI;Bz>z{3Dd6l;cgt`$3IqD>QLl%_x_UD7G_t=Wo z%+B~GL?1)R8vNs1x$?1!b#@{y3zCD=(l!b|Afoj+xkHFwjj|^*w{j5T_2ILpKdl0K z2zs~nOPrf_leZcSNCkDdGmn+)a866yc{YA7k9A^K5pi6*KHNgog)GN!$r7kYRKSzT zi!6mDWxmi)zh_}(eBogs5Y{GCW}sqwW*XklBRLh$`Mdo$D;-l#IK8UO{rY!GLGZa) zU9_g?o%M#ZK=Vw&G{z6dv*W+y{k>`J`p=8|%!=Yd$(r>FG0-23HwKMcL{M?*)=M(J zF#$oPGApNOx%Vc(KRKei5dAa7Qy*Ou}*)$U_?-$cUFzP{Sob9*Id%yn@au4J7DOVxOvci z8f@B}JFY73HOKY%@#A{8t<~tRQX5|7IugdzS}*|vBHnGJKR-ZwMJ=Us9umUK3LEW~ zlZop;bSO@69fL)6a%`jVT6uGw7n>xje?-##7<<2Mgy%DEBrG$l6JkYBNbP{Br&|6l zu*vd!Lm^4>JK*L<@8-ekY1Vz>*6gp&)mP5>U2jI5Dsbx_2Gs zItr`TgKq>Gi}4-xM{_|`^ObjLt|xk5!7LC07?3Gz7Ma-3mMSoax5j%{7owA%@g{=8 zf!F4cucSmCheTR+#z6ro{Br_z+|T9NIA)=rOef#qP?(%g+WU1Si!^u7e0Nh9YdA_i zx$ZmSVU)l#Mm>$W3jW2&NsAaUpQp1&2B{ON(G{2kV3usnKteI+aZ_doA*~}1r^}Aa z?e7*M% zghS?O?}sHbOq#s5+C2aVNb3^CWXXYO0bqy1946dwlhxVUNNCdfm!ouNK72Y2L}qE} za~7q5+0HTRjbK#3&~W$A!HF*Q<13=mFS^#o;s#Lt8cVF{%buB0c=fqcq>cxl-FrlZ8)ikDTVB%C1xlAyi zHMIMtwvrVO-sq<&G5t{{Bs=faGx{%+GDF`HJJIW{(y?@;u7@G)OZAsSaqX0(muYs%fzDCu|3Mkulfc5Y9_g+Iym#a+gnF*dYWN?{;^E#=gv46*K3VO4~FRe zgL0Y3+oTovtE_qpjw}342lUO~)3Hio{$^rud)iX^X&<(H z*x!UAYP8OL`@>cC73`nbv+ud*t#cQE{HVWZv+GYjbTx{9eUy5{dI?`3|7Uxj-8!Sc z)jt_wi9hQ936Q=~eG2w#5&nnzxW#4paP&@apx zX8{{K8<)$9`F=RU0>+wN(4i7erZc7Fe(~%65O%Z!nXHmC4;!4VjSnGuQnUK9;!Zlw zs2>Y_cWVCMR9{Mgc&XX1&9KRtBgY#>qq;=dWJ-}M*_90pg`(rW!_rGz^G|v)CvFm@ zz>@bAdlQ-S=P$qQ`&+mWvsPL$R&^x}I6ONTv5x)q5GC+T^OWjaBzL8Ioa(`x81DE> zjjJ}C1L`>lX<jl*wD0m=|oJrn^Vx%eR!?}8~4`y zW9302Ql+wQJ-f7W*UK7|FOyTzrBbC)$88`@EZn+A09%I*1()D!*6PI50`|IdfiZQC zcdjj4E}iKsK9Q@l56d4+FPMs^EA(Ip{;JfE`2!-QYP)q}NpWMD7QFsGGNpqvQG{#7 zOI`SOyxPq#W3k(p56-!;VVHktGg&CSpjT$FYD-Kv({GEK+ApGdR5x%lyf-j4#0 z;@bRZHX%*~57`Ued5dcnNx)oufB5N5&+2!1{%CRXX%?S@0wesXMfpPFZd+5X3${N9 zReIUOCmQxxV}wwyRiKqWJ_4^dC2_vxjJjFziQgH@MQ-p6Zv*8B4<{TMr#ICCVg0;1 zgx9wAi{Vm7;_F2v92@bAAjrQQS2Gqr=iYBTmxev6R&_D|tYfYX8ha${J{%-L7eoN7 z{bx_usk&esz4U8~N}x79!|yM95A$pIx2`pHGDW*xExU;SxZjI`y9HlIL5)kf#Etwi z1Zi%mYoeTxY+G4^-@HEK6feaskLShPq%$j7byko_VyPFCaDpptX_sc#K+nT=v93jb zZ=*wTJ1(L1r?1>u6w^r!`&3Q8VsB-73{)WL$u2r9D2@z??qB-G~Z8@QT%_A^+ZuT1;c*US;X~#dS`8I65wE83R713{Z@Lb5sI;zr5 zlKZXNa<@U(9CmYbxR(0Ri*J1DE-%e0q7(V~>(#eK6MZlI{E$tUy@eE^qZQ)~0x~OG ze>~Nt01Ghh6V$ z57ivHnin?Sw4{app;z^U59M;Zk{a2`wXG>!#3}>DIJzaa2Jw798*%ITBdgWPG8oer2QiAp4Asw*i)8U{^P7kceiR74iH0i49 z+5HgusMm49x9?#<(XZR8(2xUT?l6)^`4*8A+wTwF`7ivgK@0H8SN7tsqF!>#X5g-x zvg%7s40IDmn3&Fmpjy$F7az8K=#51U^*3y^GG>nH^Y_9jtC!t7KC zS>!_S4po(4?!&`i<)B}4(3Jog1RjZuZvdUNO8-_ren#X({g5ekiNcBup%FnVEdw|F zS#gN^%8Xit?l>Rk>W{QxIif|o$OOXY&cx-VtCX~eGh|oes1q(f0lrKUP&81Eu-_#L zlC$IIUtfAu?Vz__zmfz5#*8l6ypw2l(|@p#z9hQsqr-BewHw@8h$ zZ@WH6U@3I^-q6lUzdVaKN~$-S^e)E_TUg)ZvurzcwbR2R$%NQi$x*a47&&2cNeYEF z&k*B(!2-EZXrs1BEqh$BRloVwe?*Ea8y3dXPZV;FbNRn`0Qbg#T;=$-$U>karLnwj z&m$*t->fNbm3%JXnI1|qV;=;2foTz(Pqx#fd>fT+=dLK46Z61hp7d^b^4s7BPlD*T z24WDi46@T-V=r~fsviNN=sUQ$SJD&~9tYoOiIgFiS>0tS8cxianxmdoI?}4E%#K zg$o}ZB%$oB+*}zRLJfq?psV*&tg{=Xv$gyVk~+p>)QiQ=I$oo&7oQqP`zyR$c1z@l z#i~;-X1m&q;66N1)bLuIy-)Q19epVh`OIz+6xmTrroi$Qt2cpz(8askjAVCq$3Nk9 z=1lbUs(H})LxDpvG|sONY(+G0!+;1gx~!mlUzzQwV$z{)*G&n+An0~+#C+7cne*y5 z|Lm(zfC%<#yBMV64*#G)tyOzL+HJ{LFfFfuLnEX zzD$vdHdc+E;FQ?8nVo$IL?@i4*K&*bHc~!H0gDXRbs-A@tS9XE*)Zey>m<)Tpj>3euo z6J@93c6CJhS>FTsU{TQOiL?#S!n%w}U{_~jRFHRSye`K|B#G=u;q65N(!U89H^JHL zb6s_ReifPF7K;3YRTbJ5tj>;WbX`HZ(Mgg+Nl-6gjfG<|y$I6HPA5h=C7&cZ0!4@8 zm+!o?I@9ke&LVKsmxSxnWKtT39pN{})1_vRi@me4A&B?Qazq(}zi@C=!qr{w{H=vS zG9lIdzq$+ZK*!96yQ@2&Amq>QLJ7zU;Tu#(`h9w#B-mKl;gq<&Ur7P>^6=CN|Htr; zFYGnpX!cM3%Mq=i7>PlHH90k^8)~IpoIxp=@xVUWH9ldiU^c~;4hX)iMzd15H1kq| zfe6*|7HZ9KBgx#T74E_!z+u&XU(`4XI2*blX5)3A)@FMyD5kc(IS`gE4GlXc4Fw!L z~yX(9Dd~TlbL6w0tL(xtCp@E`f{ABz-nn_$Z_?t$+-M+oe^c$k20SuCr z$9V*u@v^neq(H)o13n?b>O?Ei`j!pu+Gl$OPo64<1bH(IKWmW?PG`rZPHRcNnbxq^Dmc+RmVMWKhqx53pne#I+gSEeAJ+;_emT%=VY*|5vsSaufE z!YG{HE2!p`_2)dGbu7IAwWHDSCz@d3p8jMhCvkT9{dc1>$`Q2svR9!vjNQ?;7SAfw zi!!!|uX`+NZt7QjSZQ$zPKd_fKE)o}jA_c_mg)M9&uJ#3&T{&=G9Pdqi)R`=T$04H z3M=BtE@&7lT~YbRl36?=EhM+rU9#>h*HLe-Q}kM^x~!g1ezB>XuPM=lp3Hq+QUkS< zKK3b{P^~$;m|g!uo+>tC7iexke!b6QeSi| z#Z1?`ZU=`sM7c9|y6-E@1&1MqCJ(auHJF-~2(1r3`Psa~6d3iP@ISjybYglb+bN3FK5p7CphdK= z~;i=X$0>2CS6=#ZWVqq|oEVj%b_IrgzEsrrt72DnvbfX)krpXLEf`>= zi^|C0jXn6ztva%pKV2|J5%)~I!@S=paX=>XmJ9;PE^OylN!%6n?Y1gTPz?>NuqP3x z-01V@GU<0|fl;$AH)Og!N4V_9T-PDj5A{kPcVk4ul9A@>a@}0(SdwiGgY`XiPq2=V zz~#A2V?yCkI^hHy6$ZmQKA#Yz2=zAJ;(-5T=I}FqPXDCo3(W~77!PP}5PrdgBfbP> z;1`@aLT&H)T5q*kSf2N?!`b_qSP%zeRAI=OFXiVF656n2VfQjaJ_nc(H-&pm!TEf5n< z5aF)%T6p47-GG`?<_{n6OLrw$<;u0v%}aLF!p<~*;hZEf!qkpK*>2rrE(-(7Gp5JU!G}M==MK?-DC;dO#Udzr9gw zI9?u^P#rdUf1&z#^O{cVZJ|Iyds^9$fphBv$@yD>@70j2u|l%62ihQ0QJS*Iqi&Mm z!M8&3(p5)@gox^~4|bY-G0%QCZaKw-@WS!wdN_Zy2*mkMtFT~^Ba9s8V@IES zF{zWuchdg+fvi=rtXpeay4{C=tbskTUJB-)s5%~3%)P8QH*|$w{TeTKdgEpR*ubL1 z&9P{s{GQQlQN~b(l(&k18OX@0r`~V=e>whS~q$DhA>jLbaT2s%yhNaSUlCmK( zN^U0S!LaZWsMWLwwI-1lW0f-z#q5eq)D?X`F1%K-k!drqHh?ktNXwGni5~iXBy_b+ z{kc+SurT`m%>)BNU?l#DrnT)ZmeD<{sq!mgBs5Xd7#zGo1#&6inc^IidO!a)e3f9j zyDrWrP|m{3o3x!%zjv!a=$VL?l9Bh|sQ24TUKZC6S~l`K?DtHCuwDJuGe17R|5tk9 zzmOEdNyr6b49YMmg&q#NR)GCO0(u0APs-0lp=e~62iTqm5WOG-%>2Ae?vvXxC=`aY zd>HaDc!dj?`GyJV#q}T(qa!k5(D&sd2-<=?r4$US4nWZT9HjNL4~?K34Wwn73{wU~ z62ZaSrCA^iSokwDd(G%L9g^3MDD&2z@41;d6U>ik!Xy)n?9J5;PPKz-ssWVGzS{D} z*@e&kNnlyIL7Qi{rT{nK*Qor# zmPoAs+1K#=>^ko-`7{P_PXZn9U#A4nJIUxc8NYWj5ixQwxi0`bTs-{jT*B Date: Wed, 1 Oct 2025 11:42:47 -0400 Subject: [PATCH 8/9] Doc review - part 2 Signed-off-by: Fanit Kolchina --- .../document-apis/pull-based-ingestion.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/_api-reference/document-apis/pull-based-ingestion.md b/_api-reference/document-apis/pull-based-ingestion.md index 668091b24c8..b4be9b63b2e 100644 --- a/_api-reference/document-apis/pull-based-ingestion.md +++ b/_api-reference/document-apis/pull-based-ingestion.md @@ -26,7 +26,7 @@ Before using pull-based ingestion, ensure that the following prerequisites are m ## Creating an index for pull-based ingestion -To ingest data from a streaming source, first create an index with pull-based ingestion settings. The following request creates an index that pulls data from a Kafka topic in the segment replication mode. For other available modes, see [Ingestion modes](#ingestion-modes). +To ingest data from a streaming source, first create an index with pull-based ingestion settings. The following request creates an index that pulls data from a Kafka topic in the segment replication mode. For other available modes, see [Ingestion modes](#ingestion-modes): ```json PUT /my-index @@ -74,7 +74,7 @@ The `ingestion_source` parameters control how OpenSearch pulls data from the str | `poll.timeout` | The maximum time to wait for data in each poll operation. Optional. | | `num_processor_threads` | The number of threads for processing ingested data. Optional. Default is 1. | | `internal_queue_size` | The size of the internal blocking queue for advanced tuning. Valid values are from 1 to 100,000, inclusive. Optional. Default is 100. | -| `all_active` | Whether to enable the all-active ingestion mode. Cannot be enabled for indexes that use segment replication mode. Default is `false`. See [Ingestion modes](#ingestion-modes) | +| `all_active` | Whether to enable the all-active ingestion mode. Cannot be enabled for indexes that use segment replication mode. Default is `false`. See [Ingestion modes](#ingestion-modes). | | `param` | Source-specific configuration parameters. Required.
 • The `ingest-kafka` plugin requires:
  - `topic`: The Kafka topic to consume from
  - `bootstrap_servers`: The Kafka server addresses
  Optionally, you can provide additional standard Kafka consumer parameters (such as `fetch.min.bytes`). These parameters are passed directly to the Kafka consumer.
 • The `ingest-kinesis` plugin requires:
  - `stream`: The Kinesis stream name
  - `region`: The AWS Region
  - `access_key`: The AWS access key
  - `secret_key`: The AWS secret key
  Optionally, you can provide an `endpoint_override`. | @@ -84,18 +84,20 @@ Pull-based ingestion supports the following modes. #### Segment replication mode +In segment replication mode, the primary shards ingest events from a streaming source and index the documents. The pull-based index is configured to use [segment replication]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/segment-replication/index/) to copy over the segment files from primary to replica shards, as shown in the following image. + ![Pull-based ingestion segment replication mode]({{site.url}}{{site.baseurl}}/images/pull-based-ingestion/pull-based-segrep-mode.png){: width="50%" } -The primary shards ingest events from a streaming source and index the documents. The pull-based index is configured to use [segment replication]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/segment-replication/index/) to copy over the segment files from primary to replica shards. We recommend using this mode with a [remote-backed storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/). +{: .tip} #### All-active mode +Enabling all-active mode allows both primary and replica shards to independently ingest and index events from the streaming source, as shown in the following image. + ![Pull-based ingestion all active mode]({{site.url}}{{site.baseurl}}/images/pull-based-ingestion/pull-based-all-active-mode.png){: width="50%" } -Enabling all-active mode allows both primary and replica shards to independently ingest and index events from the streaming source. -There is no replication or coordination between the shards, although replica shards may fetch segment files from the primary shard during bootstrapping if a local copy is unavailable. -This mode is currently not supported along with segment replication. +There is no replication or coordination between the shards, although replica shards may fetch segment files from the primary shard during bootstrapping if a local copy is unavailable. This mode is currently not supported along with segment replication. ### Stream position From 969fd7ab3b845f4cad7a0966f308bbac7f8d6af1 Mon Sep 17 00:00:00 2001 From: Nathan Bower Date: Wed, 1 Oct 2025 11:52:57 -0400 Subject: [PATCH 9/9] Apply suggestions from code review Signed-off-by: Nathan Bower --- _api-reference/document-apis/pull-based-ingestion.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_api-reference/document-apis/pull-based-ingestion.md b/_api-reference/document-apis/pull-based-ingestion.md index b4be9b63b2e..d8921c4fa71 100644 --- a/_api-reference/document-apis/pull-based-ingestion.md +++ b/_api-reference/document-apis/pull-based-ingestion.md @@ -88,7 +88,7 @@ In segment replication mode, the primary shards ingest events from a streaming s ![Pull-based ingestion segment replication mode]({{site.url}}{{site.baseurl}}/images/pull-based-ingestion/pull-based-segrep-mode.png){: width="50%" } -We recommend using this mode with a [remote-backed storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/). +We recommend using this mode with [remote-backed storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/). {: .tip} #### All-active mode @@ -97,7 +97,7 @@ Enabling all-active mode allows both primary and replica shards to independently ![Pull-based ingestion all active mode]({{site.url}}{{site.baseurl}}/images/pull-based-ingestion/pull-based-all-active-mode.png){: width="50%" } -There is no replication or coordination between the shards, although replica shards may fetch segment files from the primary shard during bootstrapping if a local copy is unavailable. This mode is currently not supported along with segment replication. +There is no replication or coordination between the shards, although replica shards may fetch segment files from the primary shard during bootstrapping if a local copy is unavailable. This mode is currently not supported with segment replication. ### Stream position