Skip to content

Commit a9f1233

Browse files
nathaliellenaakolchfa-awsnatebower
authored
Add streaming support doc (#11170)
* Add streaming doc Signed-off-by: Nathalie Jonathan <[email protected]> * Add setup info Signed-off-by: Nathalie Jonathan <[email protected]> * Fix api name Signed-off-by: Nathalie Jonathan <[email protected]> * Update setup information Signed-off-by: Nathalie Jonathan <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Add response body fields Signed-off-by: Nathalie Jonathan <[email protected]> * Clean up Signed-off-by: Nathalie Jonathan <[email protected]> * Apply suggestions from code review Signed-off-by: kolchfa-aws <[email protected]> * Apply suggestions from code review Signed-off-by: Nathan Bower <[email protected]> --------- Signed-off-by: Nathalie Jonathan <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Signed-off-by: Nathan Bower <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]>
1 parent f5c3d3f commit a9f1233

File tree

10 files changed

+546
-6
lines changed

10 files changed

+546
-6
lines changed

‎_ml-commons-plugin/api/agent-apis/delete-agent.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ layout: default
33
title: Delete agent
44
parent: Agent APIs
55
grand_parent: ML Commons APIs
6-
nav_order: 50
6+
nav_order: 40
77
---
88

99
# Delete Agent API

‎_ml-commons-plugin/api/agent-apis/execute-stream-agent.md‎

Lines changed: 285 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,285 @@
1+
---
2+
layout: default
3+
title: Execute stream agent
4+
parent: Agent APIs
5+
grand_parent: ML Commons APIs
6+
nav_order: 25
7+
---
8+
9+
# Execute Stream Agent API
10+
**Introduced 3.3**
11+
{: .label .label-purple }
12+
13+
This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, join the discussion on the [OpenSearch forum](https://forum.opensearch.org/).
14+
{: .warning}
15+
16+
The Execute Stream Agent API provides the same functionality as the [Execute Agent API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/execute-agent/) but returns responses in a streaming format, delivering data in chunks as it becomes available. This streaming approach is particularly beneficial for large language model interactions with lengthy responses, allowing you to see partial results immediately rather than waiting for the complete response.
17+
18+
This API currently supports conversational agents with the following remote model types:
19+
- [OpenAI Chat Completion](https://platform.openai.com/docs/api-reference/completions)
20+
- [Amazon Bedrock Converse Stream](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ConverseStream.html)
21+
22+
## Endpoint
23+
24+
```json
25+
POST /_plugins/_ml/agents/<agent_id>/_execute/stream
26+
```
27+
28+
## Prerequisites
29+
30+
Before using this API, ensure that you have fulfilled the following prerequisites.
31+
32+
### Set up your cluster
33+
34+
Follow these steps to set up your cluster.
35+
36+
#### Step 1: Install the required plugins
37+
38+
The Execute Stream Agent API depends on the following plugins, which are included in the OpenSearch distribution but must be explicitly installed as follows:
39+
40+
```bash
41+
bin/opensearch-plugin install transport-reactor-netty4
42+
bin/opensearch-plugin install arrow-flight-rpc
43+
```
44+
45+
For more information, see [Installing plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/).
46+
47+
#### Step 2: Configure OpenSearch settings
48+
49+
Add these settings to your `opensearch.yml` file or Docker Compose configuration:
50+
51+
```yaml
52+
opensearch.experimental.feature.transport.stream.enabled: true
53+
54+
# Choose one based on your security settings
55+
http.type: reactor-netty4 # security disabled
56+
http.type: reactor-netty4-secure # security enabled
57+
58+
# Multi-node cluster settings (if applicable)
59+
# Use network.host IP for opensearch.yml or node name for Docker
60+
arrow.flight.publish_host: <ip>
61+
arrow.flight.bind_host: <ip>
62+
63+
# Security-enabled cluster settings (if applicable)
64+
transport.stream.type.default: FLIGHT-SECURE
65+
flight.ssl.enable: true
66+
```
67+
{% include copy.html %}
68+
69+
For more information about enabling experimental features, see [Experimental feature flags]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/experimental/).
70+
71+
#### Step 3: Configure JVM options
72+
73+
Add these settings to your `jvm.options` file:
74+
75+
```yaml
76+
-Dio.netty.allocator.numDirectArenas=1
77+
-Dio.netty.noUnsafe=false
78+
-Dio.netty.tryUnsafe=true
79+
-Dio.netty.tryReflectionSetAccessible=true
80+
--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED
81+
```
82+
{% include copy.html %}
83+
84+
### Configure the necessary APIs
85+
86+
Configure the API using the following steps.
87+
88+
#### Step 1: Enable the streaming feature flag
89+
90+
To enable the streaming feature flag, update the cluster settings as follows:
91+
92+
```json
93+
PUT _cluster/settings
94+
{
95+
"persistent" : {
96+
"plugins.ml_commons.stream_enabled": true
97+
}
98+
}
99+
```
100+
{% include copy-curl.html %}
101+
102+
#### Step 2: Register a compatible externally hosted model
103+
104+
To register an OpenAI Chat Completion model, send the following request:
105+
106+
```json
107+
POST /_plugins/_ml/models/_register
108+
{
109+
"name": "OpenAI gpt 3.5 turbo",
110+
"function_name": "remote",
111+
"description": "OpenAI model",
112+
"connector": {
113+
"name": "OpenAI Chat Connector",
114+
"description": "The connector to OpenAI model service for GPT 3.5",
115+
"version": 1,
116+
"protocol": "http",
117+
"parameters": {
118+
"endpoint": "api.openai.com",
119+
"model": "gpt-3.5-turbo"
120+
},
121+
"credential": {
122+
"openAI_key": "<your_api_key>"
123+
},
124+
"actions": [
125+
{
126+
"action_type": "predict",
127+
"method": "POST",
128+
"url": "https://${parameters.endpoint}/v1/chat/completions",
129+
"headers": {
130+
"Authorization": "Bearer ${credential.openAI_key}"
131+
},
132+
"request_body": "{ \"model\": \"${parameters.model}\", \"messages\": [{\"role\":\"developer\",\"content\":\"${parameters.system_prompt}\"},${parameters._chat_history:-}{\"role\":\"user\",\"content\":\"${parameters.prompt}\"}${parameters._interactions:-}]${parameters.tool_configs:-} }"
133+
}
134+
]
135+
}
136+
}
137+
```
138+
{% include copy-curl.html %}
139+
140+
To register an Amazon Bedrock Converse Stream model, send the following request:
141+
142+
```json
143+
POST /_plugins/_ml/models/_register
144+
{
145+
"name": "Amazon Bedrock Converse Stream model",
146+
"function_name": "remote",
147+
"description": "Amazon Bedrock Claude model",
148+
"connector": {
149+
"name": "Amazon Bedrock Converse",
150+
"description": "The connector to Amazon Bedrock Converse",
151+
"version": 1,
152+
"protocol": "aws_sigv4",
153+
"credential": {
154+
"access_key": "<your_aws_access_key>",
155+
"secret_key": "<your_aws_secret_key>",
156+
"session_token": "<your_aws_session_token>"
157+
},
158+
"parameters": {
159+
"region": "<your_aws_region>",
160+
"service_name": "bedrock",
161+
"model": "us.anthropic.claude-3-7-sonnet-20250219-v1:0"
162+
},
163+
"actions": [
164+
{
165+
"action_type": "predict",
166+
"method": "POST",
167+
"headers": {
168+
"content-type": "application/json"
169+
},
170+
"url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/converse",
171+
"request_body": "{ \"system\": [{\"text\": \"${parameters.system_prompt}\"}], \"messages\": [${parameters._chat_history:-}{\"role\":\"user\",\"content\":[{\"text\":\"${parameters.prompt}\"}]}${parameters._interactions:-}]${parameters.tool_configs:-} }"
172+
}
173+
]
174+
}
175+
}
176+
```
177+
{% include copy-curl.html %}
178+
179+
#### Step 3: Register a conversational agent
180+
181+
When registering your agent, you must include the `_llm_interface` parameter that corresponds to your model type:
182+
- OpenAI Chat Completion: `openai/v1/chat/completions`
183+
- Amazon Bedrock Converse Stream: `bedrock/converse/claude`
184+
185+
To register your agent, send the following request:
186+
187+
```json
188+
POST /_plugins/_ml/agents/_register
189+
{
190+
"name": "Chat Agent with RAG",
191+
"type": "conversational",
192+
"description": "This is a test agent",
193+
"llm": {
194+
"model_id": "<model_id_from_step_2>",
195+
"parameters": {
196+
"max_iteration": 5,
197+
"system_prompt": "You are a helpful assistant. You are able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics.\nIf the question is complex, you will split it into several smaller questions, and solve them one by one. For example, the original question is:\nhow many orders in last three month? Which month has highest?\nYou will spit into several smaller questions:\n1.Calculate total orders of last three month.\n2.Calculate monthly total order of last three month and calculate which months order is highest. You MUST use the available tools everytime to answer the question",
198+
"prompt": "${parameters.question}"
199+
}
200+
},
201+
"memory": {
202+
"type": "conversation_index"
203+
},
204+
"parameters": {
205+
"_llm_interface": "openai/v1/chat/completions"
206+
},
207+
"tools": [
208+
{
209+
"type": "IndexMappingTool",
210+
"name": "DemoIndexMappingTool",
211+
"parameters": {
212+
"index": "${parameters.index}",
213+
"input": "${parameters.question}"
214+
}
215+
},
216+
{
217+
"type": "ListIndexTool",
218+
"name": "RetrieveIndexMetaTool",
219+
"description": "Use this tool to get OpenSearch index information: (health, status, index, uuid, primary count, replica count, docs.count, docs.deleted, store.size, primary.store.size)."
220+
}
221+
],
222+
"app_type": "my_app"
223+
}
224+
```
225+
{% include copy-curl.html %}
226+
227+
## Example request
228+
229+
```json
230+
POST /_plugins/_ml/agents/<agent_id>/_execute/stream
231+
{
232+
"parameters": {
233+
"question": "How many indices are in my cluster?"
234+
}
235+
}
236+
```
237+
{% include copy-curl.html %}
238+
239+
## Example response
240+
241+
```json
242+
data: {"inference_results":[{"output":[{"name":"memory_id","result":"LvU1iJkBCzHrriq5hXbN"},{"name":"parent_interaction_id","result":"L_U1iJkBCzHrriq5hXbs"},{"name":"response","dataAsMap":{"content":"[{\"index\":0.0,\"id\":\"call_HjpbrbdQFHK0omPYa6m2DCot\",\"type\":\"function\",\"function\":{\"name\":\"RetrieveIndexMetaTool\",\"arguments\":\"\"}}]","is_last":false}}]}]}
243+
244+
data: {"inference_results":[{"output":[{"name":"memory_id","result":"LvU1iJkBCzHrriq5hXbN"},{"name":"parent_interaction_id","result":"L_U1iJkBCzHrriq5hXbs"},{"name":"response","dataAsMap":{"content":"[{\"index\":0.0,\"function\":{\"arguments\":\"{}\"}}]","is_last":false}}]}]}
245+
246+
data: {"inference_results":[{"output":[{"name":"memory_id","result":"LvU1iJkBCzHrriq5hXbN"},{"name":"parent_interaction_id","result":"L_U1iJkBCzHrriq5hXbs"},{"name":"response","dataAsMap":{"content":"{\"choices\":[{\"message\":{\"tool_calls\":[{\"type\":\"function\",\"function\":{\"name\":\"RetrieveIndexMetaTool\",\"arguments\":\"{}\"},\"id\":\"call_HjpbrbdQFHK0omPYa6m2DCot\"}]},\"finish_reason\":\"tool_calls\"}]}","is_last":false}}]}]}
247+
248+
data: {"inference_results":[{"output":[{"name":"memory_id","result":"LvU1iJkBCzHrriq5hXbN"},{"name":"parent_interaction_id","result":"L_U1iJkBCzHrriq5hXbs"},{"name":"response","dataAsMap":{"content":"","is_last":false}}]}]}
249+
250+
data: {"inference_results":[{"output":[{"name":"memory_id","result":"LvU1iJkBCzHrriq5hXbN"},{"name":"parent_interaction_id","result":"L_U1iJkBCzHrriq5hXbs"},{"name":"response","dataAsMap":{"content":"row,health,status,index,uuid,pri(number of primary shards),rep(number of replica shards),docs.count(number of available documents),docs.deleted(number of deleted documents),store.size(store size of primary and replica shards),pri.store.size(store size of primary shards)\n1,green,open,.plugins-ml-model-group,Msb1Y4W5QeiLs5yUQi-VRg,1,1,2,0,17.1kb,5.9kb\n2,green,open,.plugins-ml-memory-message,1IWd1HPeSWmM29qE6rcj_A,1,1,658,0,636.4kb,313.5kb\n3,green,open,.plugins-ml-memory-meta,OETb21fqQJa3Y2hGQbknCQ,1,1,267,7,188kb,93.9kb\n4,green,open,.plugins-ml-config,0mnOWX5gSX2s-yP27zPFNw,1,1,1,0,8.1kb,4kb\n5,green,open,.plugins-ml-model,evYOOKN4QPqtmUjxsDwJYA,1,1,5,5,421.5kb,210.7kb\n6,green,open,.plugins-ml-agent,I0SpBovjT3C6NABCBzGiiQ,1,1,6,0,205.5kb,111.3kb\n7,green,open,.plugins-ml-task,_Urzn9gdSuCRqUaYAFaD_Q,1,1,100,4,136.1kb,45.3kb\n8,green,open,top_queries-2025.09.26-00444,jb7Q1FiLSl-wTxjdSUKs_w,1,1,1736,126,1.8mb,988kb\n9,green,open,.plugins-ml-connector,YaJORo4jT0Ksp24L5cW1uA,1,1,2,0,97.8kb,48.9kb\n","is_last":false}}]}]}
251+
252+
data: {"inference_results":[{"output":[{"name":"memory_id","result":"LvU1iJkBCzHrriq5hXbN"},{"name":"parent_interaction_id","result":"L_U1iJkBCzHrriq5hXbs"},{"name":"response","dataAsMap":{"content":"There","is_last":false}}]}]}
253+
254+
data: {"inference_results":[{"output":[{"name":"memory_id","result":"LvU1iJkBCzHrriq5hXbN"},{"name":"parent_interaction_id","result":"L_U1iJkBCzHrriq5hXbs"},{"name":"response","dataAsMap":{"content":" are","is_last":false}}]}]}
255+
256+
data: {"inference_results":[{"output":[{"name":"memory_id","result":"LvU1iJkBCzHrriq5hXbN"},{"name":"parent_interaction_id","result":"L_U1iJkBCzHrriq5hXbs"},{"name":"response","dataAsMap":{"content":" ","is_last":false}}]}]}
257+
258+
data: {"inference_results":[{"output":[{"name":"memory_id","result":"LvU1iJkBCzHrriq5hXbN"},{"name":"parent_interaction_id","result":"L_U1iJkBCzHrriq5hXbs"},{"name":"response","dataAsMap":{"content":"9","is_last":false}}]}]}
259+
260+
data: {"inference_results":[{"output":[{"name":"memory_id","result":"LvU1iJkBCzHrriq5hXbN"},{"name":"parent_interaction_id","result":"L_U1iJkBCzHrriq5hXbs"},{"name":"response","dataAsMap":{"content":" indices","is_last":false}}]}]}
261+
262+
data: {"inference_results":[{"output":[{"name":"memory_id","result":"LvU1iJkBCzHrriq5hXbN"},{"name":"parent_interaction_id","result":"L_U1iJkBCzHrriq5hXbs"},{"name":"response","dataAsMap":{"content":" in","is_last":false}}]}]}
263+
264+
data: {"inference_results":[{"output":[{"name":"memory_id","result":"LvU1iJkBCzHrriq5hXbN"},{"name":"parent_interaction_id","result":"L_U1iJkBCzHrriq5hXbs"},{"name":"response","dataAsMap":{"content":" your","is_last":false}}]}]}
265+
266+
data: {"inference_results":[{"output":[{"name":"memory_id","result":"LvU1iJkBCzHrriq5hXbN"},{"name":"parent_interaction_id","result":"L_U1iJkBCzHrriq5hXbs"},{"name":"response","dataAsMap":{"content":" cluster","is_last":false}}]}]}
267+
268+
data: {"inference_results":[{"output":[{"name":"memory_id","result":"LvU1iJkBCzHrriq5hXbN"},{"name":"parent_interaction_id","result":"L_U1iJkBCzHrriq5hXbs"},{"name":"response","dataAsMap":{"content":".","is_last":false}}]}]}
269+
270+
data: {"inference_results":[{"output":[{"name":"memory_id","result":"LvU1iJkBCzHrriq5hXbN"},{"name":"parent_interaction_id","result":"L_U1iJkBCzHrriq5hXbs"},{"name":"response","dataAsMap":{"content":"","is_last":true}}]}]}
271+
```
272+
273+
## Response body fields
274+
275+
The following table lists all response body fields.
276+
277+
| Field | Data type | Description |
278+
| :--- | :--- |:------------------------------------------------------------------------------------------------------------|
279+
| `inference_results` | Array | Contains the streaming response data returned by the agent. |
280+
| `inference_results.output` | Array | Contains output objects for each inference result. |
281+
| `inference_results.output.name` | String | The name of the output field. Can be `memory_id`, `parent_interaction_id`, or `response`. |
282+
| `inference_results.output.result` | String | The values of the `memory_id` and `parent_interaction_id` fields. |
283+
| `inference_results.output.dataAsMap` | Object | Contains the response content and metadata (present only for a `response` output). |
284+
| `inference_results.output.dataAsMap.content` | String | The agent's response content, which can include tool calls, tool results, or final text output. |
285+
| `inference_results.output.dataAsMap.is_last` | Boolean | Indicates whether this is the final chunk in the stream: `true` for the last chunk, `false` if there are more chunks. |

‎_ml-commons-plugin/api/agent-apis/get-agent.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ layout: default
33
title: Get agent
44
parent: Agent APIs
55
grand_parent: ML Commons APIs
6-
nav_order: 20
6+
nav_order: 30
77
---
88

99
# Get Agent API

‎_ml-commons-plugin/api/agent-apis/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ ML Commons supports the following agent-level APIs:
2020
- [Register agent]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/register-agent/)
2121
- [Update agent]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/update-agent/)
2222
- [Execute agent]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/execute-agent/)
23+
- [Execute stream agent]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/execute-stream-agent/)
2324
- [Get agent]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/get-agent/)
2425
- [Search agent]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/search-agent/)
2526
- [Delete agent]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/delete-agent/)

‎_ml-commons-plugin/api/agent-apis/search-agent.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ layout: default
33
title: Search agent
44
parent: Agent APIs
55
grand_parent: ML Commons APIs
6-
nav_order: 30
6+
nav_order: 35
77
---
88

99
# Search Agent API

‎_ml-commons-plugin/api/model-apis/batch-predict.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ layout: default
33
title: Batch predict
44
parent: Model APIs
55
grand_parent: ML Commons APIs
6-
nav_order: 65
6+
nav_order: 70
77
---
88

99
# Batch Predict API

‎_ml-commons-plugin/api/model-apis/delete-model.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ layout: default
33
title: Delete model
44
parent: Model APIs
55
grand_parent: ML Commons APIs
6-
nav_order: 50
6+
nav_order: 47
77
---
88

99
# Delete Model API

‎_ml-commons-plugin/api/model-apis/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ ML Commons supports the following model-level CRUD APIs:
2626
Predict APIs are used to invoke machine learning (ML) models. ML Commons supports the following Predict APIs:
2727

2828
- [Predict]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/train-predict/predict/)
29+
- [Predict Stream]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/train-predict/predict-stream/)
2930
- [Batch Predict]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/model-apis/batch-predict/)
3031

3132
# Train API

0 commit comments

Comments
 (0)