Add documentation for agentic search processors #11117
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| --- | ||
| layout: default | ||
| title: Agentic | ||
| parent: Specialized queries | ||
| nav_order: 2 | ||
| --- | ||
|
|
||
| # Agentic query | ||
|
Check failure on line 8 in _query-dsl/specialized/agentic.md
|
||
| **Introduced 3.2** | ||
| {: .label .label-purple } | ||
|
|
||
| Use the `agentic` query to ask questions in natural language and have OpenSearch automatically plan and execute the retrieval. The `agentic` query works in conjunction with a preconfigured agent that reads the question, plans the search, and returns relevant results. For more information about agentic search, see [Agentic search]({{site.url}}{{site.baseurl}}/vector-search/ai-search/agentic-search/). | ||
|
Check failure on line 12 in _query-dsl/specialized/agentic.md
|
||
|
|
||
| **Prerequisite**<br> | ||
| Before using an `agentic` query, you must configure an agent with the [`QueryPlanningTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/query-planning-tool/) and create a search pipeline with an `agentic_query_translator` search request processor. | ||
| {: .note} | ||
|
|
||
| ## Request body fields | ||
|
|
||
| The `agentic` query accepts the following fields. | ||
|
|
||
| Field | Data type | Required/Optional | Description | ||
| :--- | :--- | :--- | :--- | ||
| `query_text` | String | Required | The natural language question or request. | ||
| `query_fields` | Array | Optional | A list of fields that the agent should consider when generating the search query. If not provided, the agent considers all available fields in the index mapping. | ||
|
|
||
| ## Example | ||
|
|
||
| The following example uses an `agentic` query that asks a natural language question about flowers in an `iris` dataset. In this example, `query_text` contains the natural language question, `query_fields` specifies the fields to use when generating the query, and the `search-pipeline` query parameter specifies the search pipeline containing the agentic query translator processor: | ||
|
Check failure on line 29 in _query-dsl/specialized/agentic.md
|
||
|
|
||
| ```json | ||
| GET /iris-index/_search?search_pipeline=agentic-pipeline | ||
| { | ||
| "query": { | ||
| "agentic": { | ||
| "query_text": "List all the flowers present", | ||
| "query_fields": ["species", "petal_length_in_cm"] | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| When executed, the agentic search request performs the following steps: | ||
|
Check failure on line 44 in _query-dsl/specialized/agentic.md
|
||
|
|
||
| 1. Sends the natural language question, along with the index mapping and a default prompt, to a large language model (LLM). | ||
| 2. The LLM generates a query domain-specific language (DSL) query based on the input. | ||
| 3. The generated DSL query is executed as a search request in OpenSearch. | ||
| 4. Returns the search results based on the generated query. | ||
|
|
||
| For a complete example, see [Agentic search]({{site.url}}{{site.baseurl}}/vector-search/ai-search/agentic-search/). | ||
|
Check failure on line 51 in _query-dsl/specialized/agentic.md
|
||
|
|
||
| ## Next steps | ||
|
|
||
| - Learn how to set up agentic search in [Agentic search]({{site.url}}{{site.baseurl}}/vector-search/ai-search/agentic-search/). | ||
|
Check failure on line 55 in _query-dsl/specialized/agentic.md
|
||
| - Learn about configuring agents in [Agents]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/agents/). | ||
| - Learn about the [QueryPlanningTool]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/query-planning-tool/). | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| --- | ||
| layout: default | ||
| title: Agentic context | ||
| nav_order: 2 | ||
| has_children: false | ||
| parent: Search processors | ||
| grand_parent: Search pipelines | ||
| --- | ||
|
|
||
| # Agentic context processor | ||
|
Check failure on line 10 in _search-plugins/search-pipelines/agentic-context-processor.md
|
||
| **Introduced 3.3** | ||
| {: .label .label-purple } | ||
|
|
||
| The `agentic_context` search response processor adds agent execution context information to search response extensions. This processor works in conjunction with the [agentic query translator]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/agentic-query-translator-processor/) to expose the agent's query translation process and maintain conversation continuity: | ||
|
|
||
| 1. The processor retrieves agent context information from the pipeline processing context. | ||
| 2. Based on the processor configuration, it selectively includes the agent steps summary and the query domain-specific language (DSL) query in the response. | ||
| 3. The memory ID is always included, when available, for conversation continuity. | ||
| 4. The context information is added to the search response extensions. | ||
| 5. Type validation ensures that all context attributes are strings. | ||
|
|
||
| ## Request body fields | ||
|
|
||
| The following table lists all available request fields. | ||
|
|
||
| Field | Data type | Description | ||
| :--- | :--- | :--- | ||
| `agent_steps_summary` | Boolean | Whether to include the agent's execution steps summary in the response. Default is `false`. Optional. | ||
| `dsl_query` | Boolean | Whether to include the generated DSL query in the response. Default is `false`. Optional. | ||
|
|
||
| ## Response fields | ||
|
|
||
| When enabled, the processor adds the following fields to the search response extensions. | ||
|
|
||
| Field | Description | ||
| :--- | :--- | ||
| `agent_steps_summary` | A summary of the steps that the agent took to translate the natural language query (included when `agent_steps_summary` is `true`). | ||
| `memory_id` | The conversation memory ID for maintaining context across queries. Only provide this in the `agentic` query if you want to continue the previous conversation. | ||
| `dsl_query` | The generated DSL query that was executed (included when `dsl_query` is `true`). | ||
|
|
||
| ## Example | ||
|
|
||
| The following example request creates a search pipeline with an `agentic_context` response processor: | ||
|
|
||
| ```json | ||
| PUT /_search/pipeline/agentic_pipeline | ||
| { | ||
| "request_processors": [ | ||
| { | ||
| "agentic_query_translator": { | ||
| "agent_id": "your-agent-id" | ||
| } | ||
| } | ||
| ], | ||
| "response_processors": [ | ||
| { | ||
| "agentic_context": { | ||
| "agent_steps_summary": true, | ||
| "dsl_query": true | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| Perform a search using the configured pipeline: | ||
|
|
||
| ```json | ||
| POST /your-index/_search?search_pipeline=agentic_search_pipeline | ||
| { | ||
| "query": { | ||
| "agentic": { | ||
| "query_text": "Show me shoes in white color", | ||
| "memory_id": "your memory id" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| The response contains the steps taken by the agent to translate the query, the memory ID, and the rewritten DSL query: | ||
|
|
||
| ```json | ||
| { | ||
| "took": 15, | ||
| "hits": { | ||
| "_shards": {...}, | ||
| "hits": [...] | ||
| }, | ||
| "ext": { | ||
| "agent_steps_summary": "I have these tools available: [ListIndexTool, IndexMappingTool, query_planner_tool]\\nFirst I used: ListIndexTool — input: \"\"; context gained: \"Discovered products-index which seems relevant for products and pricing context\"\\nSecond I used: IndexMappingTool — input: \"products-index\"; context gained: \"Confirmed presence of category and price fields in products-index\"\\nThird I used: query_planner_tool — qpt.question: \"Show me shoes that cost exactly 100 dollars.\"; index_name_provided: \"products-index\"\\nValidation: qpt output is valid and accurately reflects the request for shoes priced at 100 dollars.", | ||
| "memory_id": "WVhHiJkBnqovov2plcDH", | ||
| "dsl_query": "{\"query\":{\"bool\":{\"filter\":[{\"term\":{\"category\":\"shoes\"}},{\"term\":{\"price\":100.0}}]}}}" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Related pages | ||
|
|
||
| - [Agentic search queries]({{site.url}}{{site.baseurl}}/vector-search/ai-search/agentic-search) | ||
| - [Agents]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/agents/index/) | ||
| - [Agentic query translator processor]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/agentic-query-translator-processor/) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,132 @@ | ||
| --- | ||
| layout: default | ||
| title: Agentic query translator | ||
| nav_order: 5 | ||
| has_children: false | ||
| parent: Search processors | ||
| grand_parent: Search pipelines | ||
| --- | ||
|
|
||
| # Agentic query translator processor | ||
| **Introduced 3.2** | ||
| {: .label .label-purple } | ||
|
|
||
| The `agentic_query_translator` search request processor enables natural language search by translating user queries into OpenSearch query domain-specific language (DSL) queries using machine learning (ML) agents. It works with [agentic search queries]({{site.url}}{{site.baseurl}}/vector-search/ai-search/agentic-search) to provide conversational search capabilities: | ||
|
|
||
| 1. The processor sends the user's natural language query to the specified ML agent. | ||
| 2. The agent translates the query into OpenSearch DSL. | ||
| 3. The original query is replaced with the generated DSL query. | ||
|
|
||
| This processor only works with the `agentic` query type as the top-level query. | ||
| {: .note} | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| Before using the `agentic_query_translator` processor, you must have either a conversational or flow agent configured. For more information, see [Agents]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/agents/index/). | ||
|
|
||
| ## Request body fields | ||
|
|
||
| The following table lists all available request fields. | ||
|
|
||
| Field | Data type | Description | ||
| :--- | :--- | :--- | ||
| `agent_id` | String | The ID of the ML agent that will translate natural language queries into DSL queries. Required. | ||
|
|
||
|
|
||
| ## Example | ||
|
|
||
| The following example request creates a search pipeline with an `agentic_query_translator` processor: | ||
|
|
||
| ```json | ||
| PUT /_search/pipeline/agentic_search_pipeline | ||
| { | ||
| "request_processors": [ | ||
| { | ||
| "agentic_query_translator": { | ||
| "agent_id": "your-agent-id-here" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| ## Usage | ||
|
|
||
| To use the processor, run an `agentic` query: | ||
|
|
||
| ```json | ||
| POST /your-index/_search?search_pipeline=agentic_search_pipeline | ||
| { | ||
| "query": { | ||
| "agentic": { | ||
| "query_text": "Show me shoes in white color" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| The response contains the matching documents: | ||
|
|
||
| ```json | ||
| { | ||
| "took": 6031, | ||
| "timed_out": false, | ||
| "_shards": { | ||
| "total": 8, | ||
| "successful": 8, | ||
| "skipped": 0, | ||
| "failed": 0 | ||
| }, | ||
| "hits": { | ||
| "total": { | ||
| "value": 8, | ||
| "relation": "eq" | ||
| }, | ||
| "max_score": 0.0, | ||
| "hits": [ | ||
| { | ||
| "_index": "products-index", | ||
| "_id": "43", | ||
| "_score": 0.0, | ||
| "_source": { | ||
| "product_name": "Nike Air Max white", | ||
| "description": "Red cushioned sneakers", | ||
| "price": 140.0, | ||
| "currency": "USD", | ||
| "in_stock": true, | ||
| "color": "white", | ||
| "size": "10", | ||
| "product_id": "P6001", | ||
| "category": "shoes", | ||
| "brand": "Nike" | ||
| } | ||
| }, | ||
| { | ||
| "_index": "products-index", | ||
| "_id": "45", | ||
| "_score": 0.0, | ||
| "_source": { | ||
| "product_name": "Adidas Superstar white", | ||
| "description": "Classic black sneakers", | ||
| "price": 100.0, | ||
| "currency": "USD", | ||
| "in_stock": true, | ||
| "color": "white", | ||
| "size": "8", | ||
| "product_id": "P6003", | ||
| "category": "shoes", | ||
| "brand": "Adidas" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Related pages | ||
|
|
||
| - [Agentic search queries]({{site.url}}{{site.baseurl}}/vector-search/ai-search/agentic-search) | ||
| - [Agents]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/agents/index/) | ||
| - [Agentic context processor]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/agentic-context-processor/) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How is memory ID provided? Do we need to mention that memory ID will always be provided as long as this response processor is added?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Memory id is not provided in the processor but in the query side. Can be taken care in the agentic search doc PR. I can add about the response that memory id would be present always