Merge branch 'main' into span-queries

Author: kolchfa-aws

.github/vale/styles/Vocab/OpenSearch/Products/accept.txt

@@ -6,11 +6,13 @@ Amazon OpenSearch Serverless
 Amazon OpenSearch Service
 Amazon Bedrock
 Amazon SageMaker
+AWS Secrets Manager
 Ansible
 Anthropic Claude
 Auditbeat
 AWS Cloud
 Cohere Command
+Cohere Embed
 Cohere Embed English
 Cohere Embed Multilingual
 Cognito
@@ -50,6 +52,7 @@ Kerberos
 Kibana
 Kubernetes
 Lambda
+Langflow
 Linux
 Log4j
 Logstash
@@ -93,6 +96,7 @@ Ruby
 Simple Schema for Observability
 Tableau
 Textract
+Titan
 Titan Multimodal Embeddings
 Titan Text Embeddings
 TorchScript

.github/workflows/automerge-backport.yml

@@ -25,7 +25,7 @@ jobs:
           MERGE_LABELS: "backport-automerge,!On hold"
           MERGE_FILTER_AUTHOR: "opensearch-trigger-bot[bot]"
           MERGE_REQUIRED_APPROVALS: "1"
-          MERGE_RETRIES: "20"
+          MERGE_RETRIES: "30"
           MERGE_RETRY_SLEEP: "10000"
           MERGE_ERROR_FAIL: "true"
           MERGE_FORKS: "false"

.vale.ini

@@ -12,8 +12,8 @@ BlockIgnores = {%-?\s*comment[.|\s|\S]*?endcomment\s*-?%}, \
     {%\s+[^%]*%} 
 
 # ignore variables
-TokenIgnores = [a-zA-Z_]+((?:_|\.)[a-zA-Z]+)+
-         
+TokenIgnores = [_a-zA-Z][_a-zA-Z0-9]*(?:[_\.][_a-zA-Z0-9]+)+
+
 # override Vale spelling               
 Vale.Spelling = NO 
 Vale.Repetition = NO

DEVELOPER_GUIDE.md

@@ -14,6 +14,7 @@
 The `.md` documents in this repository are rendered into HTML pages using [Jekyll](https://jekyllrb.com/). These HTML pages are hosted on [opensearch.org](https://opensearch.org/docs/latest/).
 
 ## Starting the Jekyll server locally
+
 You can run the Jekyll server locally to view the rendered HTML pages using the following steps:
 
 1. Install [Ruby](https://www.ruby-lang.org/en/documentation/installation/) 3.1.0 or later for your operating system.
@@ -22,6 +23,7 @@ You can run the Jekyll server locally to view the rendered HTML pages using the
 4. Open your browser and navigate to `http://localhost:4000` to view the rendered HTML pages.
 
 ## Using the `spec-insert` Jekyll plugin
+
 The `spec-insert` Jekyll plugin is used to insert API components into Markdown files. The plugin downloads the [latest OpenSearch specification](https://github.com/opensearch-project/opensearch-api-specification) and renders the API components from the spec. This aims to reduce the manual effort required to keep the documentation up to date.
 
 To use this plugin, make sure that you have installed Ruby 3.1.0 or later and the required gems by running `bundle install`.
@@ -52,7 +54,7 @@ If you are working on multiple Markdown files and do not want to keep running th
 bundle exec jekyll spec-insert -W
 ```
 
-By default, when the plugin encounters an error when processing a file, the plugin prints out the error than moves on to the next file. If you want it to short-circuit when an error occurs, add the `--fail-on-error` (or `-F`) flag to the command:
+By default, when the plugin encounters an error when processing a file, the plugin prints out the error then moves on to the next file. If you want it to short-circuit when an error occurs, add the `--fail-on-error` (or `-F`) flag to the command:
 
 ```shell
 bundle exec jekyll spec-insert -F
@@ -67,24 +69,28 @@ bundle exec jekyll spec-insert --refresh-spec
 ```
 
 ### Ignoring files and folders
+
 The `spec-insert` plugin ignores all files and folders listed in the [./_config.yml#exclude](./_config.yml) list, which is also the list of files and folders that Jekyll ignores.
 
 ### Configuration
+
 You can update the configuration settings for this plugin through the [config.yml](./spec-insert/config.yml) file. 
 
-_Note that tests for this plugin use a mock configuration [file](./spec-insert/spec/mock_config.yml) to assure that the tests still pass when the config file is altered. The expected output for the tests is based on the mock configuration file and will look different from the actual output when the plugin is run._
+**Note:** The tests for this plugin use a mock configuration [file](./spec-insert/spec/mock_config.yml) to assure that the tests still pass when the config file is altered. The expected output for the tests is based on the mock configuration file and will look different from the actual output when the plugin is run.
 
 ## CI/CD
 The `spec-insert` plugin is run as part of the CI/CD pipeline to ensure that the API components are up to date in the documentation. This is performed through the [update-api-components.yml](.github/workflows/update-api-components.yml) GitHub Actions workflow, which creates a pull request containing the updated API components every Sunday.
 
 ## Spec insert components
 All spec insert components accept the following arguments:
+
 - `api` (String; required): The name of the API to render the component from. This is equivalent to the `x-operation-group` field in the OpenSearch OpenAPI Spec.
 - `component` (String; required): The name of the component to render, such as `query_parameters`, `path_parameters`, or `endpoints`.
 - `omit_header` (Boolean; Default is `false`): If set to `true`, the markdown header of the component will not be rendered.
 
 ### Endpoints
 To insert endpoints for the `search` API, use the following snippet:
+
 ```markdown
 <!-- spec_insert_start
 api: search
@@ -104,10 +110,13 @@ component: path_parameters
 -->
 <!-- spec_insert_end -->
 ```
+
 This table accepts the same arguments as the query parameters table except the `include_global` argument.
 
 ### Query parameters
+
 To insert the API query parameters table of the `cat.indices` API, use the following snippet:
+
 ```markdown
 <!-- spec_insert_start
 api: cat.indices
@@ -144,3 +153,28 @@ pretty: true
 -->
 <!-- spec_insert_end -->
 ```
+
+### Request and response bodies (Beta)
+
+To insert the request and response body tables of the `indices.create` API, use the following snippet:
+
+```markdown
+<!-- spec_insert_start
+api: indices.create
+component: request_body_parameters // or response_body_parameters
+-->
+<!-- spec_insert_end -->
+```
+
+**Note:**: These components are still a work in progress and may not render correctly for all APIs.
+
+## Spec insert coverage report
+To generate a coverage report of the API components that are being used in the documentation, run the following command:
+
+```shell
+cd spec-insert
+bundle exec rake generate_utilization_coverage
+```
+
+The coverage report will be generated in the `spec-insert/utilization_coverage.md` by default.
+

FORMATTING_GUIDE.md

@@ -9,6 +9,7 @@ This guide provides an overview of the formatted elements commonly used in the O
 * [Adding pages or sections](#adding-pages-or-sections)
 * [Buttons](#buttons)
 * [Callouts](#callouts)
+* [Cards](#cards)
 * [Collapsible blocks](#collapsible-blocks)
 * [Dashes](#dashes)
 * [Horizontal rule](#horizontal-rule)
@@ -22,6 +23,7 @@ This guide provides an overview of the formatted elements commonly used in the O
   * [Nested lists](#nested-lists) 
   * [Lists with code snippets or images](#lists-with-code-snippets-or-images)
 * [Math](#math)
+* [Steps](#steps)
 * [Tables](#tables)
 * [Text style](#text-style)
 * [Variables in curly braces](#variables-in-curly-braces)
@@ -109,6 +111,27 @@ For a callout with multiple paragraphs or lists, use `>`:
 
 ```
 
+## Cards
+
+To add a card to a page, specify it in the front matter as follows. The `description`, `link`, and `list` are optional. Use relative links. You can optionally style the text using HTML tags:
+
+```yaml
+tutorial_cards:
+  - heading: "Getting started with semantic and hybrid search"
+    description: "Learn how to implement semantic and hybrid search"
+    link: "/vector-search/tutorials/neural-search-tutorial/"
+    list:
+      - "<b>Platform:</b> OpenSearch"
+      - "<b>Model:</b> Anthropic Claude" 
+      - "<b>Deployment:</b> Amazon Bedrock"  
+```
+
+Insert an include in the page body where you want the cards to appear:
+
+```
+{% include cards.html cards=page.tutorial_cards %}  
+```
+
 ## Collapsible blocks
 
 To insert an open collapsible block, use the `<details>` element as follows:
@@ -405,6 +428,26 @@ Alternatively, you can use double dollar signs (`$$`) for both display and inlin
 The probability of selecting pair $$i$$ is proportional to $$1 \over i^\alpha$$.
 ```
 
+## Steps
+
+To insert steps, specify them in the front matter as follows. Steps are automatically numbered. Use relative links. The `description` and `link` are optional:
+
+```yaml
+steps:
+  - heading: "Create an OpenSearch index"
+    description: "Create an OpenSearch index to store your embeddings."
+    link: "/vector-search/creating-vector-index/#storing-raw-vectors-or-embeddings-generated-outside-of-opensearch"
+  - heading: "Ingest embeddings"
+    description: "Ingest your embeddings into the index."
+    link: "/vector-search/ingesting-data/#raw-vector-ingestion"
+```
+
+Insert an include in the page body where you want the steps to appear:
+
+```
+{% include list.html list_items=page.steps%}
+```
+
 ## Tables
 
 Markdown table columns are automatically sized, and there is no need to specify a different number of dashes in the formatting. 

_about/breaking-changes.md

@@ -41,4 +41,121 @@ A Lucene upgrade forced OpenSearch to drop support for JDK 8. As a consequence,
 
 ### Wildcard query behavior for text fields
 
-OpenSearch 2.5 contains a bug fix to correct the behavior of the `case_insensitive` parameter for the `wildcard` query on text fields. As a result, a wildcard query on text fields that ignored case sensitivity and erroneously returned results prior to the bug fix will not return the same results. For more information, see issue [#8711](https://github.com/opensearch-project/OpenSearch/issues/8711).
+OpenSearch 2.5 contains a bug fix that corrects the behavior of the `case_insensitive` parameter for the `wildcard` query on text fields. As a result, a wildcard query on text fields that ignored case sensitivity and erroneously returned results prior to the bug fix will not return the same results. For more information, see issue [#8711](https://github.com/opensearch-project/OpenSearch/issues/8711).
+
+## 3.0.0
+
+### JDK requirement
+
+The minimum supported JDK version is JDK 21.
+
+### System index access
+
+Access to system indexes through the REST API is no longer provided. This functionality has been deprecated since OpenSearch 1.x. For more information, see issue [#7936](https://github.com/opensearch-project/OpenSearch/issues/7936).
+
+### Document ID length limits
+
+The document ID length limit of 512 bytes is now consistently enforced across all APIs, including the Bulk API. Previously, the Bulk API allowed document IDs longer than 512 bytes. For more information, see issue [#6595](https://github.com/opensearch-project/OpenSearch/issues/6595).
+
+### Node role configuration
+
+The configuration of empty node roles using environment variables has been fixed. Setting `node.roles=` using environment variables now properly configures a coordinating-only node, consistent with the `opensearch.yml` configuration. For more information, see issue [#3412](https://github.com/opensearch-project/OpenSearch/issues/3412).
+
+### JSON processing limits
+
+New default limits have been introduced for JSON processing (using the Jackson library) throughout OpenSearch:
+
+- The maximum nesting depth of JSON objects and arrays is limited to 1,000 levels.
+- The maximum length of JSON property names is limited to 50,000 units (bytes or chars, depending on the input source).
+
+These limits help prevent potential memory issues and denial-of-service attacks. For more information, see issue [#11278](https://github.com/opensearch-project/OpenSearch/issues/11278).
+
+### Nested query depth
+
+A new `index.query.max_nested_depth` setting has been introduced with a default value of `20` and a minimum value of `1`, limiting the maximum number of nesting levels for `nested` queries. For more information, see issue [#3268](https://github.com/opensearch-project/OpenSearch/issues/3268).
+
+### Thread pool settings
+
+The following deprecated thread pool settings have been removed:
+- `thread_pool.test.max_queue_size`
+- `thread_pool.test.min_queue_size`
+For more information, see issue [#2595](https://github.com/opensearch-project/OpenSearch/issues/2595).
+
+### Index store settings
+
+The `index.store.hybrid.mmap.extensions` setting has been removed as part of improvements to `hybridfs` file handling. For more information, see pull request [#9392](https://github.com/opensearch-project/OpenSearch/pull/9392).
+
+### Transport Nio plugin
+
+The `transport-nio` plugin has been removed. Netty remains the standard network framework for both node-to-node and client-to-server communication. For more information, see issue [#16887](https://github.com/opensearch-project/OpenSearch/issues/16887).
+
+### Nodes API response format
+
+The format of indexing buffer values in the Nodes API response has changed:
+
+- `total_indexing_buffer_in_bytes` now displays raw bytes (for example, `53687091`).
+- `total_indexing_buffer` now displays human-readable format (for example, `51.1mb`).
+
+For more information, see pull request [#17070](https://github.com/opensearch-project/OpenSearch/pull/17070).
+
+### PathHierarchy tokenizer
+
+The camel case `PathHierarchy` tokenizer name has been deprecated in favor of the snake case `path_hierarchy`. For more information, see pull request [#10894](https://github.com/opensearch-project/OpenSearch/pull/10894).
+
+### Security plugin
+
+The Blake2b hash implementation now uses the salt parameter correctly, which will result in different (though correct) hash values compared to previous versions. For more information, see pull request [#5089](https://github.com/opensearch-project/security/pull/5089).
+
+### k-NN plugin
+
+The following deprecated settings have been removed from the k-NN plugin:
+
+- `knn.plugin.enabled` setting
+- `index.knn.algo_param.ef_construction` index setting
+- `index.knn.algo_param.m` index setting
+- `index.knn.space_type` index setting
+
+The NMSLIB engine is now deprecated. We recommend using the Faiss or Lucene engines instead.
+
+For more information, see pull request [#2564](https://github.com/opensearch-project/k-NN/pull/2564).
+
+### Performance Analyzer plugin
+
+The `performance-analyzer-rca` agent has been removed. We recommend transitioning to the [Telemetry plugin](https://github.com/opensearch-project/performance-analyzer/issues/585) for performance monitoring and analysis. The Telemetry plugin, using the OpenTelemetry framework, allows for seamless integration with lightweight open-source agents in order to publish performance metrics to observability stores. For more information, see issue [#591](https://github.com/opensearch-project/performance-analyzer-rca/issues/591).
+
+### SQL plugin
+
+- The OpenSearch query domain-specific language (DSL) response format has been removed.
+- `DELETE` statement support has been removed.
+- The `plugins.sql.delete.enabled` setting has been removed.
+- The legacy Spark Connector module has been deprecated. For information about connecting to Spark, see [`async-query-core`](https://github.com/opensearch-project/sql/blob/main/async-query-core/README.md).
+- Deprecated OpenDistro endpoints and legacy settings with the `opendistro` prefix have been removed.
+- The `plugins.sql.pagination.api` has been removed and the Scroll API has been deprecated. Pagination now defaults to Point in Time.
+
+For more information, see issue [#3248](https://github.com/opensearch-project/sql/issues/3248).
+
+### OpenSearch Dashboards
+
+- Discover experience:
+
+    - The `discover:newExperience` setting has been removed.
+    - The DataGrid table feature has been removed.
+
+    For more information, see pull request [#9511](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/9511).
+
+- Visualizations: The `dashboards-visualizations` plugin (including Gantt chart visualization) has been removed. We recommend transitioning to:
+
+    - Vega visualization for flexible visualization needs.
+    - Trace analytics for trace-related use cases.
+
+    For more information, see issue [#430](https://github.com/opensearch-project/dashboards-visualizations/issues/430).
+
+### Dashboards Observability plugin
+
+The legacy notebooks feature has been removed from `dashboards-observability`. Key changes include the following:
+
+- Legacy notebooks (previously stored in the `.opensearch-observability` index) are no longer supported.
+- Only notebooks stored in the `.kibana` index (introduced in version 2.17) are supported.
+- You must migrate your notebooks to the new storage system before upgrading to version 3.0.
+
+For more information, see issue [#2350](https://github.com/opensearch-project/dashboards-observability/issues/2350).