Merge branch 'main' into top-n

kolchfa-aws · web-flow · commit 74c1ab1b564b · 2024-09-04T15:02:54.000-04:00
diff --git a/_benchmark/user-guide/understanding-workloads/choosing-a-workload.md b/_benchmark/user-guide/understanding-workloads/choosing-a-workload.md
@@ -18,7 +18,7 @@ Consider the following criteria when deciding which workload would work best for
 
 - The cluster's use case. 
 - The data types that your cluster uses compared to the data structure of the documents contained in the workload. Each workload contains an example document so that you can compare data types, or you can view the index mappings and data types in the `index.json` file.
-- The query types most commonly used inside your cluster. The `operations/default.json` file contains information about the query types and workload operations. 
+- The query types most commonly used inside your cluster. The `operations/default.json` file contains information about the query types and workload operations. For a list of common operations, see [Common operations]({{site.url}}{{site.baseurl}}/benchmark/user-guide/understanding-workloads/common-operations/).
 
 ## General search clusters
 
diff --git a/_benchmark/user-guide/understanding-workloads/common-operations.md b/_benchmark/user-guide/understanding-workloads/common-operations.md
@@ -0,0 +1,181 @@
+---
+layout: default
+title: Common operations
+nav_order: 16
+grand_parent: User guide
+parent: Understanding workloads
+---
+
+# Common operations
+
+[Test procedures]({{site.url}}{{site.baseurl}}/benchmark/user-guide/understanding-workloads/anatomy-of-a-workload#_operations-and-_test-procedures) use a variety of operations, found inside the `operations` directory of a workload. This page details the most common operations found inside OpenSearch Benchmark workloads.
+
+- [Common operations](#common-operations)
+  - [bulk](#bulk)
+  - [create-index](#create-index)
+  - [delete-index](#delete-index)
+  - [cluster-health](#cluster-health)
+  - [refresh](#refresh)
+  - [search](#search)
+
+<!-- vale off -->
+## bulk
+<!-- vale on -->
+
+The `bulk` operation type allows you to run [bulk](/api-reference/document-apis/bulk/) requests as a task. 
+
+The following example shows a `bulk` operation type with a `bulk-size` of `5000` documents:
+
+```yml
+{
+  "name": "index-append",
+  "operation-type": "bulk",
+  "bulk-size": 5000
+}
+```
+
+
+<!-- vale off -->
+## create-index
+<!-- vale on -->
+
+The `create-index` operation runs the [Create Index API](/api-reference/index-apis/create-index/). It supports the following two modes of index creation:
+
+- Creating all indexes specified in the workloads `indices` section
+- Creating one specific index defined within the operation itself
+
+The following example creates all indexes defined in the `indices` section of the workload. It uses all of the index settings defined in the workload but overrides the number of shards:
+
+```yml
+{
+  "name": "create-all-indices",
+  "operation-type": "create-index",
+  "settings": {
+    "index.number_of_shards": 1
+  },
+  "request-params": {
+    "wait_for_active_shards": "true"
+  }
+}
+```
+
+The following example creates a new index with all index settings specified in the operation body:
+
+```yml
+{
+  "name": "create-an-index",
+  "operation-type": "create-index",
+  "index": "people",
+  "body": {
+    "settings": {
+      "index.number_of_shards": 0
+    },
+    "mappings": {
+      "docs": {
+        "properties": {
+          "name": {
+            "type": "text"
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+
+
+<!-- vale off -->
+## delete-index
+<!-- vale on -->
+
+The `delete-index` operation runs the [Delete Index API](api-reference/index-apis/delete-index/). Like with the [`create-index`](#create-index) operation, you can delete all indexes found in the `indices` section of the workload or delete one or more indexes based on the string passed in the `index` setting.
+
+The following example deletes all indexes found in the `indices` section of the workload:
+
+```yml
+{
+  "name": "delete-all-indices",
+  "operation-type": "delete-index"
+}
+```
+
+The following example deletes all `logs_*` indexes:
+
+```yml
+{
+  "name": "delete-logs",
+  "operation-type": "delete-index",
+  "index": "logs-*",
+  "only-if-exists": false,
+  "request-params": {
+    "expand_wildcards": "all",
+    "allow_no_indices": "true",
+    "ignore_unavailable": "true"
+  }
+}
+```
+
+<!-- vale off -->
+## cluster-health
+<!-- vale on -->
+
+The `cluster-health` operation runs the [Cluster Health API](api-reference/cluster-api/cluster-health/), which checks the cluster health status and returns the expected status according to the parameters set for `request-params`. If an unexpected cluster health status is returned, the operation reports a failure. You can use the `--on-error` option in the OpenSearch Benchmark `execute-test` command to control how OpenSearch Benchmark behaves when the health check fails.
+
+The following example creates a `cluster-health` operation that checks for a `green` health status on any `log-*` indexes:
+
+```yml
+{
+  "name": "check-cluster-green",
+  "operation-type": "cluster-health",
+  "index": "logs-*",
+  "request-params": {
+    "wait_for_status": "green",
+    "wait_for_no_relocating_shards": "true"
+  },
+  "retry-until-success": true
+}
+
+```
+
+<!-- vale off -->
+## refresh
+<!-- vale on -->
+
+The `refresh` operation runs the Refresh API. The `operation` returns no metadata.
+
+
+The following example refreshes all `logs-*` indexes:
+
+```yml
+{
+ "name": "refresh",
+ "operation-type": "refresh",
+ "index": "logs-*"
+}
+```
+
+
+<!-- vale off -->
+## search
+<!-- vale on -->
+
+The `search` operation runs the [Search API](/api-reference/search/), which you can use to run queries in OpenSearch Benchmark indexes.
+
+The following example runs a `match_all` query inside the `search` operation:
+
+```yml
+{
+  "name": "default",
+  "operation-type": "search",
+  "body": {
+    "query": {
+      "match_all": {}
+    }
+  },
+  "request-params": {
+    "_source_include": "some_field",
+    "analyze_wildcard": "false"
+  }
+}
+```
diff --git a/_data-prepper/pipelines/configuration/sinks/s3.md b/_data-prepper/pipelines/configuration/sinks/s3.md
@@ -173,14 +173,14 @@ When you provide your own Avro schema, that schema defines the final structure o
 
 In cases where your data is uniform, you may be able to automatically generate a schema. Automatically generated schemas are based on the first event that the codec receives. The schema will only contain keys from this event, and all keys must be present in all events in order to automatically generate a working schema. Automatically generated schemas make all fields nullable. Use the `include_keys` and `exclude_keys` sink configurations to control which data is included in the automatically generated schema.
 
-Avro fields should use a null [union](https://avro.apache.org/docs/1.10.2/spec.html#Unions) because this will allow missing values. Otherwise, all required fields must be present for each event. Use non-nullable fields only when you are certain they exist.
+Avro fields should use a null [union](https://avro.apache.org/docs/1.12.0/specification/#unions) because this will allow missing values. Otherwise, all required fields must be present for each event. Use non-nullable fields only when you are certain they exist.
 
 Use the following options to configure the codec.
 
 Option | Required | Type | Description
 :--- | :--- | :--- | :---
-`schema` | Yes | String | The Avro [schema declaration](https://avro.apache.org/docs/1.2.0/spec.html#schemas). Not required if `auto_schema` is set to true.
-`auto_schema` | No | Boolean | When set to `true`, automatically generates the Avro [schema declaration](https://avro.apache.org/docs/1.2.0/spec.html#schemas) from the first event.
+`schema` | Yes | String | The Avro [schema declaration](https://avro.apache.org/docs/1.12.0/specification/#schema-declaration). Not required if `auto_schema` is set to true.
+`auto_schema` | No | Boolean | When set to `true`, automatically generates the Avro [schema declaration](https://avro.apache.org/docs/1.12.0/specification/#schema-declaration) from the first event.
  
 ### `ndjson` codec
 
@@ -208,8 +208,8 @@ Use the following options to configure the codec.
 
 Option | Required | Type | Description
 :--- | :--- | :--- | :---
-`schema` | Yes | String | The Avro [schema declaration](https://avro.apache.org/docs/current/specification/#schema-declaration). Not required if `auto_schema` is set to true.
-`auto_schema` | No | Boolean | When set to `true`, automatically generates the Avro [schema declaration](https://avro.apache.org/docs/current/specification/#schema-declaration) from the first event.
+`schema` | Yes | String | The Avro [schema declaration](https://avro.apache.org/docs/1.12.0/specification/#schema-declaration). Not required if `auto_schema` is set to true.
+`auto_schema` | No | Boolean | When set to `true`, automatically generates the Avro [schema declaration](https://avro.apache.org/docs/1.12.0/specification/#schema-declaration) from the first event.
 
 ### Setting a schema with Parquet
 
diff --git a/_field-types/supported-field-types/binary.md b/_field-types/supported-field-types/binary.md
@@ -50,5 +50,5 @@ The following table lists the parameters accepted by binary field types. All par
 
 Parameter | Description 
 :--- | :--- 
-`doc_values` | A Boolean value that specifies whether the field should be stored on disk so that it can be used for aggregations, sorting, or scripting. Optional. Default is `true`.
-`store` | A Boolean value that specifies whether the field value should be stored and can be retrieved separately from the _source field. Optional. Default is `false`.
+`doc_values` | A Boolean value that specifies whether the field should be stored on disk so that it can be used for aggregations, sorting, or scripting. Optional. Default is `false`.
+`store` | A Boolean value that specifies whether the field value should be stored and can be retrieved separately from the _source field. Optional. Default is `false`.
diff --git a/_install-and-configure/configuring-opensearch/network-settings.md b/_install-and-configure/configuring-opensearch/network-settings.md
@@ -51,7 +51,7 @@ OpenSearch supports the following advanced network settings for transport commun
 
 ## Selecting the transport
 
-The default OpenSearch transport is provided by the `transport-netty4` module and uses the [Netty 4](https://netty.io/) engine for both internal TCP-based communication between nodes in the cluster and external HTTP-based communication with clients. This communication is fully asynchronous and non-blocking. However, there are other transport plugins available that can be used interchangeably:
+The default OpenSearch transport is provided by the `transport-netty4` module and uses the [Netty 4](https://netty.io/) engine for both internal TCP-based communication between nodes in the cluster and external HTTP-based communication with clients. This communication is fully asynchronous and non-blocking. The following table lists other available transport plugins that can be used interchangeably.
 
 Plugin | Description
 :---------- | :--------
diff --git a/_install-and-configure/configuring-opensearch/security-settings.md b/_install-and-configure/configuring-opensearch/security-settings.md
@@ -9,7 +9,7 @@ nav_order: 40
 
 The Security plugin provides a number of YAML configuration files that are used to store the necessary settings that define the way the Security plugin manages users, roles, and activity within the cluster. For a full list of the Security plugin configuration files, see [Modifying the YAML files]({{site.url}}{{site.baseurl}}/security/configuration/yaml/).
 
-The following sections describe security-related settings in `opensearch.yml`. To learn more about static and dynamic settings, see [Configuring OpenSearch]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index/).
+The following sections describe security-related settings in `opensearch.yml`. You can find the `opensearch.yml` in the `<OPENSEARCH_HOME>/config/opensearch.yml`. To learn more about static and dynamic settings, see [Configuring OpenSearch]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index/).
 
 ## Common settings
 
diff --git a/_security/configuration/index.md b/_security/configuration/index.md
@@ -28,4 +28,4 @@ The Security plugin has several default users, roles, action groups, permissions
 {: .note }
 
 For a full list of `opensearch.yml` Security plugin settings, Security plugin settings, see [Security settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/security-settings/).
-{: .note}
+{: .note}
diff --git a/_security/configuration/security-admin.md b/_security/configuration/security-admin.md
@@ -23,13 +23,13 @@ The `securityadmin.sh` script requires SSL/TLS HTTP to be enabled for your OpenS
 
 ## A word of caution
 
-If you make changes to the configuration files in `config/opensearch-security`, OpenSearch does _not_ automatically apply these changes. Instead, you must run `securityadmin.sh` to load the updated files into the index.
+If you make changes to the configuration files in `config/opensearch-security`, OpenSearch does _not_ automatically apply these changes. Instead, you must run `securityadmin.sh` to load the updated files into the index. The `securityadmin.sh` file can be found in `<OPENSEARCH_HOME>/plugins/opensearch-security/tools/securityadmin.[sh|bat]`.
 
 Running `securityadmin.sh` **overwrites** one or more portions of the `.opendistro_security` index. Run it with extreme care to avoid losing your existing resources. Consider the following example:
 
 1. You initialize the `.opendistro_security` index.
 1. You create ten users using the REST API.
-1. You decide to create a new [reserved user]({{site.url}}{{site.baseurl}}/security/access-control/api/#reserved-and-hidden-resources) using `internal_users.yml`.
+1. You decide to create a new [reserved user]({{site.url}}{{site.baseurl}}/security/access-control/api/#reserved-and-hidden-resources) using `internal_users.yml`, found in `<OPENSEARCH_HOME>/config/opensearch-security/` directory.
 1. You run `securityadmin.sh` again to load the new reserved user into the index.
 1. You lose all ten users that you created using the REST API.
 
diff --git a/_security/configuration/yaml.md b/_security/configuration/yaml.md
@@ -17,7 +17,7 @@ The approach we recommend for using the YAML files is to first configure [reserv
 
 ## action_groups.yml
 
-This file contains any initial action groups that you want to add to the Security plugin.
+This file contains any role mappings required for your security configuration. You can find the `role_mapping.yml` file in `<OPENSEARCH_HOME>/config/opensearch-security/roles_mapping.yml`.
 
 Aside from some metadata, the default file is empty, because the Security plugin has a number of static action groups that it adds automatically. These static action groups cover a wide variety of use cases and are a great way to get started with the plugin.
 
@@ -43,6 +43,8 @@ _meta:
 
 You can use `allowlist.yml` to add any endpoints and HTTP requests to a list of allowed endpoints and requests. If enabled, all users except the super admin are allowed access to only the specified endpoints and HTTP requests, and all other HTTP requests associated with the endpoint are denied. For example, if GET `_cluster/settings` is added to the allow list, users cannot submit PUT requests to `_cluster/settings` to update cluster settings.
 
+You can find the `allowlist.yml` file in `<OPENSEARCH_HOME>/config/opensearch-security/allowlist.yml`.
+
 Note that while you can configure access to endpoints this way, for most cases, it is still best to configure permissions using the Security plugin's users and roles, which have more granular settings.
 
 ```yml
@@ -92,7 +94,7 @@ requests: # Only allow GET requests to /sample-index1/_doc/1 and /sample-index2/
 
 ## internal_users.yml
 
-This file contains any initial users that you want to add to the Security plugin's internal user database.
+This file contains any initial users that you want to add to the Security plugin's internal user database. You can find this file in ``<OPENSEARCH_HOME>/config/opensearch-security/internal_users.yml`.
 
 The file format requires a hashed password. To generate one, run `plugins/opensearch-security/tools/hash.sh -p <new-password>`. If you decide to keep any of the demo users, *change their passwords* and re-run [securityadmin.sh]({{site.url}}{{site.baseurl}}/security/configuration/security-admin/) to apply the new passwords.
 
@@ -313,7 +315,7 @@ admin_tenant:
 
 ## opensearch.yml
 
-In addition to many OpenSearch settings, this file contains paths to TLS certificates and their attributes, such as distinguished names and trusted certificate authorities.
+In addition to many OpenSearch settings, the `opensearch.yml` file contains paths to TLS certificates and their attributes, such as distinguished names and trusted certificate authorities. You can find this file in  `<OPENSEARCH_HOME>/config/`.
 
 ```yml
 plugins.security.ssl.transport.pemcert_filepath: esnode.pem
diff --git a/_tuning-your-cluster/availability-and-recovery/snapshots/searchable_snapshot.md b/_tuning-your-cluster/availability-and-recovery/snapshots/searchable_snapshot.md
@@ -108,4 +108,5 @@ The following are known limitations of the searchable snapshots feature:
 - Many remote object stores charge on a per-request basis for retrieval, so users should closely monitor any costs incurred.
 - Searching remote data can impact the performance of other queries running on the same node. We recommend that users provision dedicated nodes with the `search` role for performance-critical applications.
 - For better search performance, consider [force merging]({{site.url}}{{site.baseurl}}/api-reference/index-apis/force-merge/) indexes into a smaller number of segments before taking a snapshot. For the best performance, at the cost of using compute resources prior to snapshotting, force merge your index into one segment.
-- We recommend configuring a maximum ratio of remote data to local disk cache size using the `cluster.filecache.remote_data_ratio` setting. A ratio of 5 is a good starting point for most workloads to ensure good query performance. If the ratio is too large, then there may not be sufficient disk space to handle the search workload. For more details on the maximum ratio of remote data, see issue [#11676](https://github.com/opensearch-project/OpenSearch/issues/11676). 
+- We recommend configuring a maximum ratio of remote data to local disk cache size using the `cluster.filecache.remote_data_ratio` setting. A ratio of 5 is a good starting point for most workloads to ensure good query performance. If the ratio is too large, then there may not be sufficient disk space to handle the search workload. For more details on the maximum ratio of remote data, see issue [#11676](https://github.com/opensearch-project/OpenSearch/issues/11676).
+- k-NN native-engine-based indexes using `faiss` and `nmslib` engines are incompatible with searchable snapshots.
