Skip to content

Create documentation for snapshots with hashed prefix path type #8196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Sep 11, 2024
Merged

Create documentation for snapshots with hashed prefix path type #8196

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f18a5e3
Create documentation for snapshots with hashed prefix path type
ashking94 Sep 9, 2024
656459a
Add documentation on new cluster settings for fixed prefix
ashking94 Sep 9, 2024
ce762a0
Update create-repository.md
Naarcha-AWS Sep 10, 2024
2d5e231
Update create-repository.md
Naarcha-AWS Sep 10, 2024
6e97500
Apply suggestions from code review
Naarcha-AWS Sep 10, 2024
5e9e2aa
Apply suggestions from code review
Naarcha-AWS Sep 10, 2024
5e7ca66
Apply suggestions from code review
Naarcha-AWS Sep 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions _api-reference/snapshots/create-repository.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,15 @@
Request field | Description
:--- | :---
`prefix_mode_verification` | When enabled, adds a hashed value of a random seed to the prefix for repository verification. For remote-store-enabled clusters, you can add the `setting.prefix_mode_verification` setting to the node attributes for the supplied repository. This field works with both new and existing repositories. Optional.
`shard_path_type` | Controls the path stucture of shard-level blobs. Supported values are `FIXED`, `HASHED_PREFIX`, or `HASHED_INFIX`. For more information about each value, see [shard_path_type values](#shard_path_type-values)/. Default is `FIXED`. Optional.

Check failure on line 46 in _api-reference/snapshots/create-repository.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: stucture. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: stucture. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_api-reference/snapshots/create-repository.md", "range": {"start": {"line": 46, "column": 39}}}, "severity": "ERROR"}

Check failure on line 46 in _api-reference/snapshots/create-repository.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.HeadingCapitalization] 'values' is a heading and should be in sentence case.

Raw Output:
{"message": "[OpenSearch.HeadingCapitalization] 'values' is a heading and should be in sentence case.", "location": {"path": "_api-reference/snapshots/create-repository.md", "range": {"start": {"line": 46, "column": 221}}}, "severity": "ERROR"}

#### shard_path_type values

The following values are supported in the `shard_path_type` setting:

- `FIXED`: Keeps the path structure in the existing hierarchical manner, such as, `<ROOT>/<BASE_PATH>/indices/<index-id>/0/<SHARD_BLOBS>`
- `HASHED_PREFIX`: Prepends a hashed prefix at the start of path for each unique shard ID, for example, `<ROOT>/<HASH-OF-INDEX-ID-AND-SHARD-ID>/<BASE_PATH>/indices/<index-id>/0/<SHARD_BLOBS>`.
- `HASHED_INFIX`: Appends a hashed prefix after the base path for each unique shard ID, for example, `<ROOT>/<BASE-PATH>/<HASH-OF-INDEX-ID-AND-SHARD-ID>/indices/<index-id>/0/<SHARD_BLOBS>`. The hash method used is `fnv_1a_composite_1, which uses the `FNV1a` hash function and generates a custom-encoded 64-bit hash value that scales well with most remote store options. `FNV1a` takes the most significant 6 bits to create a url-safe base64 character and the next 14 bits to create a binary string.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line above: Same question re: fnv_1a_composite_1. I believe it may have been in all caps in a previous PR. Also, it looks like you have an unclosed tic mark here.

### fs repository

Expand Down
6 changes: 6 additions & 0 deletions _install-and-configure/configuring-opensearch/index-settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,12 @@ OpenSearch supports the following dynamic cluster-level index settings:

- `cluster.remote_store.segment.transfer_timeout` (Time unit): Controls the maximum amount of time to wait for all new segments to update after refresh to the remote store. If the upload does not complete within a specified amount of time, it throws a `SegmentUploadFailedException` error. Default is `30m`. It has a minimum constraint of `10m`.

- `cluster.remote_store.translog.path.prefix` (String): Controls the fixed path prefix for translog data on a remote store enabled cluster. This setting is effective when `cluster.remote_store.index.path.type` setting is either `hashed_prefix` or `hashed_infix`. This defaults to `""` (empty string).

- `cluster.remote_store.segments.path.prefix` (String): Controls the fixed path prefix for segments data on a remote store enabled cluster. This setting is effective when `cluster.remote_store.index.path.type` setting is either `hashed_prefix` or `hashed_infix`. This defaults to `""` (empty string).

- `cluster.snapshot.shard.path.prefix` (String): Controls the fixed path prefix for snapshot shard level blobs. This setting is effective when repository `shard_path_type` setting is either `hashed_prefix` or `hashed_infix`. This defaults to `""` (empty string).

## Index-level index settings

You can specify index settings at index creation. There are two types of index settings:
Expand Down
Loading