Skip to content

Create documentation for snapshots with hashed prefix path type #8196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Sep 11, 2024
Merged

Create documentation for snapshots with hashed prefix path type #8196

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f18a5e3
Create documentation for snapshots with hashed prefix path type
ashking94 Sep 9, 2024
656459a
Add documentation on new cluster settings for fixed prefix
ashking94 Sep 9, 2024
ce762a0
Update create-repository.md
Naarcha-AWS Sep 10, 2024
2d5e231
Update create-repository.md
Naarcha-AWS Sep 10, 2024
6e97500
Apply suggestions from code review
Naarcha-AWS Sep 10, 2024
5e9e2aa
Apply suggestions from code review
Naarcha-AWS Sep 10, 2024
5e7ca66
Apply suggestions from code review
Naarcha-AWS Sep 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions _api-reference/snapshots/create-repository.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,14 @@
Request field | Description
:--- | :---
`prefix_mode_verification` | When enabled, adds a hashed value of a random seed to the prefix for repository verification. For remote-store-enabled clusters, you can add the `setting.prefix_mode_verification` setting to the node attributes for the supplied repository. This field works with both new and existing repositories. Optional.
`shard_path_type` | This setting is used to control the path stucture of shard level blobs. Defaults to fixed. There are 2 other possible values - 1. `hashed_prefix` 2. `hashed_infix`. More details are shared below. Optional.

Check failure on line 46 in _api-reference/snapshots/create-repository.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: stucture. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: stucture. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_api-reference/snapshots/create-repository.md", "range": {"start": {"line": 46, "column": 62}}}, "severity": "ERROR"}

Check warning on line 46 in _api-reference/snapshots/create-repository.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.DirectionAboveBelow] Use 'following or later' instead of 'below' for versions or orientation within a document. Use 'above' and 'below' only for physical space or screen descriptions.

Raw Output:
{"message": "[OpenSearch.DirectionAboveBelow] Use 'following or later' instead of 'below' for versions or orientation within a document. Use 'above' and 'below' only for physical space or screen descriptions.", "location": {"path": "_api-reference/snapshots/create-repository.md", "range": {"start": {"line": 46, "column": 210}}}, "severity": "WARNING"}

##### shard_path_type values

Check failure on line 48 in _api-reference/snapshots/create-repository.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.HeadingCapitalization] 'values' is a heading and should be in sentence case.

Raw Output:
{"message": "[OpenSearch.HeadingCapitalization] 'values' is a heading and should be in sentence case.", "location": {"path": "_api-reference/snapshots/create-repository.md", "range": {"start": {"line": 48, "column": 23}}}, "severity": "ERROR"}
1. FIXED - This keep the path structure in the existing hierarchical manner. eg - `<ROOT>/<BASE_PATH>/indices/<index-id>/0/<SHARD_BLOBS>`

Check warning on line 49 in _api-reference/snapshots/create-repository.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.LatinismsSubstitution] Use 'for example or such as' instead of 'eg'.

Raw Output:
{"message": "[OpenSearch.LatinismsSubstitution] Use 'for example or such as' instead of 'eg'.", "location": {"path": "_api-reference/snapshots/create-repository.md", "range": {"start": {"line": 49, "column": 78}}}, "severity": "WARNING"}

Check failure on line 49 in _api-reference/snapshots/create-repository.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: eg. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: eg. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_api-reference/snapshots/create-repository.md", "range": {"start": {"line": 49, "column": 78}}}, "severity": "ERROR"}
2. HASHED_PREFIX - This prepends a hashed prefix at the start of path for each unique shard id. eg - `<ROOT>/<HASH-OF-INDEX-ID-AND-SHARD-ID>/<BASE_PATH>/indices/<index-id>/0/<SHARD_BLOBS>`

Check warning on line 50 in _api-reference/snapshots/create-repository.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.LatinismsSubstitution] Use 'for example or such as' instead of 'eg'.

Raw Output:
{"message": "[OpenSearch.LatinismsSubstitution] Use 'for example or such as' instead of 'eg'.", "location": {"path": "_api-reference/snapshots/create-repository.md", "range": {"start": {"line": 50, "column": 97}}}, "severity": "WARNING"}

Check failure on line 50 in _api-reference/snapshots/create-repository.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: eg. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: eg. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_api-reference/snapshots/create-repository.md", "range": {"start": {"line": 50, "column": 97}}}, "severity": "ERROR"}
3. HASHED_INFIX - This appends a hashed prefix after the base path for each unique shard id. eg - `<ROOT>/<BASE-PATH>/<HASH-OF-INDEX-ID-AND-SHARD-ID>/indices/<index-id>/0/<SHARD_BLOBS>`

Check warning on line 51 in _api-reference/snapshots/create-repository.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.LatinismsSubstitution] Use 'for example or such as' instead of 'eg'.

Raw Output:
{"message": "[OpenSearch.LatinismsSubstitution] Use 'for example or such as' instead of 'eg'.", "location": {"path": "_api-reference/snapshots/create-repository.md", "range": {"start": {"line": 51, "column": 94}}}, "severity": "WARNING"}

Check failure on line 51 in _api-reference/snapshots/create-repository.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: eg. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: eg. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_api-reference/snapshots/create-repository.md", "range": {"start": {"line": 51, "column": 94}}}, "severity": "ERROR"}

Note - The hash method used is `fnv_1a_composite_1`. It uses the FNV1a hash function and generates a custom encoded hash value that scales well with most remote store options. The FNV1a function generates 64-bit value. The custom encoding uses the most significant 6 bits to create a url-safe base64 character and the next 14 bits to create a binary string.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line above: Same question re: fnv_1a_composite_1. I believe it may have been in all caps in a previous PR. Also, it looks like you have an unclosed tic mark here.

### fs repository

Expand Down
6 changes: 6 additions & 0 deletions _install-and-configure/configuring-opensearch/index-settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,12 @@ OpenSearch supports the following dynamic cluster-level index settings:

- `cluster.remote_store.segment.transfer_timeout` (Time unit): Controls the maximum amount of time to wait for all new segments to update after refresh to the remote store. If the upload does not complete within a specified amount of time, it throws a `SegmentUploadFailedException` error. Default is `30m`. It has a minimum constraint of `10m`.

- `cluster.remote_store.translog.path.prefix` (String): Controls the fixed path prefix for translog data on a remote store enabled cluster. This setting is effective when `cluster.remote_store.index.path.type` setting is either `hashed_prefix` or `hashed_infix`. This defaults to `""` (empty string).

- `cluster.remote_store.segments.path.prefix` (String): Controls the fixed path prefix for segments data on a remote store enabled cluster. This setting is effective when `cluster.remote_store.index.path.type` setting is either `hashed_prefix` or `hashed_infix`. This defaults to `""` (empty string).

- `cluster.snapshot.shard.path.prefix` (String): Controls the fixed path prefix for snapshot shard level blobs. This setting is effective when repository `shard_path_type` setting is either `hashed_prefix` or `hashed_infix`. This defaults to `""` (empty string).

## Index-level index settings

You can specify index settings at index creation. There are two types of index settings:
Expand Down
Loading