diff --git a/src/current/_includes/v25.1/backups/external-storage-check-tip.md b/src/current/_includes/v25.1/backups/external-storage-check-tip.md new file mode 100644 index 00000000000..b8cb7561f4c --- /dev/null +++ b/src/current/_includes/v25.1/backups/external-storage-check-tip.md @@ -0,0 +1 @@ +You can test the connection from each node in the cluster to your external storage with the [`CHECK EXTERNAL CONNECTION`]({% link {{ page.version.version }}/check-external-connection.md %}) statement. \ No newline at end of file diff --git a/src/current/_includes/v25.1/sidebar-data/sql.json b/src/current/_includes/v25.1/sidebar-data/sql.json index 148d7a6a3a7..2d2ea16d846 100644 --- a/src/current/_includes/v25.1/sidebar-data/sql.json +++ b/src/current/_includes/v25.1/sidebar-data/sql.json @@ -160,6 +160,12 @@ "/${VERSION}/cancel-session.html" ] }, + { + "title": "CHECK EXTERNAL CONNECTION", + "urls": [ + "/${VERSION}/check-external-connection.html" + ] + }, { "title": "COMMENT ON", "urls": [ diff --git a/src/current/_includes/v25.2/backups/external-storage-check-tip.md b/src/current/_includes/v25.2/backups/external-storage-check-tip.md new file mode 100644 index 00000000000..b8cb7561f4c --- /dev/null +++ b/src/current/_includes/v25.2/backups/external-storage-check-tip.md @@ -0,0 +1 @@ +You can test the connection from each node in the cluster to your external storage with the [`CHECK EXTERNAL CONNECTION`]({% link {{ page.version.version }}/check-external-connection.md %}) statement. \ No newline at end of file diff --git a/src/current/_includes/v25.2/sidebar-data/sql.json b/src/current/_includes/v25.2/sidebar-data/sql.json index b4b9e2a71b6..8759eae97e6 100644 --- a/src/current/_includes/v25.2/sidebar-data/sql.json +++ b/src/current/_includes/v25.2/sidebar-data/sql.json @@ -166,6 +166,12 @@ "/${VERSION}/cancel-session.html" ] }, + { + "title": "CHECK EXTERNAL CONNECTION", + "urls": [ + "/${VERSION}/check-external-connection.html" + ] + }, { "title": "COMMENT ON", "urls": [ diff --git a/src/current/_includes/v25.3/backups/external-storage-check-tip.md b/src/current/_includes/v25.3/backups/external-storage-check-tip.md new file mode 100644 index 00000000000..b8cb7561f4c --- /dev/null +++ b/src/current/_includes/v25.3/backups/external-storage-check-tip.md @@ -0,0 +1 @@ +You can test the connection from each node in the cluster to your external storage with the [`CHECK EXTERNAL CONNECTION`]({% link {{ page.version.version }}/check-external-connection.md %}) statement. \ No newline at end of file diff --git a/src/current/_includes/v25.3/sidebar-data/sql.json b/src/current/_includes/v25.3/sidebar-data/sql.json index b4b9e2a71b6..8759eae97e6 100644 --- a/src/current/_includes/v25.3/sidebar-data/sql.json +++ b/src/current/_includes/v25.3/sidebar-data/sql.json @@ -166,6 +166,12 @@ "/${VERSION}/cancel-session.html" ] }, + { + "title": "CHECK EXTERNAL CONNECTION", + "urls": [ + "/${VERSION}/check-external-connection.html" + ] + }, { "title": "COMMENT ON", "urls": [ diff --git a/src/current/v25.1/backup-validation.md b/src/current/v25.1/backup-validation.md index a4f00b7754a..44fde6cfdb4 100644 --- a/src/current/v25.1/backup-validation.md +++ b/src/current/v25.1/backup-validation.md @@ -15,6 +15,10 @@ You can validate a backup of a [cluster]({% link {{ page.version.version }}/back The options that give the most validation coverage will increase the runtime of the check. That is, `verify_backup_table_data` will take a longer time to validate a backup compared to `check_files` or `schema_only` alone. Despite that, each of these validation options provide a quicker way to validate a backup over running a "regular" restore. +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + ## Recommendations Cockroach Labs recommends implementing the following validation plan to test your backups: diff --git a/src/current/v25.1/backup.md b/src/current/v25.1/backup.md index e45b569f058..b93d5e3e8f6 100644 --- a/src/current/v25.1/backup.md +++ b/src/current/v25.1/backup.md @@ -44,6 +44,10 @@ To view the contents of an backup created with the `BACKUP` statement, use [`SHO - Modifying backup files in the storage location could invalidate a backup, and therefore, prevent a restore. In v22.1 and later, **we recommend enabling [object locking]({% link {{ page.version.version }}/use-cloud-storage.md %}#immutable-storage) in your cloud storage bucket.** - While Cockroach Labs actively tests Amazon S3, Google Cloud Storage, and Azure Storage, we **do not** test [S3-compatible services]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., [MinIO](https://min.io/), [Red Hat Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/)). +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + ## Required privileges {% include {{ page.version.version }}/backups/updated-backup-privileges.md %} diff --git a/src/current/v25.1/check-external-connection.md b/src/current/v25.1/check-external-connection.md new file mode 100644 index 00000000000..b629babe4c4 --- /dev/null +++ b/src/current/v25.1/check-external-connection.md @@ -0,0 +1,93 @@ +--- +title: CHECK EXTERNAL CONNECTION +summary: Test the connection of each node to your cloud storage location. +toc: true +--- + +{% include_cached new-in.html version="v25.1" %} The `CHECK EXTERNAL CONNECTION` tests the connection from each node in the cluster to an external cloud storage location. `CHECK EXTERNAL CONNECTION` will measure the time it takes each node to write a file, read it, and delete it from the specified storage location. You can adjust the number and concurrency of the test runs as well as the size of the file to write and read for each test. + +{{site.data.alerts.callout_info}} +You can use the `CHECK EXTERNAL CONNECTION` to test the connection to [**cloud storage**]({% link {{ page.version.version }}/use-cloud-storage.md %}) locations. +{{site.data.alerts.end}} + +## Synopsis + +
+{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/check_external_connection.html %} +
+ +## Parameters + +Parameter | Description +----------+------------ +`connection_uri` | The URI to the external storage. Specify the [provider's URI]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., `gs://bucket_name?AUTH...`) or a user-defined [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) (e.g., `external://gcs`). + +## Options + +Option | Value | Description +--------+-------+------------ +`concurrently` | `INT` | Run multiple connection tests concurrently. If you also set the `time` option, it will run the specified number of concurrent tests until the time has elapsed. By default, only `1` connection test will run. +`time` | `STRING` | Run the test repeatedly until the duration has elapsed. +`transfer` | `STRING` | The size of the file that is written and read during each iteration of the connection test. By default, this will transfer a `32MiB` file. + +## Responses + +Field | Value | Description +------|-------|------------ +`node` | `INT` | The node ID. +`locality` | `STRING` | The [locality]({% link {{ page.version.version }}/cockroach-start.md %}#locality) of the node. +`ok` | `BOOL` | The success of the test run. +`error` | `STRING` | Errors encountered during the test run. +`transferred` | `STRING` | The size of the file transferred during the test. +`read_speed` | `STRING` | The speed at which the node read the test file. +`write_speed` | `STRING` | The speed at which the node wrote the test file. +`can_delete` | `BOOL` | The success of file deletion. + +## Examples + +Specify the connection URI to the [external storage location]({% link {{ page.version.version }}/use-cloud-storage.md %}), or a created [external connection]({% link {{ page.version.version }}/create-external-connection.md %}): + +{% include_cached copy-clipboard.html %} +~~~ sql +CHECK EXTERNAL CONNECTION 'external://cloud-storage'; +~~~ + +~~~ + node | locality | ok | error | transferred | read_speed | write_speed | can_delete +-------+-------------------------------------------+----+-------+-------------+-------------+-------------+------------- + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 66.17 MiB/s | 37.52 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 41.77 MiB/s | 33.55 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 14.23 MiB/s | 37.12 MiB/s | t +~~~ + +To modify the testing parameters, use one or a combination of the options: `concurrently`, `time`, `transfer`. For details on each, refer to [Options](#options). + +{% include_cached copy-clipboard.html %} +~~~ sql +CHECK EXTERNAL CONNECTION 'external://cloud-storage' WITH transfer = '50MiB', concurrently = 5, time = '1ms'; +~~~ +~~~ + node | locality | ok | error | transferred | read_speed | write_speed | can_delete +-------+-------------------------------------------+----+-------+-------------+-------------+-------------+------------- + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.85 MiB/s | 34.99 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.26 MiB/s | 34.91 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.69 MiB/s | 32.30 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.51 MiB/s | 33.02 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.29 MiB/s | 31.45 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.61 MiB/s | 32.58 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 61.04 MiB/s | 29.63 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 47.69 MiB/s | 34.04 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.66 MiB/s | 30.39 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.77 MiB/s | 29.64 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 44.95 MiB/s | 34.41 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 46.77 MiB/s | 33.31 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.64 MiB/s | 28.96 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.99 MiB/s | 26.65 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 15.14 MiB/s | 33.45 MiB/s | t +~~~ + +## See also + +- [Use Cloud Storage]({% link {{ page.version.version }}/use-cloud-storage.md %}) +- [Backup and Restore Overview]({% link {{ page.version.version }}/backup-and-restore-overview.md %}) +- [`CREATE EXTERNAL CONNECTION`]({% link {{ page.version.version }}/create-external-connection.md %}) \ No newline at end of file diff --git a/src/current/v25.1/cloud-storage-authentication.md b/src/current/v25.1/cloud-storage-authentication.md index 1357e00a422..8d225a6e2fb 100644 --- a/src/current/v25.1/cloud-storage-authentication.md +++ b/src/current/v25.1/cloud-storage-authentication.md @@ -11,6 +11,10 @@ Cockroach Labs supports different levels of authentication to cloud storage. Whe We recommend using IAM roles for users to authenticate to cloud storage resources. For more detail, see the assume role and workload identity sections for [Amazon S3]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#set-up-amazon-s3-assume-role) and [Google Cloud Storage](cloud-storage-authentication.html?filters=gcs#set-up-google-cloud-storage-assume-role). {{site.data.alerts.end}} +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} +
diff --git a/src/current/v25.1/use-cloud-storage.md b/src/current/v25.1/use-cloud-storage.md index b7acaac9b10..07c942e4a75 100644 --- a/src/current/v25.1/use-cloud-storage.md +++ b/src/current/v25.1/use-cloud-storage.md @@ -59,6 +59,10 @@ You can disable the use of implicit credentials when accessing external cloud st ### Example file URLs +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + Example URLs for [`BACKUP`]({% link {{ page.version.version }}/backup.md %}), [`RESTORE`]({% link {{ page.version.version }}/restore.md %}), or [`EXPORT`]({% link {{ page.version.version }}/export.md %}) given a bucket or container name of `acme-co` and an `employees` subdirectory: Location | Example diff --git a/src/current/v25.2/backup-validation.md b/src/current/v25.2/backup-validation.md index a4f00b7754a..44fde6cfdb4 100644 --- a/src/current/v25.2/backup-validation.md +++ b/src/current/v25.2/backup-validation.md @@ -15,6 +15,10 @@ You can validate a backup of a [cluster]({% link {{ page.version.version }}/back The options that give the most validation coverage will increase the runtime of the check. That is, `verify_backup_table_data` will take a longer time to validate a backup compared to `check_files` or `schema_only` alone. Despite that, each of these validation options provide a quicker way to validate a backup over running a "regular" restore. +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + ## Recommendations Cockroach Labs recommends implementing the following validation plan to test your backups: diff --git a/src/current/v25.2/backup.md b/src/current/v25.2/backup.md index e45b569f058..b93d5e3e8f6 100644 --- a/src/current/v25.2/backup.md +++ b/src/current/v25.2/backup.md @@ -44,6 +44,10 @@ To view the contents of an backup created with the `BACKUP` statement, use [`SHO - Modifying backup files in the storage location could invalidate a backup, and therefore, prevent a restore. In v22.1 and later, **we recommend enabling [object locking]({% link {{ page.version.version }}/use-cloud-storage.md %}#immutable-storage) in your cloud storage bucket.** - While Cockroach Labs actively tests Amazon S3, Google Cloud Storage, and Azure Storage, we **do not** test [S3-compatible services]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., [MinIO](https://min.io/), [Red Hat Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/)). +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + ## Required privileges {% include {{ page.version.version }}/backups/updated-backup-privileges.md %} diff --git a/src/current/v25.2/check-external-connection.md b/src/current/v25.2/check-external-connection.md new file mode 100644 index 00000000000..3037bb413d4 --- /dev/null +++ b/src/current/v25.2/check-external-connection.md @@ -0,0 +1,93 @@ +--- +title: CHECK EXTERNAL CONNECTION +summary: Test the connection of each node to your cloud storage location. +toc: true +--- + +The `CHECK EXTERNAL CONNECTION` tests the connection from each node in the cluster to an external cloud storage location. `CHECK EXTERNAL CONNECTION` will measure the time it takes each node to write a file, read it, and delete it from the specified storage location. You can adjust the number and concurrency of the test runs as well as the size of the file to write and read for each test. + +{{site.data.alerts.callout_info}} +You can use the `CHECK EXTERNAL CONNECTION` to test the connection to [**cloud storage**]({% link {{ page.version.version }}/use-cloud-storage.md %}) locations. +{{site.data.alerts.end}} + +## Synopsis + +
+{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/check_external_connection.html %} +
+ +## Parameters + +Parameter | Description +----------+------------ +`connection_uri` | The URI to the external storage. Specify the [provider's URI]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., `gs://bucket_name?AUTH...`) or a user-defined [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) (e.g., `external://gcs`). + +## Options + +Option | Value | Description +--------+-------+------------ +`concurrently` | `INT` | Run multiple connection tests concurrently. If you also set the `time` option, it will run the specified number of concurrent tests until the time has elapsed. By default, only `1` connection test will run. +`time` | `STRING` | Run the test repeatedly until the duration has elapsed. +`transfer` | `STRING` | The size of the file that is written and read during each iteration of the connection test. By default, this will transfer a `32MiB` file. + +## Responses + +Field | Value | Description +------|-------|------------ +`node` | `INT` | The node ID. +`locality` | `STRING` | The [locality]({% link {{ page.version.version }}/cockroach-start.md %}#locality) of the node. +`ok` | `BOOL` | Whether the test run succeeded. +`error` | `STRING` | Errors encountered during the test run. +`transferred` | `STRING` | The size of the file transferred during the test. +`read_speed` | `STRING` | The speed at which the node read the test file. +`write_speed` | `STRING` | The speed at which the node wrote the test file. +`can_delete` | `BOOL` | Whether file deletion succeeded. + +## Test an external connection + +Specify the connection URI to the [external storage location]({% link {{ page.version.version }}/use-cloud-storage.md %}), or a created [external connection]({% link {{ page.version.version }}/create-external-connection.md %}): + +{% include_cached copy-clipboard.html %} +~~~ sql +CHECK EXTERNAL CONNECTION 'external://cloud-storage'; +~~~ + +~~~ + node | locality | ok | error | transferred | read_speed | write_speed | can_delete +-------+-------------------------------------------+----+-------+-------------+-------------+-------------+------------- + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 66.17 MiB/s | 37.52 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 41.77 MiB/s | 33.55 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 14.23 MiB/s | 37.12 MiB/s | t +~~~ + +To modify the testing parameters, use one or a combination of the options: `concurrently`, `time`, `transfer`. For details on each, refer to [Options](#options). + +{% include_cached copy-clipboard.html %} +~~~ sql +CHECK EXTERNAL CONNECTION 'external://cloud-storage' WITH transfer = '50MiB', concurrently = 5, time = '1ms'; +~~~ +~~~ + node | locality | ok | error | transferred | read_speed | write_speed | can_delete +-------+-------------------------------------------+----+-------+-------------+-------------+-------------+------------- + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.85 MiB/s | 34.99 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.26 MiB/s | 34.91 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.69 MiB/s | 32.30 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.51 MiB/s | 33.02 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.29 MiB/s | 31.45 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.61 MiB/s | 32.58 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 61.04 MiB/s | 29.63 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 47.69 MiB/s | 34.04 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.66 MiB/s | 30.39 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.77 MiB/s | 29.64 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 44.95 MiB/s | 34.41 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 46.77 MiB/s | 33.31 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.64 MiB/s | 28.96 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.99 MiB/s | 26.65 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 15.14 MiB/s | 33.45 MiB/s | t +~~~ + +## See also + +- [Use Cloud Storage]({% link {{ page.version.version }}/use-cloud-storage.md %}) +- [Backup and Restore Overview]({% link {{ page.version.version }}/backup-and-restore-overview.md %}) +- [`CREATE EXTERNAL CONNECTION`]({% link {{ page.version.version }}/create-external-connection.md %}) \ No newline at end of file diff --git a/src/current/v25.2/cloud-storage-authentication.md b/src/current/v25.2/cloud-storage-authentication.md index 32f07e11c26..6a81d6ec648 100644 --- a/src/current/v25.2/cloud-storage-authentication.md +++ b/src/current/v25.2/cloud-storage-authentication.md @@ -11,6 +11,10 @@ Cockroach Labs supports different levels of authentication to cloud storage. Whe We recommend using IAM roles for users to authenticate to cloud storage resources. For more detail, see the assume role and workload identity sections for [Amazon S3]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#set-up-amazon-s3-assume-role) and [Google Cloud Storage](cloud-storage-authentication.html?filters=gcs#set-up-google-cloud-storage-assume-role). {{site.data.alerts.end}} +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} +
diff --git a/src/current/v25.2/use-cloud-storage.md b/src/current/v25.2/use-cloud-storage.md index b7acaac9b10..07c942e4a75 100644 --- a/src/current/v25.2/use-cloud-storage.md +++ b/src/current/v25.2/use-cloud-storage.md @@ -59,6 +59,10 @@ You can disable the use of implicit credentials when accessing external cloud st ### Example file URLs +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + Example URLs for [`BACKUP`]({% link {{ page.version.version }}/backup.md %}), [`RESTORE`]({% link {{ page.version.version }}/restore.md %}), or [`EXPORT`]({% link {{ page.version.version }}/export.md %}) given a bucket or container name of `acme-co` and an `employees` subdirectory: Location | Example diff --git a/src/current/v25.3/backup-validation.md b/src/current/v25.3/backup-validation.md index a4f00b7754a..44fde6cfdb4 100644 --- a/src/current/v25.3/backup-validation.md +++ b/src/current/v25.3/backup-validation.md @@ -15,6 +15,10 @@ You can validate a backup of a [cluster]({% link {{ page.version.version }}/back The options that give the most validation coverage will increase the runtime of the check. That is, `verify_backup_table_data` will take a longer time to validate a backup compared to `check_files` or `schema_only` alone. Despite that, each of these validation options provide a quicker way to validate a backup over running a "regular" restore. +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + ## Recommendations Cockroach Labs recommends implementing the following validation plan to test your backups: diff --git a/src/current/v25.3/backup.md b/src/current/v25.3/backup.md index e45b569f058..b93d5e3e8f6 100644 --- a/src/current/v25.3/backup.md +++ b/src/current/v25.3/backup.md @@ -44,6 +44,10 @@ To view the contents of an backup created with the `BACKUP` statement, use [`SHO - Modifying backup files in the storage location could invalidate a backup, and therefore, prevent a restore. In v22.1 and later, **we recommend enabling [object locking]({% link {{ page.version.version }}/use-cloud-storage.md %}#immutable-storage) in your cloud storage bucket.** - While Cockroach Labs actively tests Amazon S3, Google Cloud Storage, and Azure Storage, we **do not** test [S3-compatible services]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., [MinIO](https://min.io/), [Red Hat Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/)). +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + ## Required privileges {% include {{ page.version.version }}/backups/updated-backup-privileges.md %} diff --git a/src/current/v25.3/check-external-connection.md b/src/current/v25.3/check-external-connection.md new file mode 100644 index 00000000000..3037bb413d4 --- /dev/null +++ b/src/current/v25.3/check-external-connection.md @@ -0,0 +1,93 @@ +--- +title: CHECK EXTERNAL CONNECTION +summary: Test the connection of each node to your cloud storage location. +toc: true +--- + +The `CHECK EXTERNAL CONNECTION` tests the connection from each node in the cluster to an external cloud storage location. `CHECK EXTERNAL CONNECTION` will measure the time it takes each node to write a file, read it, and delete it from the specified storage location. You can adjust the number and concurrency of the test runs as well as the size of the file to write and read for each test. + +{{site.data.alerts.callout_info}} +You can use the `CHECK EXTERNAL CONNECTION` to test the connection to [**cloud storage**]({% link {{ page.version.version }}/use-cloud-storage.md %}) locations. +{{site.data.alerts.end}} + +## Synopsis + +
+{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/check_external_connection.html %} +
+ +## Parameters + +Parameter | Description +----------+------------ +`connection_uri` | The URI to the external storage. Specify the [provider's URI]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., `gs://bucket_name?AUTH...`) or a user-defined [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) (e.g., `external://gcs`). + +## Options + +Option | Value | Description +--------+-------+------------ +`concurrently` | `INT` | Run multiple connection tests concurrently. If you also set the `time` option, it will run the specified number of concurrent tests until the time has elapsed. By default, only `1` connection test will run. +`time` | `STRING` | Run the test repeatedly until the duration has elapsed. +`transfer` | `STRING` | The size of the file that is written and read during each iteration of the connection test. By default, this will transfer a `32MiB` file. + +## Responses + +Field | Value | Description +------|-------|------------ +`node` | `INT` | The node ID. +`locality` | `STRING` | The [locality]({% link {{ page.version.version }}/cockroach-start.md %}#locality) of the node. +`ok` | `BOOL` | Whether the test run succeeded. +`error` | `STRING` | Errors encountered during the test run. +`transferred` | `STRING` | The size of the file transferred during the test. +`read_speed` | `STRING` | The speed at which the node read the test file. +`write_speed` | `STRING` | The speed at which the node wrote the test file. +`can_delete` | `BOOL` | Whether file deletion succeeded. + +## Test an external connection + +Specify the connection URI to the [external storage location]({% link {{ page.version.version }}/use-cloud-storage.md %}), or a created [external connection]({% link {{ page.version.version }}/create-external-connection.md %}): + +{% include_cached copy-clipboard.html %} +~~~ sql +CHECK EXTERNAL CONNECTION 'external://cloud-storage'; +~~~ + +~~~ + node | locality | ok | error | transferred | read_speed | write_speed | can_delete +-------+-------------------------------------------+----+-------+-------------+-------------+-------------+------------- + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 66.17 MiB/s | 37.52 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 41.77 MiB/s | 33.55 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 14.23 MiB/s | 37.12 MiB/s | t +~~~ + +To modify the testing parameters, use one or a combination of the options: `concurrently`, `time`, `transfer`. For details on each, refer to [Options](#options). + +{% include_cached copy-clipboard.html %} +~~~ sql +CHECK EXTERNAL CONNECTION 'external://cloud-storage' WITH transfer = '50MiB', concurrently = 5, time = '1ms'; +~~~ +~~~ + node | locality | ok | error | transferred | read_speed | write_speed | can_delete +-------+-------------------------------------------+----+-------+-------------+-------------+-------------+------------- + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.85 MiB/s | 34.99 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.26 MiB/s | 34.91 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.69 MiB/s | 32.30 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.51 MiB/s | 33.02 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.29 MiB/s | 31.45 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.61 MiB/s | 32.58 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 61.04 MiB/s | 29.63 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 47.69 MiB/s | 34.04 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.66 MiB/s | 30.39 MiB/s | t + 1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.77 MiB/s | 29.64 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 44.95 MiB/s | 34.41 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 46.77 MiB/s | 33.31 MiB/s | t + 2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.64 MiB/s | 28.96 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.99 MiB/s | 26.65 MiB/s | t + 3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 15.14 MiB/s | 33.45 MiB/s | t +~~~ + +## See also + +- [Use Cloud Storage]({% link {{ page.version.version }}/use-cloud-storage.md %}) +- [Backup and Restore Overview]({% link {{ page.version.version }}/backup-and-restore-overview.md %}) +- [`CREATE EXTERNAL CONNECTION`]({% link {{ page.version.version }}/create-external-connection.md %}) \ No newline at end of file diff --git a/src/current/v25.3/cloud-storage-authentication.md b/src/current/v25.3/cloud-storage-authentication.md index 32f07e11c26..6a81d6ec648 100644 --- a/src/current/v25.3/cloud-storage-authentication.md +++ b/src/current/v25.3/cloud-storage-authentication.md @@ -11,6 +11,10 @@ Cockroach Labs supports different levels of authentication to cloud storage. Whe We recommend using IAM roles for users to authenticate to cloud storage resources. For more detail, see the assume role and workload identity sections for [Amazon S3]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#set-up-amazon-s3-assume-role) and [Google Cloud Storage](cloud-storage-authentication.html?filters=gcs#set-up-google-cloud-storage-assume-role). {{site.data.alerts.end}} +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} +
diff --git a/src/current/v25.3/use-cloud-storage.md b/src/current/v25.3/use-cloud-storage.md index b7acaac9b10..07c942e4a75 100644 --- a/src/current/v25.3/use-cloud-storage.md +++ b/src/current/v25.3/use-cloud-storage.md @@ -59,6 +59,10 @@ You can disable the use of implicit credentials when accessing external cloud st ### Example file URLs +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/backups/external-storage-check-tip.md %} +{{site.data.alerts.end}} + Example URLs for [`BACKUP`]({% link {{ page.version.version }}/backup.md %}), [`RESTORE`]({% link {{ page.version.version }}/restore.md %}), or [`EXPORT`]({% link {{ page.version.version }}/export.md %}) given a bucket or container name of `acme-co` and an `employees` subdirectory: Location | Example