From 5f293281a0bcae69764a34ab7abf90b5dacf72a7 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Fri, 6 Jun 2025 12:44:15 +0300 Subject: [PATCH 01/13] initial commit that fixes a toc issue, next is the decrypt topic creation fixed an issue that has been bugging me in tocs first. --- .../docs/advanced-topics/{index.md => adv-top-overview.md} | 2 +- .../docs/architecture/{index.md => architecture.md} | 0 contrib/pg_tde/documentation/docs/wal-encryption.md | 2 +- contrib/pg_tde/documentation/mkdocs.yml | 4 ++-- 4 files changed, 4 insertions(+), 4 deletions(-) rename contrib/pg_tde/documentation/docs/advanced-topics/{index.md => adv-top-overview.md} (63%) rename contrib/pg_tde/documentation/docs/architecture/{index.md => architecture.md} (100%) diff --git a/contrib/pg_tde/documentation/docs/advanced-topics/index.md b/contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md similarity index 63% rename from contrib/pg_tde/documentation/docs/advanced-topics/index.md rename to contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md index 9df169821bcd3..f52e01b97b378 100644 --- a/contrib/pg_tde/documentation/docs/advanced-topics/index.md +++ b/contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md @@ -4,4 +4,4 @@ This section covers the internal components and tools that power `pg_tde`. Use it to understand how encryption is implemented, fine-tune a configuration, leverage advanced CLI tools and functions for diagnostics and customization. -[Architecture](../architecture/index.md){.md-button} [GUC Variables](../variables.md){.md-button} [Functions](../functions.md){.md-button} +[Architecture](../architecture/architecture.md){.md-button} [GUC Variables](../variables.md){.md-button} [Functions](../functions.md){.md-button} diff --git a/contrib/pg_tde/documentation/docs/architecture/index.md b/contrib/pg_tde/documentation/docs/architecture/architecture.md similarity index 100% rename from contrib/pg_tde/documentation/docs/architecture/index.md rename to contrib/pg_tde/documentation/docs/architecture/architecture.md diff --git a/contrib/pg_tde/documentation/docs/wal-encryption.md b/contrib/pg_tde/documentation/docs/wal-encryption.md index 21f37ed73ae85..d63e6b7de2075 100644 --- a/contrib/pg_tde/documentation/docs/wal-encryption.md +++ b/contrib/pg_tde/documentation/docs/wal-encryption.md @@ -88,4 +88,4 @@ Before turning WAL encryption on, you must follow the steps below to create your Now WAL files start to be encrypted for both encrypted and unencrypted tables. For more technical references related to architecture, variables or functions, see: -[Technical Reference](advanced-topics/index.md){.md-button} +[Technical Reference](advanced-topics/adv-top-overview.md){.md-button} diff --git a/contrib/pg_tde/documentation/mkdocs.yml b/contrib/pg_tde/documentation/mkdocs.yml index b087c856aca9d..5025d73a37a91 100644 --- a/contrib/pg_tde/documentation/mkdocs.yml +++ b/contrib/pg_tde/documentation/mkdocs.yml @@ -176,8 +176,8 @@ nav: - "3. Validate Encryption with pg_tde": test.md - "4. Configure WAL Encryption (tech preview)": wal-encryption.md - "Technical Reference": - - "Technical Reference": advanced-topics/index.md - - "Architecture": architecture/index.md + - "Overview": advanced-topics/adv-top-overview.md + - "Architecture": architecture/architecture.md - "GUC Variables": variables.md - "Functions": functions.md - "TDE Operations": From c57e70805689ee6e701fe6b4c8094ed9e719f2d7 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Fri, 6 Jun 2025 15:58:55 +0300 Subject: [PATCH 02/13] created decrypt backup topic added basic information and updated ToC, it is under TDE Operations, and updated FAQ response to include link to new topic --- contrib/pg_tde/documentation/docs/faq.md | 2 +- .../docs/how-to/decrypt-backups.md | 38 +++++++++++++++++++ contrib/pg_tde/documentation/mkdocs.yml | 1 + 3 files changed, 40 insertions(+), 1 deletion(-) create mode 100644 contrib/pg_tde/documentation/docs/how-to/decrypt-backups.md diff --git a/contrib/pg_tde/documentation/docs/faq.md b/contrib/pg_tde/documentation/docs/faq.md index 902e0a18b49fc..b23316309a4fd 100644 --- a/contrib/pg_tde/documentation/docs/faq.md +++ b/contrib/pg_tde/documentation/docs/faq.md @@ -157,7 +157,7 @@ WAL files are encrypted globally across the entire PostgreSQL cluster using the Since the encryption happens on the database level, it makes no difference for your tools and applications. They work with the data in the same way. -To restore from an encrypted backup, you must have the same principal encryption key, which was used to encrypt files in your backup. +To restore from an encrypted backup, you must have the same principal encryption key, which was used to encrypt files in your backup. For more information, see [Handling Key Rotation and Encrypted Backups](how-to/decrypt-backups.md). ## I'm using OpenSSL in FIPS mode and need to use `pg_tde`. Does `pg_tde` comply with FIPS requirements? Can I use my own FIPS-mode OpenSSL library with `pg_tde`? diff --git a/contrib/pg_tde/documentation/docs/how-to/decrypt-backups.md b/contrib/pg_tde/documentation/docs/how-to/decrypt-backups.md new file mode 100644 index 0000000000000..8a47becff4ac1 --- /dev/null +++ b/contrib/pg_tde/documentation/docs/how-to/decrypt-backups.md @@ -0,0 +1,38 @@ +# Handling Key Rotation and Encrypted Backups + +Backups created with `pg_tde` are encrypted using the *active principal key* at the time of backup. If this key is rotated and the original version is not retained, those backups **can no longer be decrypted**. This makes **secure key storage** essential for reliable disaster recovery. + +## Why you must store old principal keys + +Each time you rotate the principal key, you create a potential split in decryption compatibility: + +- New backups will be encrypted with the new key. +- Old backups **require** the key that was active at the time they were created. + +!!! important + Losing access to an old principal key means **permanent data loss** for the associated backups. + +## How to store principal keys securely + +We recommend using a dedicated [Key Management System (KMS)](../global-key-provider-configuration/index.md). The KMS provides: + +- Secure, access-controlled key storage. +- An audit trail for compliance and traceability. +- Support for versioning and key rotation policies. + +## Example: Using HashiCorp Vault to store rotated keys + +Here’s a basic workflow using the **KV v2 secrets engine**: + +1. Enable the secrets engine by following the [Vault KV documentation](https://developer.hashicorp.com/vault/docs/commands/kv). +2. Store your current principal key. +3. When rotating, generate and store a new principal key under a new version or path. + +## How to Restore a Backup Using an Old Key + +To decrypt a backup made with a previous principal key: + +1. Retrieve the correct principal key used at the time of backup from your key management system. +2. Configure `pg_tde` to use this key temporarily. +3. Restore the backup as usual. +4. (Optional) After the restore, re-encrypt the data using the latest principal key. diff --git a/contrib/pg_tde/documentation/mkdocs.yml b/contrib/pg_tde/documentation/mkdocs.yml index 5025d73a37a91..c40040d2e352b 100644 --- a/contrib/pg_tde/documentation/mkdocs.yml +++ b/contrib/pg_tde/documentation/mkdocs.yml @@ -189,6 +189,7 @@ nav: - "Uninstall pg_tde": how-to/uninstall.md - "Configure Multi-tenancy": how-to/multi-tenant-setup.md - "Decrypt an Encrypted Table": how-to/decrypt.md + - "Handling Key Rotation and Encrypted Backups": how-to/decrypt-backups.md - faq.md - "Release Notes": - "pg_tde release notes": release-notes/release-notes.md From 3d11bca62f79ab6aa8a43fda13a52316b2414111 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Mon, 9 Jun 2025 15:11:50 +0300 Subject: [PATCH 03/13] renamed and rewrote decrypt to restore Restoring is more correct in the context of getting an older backup's info. Rewrote the chapter as I added two new question/answers to FAQ with links to our docs, fixed FAQ a bit with better wording. --- contrib/pg_tde/documentation/docs/faq.md | 16 ++++++-- .../docs/how-to/decrypt-backups.md | 38 ------------------- .../docs/how-to/restore-backups.md | 16 ++++++++ contrib/pg_tde/documentation/mkdocs.yml | 2 +- 4 files changed, 29 insertions(+), 43 deletions(-) delete mode 100644 contrib/pg_tde/documentation/docs/how-to/decrypt-backups.md create mode 100644 contrib/pg_tde/documentation/docs/how-to/restore-backups.md diff --git a/contrib/pg_tde/documentation/docs/faq.md b/contrib/pg_tde/documentation/docs/faq.md index b23316309a4fd..e1256e8b59f51 100644 --- a/contrib/pg_tde/documentation/docs/faq.md +++ b/contrib/pg_tde/documentation/docs/faq.md @@ -133,17 +133,17 @@ Since the `SET ACCESS METHOD` command drops hint bits and this may affect the pe You must restart the database in the following cases to apply the changes: * after you enabled the `pg_tde` extension -* to turn on / off the WAL encryption +* to turn WAL encryption on or off After that, no database restart is required. When you create or alter the table using the `tde_heap` access method, the files are marked as those that require encryption. The encryption happens at the storage manager level, before a transaction is written to disk. Read more about [how tde_heap works](index/table-access-method.md#how-tde_heap-works). ## What happens to my data if I lose a principal key? -If you lose encryption keys, especially, the principal key, the data is lost. That's why it's critical to back up your encryption keys securely and use the Key Management service for key management. +If you lose encryption keys, especially the principal key, the data is **lost**. It is critical to back up your encryption keys securely and use the Key Management service for key management. For more information, see the [Configure Key Management (KMS)](../docs/global-key-provider-configuration/index.md) topic. ## Can I use `pg_tde` in a multi-tenant setup? -Multi-tenancy is the type of architecture where multiple users, or tenants, share the same resource. It can be a database, a schema or an entire cluster. +Yes. Multi-tenancy is the type of architecture where multiple users, or tenants, share the same resource. It can be a database, a schema or an entire cluster. In `pg_tde`, multi-tenancy is supported via a separate principal key per database. This means that a database owner can decide what tables to encrypt within a database. The same database can have both encrypted and non-encrypted tables. @@ -157,7 +157,15 @@ WAL files are encrypted globally across the entire PostgreSQL cluster using the Since the encryption happens on the database level, it makes no difference for your tools and applications. They work with the data in the same way. -To restore from an encrypted backup, you must have the same principal encryption key, which was used to encrypt files in your backup. For more information, see [Handling Key Rotation and Encrypted Backups](how-to/decrypt-backups.md). +To restore from an encrypted backup, you must have the same principal encryption key, which was used to encrypt files in your backup. See the [Restore an encrypted pg_tde backup](../docs/how-to/restore-backups.md) topic for more details. + +## What if the address of the key provider changed for one of my old backups? + +You can use the [pg_tde_change_key_provider](../docs/command-line-tools/pg-tde-change-key-provider.md) tool to correct the configuration. + +## How can I store an old key securely? + +To store an old key securely, see the [Configure Key Management (KMS)](../docs/global-key-provider-configuration/index.md) topic. ## I'm using OpenSSL in FIPS mode and need to use `pg_tde`. Does `pg_tde` comply with FIPS requirements? Can I use my own FIPS-mode OpenSSL library with `pg_tde`? diff --git a/contrib/pg_tde/documentation/docs/how-to/decrypt-backups.md b/contrib/pg_tde/documentation/docs/how-to/decrypt-backups.md deleted file mode 100644 index 8a47becff4ac1..0000000000000 --- a/contrib/pg_tde/documentation/docs/how-to/decrypt-backups.md +++ /dev/null @@ -1,38 +0,0 @@ -# Handling Key Rotation and Encrypted Backups - -Backups created with `pg_tde` are encrypted using the *active principal key* at the time of backup. If this key is rotated and the original version is not retained, those backups **can no longer be decrypted**. This makes **secure key storage** essential for reliable disaster recovery. - -## Why you must store old principal keys - -Each time you rotate the principal key, you create a potential split in decryption compatibility: - -- New backups will be encrypted with the new key. -- Old backups **require** the key that was active at the time they were created. - -!!! important - Losing access to an old principal key means **permanent data loss** for the associated backups. - -## How to store principal keys securely - -We recommend using a dedicated [Key Management System (KMS)](../global-key-provider-configuration/index.md). The KMS provides: - -- Secure, access-controlled key storage. -- An audit trail for compliance and traceability. -- Support for versioning and key rotation policies. - -## Example: Using HashiCorp Vault to store rotated keys - -Here’s a basic workflow using the **KV v2 secrets engine**: - -1. Enable the secrets engine by following the [Vault KV documentation](https://developer.hashicorp.com/vault/docs/commands/kv). -2. Store your current principal key. -3. When rotating, generate and store a new principal key under a new version or path. - -## How to Restore a Backup Using an Old Key - -To decrypt a backup made with a previous principal key: - -1. Retrieve the correct principal key used at the time of backup from your key management system. -2. Configure `pg_tde` to use this key temporarily. -3. Restore the backup as usual. -4. (Optional) After the restore, re-encrypt the data using the latest principal key. diff --git a/contrib/pg_tde/documentation/docs/how-to/restore-backups.md b/contrib/pg_tde/documentation/docs/how-to/restore-backups.md new file mode 100644 index 0000000000000..c5d0f38383d55 --- /dev/null +++ b/contrib/pg_tde/documentation/docs/how-to/restore-backups.md @@ -0,0 +1,16 @@ +# Restoring encrypted backups + +To restore an encrypted backup created with an earlier key, ensure the key is still available in your Key Management System (KMS). The backup remains **encrypted** on disk, `pg_tde` uses the correct key to transparently access the data. + +Each encrypted table stores metadata that identifies which encryption key was used. When the encrypted data is read, `pg_tde` retrieves the correct key from the configured KMS. + +## How `pg_tde` uses old keys to load backups + +* **KMS**: stores the key that was active when the backup was made. +* **Encrypted backup**: contains data encrypted with the key available at that time. +* **Current server**: `pg_tde` automatically loads the backup if the key is accessible through the KMS. + +At runtime, `pg_tde` reads the key ID from the table metadata and fetches the corresponding key from the KMS. + +!!! note + If the key was deleted or the KMS configuration changed, the backup **cannot** be read. Use [pg_tde_change_key_provider](../command-line-tools/pg-tde-change-key-provider.md) to update KMS credentials or endpoints if needed. diff --git a/contrib/pg_tde/documentation/mkdocs.yml b/contrib/pg_tde/documentation/mkdocs.yml index c40040d2e352b..b7bb0d8160044 100644 --- a/contrib/pg_tde/documentation/mkdocs.yml +++ b/contrib/pg_tde/documentation/mkdocs.yml @@ -189,7 +189,7 @@ nav: - "Uninstall pg_tde": how-to/uninstall.md - "Configure Multi-tenancy": how-to/multi-tenant-setup.md - "Decrypt an Encrypted Table": how-to/decrypt.md - - "Handling Key Rotation and Encrypted Backups": how-to/decrypt-backups.md + - "Restore an encrypted pg_tde backup": how-to/restore-backups.md - faq.md - "Release Notes": - "pg_tde release notes": release-notes/release-notes.md From f6e48d47479bb960baf5361c4ba9ac90314eecbb Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Wed, 25 Jun 2025 15:21:52 +0300 Subject: [PATCH 04/13] Delete contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md removed, not needed in PR --- .../documentation/docs/advanced-topics/adv-top-overview.md | 7 ------- 1 file changed, 7 deletions(-) delete mode 100644 contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md diff --git a/contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md b/contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md deleted file mode 100644 index f52e01b97b378..0000000000000 --- a/contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md +++ /dev/null @@ -1,7 +0,0 @@ -# Technical Reference - -This section covers the internal components and tools that power `pg_tde`. - -Use it to understand how encryption is implemented, fine-tune a configuration, leverage advanced CLI tools and functions for diagnostics and customization. - -[Architecture](../architecture/architecture.md){.md-button} [GUC Variables](../variables.md){.md-button} [Functions](../functions.md){.md-button} From a9364d2cba66a8aa1fa9de2cc49f25f691634529 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Wed, 25 Jun 2025 15:25:23 +0300 Subject: [PATCH 05/13] Delete contrib/pg_tde/documentation/docs/architecture/architecture.md not required in PR --- .../docs/architecture/architecture.md | 373 ------------------ 1 file changed, 373 deletions(-) delete mode 100644 contrib/pg_tde/documentation/docs/architecture/architecture.md diff --git a/contrib/pg_tde/documentation/docs/architecture/architecture.md b/contrib/pg_tde/documentation/docs/architecture/architecture.md deleted file mode 100644 index d2061a8f7058c..0000000000000 --- a/contrib/pg_tde/documentation/docs/architecture/architecture.md +++ /dev/null @@ -1,373 +0,0 @@ -# Architecture - -`pg_tde` is a **customizable, complete, data at rest encryption extension** for PostgreSQL. - -Let's break down what it means. - -**Customizable** means that `pg_tde` aims to support many different use cases: - -* Encrypting either every table in every database or only some tables in some databases -* Encryption keys can be stored on various external key storage servers including Hashicorp Vault, KMIP servers. -* Using one key for everything or different keys for different databases -* Storing every key on the same key storage, or using different storages for different databases -* Handling permissions: who can manage database specific or global permissions, who can create encrypted or not encrypted tables - -**Complete** means that `pg_tde` aims to encrypt data at rest. - -**Data at rest** means everything written to the disk. This includes the following: - -* Table data files -* Indexes -* Sequences -* Temporary tables -* Write Ahead Log (WAL) - -**Extension** means that `pg_tde` should be implemented only as an extension, possibly compatible with any PostgreSQL distribution, including the open source community version. This requires changes in the PostgreSQL core to make it more extensible. Therefore, `pg_tde` currently works only with the [Percona Server for PostgreSQL](https://docs.percona.com/postgresql/17/index.html) - a binary replacement of community PostgreSQL and included in Percona Distribution for PostgreSQL. - -## Main components - -The main components of `pg_tde` are the following: - -* **Core server changes** focus on making the server more extensible, allowing the main logic of `pg_tde` to remain separate, as an extension. Core changes also add encryption-awareness to some command line tools that have to work directly with encrypted tables or encrypted WAL files. - - [Percona Server for PostgreSQL location](https://github.com/percona/postgres/tree/{{tdebranch}}) - -* The **`pg_tde` extension itself** implements the encryption code by hooking into the extension points introduced in the core changes, and the already existing extension points in the PostgreSQL server. - - Everything is controllable with GUC variables and SQL statements, similar to other extensions. - -* The **keyring API / libraries** implement the key storage logic with different key providers. The API is internal only, the keyring libraries are part of the main library for simplicity. -In the future these could be extracted into separate shared libraries with an open API, allowing the use of third-party providers. - -## Encryption architecture - -### Two-key hierarchy - -`pg_tde` uses two kinds of keys for encryption: - -* Internal keys to encrypt the data. They are stored in PostgreSQL's data directory under `$PGDATA/pg_tde`. -* Higher-level keys to encrypt internal keys. These keys are called *principal keys*. They are stored externally, in a Key Management System (KMS) using the key provider API. - -`pg_tde` uses one principal key per database. Every internal key for the given database is encrypted using this principal key. - -Internal keys are used for specific database files: each file with a different [Object Identifier (OID)](https://www.postgresql.org/docs/current/datatype-oid.html) has a different internal key. - -This means that, for example, a table with 4 indexes will have at least 5 internal keys - one for the table, and one for each index. - -If a table has additional associated relations, such as sequences or a TOAST table, those relations will also have separate keys. - -### Encryption algorithm - -`pg_tde` currently uses the following encryption algorithms: - -* `AES-128-CBC` for encrypting database files; encrypted with internal keys. - -* `AES-128-CTR` for WAL encryption; encrypted with internal keys. - -* `AES-128-GCM` for encrypting internal keys; encrypted with the principal key. - -Support for other cipher lengths / algorithms is planned in the future. - -### Encryption workflow - -`pg_tde` makes it possible to encrypt everything or only some tables in some databases. - -To support this without metadata changes, encrypted tables are labeled with a `tde_heap` access method marker. - -The `tde_heap` access method is the same as the `heap` one. It uses the same functions internally without any changes, but with the different name and ID. In such a way `pg_tde` knows that `tde_heap` tables are encrypted and `heap` tables are not. - -The initial decision what to encrypt is made using the `postgres` event trigger mechanism: if a `CREATE TABLE` or `ALTER TABLE` statement uses the `tde_heap` clause, the newly created data files are marked as encrypted. Then file operations encrypt or decrypt the data. - -Later decisions are made using a slightly modified Storage Manager (SMGR) API: when a database file is re-created with a different ID as a result of a `TRUNCATE` or a `VACUUM FULL` command, the newly created file inherits the encryption information and is either encrypted or not. - -### WAL encryption - -WAL encryption is controlled globally via a global GUC variable, `pg_tde.wal_encrypt`, that requires a server restart. - -WAL keys also contain the [LSN](https://www.postgresql.org/docs/17/wal-internals.html) of the first WAL write after key creation. This allows `pg_tde` to know which WAL ranges are encrypted or not and with which key. - -The setting only controls writes so that only WAL writes are encrypted when WAL encryption is enabled. This means that WAL files can contain both encrypted and unencrpyted data, depending on what the status of this variable was when writing the data. - -`pg_tde` keeps track of the encryption status of WAL records using internal keys. When the server is restarted it writes a new internal key if WAL encryption is enabled, or if it is disabled and was previously enabled it writes a dummy key signalling that WAL encryption ended. - -With this information the WAL reader code can decide if a specific WAL record has to be decrypted or not and which key it should use to decrypt it. - -### Encrypting other access methods - -Currently `pg_tde` only encrypts `heap` tables and other files such as indexes, TOAST tables, sequences that are related to the `heap` tables. - -Indexes include any kind of index that goes through the SMGR API, not just the built-in indexes in PostgreSQL. - -In theory, it is also possible to encrypt any other table access method that goes through the SMGR API by similarly providing a marker access method to it and extending the event triggers. - -### Storage Manager (SMGR) API - -`pg_tde` relies on a slightly modified version of the SMGR API. These modifications include: - -* Making the API generally extensible, where extensions can inject custom code into the storage manager -* Adding tracking information for files. When a new file is created for an existing relation, references to the existing file are also passed to the SMGR functions - -With these modifications, the `pg_tde` extension can implement an additional layer on top of the normal Magnetic Disk SMGR API: if the related table is encrypted, `pg_tde` encrypts a file before writing it to the disk and, similarly, decrypts it after reading when needed. - -## Key and key provider management - -### Principal key rotation - -You can rotate principal keys to comply with common policies and to handle situations with potentially exposed principal keys. - -Rotation means that `pg_tde` generates a new version of the principal key, and re-encrypts the associated internal keys with the new key. The old principal key is kept as is at the same location, because it may still be needed to decrypt backups or other databases. - -### Internal key regeneration - -Internal keys for tables, indexes and other data files are fixed once a file is created. There's no way to re-encrypt a file. - -There are workarounds for this, because operations that move the table data to a new file, such as `VACUUM FULL` or an `ALTER TABLE` that rewrites the file will create a new key for the new file, essentially rotating the internal key. This however means taking an exclusive lock on the table for the duration of the operation, which might not be desirable for huge tables. - -WAL internal keys are also fixed to the respective ranges. To generate a new WAL key you need to restart the database. - -### Internal key storage - -Internal keys and `pg_tde` metadata in general are kept in a single `$PGDATA/pg_tde` directory. This directory stores separate files for each database, such as: - -* Encrypted internal keys and internal key mapping to tables -* Information about the key providers - -Also, the `$PGDATA/pg_tde` directory has a special global section marked with the OID `1664`, which includes the global key providers and global internal keys. - -The global section is used for WAL encryption. Specific databases can use the global section too, for scenarios where users configure individual principal keys for databases but use the same global key provider. For this purpose, you must enable the global provider inheritance. - -The global default principal key uses the special OID `1663`. - -### Key providers (principal key storage) - -Principal keys are stored externally in a Key Management Services (KMS). In `pg_tde`a KMS is defined as an external key provider. - -The following key providers are supported: - -* [HashiCorp Vault](https://developer.hashicorp.com/vault/docs/what-is-vault) KV2 secrets engine -* [OpenBao](https://openbao.org/) implementation of Vault -* KMIP compatible servers -* A local file storage. This storage is intended only for development and testing and is not recommended for production use. - -For each key provider `pg_tde` requires a detailed configuration including the address of the service and the authentication information. - -With these details `pg_tde` does the following based on user operations: - -* Uploads a new principal key to it after this key is created -* Retrieves the principal key from the service when it is required for decryption - -Retreival of the principal key is cached so it only happens when necessary. - -### Key provider management - -Key provider configuration or location may change. For example, a service is moved to a new address or the principal key must be moved to a different key provider type. `pg_tde` supports both these scenarios enabling you to manage principal keys using simple [SQL functions](../functions.md#key-provider-management). - -In certain cases you can't use SQL functions to manage key providers. For example, if the key provider changed while the server wasn't running and is therefore unaware of these changes. The startup can fail if it needs to access the encryption keys. - -For such situations, `pg_tde` also provides [command line tools](../command-line-tools/index.md) to recover the database. - -### Sensitive key provider information - -!!! important - - Authentication details for key providers are sensitive and must be protected. - Do not store these credentials in the `$PGDATA` directory alongside the database. Instead, ensure they are stored in a secure location with strict file system permissions to prevent unauthorized access. - -## User interface - -### Setting up `pg_tde` - -To use `pg_tde`, users are required to: - -* Add `pg_tde` to the `shared_preload_libraries` in `postgresql.conf` as this is required for the SMGR extensions -* Execute `CREATE EXTENSION pg_tde` in the databases where they want to use encryption -* Optionally, enable `pg_tde.wal_encrypt` in `postgresql.conf` -* Optionally, disable `pg_tde.inherit_global_providers` in `postgresql.conf` (enabled by default) - -### Adding providers - -Keyring providers can be added to either the global or to the database specific scope. - -If `pg_tde.inherit_global_providers` is `on`, global providers are visible for all databases, and can be used. -If `pg_tde.inherit_global_providers` is `off`, global providers are only used for WAL encryption. - -To add a global provider: - -```sql -pg_tde_add_global_key_provider_('provider_name', ... details ...) -``` - -To add a database specific provider: - -```sql -pg_tde_add_database_key_provider_('provider_name', ... details ...) -``` - -### Changing providers - -To change a value of a global provider: - -```sql -pg_tde_change_global_key_provider_('provider_name', ... details ...) -``` - -To change a value of a database specific provider: - -```sql -pg_tde_change_database_key_provider_('provider_name', ... details ...) -``` - -These functions also allow changing the type of a provider. - -The functions however do not migrate any data. They are expected to be used during infrastructure migration, for example when the address of a server changes. - -Note that in these functions do not verify the parameters. For that, see `pg_tde_verify_key`. - -### Changing providers from the command line - -To change a provider from a command line, `pg_tde` provides the `pg_tde_change_key_provider` command line tool. - -This tool work similarly to the above functions, with the following syntax: - -```bash -pg_tde_change_key_provider ... details ... -``` - -Note that since this tool is expected to be offline, it bypasses all permission checks! - -This is also the reason why it requires a `dbOid` instead of a name, as it has no way to process the catalog and look up names. - -### Deleting providers - -Providers can be deleted by using the - -```sql -pg_tde_delete_database_key_provider(provider_name) -pg_tde_delete_global_key_provider(provider_name) -``` - -functions. - -For database specific providers, the function first checks if the provider is used or not, and the provider is only deleted if it's not used. - -For global providers, the function checks if the provider is used anywhere, WAL or any specific database, and returns an error if it is. - -This somewhat goes against the principle that `pg_tde` should not interact with other databases than the one the user is connected to, but on the other hand, it only does this lookup in the internal `pg_tde` metadata, not in postgres catalogs, so it is a gray zone. Making this check makes more sense than potentially making some databases inaccessible. - -### Listing/querying providers - -`pg_tde` provides 2 functions to show providers: - -* `pg_tde_list_all_database_key_providers()` -* `pg_tde_list_all_global_key_providers()` - -These functions return a list of provider names, type and configuration. - -### Provider permissions - -`pg_tde` implements access control based on execution rights on the administration functions. - -For keys and providers administration, it provides two pair of functions: - -```sql -pg_tde_GRANT_database_key_management_TO_role -pg_tde_REVOKE_database_key_management_FROM_role -``` - -### Creating and rotating keys - -Principal keys can be created or rotated using the following functions: - -```sql -pg_tde_set_key_using_(global/database)_key_provider('key-name', 'provider-name', ensure_new_key) -pg_tde_set_server_key_using_(global/database)_key_provider('key-name', 'provider-name', ensure_new_key) -pg_tde_set_default_key_using_(global/database)_key_provider('key-name', 'provider-name', ensure_new_key) -``` - -`ensure_new_key` is a boolean parameter defaulting to false. If it is `true` the function might return an error instead of setting the key if it already exists on the provider. - -### Default principal key - -With `pg_tde.inherit_global_key_providers`, it is also possible to set up a default global principal key, which will be used by any database which has the `pg_tde` extension enabled, but doesn't have a database specific principal key configured using `pg_tde_set_key_using_(global/database)_key_provider`. - -With this feature, it is possible for the entire database server to easily use the same principal key for all databases, completely disabling multi-tenency. - -A default key can be managed with the following functions: - -```sql -pg_tde_set_default_key_using_global_key_provider('key-name', 'provider-name', 'true/false') -``` - -`DROP` is only possible if there's no table currently using the default principal key. - -Changing the default principal key will rotate the encryption of internal keys for all databases using the current default principal key. - -### Current key details - -`pg_tde_key_info()` returns the name of the current principal key, and the provider it uses. - -`pg_tde_server_key_info()` does the same for the server key. - -`pg_tde_default_key_info()` does the same for the default key. - -`pg_tde_verify_key()` checks that the key provider is accessible, that the current principal key can be downloaded from it, and that it is the same as the current key stored in memory - if any of these fail, it reports an appropriate error. - -### Key permissions - -Users with management permissions to a specific database `(pg_tde_(grant/revoke)_(global/databse)_key_management_(to/from)_role)` can change the keys for the database, and use the current key functions. This includes creating keys using global providers, if `pg_tde.inherit_global_providers` is enabled. - -Also the `pg_tde_(grant/revoke)_database_key_management_to_role` function deals with only the specific permission for the above function: it allows a user to change the key for the database, but not to modify the provider configuration. - -### Creating encrypted tables - -To create an encrypted table or modify an existing table to be encrypted, use the following commands: - -```sql -CREATE TABLE t1(a INT) USING tde_heap; -ALTER TABLE t1 SET ACCESS METHOD tde_heap; -``` - -### Changing the `pg_tde.inherit_global_keys` setting - -It is possible for users to use `pg_tde` with `inherit_global_keys = on`, refer to global keys / keyrings in databases, and then change this setting to `off`. - -In this case existing references to global providers, or the global default principal key will remain working as before, but new references to the global scope can't be made. - -## Typical setup scenarios - -### Simple "one principal key" encryption - -1. Passing the option from the postgres config file the extension: `shared_preload_libraries=‘pg_tde’` -2. `CREATE EXTENSION pg_tde;` in `template1` -3. Adding a global key provider -4. Adding a default principal key using the same global provider -5. Enable WAL encryption to use the default principal key using `ALTER SYSTEM SET pg_tde.wal_encrypt=‘ON’` -6. Restart the server -7. Optionally: setting the `default_table_access_method` to `tde_heap` so that tables are encrypted by default - -Database users don't need permissions to any of the encryption functions: -encryption is managed by the admins, normal users only have to create tables with encryption, which requires no specific permissions. - -### One key storage, but different keys per database - -1. Installing the extension: `shared_preload_libraries` + `pg_tde.wal_encrypt` -2. `CREATE EXTENSION pg_tde;` in `template1` -3. Adding a global key provider -4. Changing the WAL encryption to use the proper global key provider -5. Giving users that are expected to manage database keys permissions for database specific key management, but not database specific key provider management: - specific databases HAVE to use the global key provider - -Note: setting the `default_table_access_method` to `tde_heap` is possible, but instead of `ALTER SYSTEM` only per database using `ALTER DATABASE`, after a principal key is configured for that specific database. - -Alternatively `ALTER SYSTEM` is possible, but table creation in the database will fail if there's no principal key for the database, that has to be created first. - -### Complete multi tenancy - -1. Installing the extension: `shared_preload_libraries` + `pg_tde.wal_encrypt` (that's not multi tenant currently) -2. `CREATE EXTENSION pg_tde;` in any database -3. Adding a global key provider for WAL -4. Changing the WAL encryption to use the proper global key provider - -No default configuration: key providers / principal keys are configured as a per database level, permissions are managed per database - -Same note about `default_table_access_method` as above - but in a multi tenant setup, `ALTER SYSTEM` doesn't make much sense. From c757a4ed5a70180c9c032da6bebdab7ab43cedf9 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Wed, 25 Jun 2025 15:31:40 +0300 Subject: [PATCH 06/13] Revert "Delete contrib/pg_tde/documentation/docs/architecture/architecture.md" This reverts commit a9364d2cba66a8aa1fa9de2cc49f25f691634529. --- .../docs/architecture/architecture.md | 373 ++++++++++++++++++ 1 file changed, 373 insertions(+) create mode 100644 contrib/pg_tde/documentation/docs/architecture/architecture.md diff --git a/contrib/pg_tde/documentation/docs/architecture/architecture.md b/contrib/pg_tde/documentation/docs/architecture/architecture.md new file mode 100644 index 0000000000000..d2061a8f7058c --- /dev/null +++ b/contrib/pg_tde/documentation/docs/architecture/architecture.md @@ -0,0 +1,373 @@ +# Architecture + +`pg_tde` is a **customizable, complete, data at rest encryption extension** for PostgreSQL. + +Let's break down what it means. + +**Customizable** means that `pg_tde` aims to support many different use cases: + +* Encrypting either every table in every database or only some tables in some databases +* Encryption keys can be stored on various external key storage servers including Hashicorp Vault, KMIP servers. +* Using one key for everything or different keys for different databases +* Storing every key on the same key storage, or using different storages for different databases +* Handling permissions: who can manage database specific or global permissions, who can create encrypted or not encrypted tables + +**Complete** means that `pg_tde` aims to encrypt data at rest. + +**Data at rest** means everything written to the disk. This includes the following: + +* Table data files +* Indexes +* Sequences +* Temporary tables +* Write Ahead Log (WAL) + +**Extension** means that `pg_tde` should be implemented only as an extension, possibly compatible with any PostgreSQL distribution, including the open source community version. This requires changes in the PostgreSQL core to make it more extensible. Therefore, `pg_tde` currently works only with the [Percona Server for PostgreSQL](https://docs.percona.com/postgresql/17/index.html) - a binary replacement of community PostgreSQL and included in Percona Distribution for PostgreSQL. + +## Main components + +The main components of `pg_tde` are the following: + +* **Core server changes** focus on making the server more extensible, allowing the main logic of `pg_tde` to remain separate, as an extension. Core changes also add encryption-awareness to some command line tools that have to work directly with encrypted tables or encrypted WAL files. + + [Percona Server for PostgreSQL location](https://github.com/percona/postgres/tree/{{tdebranch}}) + +* The **`pg_tde` extension itself** implements the encryption code by hooking into the extension points introduced in the core changes, and the already existing extension points in the PostgreSQL server. + + Everything is controllable with GUC variables and SQL statements, similar to other extensions. + +* The **keyring API / libraries** implement the key storage logic with different key providers. The API is internal only, the keyring libraries are part of the main library for simplicity. +In the future these could be extracted into separate shared libraries with an open API, allowing the use of third-party providers. + +## Encryption architecture + +### Two-key hierarchy + +`pg_tde` uses two kinds of keys for encryption: + +* Internal keys to encrypt the data. They are stored in PostgreSQL's data directory under `$PGDATA/pg_tde`. +* Higher-level keys to encrypt internal keys. These keys are called *principal keys*. They are stored externally, in a Key Management System (KMS) using the key provider API. + +`pg_tde` uses one principal key per database. Every internal key for the given database is encrypted using this principal key. + +Internal keys are used for specific database files: each file with a different [Object Identifier (OID)](https://www.postgresql.org/docs/current/datatype-oid.html) has a different internal key. + +This means that, for example, a table with 4 indexes will have at least 5 internal keys - one for the table, and one for each index. + +If a table has additional associated relations, such as sequences or a TOAST table, those relations will also have separate keys. + +### Encryption algorithm + +`pg_tde` currently uses the following encryption algorithms: + +* `AES-128-CBC` for encrypting database files; encrypted with internal keys. + +* `AES-128-CTR` for WAL encryption; encrypted with internal keys. + +* `AES-128-GCM` for encrypting internal keys; encrypted with the principal key. + +Support for other cipher lengths / algorithms is planned in the future. + +### Encryption workflow + +`pg_tde` makes it possible to encrypt everything or only some tables in some databases. + +To support this without metadata changes, encrypted tables are labeled with a `tde_heap` access method marker. + +The `tde_heap` access method is the same as the `heap` one. It uses the same functions internally without any changes, but with the different name and ID. In such a way `pg_tde` knows that `tde_heap` tables are encrypted and `heap` tables are not. + +The initial decision what to encrypt is made using the `postgres` event trigger mechanism: if a `CREATE TABLE` or `ALTER TABLE` statement uses the `tde_heap` clause, the newly created data files are marked as encrypted. Then file operations encrypt or decrypt the data. + +Later decisions are made using a slightly modified Storage Manager (SMGR) API: when a database file is re-created with a different ID as a result of a `TRUNCATE` or a `VACUUM FULL` command, the newly created file inherits the encryption information and is either encrypted or not. + +### WAL encryption + +WAL encryption is controlled globally via a global GUC variable, `pg_tde.wal_encrypt`, that requires a server restart. + +WAL keys also contain the [LSN](https://www.postgresql.org/docs/17/wal-internals.html) of the first WAL write after key creation. This allows `pg_tde` to know which WAL ranges are encrypted or not and with which key. + +The setting only controls writes so that only WAL writes are encrypted when WAL encryption is enabled. This means that WAL files can contain both encrypted and unencrpyted data, depending on what the status of this variable was when writing the data. + +`pg_tde` keeps track of the encryption status of WAL records using internal keys. When the server is restarted it writes a new internal key if WAL encryption is enabled, or if it is disabled and was previously enabled it writes a dummy key signalling that WAL encryption ended. + +With this information the WAL reader code can decide if a specific WAL record has to be decrypted or not and which key it should use to decrypt it. + +### Encrypting other access methods + +Currently `pg_tde` only encrypts `heap` tables and other files such as indexes, TOAST tables, sequences that are related to the `heap` tables. + +Indexes include any kind of index that goes through the SMGR API, not just the built-in indexes in PostgreSQL. + +In theory, it is also possible to encrypt any other table access method that goes through the SMGR API by similarly providing a marker access method to it and extending the event triggers. + +### Storage Manager (SMGR) API + +`pg_tde` relies on a slightly modified version of the SMGR API. These modifications include: + +* Making the API generally extensible, where extensions can inject custom code into the storage manager +* Adding tracking information for files. When a new file is created for an existing relation, references to the existing file are also passed to the SMGR functions + +With these modifications, the `pg_tde` extension can implement an additional layer on top of the normal Magnetic Disk SMGR API: if the related table is encrypted, `pg_tde` encrypts a file before writing it to the disk and, similarly, decrypts it after reading when needed. + +## Key and key provider management + +### Principal key rotation + +You can rotate principal keys to comply with common policies and to handle situations with potentially exposed principal keys. + +Rotation means that `pg_tde` generates a new version of the principal key, and re-encrypts the associated internal keys with the new key. The old principal key is kept as is at the same location, because it may still be needed to decrypt backups or other databases. + +### Internal key regeneration + +Internal keys for tables, indexes and other data files are fixed once a file is created. There's no way to re-encrypt a file. + +There are workarounds for this, because operations that move the table data to a new file, such as `VACUUM FULL` or an `ALTER TABLE` that rewrites the file will create a new key for the new file, essentially rotating the internal key. This however means taking an exclusive lock on the table for the duration of the operation, which might not be desirable for huge tables. + +WAL internal keys are also fixed to the respective ranges. To generate a new WAL key you need to restart the database. + +### Internal key storage + +Internal keys and `pg_tde` metadata in general are kept in a single `$PGDATA/pg_tde` directory. This directory stores separate files for each database, such as: + +* Encrypted internal keys and internal key mapping to tables +* Information about the key providers + +Also, the `$PGDATA/pg_tde` directory has a special global section marked with the OID `1664`, which includes the global key providers and global internal keys. + +The global section is used for WAL encryption. Specific databases can use the global section too, for scenarios where users configure individual principal keys for databases but use the same global key provider. For this purpose, you must enable the global provider inheritance. + +The global default principal key uses the special OID `1663`. + +### Key providers (principal key storage) + +Principal keys are stored externally in a Key Management Services (KMS). In `pg_tde`a KMS is defined as an external key provider. + +The following key providers are supported: + +* [HashiCorp Vault](https://developer.hashicorp.com/vault/docs/what-is-vault) KV2 secrets engine +* [OpenBao](https://openbao.org/) implementation of Vault +* KMIP compatible servers +* A local file storage. This storage is intended only for development and testing and is not recommended for production use. + +For each key provider `pg_tde` requires a detailed configuration including the address of the service and the authentication information. + +With these details `pg_tde` does the following based on user operations: + +* Uploads a new principal key to it after this key is created +* Retrieves the principal key from the service when it is required for decryption + +Retreival of the principal key is cached so it only happens when necessary. + +### Key provider management + +Key provider configuration or location may change. For example, a service is moved to a new address or the principal key must be moved to a different key provider type. `pg_tde` supports both these scenarios enabling you to manage principal keys using simple [SQL functions](../functions.md#key-provider-management). + +In certain cases you can't use SQL functions to manage key providers. For example, if the key provider changed while the server wasn't running and is therefore unaware of these changes. The startup can fail if it needs to access the encryption keys. + +For such situations, `pg_tde` also provides [command line tools](../command-line-tools/index.md) to recover the database. + +### Sensitive key provider information + +!!! important + + Authentication details for key providers are sensitive and must be protected. + Do not store these credentials in the `$PGDATA` directory alongside the database. Instead, ensure they are stored in a secure location with strict file system permissions to prevent unauthorized access. + +## User interface + +### Setting up `pg_tde` + +To use `pg_tde`, users are required to: + +* Add `pg_tde` to the `shared_preload_libraries` in `postgresql.conf` as this is required for the SMGR extensions +* Execute `CREATE EXTENSION pg_tde` in the databases where they want to use encryption +* Optionally, enable `pg_tde.wal_encrypt` in `postgresql.conf` +* Optionally, disable `pg_tde.inherit_global_providers` in `postgresql.conf` (enabled by default) + +### Adding providers + +Keyring providers can be added to either the global or to the database specific scope. + +If `pg_tde.inherit_global_providers` is `on`, global providers are visible for all databases, and can be used. +If `pg_tde.inherit_global_providers` is `off`, global providers are only used for WAL encryption. + +To add a global provider: + +```sql +pg_tde_add_global_key_provider_('provider_name', ... details ...) +``` + +To add a database specific provider: + +```sql +pg_tde_add_database_key_provider_('provider_name', ... details ...) +``` + +### Changing providers + +To change a value of a global provider: + +```sql +pg_tde_change_global_key_provider_('provider_name', ... details ...) +``` + +To change a value of a database specific provider: + +```sql +pg_tde_change_database_key_provider_('provider_name', ... details ...) +``` + +These functions also allow changing the type of a provider. + +The functions however do not migrate any data. They are expected to be used during infrastructure migration, for example when the address of a server changes. + +Note that in these functions do not verify the parameters. For that, see `pg_tde_verify_key`. + +### Changing providers from the command line + +To change a provider from a command line, `pg_tde` provides the `pg_tde_change_key_provider` command line tool. + +This tool work similarly to the above functions, with the following syntax: + +```bash +pg_tde_change_key_provider ... details ... +``` + +Note that since this tool is expected to be offline, it bypasses all permission checks! + +This is also the reason why it requires a `dbOid` instead of a name, as it has no way to process the catalog and look up names. + +### Deleting providers + +Providers can be deleted by using the + +```sql +pg_tde_delete_database_key_provider(provider_name) +pg_tde_delete_global_key_provider(provider_name) +``` + +functions. + +For database specific providers, the function first checks if the provider is used or not, and the provider is only deleted if it's not used. + +For global providers, the function checks if the provider is used anywhere, WAL or any specific database, and returns an error if it is. + +This somewhat goes against the principle that `pg_tde` should not interact with other databases than the one the user is connected to, but on the other hand, it only does this lookup in the internal `pg_tde` metadata, not in postgres catalogs, so it is a gray zone. Making this check makes more sense than potentially making some databases inaccessible. + +### Listing/querying providers + +`pg_tde` provides 2 functions to show providers: + +* `pg_tde_list_all_database_key_providers()` +* `pg_tde_list_all_global_key_providers()` + +These functions return a list of provider names, type and configuration. + +### Provider permissions + +`pg_tde` implements access control based on execution rights on the administration functions. + +For keys and providers administration, it provides two pair of functions: + +```sql +pg_tde_GRANT_database_key_management_TO_role +pg_tde_REVOKE_database_key_management_FROM_role +``` + +### Creating and rotating keys + +Principal keys can be created or rotated using the following functions: + +```sql +pg_tde_set_key_using_(global/database)_key_provider('key-name', 'provider-name', ensure_new_key) +pg_tde_set_server_key_using_(global/database)_key_provider('key-name', 'provider-name', ensure_new_key) +pg_tde_set_default_key_using_(global/database)_key_provider('key-name', 'provider-name', ensure_new_key) +``` + +`ensure_new_key` is a boolean parameter defaulting to false. If it is `true` the function might return an error instead of setting the key if it already exists on the provider. + +### Default principal key + +With `pg_tde.inherit_global_key_providers`, it is also possible to set up a default global principal key, which will be used by any database which has the `pg_tde` extension enabled, but doesn't have a database specific principal key configured using `pg_tde_set_key_using_(global/database)_key_provider`. + +With this feature, it is possible for the entire database server to easily use the same principal key for all databases, completely disabling multi-tenency. + +A default key can be managed with the following functions: + +```sql +pg_tde_set_default_key_using_global_key_provider('key-name', 'provider-name', 'true/false') +``` + +`DROP` is only possible if there's no table currently using the default principal key. + +Changing the default principal key will rotate the encryption of internal keys for all databases using the current default principal key. + +### Current key details + +`pg_tde_key_info()` returns the name of the current principal key, and the provider it uses. + +`pg_tde_server_key_info()` does the same for the server key. + +`pg_tde_default_key_info()` does the same for the default key. + +`pg_tde_verify_key()` checks that the key provider is accessible, that the current principal key can be downloaded from it, and that it is the same as the current key stored in memory - if any of these fail, it reports an appropriate error. + +### Key permissions + +Users with management permissions to a specific database `(pg_tde_(grant/revoke)_(global/databse)_key_management_(to/from)_role)` can change the keys for the database, and use the current key functions. This includes creating keys using global providers, if `pg_tde.inherit_global_providers` is enabled. + +Also the `pg_tde_(grant/revoke)_database_key_management_to_role` function deals with only the specific permission for the above function: it allows a user to change the key for the database, but not to modify the provider configuration. + +### Creating encrypted tables + +To create an encrypted table or modify an existing table to be encrypted, use the following commands: + +```sql +CREATE TABLE t1(a INT) USING tde_heap; +ALTER TABLE t1 SET ACCESS METHOD tde_heap; +``` + +### Changing the `pg_tde.inherit_global_keys` setting + +It is possible for users to use `pg_tde` with `inherit_global_keys = on`, refer to global keys / keyrings in databases, and then change this setting to `off`. + +In this case existing references to global providers, or the global default principal key will remain working as before, but new references to the global scope can't be made. + +## Typical setup scenarios + +### Simple "one principal key" encryption + +1. Passing the option from the postgres config file the extension: `shared_preload_libraries=‘pg_tde’` +2. `CREATE EXTENSION pg_tde;` in `template1` +3. Adding a global key provider +4. Adding a default principal key using the same global provider +5. Enable WAL encryption to use the default principal key using `ALTER SYSTEM SET pg_tde.wal_encrypt=‘ON’` +6. Restart the server +7. Optionally: setting the `default_table_access_method` to `tde_heap` so that tables are encrypted by default + +Database users don't need permissions to any of the encryption functions: +encryption is managed by the admins, normal users only have to create tables with encryption, which requires no specific permissions. + +### One key storage, but different keys per database + +1. Installing the extension: `shared_preload_libraries` + `pg_tde.wal_encrypt` +2. `CREATE EXTENSION pg_tde;` in `template1` +3. Adding a global key provider +4. Changing the WAL encryption to use the proper global key provider +5. Giving users that are expected to manage database keys permissions for database specific key management, but not database specific key provider management: + specific databases HAVE to use the global key provider + +Note: setting the `default_table_access_method` to `tde_heap` is possible, but instead of `ALTER SYSTEM` only per database using `ALTER DATABASE`, after a principal key is configured for that specific database. + +Alternatively `ALTER SYSTEM` is possible, but table creation in the database will fail if there's no principal key for the database, that has to be created first. + +### Complete multi tenancy + +1. Installing the extension: `shared_preload_libraries` + `pg_tde.wal_encrypt` (that's not multi tenant currently) +2. `CREATE EXTENSION pg_tde;` in any database +3. Adding a global key provider for WAL +4. Changing the WAL encryption to use the proper global key provider + +No default configuration: key providers / principal keys are configured as a per database level, permissions are managed per database + +Same note about `default_table_access_method` as above - but in a multi tenant setup, `ALTER SYSTEM` doesn't make much sense. From b1a2fe5a658ca810535041fb98cc029e12fb9d7b Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Wed, 25 Jun 2025 15:31:46 +0300 Subject: [PATCH 07/13] Revert "Delete contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md" This reverts commit f6e48d47479bb960baf5361c4ba9ac90314eecbb. --- .../documentation/docs/advanced-topics/adv-top-overview.md | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md diff --git a/contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md b/contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md new file mode 100644 index 0000000000000..f52e01b97b378 --- /dev/null +++ b/contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md @@ -0,0 +1,7 @@ +# Technical Reference + +This section covers the internal components and tools that power `pg_tde`. + +Use it to understand how encryption is implemented, fine-tune a configuration, leverage advanced CLI tools and functions for diagnostics and customization. + +[Architecture](../architecture/architecture.md){.md-button} [GUC Variables](../variables.md){.md-button} [Functions](../functions.md){.md-button} From 6abadbadd243b45e2fb359ca302a0a1d410f33b4 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Wed, 25 Jun 2025 15:34:17 +0300 Subject: [PATCH 08/13] reverted names for arch and adv topics not needed to this PR --- .../docs/advanced-topics/{adv-top-overview.md => index.md} | 2 +- .../docs/architecture/{architecture.md => index.md} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename contrib/pg_tde/documentation/docs/advanced-topics/{adv-top-overview.md => index.md} (63%) rename contrib/pg_tde/documentation/docs/architecture/{architecture.md => index.md} (100%) diff --git a/contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md b/contrib/pg_tde/documentation/docs/advanced-topics/index.md similarity index 63% rename from contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md rename to contrib/pg_tde/documentation/docs/advanced-topics/index.md index f52e01b97b378..9df169821bcd3 100644 --- a/contrib/pg_tde/documentation/docs/advanced-topics/adv-top-overview.md +++ b/contrib/pg_tde/documentation/docs/advanced-topics/index.md @@ -4,4 +4,4 @@ This section covers the internal components and tools that power `pg_tde`. Use it to understand how encryption is implemented, fine-tune a configuration, leverage advanced CLI tools and functions for diagnostics and customization. -[Architecture](../architecture/architecture.md){.md-button} [GUC Variables](../variables.md){.md-button} [Functions](../functions.md){.md-button} +[Architecture](../architecture/index.md){.md-button} [GUC Variables](../variables.md){.md-button} [Functions](../functions.md){.md-button} diff --git a/contrib/pg_tde/documentation/docs/architecture/architecture.md b/contrib/pg_tde/documentation/docs/architecture/index.md similarity index 100% rename from contrib/pg_tde/documentation/docs/architecture/architecture.md rename to contrib/pg_tde/documentation/docs/architecture/index.md From 9c5bedaecb04f0f077399fd325ebfbf23efc82f9 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Wed, 25 Jun 2025 15:36:20 +0300 Subject: [PATCH 09/13] reverted wal encrypt and mkdocs names not needed in PR --- contrib/pg_tde/documentation/docs/wal-encryption.md | 2 +- contrib/pg_tde/documentation/mkdocs.yml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/contrib/pg_tde/documentation/docs/wal-encryption.md b/contrib/pg_tde/documentation/docs/wal-encryption.md index d63e6b7de2075..21f37ed73ae85 100644 --- a/contrib/pg_tde/documentation/docs/wal-encryption.md +++ b/contrib/pg_tde/documentation/docs/wal-encryption.md @@ -88,4 +88,4 @@ Before turning WAL encryption on, you must follow the steps below to create your Now WAL files start to be encrypted for both encrypted and unencrypted tables. For more technical references related to architecture, variables or functions, see: -[Technical Reference](advanced-topics/adv-top-overview.md){.md-button} +[Technical Reference](advanced-topics/index.md){.md-button} diff --git a/contrib/pg_tde/documentation/mkdocs.yml b/contrib/pg_tde/documentation/mkdocs.yml index 054a2cf3e2a71..fb8dc32df50ae 100644 --- a/contrib/pg_tde/documentation/mkdocs.yml +++ b/contrib/pg_tde/documentation/mkdocs.yml @@ -178,8 +178,8 @@ nav: - "3. Validate Encryption with pg_tde": test.md - "4. Configure WAL Encryption (tech preview)": wal-encryption.md - "Technical Reference": - - "Overview": advanced-topics/adv-top-overview.md - - "Architecture": architecture/architecture.md + - "Overview": advanced-topics/index.md + - "Architecture": architecture/index.md - "GUC Variables": variables.md - "Functions": functions.md - "TDE Operations": From 71bd0753758012c5c8d5d278ccf02e6326533f80 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Wed, 25 Jun 2025 15:56:19 +0300 Subject: [PATCH 10/13] Update faq.md removed the FAQ as it was not PR related --- contrib/pg_tde/documentation/docs/faq.md | 24 ++++++++---------------- 1 file changed, 8 insertions(+), 16 deletions(-) diff --git a/contrib/pg_tde/documentation/docs/faq.md b/contrib/pg_tde/documentation/docs/faq.md index e1256e8b59f51..df80bea17f555 100644 --- a/contrib/pg_tde/documentation/docs/faq.md +++ b/contrib/pg_tde/documentation/docs/faq.md @@ -61,7 +61,7 @@ Thus, to protect your sensitive data, consider using TDE to encrypt it at the ta * Regular monitoring and auditing * Additional data protection for sensitive fields (e.g., application-layer encryption) -## How does `pg_tde` make my data safe? +## How does pg_tde make my data safe? `pg_tde` uses two keys to encrypt data: @@ -106,7 +106,7 @@ Consider encrypting only tables that store sensitive data. You can decide what t We advise encrypting the whole database only if all your data is sensitive, like PII, or if there is no other way to comply with data safety requirements. -## What cipher mechanisms are used by `pg_tde`? +## What cipher mechanisms are used by pg_tde? `pg_tde` currently uses a AES-CBC-128 algorithm. First the internal keys in the datafile are encrypted using the principal key with AES-CBC-128, then the file data itself is again encrypted using AES-CBC-128 with the internal key. @@ -133,17 +133,17 @@ Since the `SET ACCESS METHOD` command drops hint bits and this may affect the pe You must restart the database in the following cases to apply the changes: * after you enabled the `pg_tde` extension -* to turn WAL encryption on or off +* to turn on / off the WAL encryption After that, no database restart is required. When you create or alter the table using the `tde_heap` access method, the files are marked as those that require encryption. The encryption happens at the storage manager level, before a transaction is written to disk. Read more about [how tde_heap works](index/table-access-method.md#how-tde_heap-works). ## What happens to my data if I lose a principal key? -If you lose encryption keys, especially the principal key, the data is **lost**. It is critical to back up your encryption keys securely and use the Key Management service for key management. For more information, see the [Configure Key Management (KMS)](../docs/global-key-provider-configuration/index.md) topic. +If you lose encryption keys, especially, the principal key, the data is lost. That's why it's critical to back up your encryption keys securely and use the Key Management service for key management. -## Can I use `pg_tde` in a multi-tenant setup? +## Can I use pg_tde in a multi-tenant setup? -Yes. Multi-tenancy is the type of architecture where multiple users, or tenants, share the same resource. It can be a database, a schema or an entire cluster. +Multi-tenancy is the type of architecture where multiple users, or tenants, share the same resource. It can be a database, a schema or an entire cluster. In `pg_tde`, multi-tenancy is supported via a separate principal key per database. This means that a database owner can decide what tables to encrypt within a database. The same database can have both encrypted and non-encrypted tables. @@ -157,16 +157,8 @@ WAL files are encrypted globally across the entire PostgreSQL cluster using the Since the encryption happens on the database level, it makes no difference for your tools and applications. They work with the data in the same way. -To restore from an encrypted backup, you must have the same principal encryption key, which was used to encrypt files in your backup. See the [Restore an encrypted pg_tde backup](../docs/how-to/restore-backups.md) topic for more details. +To restore from an encrypted backup, you must have the same principal encryption key, which was used to encrypt files in your backup. -## What if the address of the key provider changed for one of my old backups? - -You can use the [pg_tde_change_key_provider](../docs/command-line-tools/pg-tde-change-key-provider.md) tool to correct the configuration. - -## How can I store an old key securely? - -To store an old key securely, see the [Configure Key Management (KMS)](../docs/global-key-provider-configuration/index.md) topic. - -## I'm using OpenSSL in FIPS mode and need to use `pg_tde`. Does `pg_tde` comply with FIPS requirements? Can I use my own FIPS-mode OpenSSL library with `pg_tde`? +## I'm using OpenSSL in FIPS mode and need to use pg_tde. Does pg_tde comply with FIPS requirements? Can I use my own FIPS-mode OpenSSL library with pg_tde? Yes. `pg_tde` works with the FIPS-compliant version of OpenSSL, whether it is provided by your operating system or if you use your own OpenSSL libraries. If you use your own libraries, make sure they are FIPS certified. From 1ea8981f23550f5d63aba20e9851ef4566c3eaac Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Wed, 25 Jun 2025 15:59:21 +0300 Subject: [PATCH 11/13] Update faq.md small updates to `pg_tde` names in faq --- contrib/pg_tde/documentation/docs/faq.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/contrib/pg_tde/documentation/docs/faq.md b/contrib/pg_tde/documentation/docs/faq.md index df80bea17f555..4c621cb0b10fb 100644 --- a/contrib/pg_tde/documentation/docs/faq.md +++ b/contrib/pg_tde/documentation/docs/faq.md @@ -61,7 +61,7 @@ Thus, to protect your sensitive data, consider using TDE to encrypt it at the ta * Regular monitoring and auditing * Additional data protection for sensitive fields (e.g., application-layer encryption) -## How does pg_tde make my data safe? +## How does `pg_tde` make my data safe? `pg_tde` uses two keys to encrypt data: @@ -106,7 +106,7 @@ Consider encrypting only tables that store sensitive data. You can decide what t We advise encrypting the whole database only if all your data is sensitive, like PII, or if there is no other way to comply with data safety requirements. -## What cipher mechanisms are used by pg_tde? +## What cipher mechanisms are used by `pg_tde`? `pg_tde` currently uses a AES-CBC-128 algorithm. First the internal keys in the datafile are encrypted using the principal key with AES-CBC-128, then the file data itself is again encrypted using AES-CBC-128 with the internal key. @@ -141,7 +141,7 @@ After that, no database restart is required. When you create or alter the table If you lose encryption keys, especially, the principal key, the data is lost. That's why it's critical to back up your encryption keys securely and use the Key Management service for key management. -## Can I use pg_tde in a multi-tenant setup? +## Can I use `pg_tde` in a multi-tenant setup? Multi-tenancy is the type of architecture where multiple users, or tenants, share the same resource. It can be a database, a schema or an entire cluster. From 33753ac192980c2ddf50a52435099cd4b1b4e889 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Wed, 25 Jun 2025 16:01:06 +0300 Subject: [PATCH 12/13] Update faq.md again used `` to update pg_tde --- contrib/pg_tde/documentation/docs/faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/pg_tde/documentation/docs/faq.md b/contrib/pg_tde/documentation/docs/faq.md index 4c621cb0b10fb..902e0a18b49fc 100644 --- a/contrib/pg_tde/documentation/docs/faq.md +++ b/contrib/pg_tde/documentation/docs/faq.md @@ -159,6 +159,6 @@ Since the encryption happens on the database level, it makes no difference for y To restore from an encrypted backup, you must have the same principal encryption key, which was used to encrypt files in your backup. -## I'm using OpenSSL in FIPS mode and need to use pg_tde. Does pg_tde comply with FIPS requirements? Can I use my own FIPS-mode OpenSSL library with pg_tde? +## I'm using OpenSSL in FIPS mode and need to use `pg_tde`. Does `pg_tde` comply with FIPS requirements? Can I use my own FIPS-mode OpenSSL library with `pg_tde`? Yes. `pg_tde` works with the FIPS-compliant version of OpenSSL, whether it is provided by your operating system or if you use your own OpenSSL libraries. If you use your own libraries, make sure they are FIPS certified. From 575580f97a190d4381a150bed76c397606e85b62 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Wed, 25 Jun 2025 16:49:50 +0300 Subject: [PATCH 13/13] Update restore-backups.md updated last note to underline that if a key is missing the backup can't be done and moved the second intro paragraph to the old backups ## topic. --- .../documentation/docs/how-to/restore-backups.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/contrib/pg_tde/documentation/docs/how-to/restore-backups.md b/contrib/pg_tde/documentation/docs/how-to/restore-backups.md index c5d0f38383d55..2abe8747a405b 100644 --- a/contrib/pg_tde/documentation/docs/how-to/restore-backups.md +++ b/contrib/pg_tde/documentation/docs/how-to/restore-backups.md @@ -2,15 +2,17 @@ To restore an encrypted backup created with an earlier key, ensure the key is still available in your Key Management System (KMS). The backup remains **encrypted** on disk, `pg_tde` uses the correct key to transparently access the data. -Each encrypted table stores metadata that identifies which encryption key was used. When the encrypted data is read, `pg_tde` retrieves the correct key from the configured KMS. +## How pg_tde uses old keys to load backups -## How `pg_tde` uses old keys to load backups - -* **KMS**: stores the key that was active when the backup was made. +* **KMS**: stores the principal key(s) that were active when the backup was made. * **Encrypted backup**: contains data encrypted with the key available at that time. * **Current server**: `pg_tde` automatically loads the backup if the key is accessible through the KMS. -At runtime, `pg_tde` reads the key ID from the table metadata and fetches the corresponding key from the KMS. +The table (internal) keys are stored within the backup. At runtime, `pg_tde` retrieves the principal key(s) from the configured KMS and uses them to decrypt the internal keys, enabling access to the encrypted table data. !!! note - If the key was deleted or the KMS configuration changed, the backup **cannot** be read. Use [pg_tde_change_key_provider](../command-line-tools/pg-tde-change-key-provider.md) to update KMS credentials or endpoints if needed. + If the required key(s) get deleted or are missing, the backup **cannot** be read. + + If the key(s) still exists but the KMS configuration has changed, you can use [`pg_tde_change_key_provider`](../command-line-tools/pg-tde-change-key-provider.md) to update the configuration. + + However, if the key(s) got deleted from all accessible KMS sources, the encrypted backup is **unrecoverable**.