PG-1257 Add functions for principal key removal

Add SQL functions that allow user to remove principal key.

* Database level principal key can be removed if there are no encrypted tables or if there is default key. For the first case we just drop key map file completely, for the second we perform key rotation.
* Default principal key can be removed if there are no databases that use it.

Readded the DELETE key function to docs based on 1257 in Architecture chapter where we also update from DROP to DELETE.

Author: artemgavrilov

contrib/pg_tde/Makefile

@@ -8,12 +8,13 @@ DATA = pg_tde--1.0-rc.sql
 # Since meson supports skipping test suites this is a make only feature
 ifndef TDE_MODE
 REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/pg_tde/pg_tde.conf
-# create_database must run after default_principal_key which must run after
-# key_provider.
 REGRESS = access_control \
 alter_index \
 cache_alloc \
 change_access_method \
+create_database \
+default_principal_key \
+delete_principal_key \
 insert_update_delete \
 key_provider \
 kmip_test \
@@ -24,9 +25,7 @@ relocate \
 tablespace \
 toast_decrypt \
 vault_v2_test \
-version \
-default_principal_key \
-create_database
+version
 TAP_TESTS = 1
 endif
 

contrib/pg_tde/documentation/docs/architecture/index.md

@@ -292,15 +292,23 @@ With `pg_tde.inherit_global_key_providers`, it is also possible to set up a defa
 
 With this feature, it is possible for the entire database server to easily use the same principal key for all databases, completely disabling multi-tenency.
 
-A default key can be managed with the following functions:
+#### Manage a default key
 
-```sql
-pg_tde_set_default_key_using_global_key_provider('key-name', 'provider-name', 'true/false')
-```
+You can manage a default key with the following functions:
+
+* `pg_tde_set_default_key_using_global_key_provider('key-name','provider-name','true/false')`
+* `pg_tde_delete_default_key()`
+
+!!! note
+    `pg_tde_delete_default_key()` is only possible if there's no table currently using the default principal key.
+    Changing the default principal key will rotate the encryption of internal keys for all databases using the current default principal key.
+
+#### Delete a key
 
-`DROP` is only possible if there's no table currently using the default principal key.
+The `pg_tde_delete_key()` function removes the principal key for the current database. If the current database has any encrypted tables, and there isn’t a default principal key configured, it reports an error instead. If there are encrypted tables, but there’s also a global default principal key, internal keys will be encrypted with the default key.
 
-Changing the default principal key will rotate the encryption of internal keys for all databases using the current default principal key.
+!!! note
+    WAL keys **cannot** be deleted, as server keys are managed separately.
 
 ### Current key details
 

contrib/pg_tde/expected/access_control.out

@@ -10,6 +10,8 @@ SET ROLE regress_pg_tde_access_control;
 -- should throw access denied
 SELECT pg_tde_set_key_using_database_key_provider('test-db-key', 'local-file-provider');
 ERROR:  permission denied for function pg_tde_set_key_using_database_key_provider
+SELECT pg_tde_delete_key();
+ERROR:  permission denied for function pg_tde_delete_key
 SELECT pg_tde_list_all_database_key_providers();
 ERROR:  permission denied for function pg_tde_list_all_database_key_providers
 SELECT pg_tde_list_all_global_key_providers();
@@ -37,6 +39,7 @@ GRANT EXECUTE ON FUNCTION pg_tde_delete_global_key_provider(TEXT) TO regress_pg_
 GRANT EXECUTE ON FUNCTION pg_tde_set_default_key_using_global_key_provider(TEXT, TEXT, BOOLEAN) TO regress_pg_tde_access_control;
 GRANT EXECUTE ON FUNCTION pg_tde_set_key_using_global_key_provider(TEXT, TEXT, BOOLEAN) TO regress_pg_tde_access_control;
 GRANT EXECUTE ON FUNCTION pg_tde_set_server_key_using_global_key_provider(TEXT, TEXT, BOOLEAN) TO regress_pg_tde_access_control;
+GRANT EXECUTE ON FUNCTION pg_tde_delete_default_key() TO regress_pg_tde_access_control;
 SET ROLE regress_pg_tde_access_control;
 SELECT pg_tde_add_database_key_provider_file('local-file-provider', '/tmp/pg_tde_test_keyring.per');
 ERROR:  must be superuser to modify key providers
@@ -56,5 +59,7 @@ SELECT pg_tde_set_default_key_using_global_key_provider('key1', 'global-file-pro
 ERROR:  must be superuser to access global key providers
 SELECT pg_tde_set_server_key_using_global_key_provider('key1', 'global-file-provider');
 ERROR:  must be superuser to access global key providers
+SELECT pg_tde_delete_default_key();
+ERROR:  must be superuser to access global key providers
 RESET ROLE;
 DROP EXTENSION pg_tde CASCADE;

contrib/pg_tde/expected/alter_index.out

@@ -67,5 +67,17 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind
 (5 rows)
 
 DROP TABLE concur_reindex_part;
+SELECT pg_tde_delete_key();
+ pg_tde_delete_key 
+-------------------
+ 
+(1 row)
+
+SELECT pg_tde_delete_database_key_provider('file-vault');
+ pg_tde_delete_database_key_provider 
+-------------------------------------
+ 
+(1 row)
+
 DROP EXTENSION pg_tde;
 RESET default_table_access_method;

contrib/pg_tde/expected/create_database.out

@@ -102,4 +102,16 @@ CREATE DATABASE new_db_file_copy TEMPLATE template_db STRATEGY FILE_COPY;
 DROP DATABASE new_db_file_copy;
 DROP DATABASE new_db;
 DROP DATABASE template_db;
+SELECT pg_tde_delete_default_key();
+ pg_tde_delete_default_key 
+---------------------------
+ 
+(1 row)
+
+SELECT pg_tde_delete_global_key_provider('global-file-vault');
+ pg_tde_delete_global_key_provider 
+-----------------------------------
+ 
+(1 row)
+
 DROP EXTENSION pg_tde;

contrib/pg_tde/expected/default_principal_key.out

@@ -33,20 +33,17 @@ SELECT key_provider_id, key_provider_name, key_name
 		FROM pg_tde_default_key_info();
  key_provider_id | key_provider_name |  key_name   
 -----------------+-------------------+-------------
-              -5 | file-provider     | default-key
+              -2 | file-provider     | default-key
 (1 row)
 
 -- fails
 SELECT pg_tde_delete_global_key_provider('file-provider');
 ERROR:  Can't delete a provider which is currently in use
 SELECT id, provider_name FROM pg_tde_list_all_global_key_providers();
- id |  provider_name   
-----+------------------
- -1 | file-keyring
- -3 | global-provider
- -4 | global-provider2
- -5 | file-provider
-(4 rows)
+ id | provider_name 
+----+---------------
+ -2 | file-provider
+(1 row)
 
 -- Should fail: no principal key for the database yet
 SELECT  key_provider_id, key_provider_name, key_name
@@ -68,7 +65,7 @@ SELECT  key_provider_id, key_provider_name, key_name
 		FROM pg_tde_key_info();
  key_provider_id | key_provider_name |  key_name   
 -----------------+-------------------+-------------
-              -5 | file-provider     | default-key
+              -2 | file-provider     | default-key
 (1 row)
 
 SELECT current_database() AS regress_database
@@ -97,7 +94,7 @@ SELECT  key_provider_id, key_provider_name, key_name
 		FROM pg_tde_key_info();
  key_provider_id | key_provider_name |  key_name   
 -----------------+-------------------+-------------
-              -5 | file-provider     | default-key
+              -2 | file-provider     | default-key
 (1 row)
 
 \c :regress_database
@@ -112,15 +109,15 @@ SELECT  key_provider_id, key_provider_name, key_name
 		FROM pg_tde_key_info();
  key_provider_id | key_provider_name |    key_name     
 -----------------+-------------------+-----------------
-              -5 | file-provider     | new-default-key
+              -2 | file-provider     | new-default-key
 (1 row)
 
 \c regress_pg_tde_other
 SELECT  key_provider_id, key_provider_name, key_name
 		FROM pg_tde_key_info();
  key_provider_id | key_provider_name |    key_name     
 -----------------+-------------------+-----------------
-              -5 | file-provider     | new-default-key
+              -2 | file-provider     | new-default-key
 (1 row)
 
 SELECT pg_buffercache_evict(bufferid) FROM pg_buffercache WHERE relfilenode = (SELECT relfilenode FROM pg_class WHERE oid = 'test_enc'::regclass);
@@ -155,6 +152,18 @@ SELECT * FROM test_enc;
 (3 rows)
 
 DROP TABLE test_enc;
+SELECT pg_tde_delete_default_key();
+ pg_tde_delete_default_key 
+---------------------------
+ 
+(1 row)
+
+SELECT pg_tde_delete_global_key_provider('file-provider');
+ pg_tde_delete_global_key_provider 
+-----------------------------------
+ 
+(1 row)
+
 DROP EXTENSION pg_tde CASCADE;
 DROP EXTENSION pg_buffercache;
 DROP DATABASE regress_pg_tde_other;

contrib/pg_tde/expected/delete_principal_key.out

@@ -0,0 +1,130 @@
+CREATE EXTENSION IF NOT EXISTS pg_tde;
+SELECT pg_tde_add_global_key_provider_file('file-provider','/tmp/pg_tde_test_keyring.per');
+ pg_tde_add_global_key_provider_file 
+-------------------------------------
+ 
+(1 row)
+
+-- Set the local key and delete it without any encrypted tables
+-- Should succeed: nothing used the key
+SELECT pg_tde_set_key_using_global_key_provider('test-db-key','file-provider');
+ pg_tde_set_key_using_global_key_provider 
+------------------------------------------
+ 
+(1 row)
+
+SELECT key_provider_id, key_provider_name, key_name FROM pg_tde_key_info();
+ key_provider_id | key_provider_name |  key_name   
+-----------------+-------------------+-------------
+              -3 | file-provider     | test-db-key
+(1 row)
+
+SELECT pg_tde_delete_key();
+ pg_tde_delete_key 
+-------------------
+ 
+(1 row)
+
+-- Set local key, encrypt a table, and delete the key
+-- Should fail: the is no default key to fallback
+SELECT pg_tde_set_key_using_global_key_provider('test-db-key','file-provider');
+ pg_tde_set_key_using_global_key_provider 
+------------------------------------------
+ 
+(1 row)
+
+CREATE TABLE test_table (id int, data text) USING tde_heap;
+SELECT pg_tde_delete_key();
+ERROR:  cannot delete principal key
+DETAIL:  There are encrypted tables in the database.
+HINT:  Set default principal key as fallback option or decrypt all tables before deleting principal key.
+-- Decrypt the table and delete the key
+-- Should succeed: there is no more encrypted tables
+ALTER TABLE test_table SET ACCESS METHOD heap;
+SELECT pg_tde_delete_key();
+ pg_tde_delete_key 
+-------------------
+ 
+(1 row)
+
+-- Set local key, encrypt the table then delete teable and key
+-- Should succeed: the table is deleted and there are no more encrypted tables
+SELECT pg_tde_set_key_using_global_key_provider('test-db-key','file-provider');
+ pg_tde_set_key_using_global_key_provider 
+------------------------------------------
+ 
+(1 row)
+
+ALTER TABLE test_table SET ACCESS METHOD tde_heap;
+DROP TABLE test_table;
+SELECT pg_tde_delete_key();
+ pg_tde_delete_key 
+-------------------
+ 
+(1 row)
+
+-- Set default key, set regular key, create table, delete regular key
+-- Should succeed: regular key will be rotated to default key
+SELECT pg_tde_set_default_key_using_global_key_provider('defalut-key','file-provider');
+ pg_tde_set_default_key_using_global_key_provider 
+--------------------------------------------------
+ 
+(1 row)
+
+SELECT pg_tde_set_key_using_global_key_provider('test-db-key','file-provider');
+ pg_tde_set_key_using_global_key_provider 
+------------------------------------------
+ 
+(1 row)
+
+CREATE TABLE test_table (id int, data text) USING tde_heap;
+SELECT pg_tde_delete_key();
+ pg_tde_delete_key 
+-------------------
+ 
+(1 row)
+
+SELECT key_provider_id, key_provider_name, key_name FROM pg_tde_key_info();
+ key_provider_id | key_provider_name |  key_name   
+-----------------+-------------------+-------------
+              -3 | file-provider     | defalut-key
+(1 row)
+
+-- Try to delete key when default key is used
+-- Should fail: table already uses the default key, so there is no key to fallback to
+SELECT pg_tde_delete_key();
+ERROR:  cannot delete principal key
+DETAIL:  There are encrypted tables in the database.
+-- Try to delete default key
+-- Should fail: default key is used by the table
+SELECT pg_tde_delete_default_key();
+ERROR:  cannot delete default principal key
+HINT:  There are encrypted tables in the database with id: 16384.
+-- Set regular principal key, delete default key
+-- Should succeed: the table will use the regular key
+SELECT pg_tde_set_key_using_global_key_provider('test-db-key','file-provider');
+ pg_tde_set_key_using_global_key_provider 
+------------------------------------------
+ 
+(1 row)
+
+SELECT pg_tde_delete_default_key();
+ pg_tde_delete_default_key 
+---------------------------
+ 
+(1 row)
+
+DROP TABLE test_table;
+SELECT pg_tde_delete_key();
+ pg_tde_delete_key 
+-------------------
+ 
+(1 row)
+
+SELECT pg_tde_delete_global_key_provider('file-provider');
+ pg_tde_delete_global_key_provider 
+-----------------------------------
+ 
+(1 row)
+
+DROP EXTENSION pg_tde;

contrib/pg_tde/expected/key_provider.out

@@ -75,8 +75,8 @@ SELECT pg_tde_add_global_key_provider_file('file-keyring2','/tmp/pg_tde_test_key
 SELECT id, provider_name FROM pg_tde_list_all_global_key_providers();
  id | provider_name 
 ----+---------------
- -1 | file-keyring
- -2 | file-keyring2
+ -4 | file-keyring
+ -5 | file-keyring2
 (2 rows)
 
 -- fails
@@ -105,8 +105,8 @@ SELECT id, provider_name FROM pg_tde_list_all_database_key_providers();
 SELECT id, provider_name FROM pg_tde_list_all_global_key_providers();
  id | provider_name 
 ----+---------------
- -1 | file-keyring
- -2 | file-keyring2
+ -4 | file-keyring
+ -5 | file-keyring2
 (2 rows)
 
 SELECT pg_tde_set_key_using_global_key_provider('test-db-key', 'file-keyring', false);
@@ -121,8 +121,8 @@ ERROR:  Can't delete a provider which is currently in use
 SELECT id, provider_name FROM pg_tde_list_all_global_key_providers();
  id | provider_name 
 ----+---------------
- -1 | file-keyring
- -2 | file-keyring2
+ -4 | file-keyring
+ -5 | file-keyring2
 (2 rows)
 
 -- works
@@ -135,7 +135,7 @@ SELECT pg_tde_delete_global_key_provider('file-keyring2');
 SELECT id, provider_name FROM pg_tde_list_all_global_key_providers();
  id | provider_name 
 ----+---------------
- -1 | file-keyring
+ -4 | file-keyring
 (1 row)
 
 -- Creating a file key provider fails if we can't open or create the file