Consistent failiure after major upgrade ( 15 -> 16 ) #1139
Description
Hi!
we manage several postgres instance, and we are planning on running a major upgrade on all of them.
We've been trying on some.
The one I'm testing with has the following manifest:
apiVersion: acid.zalan.do/v1
kind: postgresql
metadata:
creationTimestamp: "2025-06-27T09:41:46Z"
generation: 2
labels:
app.kubernetes.io/instance: postgresql
app.kubernetes.io/managed-by: libre.sh
app.kubernetes.io/name: postgres
app.libre.sh/ref: postgres_postgresql
name: postgresql--pg
namespace: drive-preprod
ownerReferences:
- apiVersion: core.libre.sh/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Postgres
name: postgresql
uid: xxx
resourceVersion: "150088394"
uid: yyy
spec:
additionalVolumes:
- mountPath: /run/etc
name: walg
targetContainers:
- postgres
- backup-exporter
volumeSource:
emptyDir: {}
allowedSourceRanges: null
env:
- name: WALE_S3_PREFIX
value: s3://drive-preprod-postgresql--pg/spilo/zzz/15
- name: AWS_S3_FORCE_PATH_STYLE
value: "true"
- name: WALG_DISABLE_S3_SSE
value: "true"
- name: USE_WALG_BACKUP
value: "true"
- name: USE_WALG_RESTORE
value: "true"
- name: BACKUP_SCHEDULE
value: 52 13 * * *
- name: BACKUP_NUM_TO_RETAIN
value: "3"
- name: WAL_S3_BUCKET
valueFrom:
secretKeyRef:
key: path
name: postgresql--pg.bucket.libre.sh
- name: AWS_ENDPOINT
valueFrom:
secretKeyRef:
key: url
name: postgresql--pg.bucket.libre.sh
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
key: username
name: postgresql--pg.bucket.libre.sh
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
key: password
name: postgresql--pg.bucket.libre.sh
numberOfInstances: 2
patroni: {}
podPriorityClassName: lsh-app-critical
postgresql:
version: "15"
teamId: lsh
users:
drive:
- superuser
- createdb
drive_reader: []
volume:
size: 5G
status:
PostgresClusterStatus: Running
At this time this was the status of the cluster:
root@postgresql--pg-0:/home/postgres# patronictl list
+ Cluster: postgresql--pg (7520563461028134983) ---------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+------------------+---------------+---------+-----------+----+-----------+
| postgresql--pg-0 | 10.233.72.107 | Replica | streaming | 2 | 0 |
| postgresql--pg-1 | 10.233.73.203 | Leader | running | 2 | |
+------------------+---------------+---------+-----------+----+-----------+
I followed the documentation to do in place major upgrade:
https://postgres-operator.readthedocs.io/en/latest/administrator/#in-place-major-version-upgrade
Changed the version in the manifest, and exec'ed into he master and ran:
python3 /scripts/inplace_upgrade.py 2
Logs of the upgrade script:
postgres@postgresql--pg-1:~$ python3 /scripts/inplace_upgrade.py 2
2025-06-27 10:04:41,095 inplace_upgrade INFO: No PostgreSQL configuration items changed, nothing to reload.
2025-06-27 10:04:41,110 inplace_upgrade INFO: establishing a new patroni heartbeat connection to postgres
2025-06-27 10:04:41,146 inplace_upgrade INFO: Cluster postgresql--pg is ready to be upgraded
2025-06-27 10:04:41,147 inplace_upgrade INFO: initdb config: [{'locale': 'en_US.utf-8'}, {'encoding': 'UTF8'}]
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf-8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /home/postgres/pgdata/pgroot/data_new ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/usr/lib/postgresql/16/bin/pg_ctl -D /home/postgres/pgdata/pgroot/data_new -l logfile start
2025-06-27 10:04:51,514 inplace_upgrade INFO: Dropping extensions from the cluster which could be incompatible
2025-06-27 10:04:51,517 inplace_upgrade INFO: Executing "DROP EXTENSION IF EXISTS pg_repack" in the database="postgres"
2025-06-27 10:04:51,520 inplace_upgrade INFO: Executing "DROP EXTENSION IF EXISTS pg_repack" in the database="drive"
2025-06-27 10:04:51,523 inplace_upgrade INFO: Executing "DROP EXTENSION IF EXISTS pg_repack" in the database="template1"
2025-06-27 10:04:51,523 inplace_upgrade INFO: Executing pg_upgrade --check
Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for system-defined composite types in user tables ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for incompatible "aclitem" data type in user tables ok
Checking for presence of required libraries ok
Checking database user is the install user ok
Checking for prepared transactions ok
Checking for new cluster tablespace directories ok
*Clusters are compatible*
2025-06-27 10:04:52,117 inplace_upgrade INFO: Dropping objects from the cluster which could be incompatible
2025-06-27 10:04:52,120 inplace_upgrade INFO: Executing "REVOKE EXECUTE ON FUNCTION pg_catalog.pg_switch_wal() FROM admin" in the database="postgres"
2025-06-27 10:04:52,121 inplace_upgrade INFO: Executing "DROP FUNCTION metric_helpers.pg_stat_statements" in the database="postgres"
2025-06-27 10:04:52,124 inplace_upgrade INFO: Executing "DROP EXTENSION IF EXISTS pg_stat_kcache" in the database="postgres"
2025-06-27 10:04:52,127 inplace_upgrade INFO: Executing "DROP EXTENSION IF EXISTS pg_stat_statements" in the database="postgres"
2025-06-27 10:04:52,129 inplace_upgrade INFO: Executing "DROP EXTENSION IF EXISTS pg_repack" in the database="postgres"
2025-06-27 10:04:52,132 inplace_upgrade INFO: Executing "REVOKE EXECUTE ON FUNCTION pg_catalog.pg_switch_wal() FROM admin" in the database="drive"
2025-06-27 10:04:52,133 inplace_upgrade INFO: Executing "DROP FUNCTION metric_helpers.pg_stat_statements" in the database="drive"
2025-06-27 10:04:52,136 inplace_upgrade INFO: Executing "DROP EXTENSION IF EXISTS pg_stat_kcache" in the database="drive"
2025-06-27 10:04:52,138 inplace_upgrade INFO: Executing "DROP EXTENSION IF EXISTS pg_stat_statements" in the database="drive"
2025-06-27 10:04:52,140 inplace_upgrade INFO: Executing "DROP EXTENSION IF EXISTS pg_repack" in the database="drive"
2025-06-27 10:04:52,143 inplace_upgrade INFO: Executing "REVOKE EXECUTE ON FUNCTION pg_catalog.pg_switch_wal() FROM admin" in the database="template1"
2025-06-27 10:04:52,145 inplace_upgrade INFO: Executing "DROP FUNCTION metric_helpers.pg_stat_statements" in the database="template1"
2025-06-27 10:04:52,149 inplace_upgrade INFO: Executing "DROP EXTENSION IF EXISTS pg_stat_kcache" in the database="template1"
2025-06-27 10:04:52,154 inplace_upgrade INFO: Executing "DROP EXTENSION IF EXISTS pg_stat_statements" in the database="template1"
2025-06-27 10:04:52,158 inplace_upgrade INFO: Executing "DROP EXTENSION IF EXISTS pg_repack" in the database="template1"
2025-06-27 10:04:52,159 inplace_upgrade INFO: Enabling maintenance mode
2025-06-27 10:04:53,211 inplace_upgrade INFO: Maintenance mode enabled
2025-06-27 10:04:53,211 inplace_upgrade INFO: Doing a clean shutdown of the cluster before pg_upgrade
2025-06-27 10:05:03,985 inplace_upgrade INFO: Latest checkpoint location: 218103848
2025-06-27 10:05:03,985 inplace_upgrade INFO: Starting rsyncd
2025-06-27 10:05:03,986 inplace_upgrade INFO: Waiting for replica nodes to catch up with primary
2025-06-27 10:05:03,992 inplace_upgrade INFO: All replicas are ready
2025-06-27 10:05:03,992 inplace_upgrade INFO: Executing CHECKPOINT on replicas postgresql--pg-0
2025-06-27 10:05:04,007 inplace_upgrade INFO: Executing pg_upgrade
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for system-defined composite types in user tables ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for incompatible "aclitem" data type in user tables ok
Creating dump of global objects ok
Creating dump of database schemas
ok
Checking for presence of required libraries ok
Checking database user is the install user ok
Checking for prepared transactions ok
Checking for new cluster tablespace directories ok
If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.
Performing Upgrade
------------------
Setting locale and encoding for new cluster ok
Analyzing all rows in the new cluster ok
Freezing all rows in the new cluster ok
Deleting files from new pg_xact ok
Copying old pg_xact to new server ok
Setting oldest XID for new cluster ok
Setting next transaction ID and epoch for new cluster ok
Deleting files from new pg_multixact/offsets ok
Copying old pg_multixact/offsets to new server ok
Deleting files from new pg_multixact/members ok
Copying old pg_multixact/members to new server ok
Setting next multixact ID and offset for new cluster ok
Resetting WAL archives ok
Setting frozenxid and minmxid counters in new cluster ok
Restoring global objects in the new cluster ok
Restoring database schemas in the new cluster
ok
Adding ".old" suffix to old global/pg_control ok
If you want to start the old cluster, you will need to remove
the ".old" suffix from /home/postgres/pgdata/pgroot/data/global/pg_control.old.
Because "link" mode was used, the old cluster cannot be safely
started once the new cluster has been started.
Linking user relation files
ok
Setting next OID for new cluster ok
Sync data directory to disk ok
Creating script to delete old cluster ok
Checking for extension updates ok
Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade.
Once you start the new server, consider running:
/usr/lib/postgresql/16/bin/vacuumdb -U postgres --all --analyze-in-stages
Running this script will delete the old cluster's data files:
./delete_old_cluster.sh
2025-06-27 10:05:45,445 inplace_upgrade INFO: Updating configuration files
2025-06-27 10:05:45,458 inplace_upgrade INFO: Writing to file /run/postgres.yml
2025-06-27 10:05:45,458 inplace_upgrade INFO: Notifying replicas postgresql--pg-0 to start rsync
2025-06-27 10:05:45,735 inplace_upgrade INFO: Waiting for replicas rsync to complete
2025-06-27 10:05:54,741 inplace_upgrade INFO: Rsync took 9.283143281936646 seconds
2025-06-27 10:05:54,741 inplace_upgrade INFO: Stopping rsyncd
2025-06-27 10:05:56,742 inplace_upgrade INFO: Removing initialize key
2025-06-27 10:05:56,790 inplace_upgrade INFO: Restarting patroni
2025-06-27 10:05:57,794 inplace_upgrade INFO: Starting the primary postgres up
2025-06-27 10:05:59,060 inplace_upgrade INFO: 200 restarted successfully
2025-06-27 10:05:59,060 inplace_upgrade INFO: Upgrade downtime: 65.84856081008911
2025-06-27 10:05:59,411 inplace_upgrade INFO: establishing a new patroni heartbeat connection to postgres
2025-06-27 10:05:59,421 inplace_upgrade INFO: Executing 'ALTER EXTENSION plpgsql UPDATE' in the database=template1
2025-06-27 10:05:59,421 inplace_upgrade INFO: Executing 'ALTER EXTENSION set_user UPDATE' in the database=template1
2025-06-27 10:05:59,425 inplace_upgrade INFO: Executing 'ALTER EXTENSION plpgsql UPDATE' in the database=drive
2025-06-27 10:05:59,426 inplace_upgrade INFO: Executing 'ALTER EXTENSION set_user UPDATE' in the database=drive
2025-06-27 10:05:59,429 inplace_upgrade INFO: Executing 'ALTER EXTENSION plpgsql UPDATE' in the database=postgres
2025-06-27 10:05:59,429 inplace_upgrade INFO: Executing 'ALTER EXTENSION pg_cron UPDATE' in the database=postgres
2025-06-27 10:05:59,429 inplace_upgrade INFO: Executing 'ALTER EXTENSION plpython3u UPDATE' in the database=postgres
2025-06-27 10:05:59,429 inplace_upgrade INFO: Executing 'ALTER EXTENSION file_fdw UPDATE' in the database=postgres
2025-06-27 10:05:59,429 inplace_upgrade INFO: Executing 'ALTER EXTENSION pg_auth_mon UPDATE' in the database=postgres
2025-06-27 10:05:59,429 inplace_upgrade INFO: Executing 'ALTER EXTENSION set_user UPDATE' in the database=postgres
2025-06-27 10:05:59,430 inplace_upgrade INFO: Resetting non-default statistics target before analyze
2025-06-27 10:05:59,430 inplace_upgrade INFO: Waiting for restart of patroni on replicas postgresql--pg-0
2025-06-27 10:05:59,435 inplace_upgrade INFO: ['postgresql--pg-0'] successfully restarted
2025-06-27 10:05:59,436 inplace_upgrade INFO: Disabling maintenance mode
2025-06-27 10:05:59,443 inplace_upgrade INFO: Rebuilding statistics (vacuumdb --analyze-in-stages)
vacuumdb: processing database "template1": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "drive": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "postgres": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "drive": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "drive": Generating default (full) optimizer statistics
vacuumdb: processing database "template1": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "postgres": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "template1": Generating default (full) optimizer statistics
vacuumdb: processing database "postgres": Generating default (full) optimizer statistics
2025-06-27 10:06:00,484 inplace_upgrade INFO: Maintenance mode disabled
2025-06-27 10:06:00,484 inplace_upgrade INFO: Total upgrade time (with analyze): 67.27327084541321
DO
NOTICE: role "admin" has already been granted membership in role "cron_admin" by role "postgres"
GRANT ROLE
DO
DO
NOTICE: extension "pg_auth_mon" already exists, skipping
CREATE EXTENSION
NOTICE: version "1.1" of extension "pg_auth_mon" is already installed
ALTER EXTENSION
GRANT
NOTICE: extension "pg_cron" already exists, skipping
CREATE EXTENSION
DO
NOTICE: version "1.6" of extension "pg_cron" is already installed
ALTER EXTENSION
ALTER POLICY
REVOKE
GRANT
REVOKE
GRANT
ALTER POLICY
REVOKE
GRANT
CREATE FUNCTION
REVOKE
GRANT
REVOKE
GRANT
NOTICE: extension "file_fdw" already exists, skipping
CREATE EXTENSION
DO
NOTICE: relation "postgres_log" already exists, skipping
CREATE TABLE
GRANT
NOTICE: column "backend_type" of relation "postgres_log" already exists, skipping
ALTER TABLE
NOTICE: column "leader_pid" of relation "postgres_log" already exists, skipping
ALTER TABLE
NOTICE: column "query_id" of relation "postgres_log" already exists, skipping
ALTER TABLE
DO
NOTICE: relation "postgres_log_0" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE: relation "postgres_log_1" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE: relation "postgres_log_2" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE: relation "postgres_log_3" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE: relation "postgres_log_4" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE: relation "postgres_log_5" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE: relation "postgres_log_6" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE: relation "postgres_log_7" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
RESET
SET
NOTICE: drop cascades to 7 other objects
DETAIL: drop cascades to function zmon_utils.get_database_cluster_information()
drop cascades to function zmon_utils.get_last_status_active_cronjobs()
drop cascades to function zmon_utils.get_replay_lag()
drop cascades to type zmon_utils.system_information
drop cascades to function zmon_utils.get_database_cluster_system_information()
drop cascades to view zmon_utils.last_status_active_cronjobs
drop cascades to view zmon_utils.replay_lag
DROP SCHEMA
NOTICE: extension "plpython3u" already exists, skipping
DO
NOTICE: language "plpythonu" does not exist, skipping
DROP LANGUAGE
NOTICE: function plpython_call_handler() does not exist, skipping
DROP FUNCTION
NOTICE: function plpython_inline_handler(internal) does not exist, skipping
DROP FUNCTION
NOTICE: function plpython_validator(oid) does not exist, skipping
DROP FUNCTION
CREATE SCHEMA
GRANT
SET
CREATE TYPE
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
REVOKE
CREATE VIEW
REVOKE
GRANT
CREATE FUNCTION
CREATE VIEW
REVOKE
REVOKE
GRANT
GRANT
You are now connected to database "template1" as user "postgres".
NOTICE: schema "user_management" already exists, skipping
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
CREATE EXTENSION
CREATE EXTENSION
NOTICE: extension "set_user" already exists, skipping
CREATE EXTENSION
NOTICE: version "4.0.1" of extension "set_user" is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
NOTICE: schema "metric_helpers" already exists, skipping
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
REVOKE
GRANT
REVOKE
GRANT
RESET
You are now connected to database "drive" as user "postgres".
NOTICE: schema "user_management" already exists, skipping
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
CREATE EXTENSION
CREATE EXTENSION
NOTICE: extension "set_user" already exists, skipping
CREATE EXTENSION
NOTICE: version "4.0.1" of extension "set_user" is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
NOTICE: schema "metric_helpers" already exists, skipping
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
REVOKE
GRANT
REVOKE
GRANT
RESET
You are now connected to database "postgres" as user "postgres".
NOTICE: schema "user_management" already exists, skipping
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
CREATE EXTENSION
CREATE EXTENSION
NOTICE: extension "set_user" already exists, skipping
CREATE EXTENSION
NOTICE: version "4.0.1" of extension "set_user" is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
NOTICE: schema "metric_helpers" already exists, skipping
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
REVOKE
GRANT
REVOKE
GRANT
RESET
2025-06-27 10:06:01,123 inplace_upgrade INFO: Removing /home/postgres/pgdata/pgroot/data_old
2025-06-27 10:06:01,160 inplace_upgrade INFO: Initiating a new backup...
csv logs of the replica:
2025-06-27 10:04:38.877 UTC,"postgres","postgres",259,"10.233.72.107:60578",685e6cb6.103,4,"idle",2025-06-27 10:04:38 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.016 user=postgres database=postgres host=10.233.72.107 port=60578",,,,,,,,,"","client backend",,0
2025-06-27 10:04:41.135 UTC,,,261,"10.233.73.203:55842",685e6cb9.105,1,"",2025-06-27 10:04:41 UTC,,0,LOG,00000,"connection received: host=10.233.73.203 port=55842",,,,,,,,,"","not initialized",,0
2025-06-27 10:04:41.142 UTC,"postgres","postgres",261,"10.233.73.203:55842",685e6cb9.105,2,"authentication",2025-06-27 10:04:41 UTC,3/263,0,LOG,00000,"connection authenticated: identity=""postgres"" method=md5 (/home/postgres/pgdata/pgroot/data/pg_hba.conf:12)",,,,,,,,,"","client backend",,-10637612914751176
2025-06-27 10:04:41.142 UTC,"postgres","postgres",261,"10.233.73.203:55842",685e6cb9.105,3,"authentication",2025-06-27 10:04:41 UTC,3/263,0,LOG,00000,"connection authorized: user=postgres database=postgres SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)",,,,,,,,,"","client backend",,-10637612914751176
2025-06-27 10:04:42.756 UTC,,,262,"127.0.0.1:36022",685e6cba.106,1,"",2025-06-27 10:04:42 UTC,,0,LOG,00000,"connection received: host=127.0.0.1 port=36022",,,,,,,,,"","not initialized",,0
2025-06-27 10:04:42.759 UTC,"postgres","postgres",262,"127.0.0.1:36022",685e6cba.106,2,"authentication",2025-06-27 10:04:42 UTC,5/62,0,LOG,00000,"connection authenticated: identity=""postgres"" method=md5 (/home/postgres/pgdata/pgroot/data/pg_hba.conf:5)",,,,,,,,,"","client backend",,-10637612914751176
2025-06-27 10:04:42.759 UTC,"postgres","postgres",262,"127.0.0.1:36022",685e6cba.106,3,"authentication",2025-06-27 10:04:42 UTC,5/62,0,LOG,00000,"connection authorized: user=postgres database=postgres SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)",,,,,,,,,"","client backend",,-10637612914751176
2025-06-27 10:04:46.338 UTC,,,263,"10.233.72.107:44638",685e6cbe.107,1,"",2025-06-27 10:04:46 UTC,,0,LOG,00000,"connection received: host=10.233.72.107 port=44638",,,,,,,,,"","not initialized",,0
2025-06-27 10:04:46.341 UTC,"postgres","postgres",263,"10.233.72.107:44638",685e6cbe.107,2,"authentication",2025-06-27 10:04:46 UTC,7/21,0,LOG,00000,"connection authenticated: identity=""postgres"" method=md5 (/home/postgres/pgdata/pgroot/data/pg_hba.conf:12)",,,,,,,,,"","client backend",,-10637612914751176
2025-06-27 10:04:46.341 UTC,"postgres","postgres",263,"10.233.72.107:44638",685e6cbe.107,3,"authentication",2025-06-27 10:04:46 UTC,7/21,0,LOG,00000,"connection authorized: user=postgres database=postgres SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)",,,,,,,,,"","client backend",,-10637612914751176
2025-06-27 10:04:46.357 UTC,"postgres","postgres",263,"10.233.72.107:44638",685e6cbe.107,4,"idle",2025-06-27 10:04:46 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.019 user=postgres database=postgres host=10.233.72.107 port=44638",,,,,,,,,"","client backend",,0
2025-06-27 10:04:52.270 UTC,,,64,,685e6c37.40,7,,2025-06-27 10:02:31 UTC,,0,LOG,00000,"received SIGHUP, reloading configuration files",,,,,,,,,"","postmaster",,0
2025-06-27 10:04:53.861 UTC,,,267,"10.233.72.107:39352",685e6cc5.10b,1,"",2025-06-27 10:04:53 UTC,,0,LOG,00000,"connection received: host=10.233.72.107 port=39352",,,,,,,,,"","not initialized",,0
2025-06-27 10:04:53.864 UTC,"postgres","postgres",267,"10.233.72.107:39352",685e6cc5.10b,2,"authentication",2025-06-27 10:04:53 UTC,7/41,0,LOG,00000,"connection authenticated: identity=""postgres"" method=md5 (/home/postgres/pgdata/pgroot/data/pg_hba.conf:12)",,,,,,,,,"","client backend",,-10637612914751176
2025-06-27 10:04:53.864 UTC,"postgres","postgres",267,"10.233.72.107:39352",685e6cc5.10b,3,"authentication",2025-06-27 10:04:53 UTC,7/41,0,LOG,00000,"connection authorized: user=postgres database=postgres SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)",,,,,,,,,"","client backend",,-10637612914751176
2025-06-27 10:04:53.876 UTC,"postgres","postgres",267,"10.233.72.107:39352",685e6cc5.10b,4,"idle",2025-06-27 10:04:53 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.015 user=postgres database=postgres host=10.233.72.107 port=39352",,,,,,,,,"","client backend",,0
2025-06-27 10:05:01.337 UTC,,,268,"10.233.72.107:39368",685e6ccd.10c,1,"",2025-06-27 10:05:01 UTC,,0,LOG,00000,"connection received: host=10.233.72.107 port=39368",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:01.341 UTC,"postgres","postgres",268,"10.233.72.107:39368",685e6ccd.10c,2,"authentication",2025-06-27 10:05:01 UTC,7/61,0,LOG,00000,"connection authenticated: identity=""postgres"" method=md5 (/home/postgres/pgdata/pgroot/data/pg_hba.conf:12)",,,,,,,,,"","client backend",,-10637612914751176
2025-06-27 10:05:01.341 UTC,"postgres","postgres",268,"10.233.72.107:39368",685e6ccd.10c,3,"authentication",2025-06-27 10:05:01 UTC,7/61,0,LOG,00000,"connection authorized: user=postgres database=postgres SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)",,,,,,,,,"","client backend",,-10637612914751176
2025-06-27 10:05:01.362 UTC,"postgres","postgres",268,"10.233.72.107:39368",685e6ccd.10c,4,"idle",2025-06-27 10:05:01 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.025 user=postgres database=postgres host=10.233.72.107 port=39368",,,,,,,,,"","client backend",,0
2025-06-27 10:05:03.963 UTC,,,175,,685e6c37.af,2,,2025-06-27 10:02:31 UTC,,0,LOG,00000,"replication terminated by primary server","End of WAL reached on timeline 2 at 0/D0000A0.",,,,,,,,"","walreceiver",,0
2025-06-27 10:05:03.964 UTC,,,175,,685e6c37.af,3,,2025-06-27 10:02:31 UTC,,0,FATAL,08006,"could not send end-of-streaming message to primary: SSL connection has been closed unexpectedly
no COPY in progress",,,,,,,,,"","walreceiver",,0
2025-06-27 10:05:04.006 UTC,,,67,,685e6c37.43,1,,2025-06-27 10:02:31 UTC,,0,LOG,00000,"restartpoint starting: immediate wait",,,,,,,,,"","checkpointer",,0
2025-06-27 10:05:04.200 UTC,,,69,,685e6c37.45,8,,2025-06-27 10:02:31 UTC,1/0,0,LOG,00000,"invalid record length at 0/D0000A0: wanted 24, got 0",,,,,,,,,"","startup",,0
2025-06-27 10:05:04.209 UTC,,,312,,685e6cd0.138,1,,2025-06-27 10:05:04 UTC,,0,FATAL,08006,"could not connect to the primary server: connection to server at ""10.233.73.203"", port 5432 failed: received invalid response to SSL negotiation: @",,,,,,,,,"","walreceiver",,0
2025-06-27 10:05:04.330 UTC,,,69,,685e6c37.45,9,,2025-06-27 10:02:31 UTC,1/0,0,LOG,00000,"waiting for WAL to become available at 0/D0000B8",,,,,,,,,"","startup",,0
2025-06-27 10:05:05.325 UTC,,,67,,685e6c37.43,2,,2025-06-27 10:02:31 UTC,,0,LOG,00000,"restartpoint complete: wrote 461 buffers (3.6%); 0 WAL file(s) added, 4 removed, 0 recycled; write=1.255 s, sync=0.038 s, total=1.320 s; sync files=156, longest=0.033 s, average=0.001 s; distance=81920 kB, estimate=81920 kB",,,,,,,,,"","checkpointer",,0
2025-06-27 10:05:05.325 UTC,,,67,,685e6c37.43,3,,2025-06-27 10:02:31 UTC,,0,LOG,00000,"recovery restart point at 0/D000028","Last completed transaction was at log time 2025-06-27 10:04:52.155176+00.",,,,,,,,"","checkpointer",,0
2025-06-27 10:05:05.326 UTC,"postgres","postgres",261,"10.233.73.203:55842",685e6cb9.105,4,"CHECKPOINT",2025-06-27 10:04:41 UTC,3/0,0,LOG,00000,"duration: 1323.583 ms statement: CHECKPOINT",,,,,,,,,"","client backend",,-2695421136582183392
2025-06-27 10:05:08.860 UTC,,,333,"10.233.72.107:53576",685e6cd4.14d,1,"",2025-06-27 10:05:08 UTC,,0,LOG,00000,"connection received: host=10.233.72.107 port=53576",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:08.863 UTC,"postgres","postgres",333,"10.233.72.107:53576",685e6cd4.14d,2,"authentication",2025-06-27 10:05:08 UTC,7/81,0,LOG,00000,"connection authenticated: identity=""postgres"" method=md5 (/home/postgres/pgdata/pgroot/data/pg_hba.conf:12)",,,,,,,,,"","client backend",,-10637612914751176
2025-06-27 10:05:08.863 UTC,"postgres","postgres",333,"10.233.72.107:53576",685e6cd4.14d,3,"authentication",2025-06-27 10:05:08 UTC,7/81,0,LOG,00000,"connection authorized: user=postgres database=postgres SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)",,,,,,,,,"","client backend",,-10637612914751176
2025-06-27 10:05:08.878 UTC,"postgres","postgres",333,"10.233.72.107:53576",685e6cd4.14d,4,"idle",2025-06-27 10:05:08 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.018 user=postgres database=postgres host=10.233.72.107 port=53576",,,,,,,,,"","client backend",,0
2025-06-27 10:05:09.179 UTC,,,69,,685e6c37.45,10,,2025-06-27 10:02:31 UTC,1/0,0,LOG,00000,"restored log file ""00000002000000000000000D"" from archive",,,,,,,,,"","startup",,0
2025-06-27 10:05:12.646 UTC,,,411,,685e6cd8.19b,1,,2025-06-27 10:05:12 UTC,,0,FATAL,08006,"could not connect to the primary server: connection to server at ""10.233.73.203"", port 5432 failed: received invalid response to SSL negotiation: @",,,,,,,,,"","walreceiver",,0
2025-06-27 10:05:12.720 UTC,,,432,"[local]",685e6cd8.1b0,1,"",2025-06-27 10:05:12 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:12.720 UTC,"standby","",432,"[local]",685e6cd8.1b0,2,"authentication",2025-06-27 10:05:12 UTC,7/101,0,LOG,00000,"replication connection authorized: user=standby",,,,,,,,,"","walsender",,-10637612914751176
2025-06-27 10:05:12.721 UTC,"standby","",432,"[local]",685e6cd8.1b0,3,"idle",2025-06-27 10:05:12 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.002 user=standby database= host=[local]",,,,,,,,,"","walsender",,0
2025-06-27 10:05:12.795 UTC,,,69,,685e6c37.45,11,,2025-06-27 10:02:31 UTC,1/0,0,LOG,00000,"waiting for WAL to become available at 0/E000018",,,,,,,,,"","startup",,0
2025-06-27 10:05:14.202 UTC,,,455,,685e6cda.1c7,1,,2025-06-27 10:05:14 UTC,,0,FATAL,08006,"could not connect to the primary server: connection to server at ""10.233.73.203"", port 5432 failed: received invalid response to SSL negotiation: @",,,,,,,,,"","walreceiver",,0
2025-06-27 10:05:14.336 UTC,,,69,,685e6c37.45,12,,2025-06-27 10:02:31 UTC,1/0,0,LOG,00000,"waiting for WAL to become available at 0/E000018",,,,,,,,,"","startup",,0
2025-06-27 10:05:16.340 UTC,,,473,"10.233.72.107:50028",685e6cdc.1d9,1,"",2025-06-27 10:05:16 UTC,,0,LOG,00000,"connection received: host=10.233.72.107 port=50028",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:16.350 UTC,"postgres","postgres",473,"10.233.72.107:50028",685e6cdc.1d9,2,"authentication",2025-06-27 10:05:16 UTC,7/102,0,LOG,00000,"connection authenticated: identity=""postgres"" method=md5 (/home/postgres/pgdata/pgroot/data/pg_hba.conf:12)",,,,,,,,,"","client backend",,0
2025-06-27 10:05:16.350 UTC,"postgres","postgres",473,"10.233.72.107:50028",685e6cdc.1d9,3,"authentication",2025-06-27 10:05:16 UTC,7/102,0,LOG,00000,"connection authorized: user=postgres database=postgres SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)",,,,,,,,,"","client backend",,0
2025-06-27 10:05:16.365 UTC,"postgres","postgres",473,"10.233.72.107:50028",685e6cdc.1d9,4,"idle",2025-06-27 10:05:16 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.025 user=postgres database=postgres host=10.233.72.107 port=50028",,,,,,,,,"","client backend",,0
2025-06-27 10:05:18.220 UTC,,,475,"[local]",685e6cde.1db,1,"",2025-06-27 10:05:18 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:18.221 UTC,"standby","",475,"[local]",685e6cde.1db,2,"authentication",2025-06-27 10:05:18 UTC,7/122,0,LOG,00000,"replication connection authorized: user=standby",,,,,,,,,"","walsender",,-10637612914751176
2025-06-27 10:05:18.221 UTC,"standby","",475,"[local]",685e6cde.1db,3,"idle",2025-06-27 10:05:18 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.001 user=standby database= host=[local]",,,,,,,,,"","walsender",,0
2025-06-27 10:05:19.080 UTC,,,479,"[local]",685e6cdf.1df,1,"",2025-06-27 10:05:19 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:19.081 UTC,"standby","",479,"[local]",685e6cdf.1df,2,"authentication",2025-06-27 10:05:19 UTC,7/123,0,LOG,00000,"replication connection authorized: user=standby",,,,,,,,,"","walsender",,0
2025-06-27 10:05:19.081 UTC,"standby","",479,"[local]",685e6cdf.1df,3,"idle",2025-06-27 10:05:19 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.001 user=standby database= host=[local]",,,,,,,,,"","walsender",,0
2025-06-27 10:05:19.183 UTC,,,501,,685e6cdf.1f5,1,,2025-06-27 10:05:19 UTC,,0,FATAL,08006,"could not connect to the primary server: connection to server at ""10.233.73.203"", port 5432 failed: received invalid response to SSL negotiation: @",,,,,,,,,"","walreceiver",,0
2025-06-27 10:05:19.337 UTC,,,69,,685e6c37.45,13,,2025-06-27 10:02:31 UTC,1/0,0,LOG,00000,"waiting for WAL to become available at 0/E000018",,,,,,,,,"","startup",,0
2025-06-27 10:05:22.720 UTC,,,525,"[local]",685e6ce2.20d,1,"",2025-06-27 10:05:22 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:22.722 UTC,"standby","",525,"[local]",685e6ce2.20d,2,"authentication",2025-06-27 10:05:22 UTC,7/124,0,LOG,00000,"replication connection authorized: user=standby",,,,,,,,,"","walsender",,0
2025-06-27 10:05:22.723 UTC,"standby","",525,"[local]",685e6ce2.20d,3,"idle",2025-06-27 10:05:22 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.002 user=standby database= host=[local]",,,,,,,,,"","walsender",,0
2025-06-27 10:05:23.863 UTC,,,526,"10.233.72.107:50164",685e6ce3.20e,1,"",2025-06-27 10:05:23 UTC,,0,LOG,00000,"connection received: host=10.233.72.107 port=50164",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:23.868 UTC,"postgres","postgres",526,"10.233.72.107:50164",685e6ce3.20e,2,"authentication",2025-06-27 10:05:23 UTC,7/125,0,LOG,00000,"connection authenticated: identity=""postgres"" method=md5 (/home/postgres/pgdata/pgroot/data/pg_hba.conf:12)",,,,,,,,,"","client backend",,0
2025-06-27 10:05:23.868 UTC,"postgres","postgres",526,"10.233.72.107:50164",685e6ce3.20e,3,"authentication",2025-06-27 10:05:23 UTC,7/125,0,LOG,00000,"connection authorized: user=postgres database=postgres SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)",,,,,,,,,"","client backend",,0
2025-06-27 10:05:23.881 UTC,"postgres","postgres",526,"10.233.72.107:50164",685e6ce3.20e,4,"idle",2025-06-27 10:05:23 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.019 user=postgres database=postgres host=10.233.72.107 port=50164",,,,,,,,,"","client backend",,0
2025-06-27 10:05:24.202 UTC,,,547,,685e6ce4.223,1,,2025-06-27 10:05:24 UTC,,0,FATAL,08006,"could not connect to the primary server: connection to server at ""10.233.73.203"", port 5432 failed: received invalid response to SSL negotiation: @",,,,,,,,,"","walreceiver",,0
2025-06-27 10:05:24.326 UTC,,,69,,685e6c37.45,14,,2025-06-27 10:02:31 UTC,1/0,0,LOG,00000,"waiting for WAL to become available at 0/E000018",,,,,,,,,"","startup",,0
2025-06-27 10:05:29.184 UTC,,,590,,685e6ce9.24e,1,,2025-06-27 10:05:29 UTC,,0,FATAL,08006,"could not connect to the primary server: connection to server at ""10.233.73.203"", port 5432 failed: received invalid response to SSL negotiation: @",,,,,,,,,"","walreceiver",,0
2025-06-27 10:05:29.338 UTC,,,69,,685e6c37.45,15,,2025-06-27 10:02:31 UTC,1/0,0,LOG,00000,"waiting for WAL to become available at 0/E000018",,,,,,,,,"","startup",,0
2025-06-27 10:05:29.379 UTC,,,615,"[local]",685e6ce9.267,1,"",2025-06-27 10:05:29 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:29.379 UTC,"standby","",615,"[local]",685e6ce9.267,2,"authentication",2025-06-27 10:05:29 UTC,7/145,0,LOG,00000,"replication connection authorized: user=standby",,,,,,,,,"","walsender",,-10637612914751176
2025-06-27 10:05:29.380 UTC,"standby","",615,"[local]",685e6ce9.267,3,"idle",2025-06-27 10:05:29 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.001 user=standby database= host=[local]",,,,,,,,,"","walsender",,0
2025-06-27 10:05:30.371 UTC,,,620,"[local]",685e6cea.26c,1,"",2025-06-27 10:05:30 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:30.371 UTC,"standby","",620,"[local]",685e6cea.26c,2,"authentication",2025-06-27 10:05:30 UTC,7/146,0,LOG,00000,"replication connection authorized: user=standby",,,,,,,,,"","walsender",,0
2025-06-27 10:05:30.372 UTC,"standby","",620,"[local]",685e6cea.26c,3,"idle",2025-06-27 10:05:30 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.001 user=standby database= host=[local]",,,,,,,,,"","walsender",,0
2025-06-27 10:05:31.340 UTC,,,622,"10.233.72.107:50170",685e6ceb.26e,1,"",2025-06-27 10:05:31 UTC,,0,LOG,00000,"connection received: host=10.233.72.107 port=50170",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:31.345 UTC,"postgres","postgres",622,"10.233.72.107:50170",685e6ceb.26e,2,"authentication",2025-06-27 10:05:31 UTC,7/147,0,LOG,00000,"connection authenticated: identity=""postgres"" method=md5 (/home/postgres/pgdata/pgroot/data/pg_hba.conf:12)",,,,,,,,,"","client backend",,0
2025-06-27 10:05:31.345 UTC,"postgres","postgres",622,"10.233.72.107:50170",685e6ceb.26e,3,"authentication",2025-06-27 10:05:31 UTC,7/147,0,LOG,00000,"connection authorized: user=postgres database=postgres SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)",,,,,,,,,"","client backend",,0
2025-06-27 10:05:31.362 UTC,"postgres","postgres",622,"10.233.72.107:50170",685e6ceb.26e,4,"idle",2025-06-27 10:05:31 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.021 user=postgres database=postgres host=10.233.72.107 port=50170",,,,,,,,,"","client backend",,0
2025-06-27 10:05:32.719 UTC,,,623,"[local]",685e6cec.26f,1,"",2025-06-27 10:05:32 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:32.720 UTC,"standby","",623,"[local]",685e6cec.26f,2,"authentication",2025-06-27 10:05:32 UTC,7/167,0,LOG,00000,"replication connection authorized: user=standby",,,,,,,,,"","walsender",,-10637612914751176
2025-06-27 10:05:32.720 UTC,"standby","",623,"[local]",685e6cec.26f,3,"idle",2025-06-27 10:05:32 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.002 user=standby database= host=[local]",,,,,,,,,"","walsender",,0
2025-06-27 10:05:33.220 UTC,,,625,"[local]",685e6ced.271,1,"",2025-06-27 10:05:33 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:33.220 UTC,"standby","",625,"[local]",685e6ced.271,2,"authentication",2025-06-27 10:05:33 UTC,7/168,0,LOG,00000,"replication connection authorized: user=standby",,,,,,,,,"","walsender",,0
2025-06-27 10:05:33.221 UTC,"standby","",625,"[local]",685e6ced.271,3,"idle",2025-06-27 10:05:33 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.001 user=standby database= host=[local]",,,,,,,,,"","walsender",,0
2025-06-27 10:05:34.081 UTC,,,627,"[local]",685e6cee.273,1,"",2025-06-27 10:05:34 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:34.082 UTC,"standby","",627,"[local]",685e6cee.273,2,"authentication",2025-06-27 10:05:34 UTC,7/169,0,LOG,00000,"replication connection authorized: user=standby",,,,,,,,,"","walsender",,0
2025-06-27 10:05:34.083 UTC,"standby","",627,"[local]",685e6cee.273,3,"idle",2025-06-27 10:05:34 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.002 user=standby database= host=[local]",,,,,,,,,"","walsender",,0
2025-06-27 10:05:34.206 UTC,,,650,,685e6cee.28a,1,,2025-06-27 10:05:34 UTC,,0,FATAL,08006,"could not connect to the primary server: connection to server at ""10.233.73.203"", port 5432 failed: received invalid response to SSL negotiation: @",,,,,,,,,"","walreceiver",,0
2025-06-27 10:05:34.344 UTC,,,69,,685e6c37.45,16,,2025-06-27 10:02:31 UTC,1/0,0,LOG,00000,"waiting for WAL to become available at 0/E000018",,,,,,,,,"","startup",,0
2025-06-27 10:05:38.861 UTC,,,671,"10.233.72.107:49354",685e6cf2.29f,1,"",2025-06-27 10:05:38 UTC,,0,LOG,00000,"connection received: host=10.233.72.107 port=49354",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:38.865 UTC,"postgres","postgres",671,"10.233.72.107:49354",685e6cf2.29f,2,"authentication",2025-06-27 10:05:38 UTC,7/170,0,LOG,00000,"connection authenticated: identity=""postgres"" method=md5 (/home/postgres/pgdata/pgroot/data/pg_hba.conf:12)",,,,,,,,,"","client backend",,0
2025-06-27 10:05:38.865 UTC,"postgres","postgres",671,"10.233.72.107:49354",685e6cf2.29f,3,"authentication",2025-06-27 10:05:38 UTC,7/170,0,LOG,00000,"connection authorized: user=postgres database=postgres SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)",,,,,,,,,"","client backend",,0
2025-06-27 10:05:38.881 UTC,"postgres","postgres",671,"10.233.72.107:49354",685e6cf2.29f,4,"idle",2025-06-27 10:05:38 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.020 user=postgres database=postgres host=10.233.72.107 port=49354",,,,,,,,,"","client backend",,0
2025-06-27 10:05:39.198 UTC,,,696,,685e6cf3.2b8,1,,2025-06-27 10:05:39 UTC,,0,FATAL,08006,"could not connect to the primary server: connection to server at ""10.233.73.203"", port 5432 failed: received invalid response to SSL negotiation: @",,,,,,,,,"","walreceiver",,0
2025-06-27 10:05:39.335 UTC,,,69,,685e6c37.45,17,,2025-06-27 10:02:31 UTC,1/0,0,LOG,00000,"waiting for WAL to become available at 0/E000018",,,,,,,,,"","startup",,0
2025-06-27 10:05:42.719 UTC,,,718,"[local]",685e6cf6.2ce,1,"",2025-06-27 10:05:42 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:42.719 UTC,"standby","",718,"[local]",685e6cf6.2ce,2,"authentication",2025-06-27 10:05:42 UTC,7/190,0,LOG,00000,"replication connection authorized: user=standby",,,,,,,,,"","walsender",,-10637612914751176
2025-06-27 10:05:42.720 UTC,"standby","",718,"[local]",685e6cf6.2ce,3,"idle",2025-06-27 10:05:42 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.001 user=standby database= host=[local]",,,,,,,,,"","walsender",,0
2025-06-27 10:05:42.827 UTC,,,719,"127.0.0.1:41884",685e6cf6.2cf,1,"",2025-06-27 10:05:42 UTC,,0,LOG,00000,"connection received: host=127.0.0.1 port=41884",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:42.830 UTC,"postgres","postgres",719,"127.0.0.1:41884",685e6cf6.2cf,2,"authentication",2025-06-27 10:05:42 UTC,7/191,0,LOG,00000,"connection authenticated: identity=""postgres"" method=md5 (/home/postgres/pgdata/pgroot/data/pg_hba.conf:5)",,,,,,,,,"","client backend",,0
2025-06-27 10:05:42.830 UTC,"postgres","postgres",719,"127.0.0.1:41884",685e6cf6.2cf,3,"authentication",2025-06-27 10:05:42 UTC,7/191,0,LOG,00000,"connection authorized: user=postgres database=postgres SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)",,,,,,,,,"","client backend",,0
2025-06-27 10:05:42.831 UTC,"postgres","postgres",262,"127.0.0.1:36022",685e6cba.106,4,"idle",2025-06-27 10:04:42 UTC,,0,LOG,00000,"disconnection: session time: 0:01:00.075 user=postgres database=postgres host=127.0.0.1 port=36022",,,,,,,,,"","client backend",,0
2025-06-27 10:05:44.217 UTC,,,748,,685e6cf8.2ec,1,,2025-06-27 10:05:44 UTC,,0,FATAL,08006,"could not connect to the primary server: connection to server at ""10.233.73.203"", port 5432 failed: received invalid response to SSL negotiation: @",,,,,,,,,"","walreceiver",,0
2025-06-27 10:05:44.353 UTC,,,69,,685e6c37.45,18,,2025-06-27 10:02:31 UTC,1/0,0,LOG,00000,"waiting for WAL to become available at 0/E000018",,,,,,,,,"","startup",,0
2025-06-27 10:05:45.735 UTC,"postgres","postgres",261,"10.233.73.203:55842",685e6cb9.105,5,"idle",2025-06-27 10:04:41 UTC,,0,LOG,00000,"disconnection: session time: 0:01:04.600 user=postgres database=postgres host=10.233.73.203 port=55842",,,,,,,,,"","client backend",,0
2025-06-27 10:05:46.337 UTC,,,770,"10.233.72.107:59940",685e6cfa.302,1,"",2025-06-27 10:05:46 UTC,,0,LOG,00000,"connection received: host=10.233.72.107 port=59940",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:46.341 UTC,"postgres","postgres",770,"10.233.72.107:59940",685e6cfa.302,2,"authentication",2025-06-27 10:05:46 UTC,3/269,0,LOG,00000,"connection authenticated: identity=""postgres"" method=md5 (/home/postgres/pgdata/pgroot/data/pg_hba.conf:12)",,,,,,,,,"","client backend",,6633926927328956866
2025-06-27 10:05:46.341 UTC,"postgres","postgres",770,"10.233.72.107:59940",685e6cfa.302,3,"authentication",2025-06-27 10:05:46 UTC,3/269,0,LOG,00000,"connection authorized: user=postgres database=postgres SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)",,,,,,,,,"","client backend",,6633926927328956866
2025-06-27 10:05:46.355 UTC,"postgres","postgres",770,"10.233.72.107:59940",685e6cfa.302,4,"idle",2025-06-27 10:05:46 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.018 user=postgres database=postgres host=10.233.72.107 port=59940",,,,,,,,,"","client backend",,0
2025-06-27 10:05:46.700 UTC,,,772,"[local]",685e6cfa.304,1,"",2025-06-27 10:05:46 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2025-06-27 10:05:46.701 UTC,"postgres","postgres",772,"[local]",685e6cfa.304,2,"authentication",2025-06-27 10:05:46 UTC,3/289,0,LOG,00000,"connection authorized: user=postgres database=postgres application_name=Patroni",,,,,,,,,"","client backend",,-10637612914751176
2025-06-27 10:05:46.704 UTC,"postgres","postgres",772,"[local]",685e6cfa.304,3,"idle",2025-06-27 10:05:46 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.003 user=postgres database=postgres host=[local]",,,,,,,,,"Patroni","client backend",,0
2025-06-27 10:05:46.704 UTC,,,64,,685e6c37.40,8,,2025-06-27 10:02:31 UTC,,0,LOG,00000,"received fast shutdown request",,,,,,,,,"","postmaster",,0
2025-06-27 10:05:46.705 UTC,,,64,,685e6c37.40,9,,2025-06-27 10:02:31 UTC,,0,LOG,00000,"aborting any active transactions",,,,,,,,,"","postmaster",,0
2025-06-27 10:05:46.705 UTC,"postgres","postgres",719,"127.0.0.1:41884",685e6cf6.2cf,4,"idle in transaction",2025-06-27 10:05:42 UTC,7/192,0,FATAL,57P01,"terminating connection due to administrator command",,,,,,,,,"","client backend",,-7707998629541637718
2025-06-27 10:05:46.705 UTC,"postgres","postgres",190,"[local]",685e6c41.be,4,"idle",2025-06-27 10:02:41 UTC,4/0,0,FATAL,57P01,"terminating connection due to administrator command",,,,,,,,,"Patroni heartbeat","client backend",,-266259238054755560
2025-06-27 10:05:46.705 UTC,"postgres","postgres",191,"[local]",685e6c41.bf,5,"idle",2025-06-27 10:02:41 UTC,8/0,0,FATAL,57P01,"terminating connection due to administrator command",,,,,,,,,"Patroni restapi","client backend",,-9107815758628649684
2025-06-27 10:05:46.705 UTC,"postgres","postgres",183,"10.233.72.107:56568",685e6c3e.b7,6,"idle",2025-06-27 10:02:38 UTC,6/0,0,FATAL,57P01,"terminating connection due to administrator command",,,,,,,,,"","client backend",,-2318070291148541119
2025-06-27 10:05:46.705 UTC,"postgres","postgres",191,"[local]",685e6c41.bf,6,"idle",2025-06-27 10:02:41 UTC,,0,LOG,00000,"disconnection: session time: 0:03:05.419 user=postgres database=postgres host=[local]",,,,,,,,,"Patroni restapi","client backend",,0
2025-06-27 10:05:46.705 UTC,"postgres","postgres",190,"[local]",685e6c41.be,5,"idle",2025-06-27 10:02:41 UTC,,0,LOG,00000,"disconnection: session time: 0:03:05.422 user=postgres database=postgres host=[local]",,,,,,,,,"Patroni heartbeat","client backend",,0
2025-06-27 10:05:46.705 UTC,"postgres","postgres",183,"10.233.72.107:56568",685e6c3e.b7,7,"idle",2025-06-27 10:02:38 UTC,,0,LOG,00000,"disconnection: session time: 0:03:08.382 user=postgres database=postgres host=10.233.72.107 port=56568",,,,,,,,,"","client backend",,0
2025-06-27 10:05:46.705 UTC,"postgres","postgres",719,"127.0.0.1:41884",685e6cf6.2cf,5,"idle in transaction",2025-06-27 10:05:42 UTC,,0,LOG,00000,"disconnection: session time: 0:00:03.878 user=postgres database=postgres host=127.0.0.1 port=41884",,,,,,,,,"","client backend",,0
2025-06-27 10:05:46.710 UTC,,,64,,685e6c37.40,10,,2025-06-27 10:02:31 UTC,,0,LOG,00000,"background worker ""bg_mon"" (PID 70) exited with exit code 1",,,,,,,,,"","postmaster",,0
2025-06-27 10:05:46.710 UTC,,,67,,685e6c37.43,4,,2025-06-27 10:02:31 UTC,,0,LOG,00000,"shutting down",,,,,,,,,"","checkpointer",,0
2025-06-27 10:05:46.715 UTC,,,64,,685e6c37.40,11,,2025-06-27 10:02:31 UTC,,0,LOG,00000,"pg_auth_mon entry for user oid : 16720 rolename_at_last_login_attempt: standby total_successful_attempts: 12; last_successful_login_at: 2025-06-27 10:05:42.719676+00; last_failed_attempt_at: 0; total_hba_conflicts: 0; other_auth_failures: 0",,,,,,,,,"","postmaster",,0
2025-06-27 10:05:46.715 UTC,,,64,,685e6c37.40,12,,2025-06-27 10:02:31 UTC,,0,LOG,00000,"pg_auth_mon entry for user oid : 10 rolename_at_last_login_attempt: postgres total_successful_attempts: 39; last_successful_login_at: 2025-06-27 10:05:46.701588+00; last_failed_attempt_at: 0; total_hba_conflicts: 0; other_auth_failures: 0",,,,,,,,,"","postmaster",,0
2025-06-27 10:05:46.719 UTC,,,64,,685e6c37.40,13,,2025-06-27 10:02:31 UTC,,0,LOG,00000,"database system is shut down",,,,,,,,,"","postmaster",,0
2025-06-27 10:06:00.534 UTC,,,804,,685e6d07.324,1,,2025-06-27 10:05:59 UTC,,0,LOG,00000,"ending log output to stderr",,"Future log output will go to log destination ""csvlog"".",,,,,,,"","postmaster",,0
2025-06-27 10:06:00.534 UTC,,,804,,685e6d07.324,2,,2025-06-27 10:05:59 UTC,,0,LOG,00000,"starting PostgreSQL 16.6 (Ubuntu 16.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit",,,,,,,,,"","postmaster",,0
2025-06-27 10:06:00.534 UTC,,,804,,685e6d07.324,3,,2025-06-27 10:05:59 UTC,,0,LOG,00000,"listening on IPv4 address ""0.0.0.0"", port 5432",,,,,,,,,"","postmaster",,0
2025-06-27 10:06:00.534 UTC,,,804,,685e6d07.324,4,,2025-06-27 10:05:59 UTC,,0,LOG,00000,"listening on IPv6 address ""::"", port 5432",,,,,,,,,"","postmaster",,0
2025-06-27 10:06:00.539 UTC,,,804,,685e6d07.324,5,,2025-06-27 10:05:59 UTC,,0,LOG,00000,"listening on Unix socket ""/var/run/postgresql/.s.PGSQL.5432""",,,,,,,,,"","postmaster",,0
2025-06-27 10:06:00.545 UTC,,,809,,685e6d08.329,1,,2025-06-27 10:06:00 UTC,,0,LOG,00000,"database system was shut down at 2025-06-27 10:05:45 UTC",,,,,,,,,"","startup",,0
2025-06-27 10:06:00.622 UTC,,,809,,685e6d08.329,2,,2025-06-27 10:06:00 UTC,,0,LOG,00000,"restored log file ""00000002.history"" from archive",,,,,,,,,"","startup",,0
2025-06-27 10:06:00.730 UTC,,,809,,685e6d08.329,3,,2025-06-27 10:06:00 UTC,,0,LOG,00000,"entering standby mode",,,,,,,,,"","startup",,0
2025-06-27 10:06:00.800 UTC,,,809,,685e6d08.329,4,,2025-06-27 10:06:00 UTC,,0,LOG,00000,"restored log file ""00000002.history"" from archive",,,,,,,,,"","startup",,0
2025-06-27 10:06:00.970 UTC,,,909,"[local]",685e6d08.38d,1,"",2025-06-27 10:06:00 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2025-06-27 10:06:00.970 UTC,"postgres","postgres",909,"[local]",685e6d08.38d,2,"",2025-06-27 10:06:00 UTC,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2025-06-27 10:06:00.977 UTC,,,911,"[local]",685e6d08.38f,1,"",2025-06-27 10:06:00 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2025-06-27 10:06:00.977 UTC,"postgres","postgres",911,"[local]",685e6d08.38f,2,"",2025-06-27 10:06:00 UTC,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2025-06-27 10:06:01.020 UTC,,,809,,685e6d08.329,5,,2025-06-27 10:06:00 UTC,,0,FATAL,XX000,"requested timeline 2 is not a child of this server's history","Latest checkpoint is at 0/10000348 on timeline 1, but in the history of the requested timeline, the server forked off from that timeline at 0/80000A0.",,,,,,,,"","startup",,0
2025-06-27 10:06:01.023 UTC,,,804,,685e6d07.324,6,,2025-06-27 10:05:59 UTC,,0,LOG,00000,"startup process (PID 809) exited with exit code 1",,,,,,,,,"","postmaster",,0
2025-06-27 10:06:01.023 UTC,,,804,,685e6d07.324,7,,2025-06-27 10:05:59 UTC,,0,LOG,00000,"aborting startup due to startup process failure",,,,,,,,,"","postmaster",,0
2025-06-27 10:06:01.023 UTC,,,804,,685e6d07.324,8,,2025-06-27 10:05:59 UTC,,0,LOG,00000,"database system is shut down",,,,,,,,,"","postmaster",,0
2025-06-27 10:06:19.813 UTC,,,932,,685e6d1b.3a4,1,,2025-06-27 10:06:19 UTC,,0,LOG,00000,"ending log output to stderr",,"Future log output will go to log destination ""csvlog"".",,,,,,,"","postmaster",,0
2025-06-27 10:06:19.813 UTC,,,932,,685e6d1b.3a4,2,,2025-06-27 10:06:19 UTC,,0,LOG,00000,"starting PostgreSQL 16.6 (Ubuntu 16.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit",,,,,,,,,"","postmaster",,0
2025-06-27 10:06:19.813 UTC,,,932,,685e6d1b.3a4,3,,2025-06-27 10:06:19 UTC,,0,LOG,00000,"listening on IPv4 address ""0.0.0.0"", port 5432",,,,,,,,,"","postmaster",,0
Stdout logs of the replica's pod:
2025-06-27 10:04:28,072 INFO: no action. I am (postgresql--pg-0), a secondary, and following a leader (postgresql--pg-1)
2025-06-27 10:04:38,076 INFO: no action. I am (postgresql--pg-0), a secondary, and following a leader (postgresql--pg-1)
2025-06-27 10:04:48,019 INFO: no action. I am (postgresql--pg-0), a secondary, and following a leader (postgresql--pg-1)
2025-06-27 10:04:52,262 INFO: PAUSE: no action. I am (postgresql--pg-0), a secondary, and following a leader (postgresql--pg-1)
2025-06-27 10:04:52,265 INFO: Changed tcp_keepalives_idle from '0' to '900'
2025-06-27 10:04:52,265 INFO: Changed tcp_keepalives_interval from '0' to '100'
2025-06-27 10:04:52,265 INFO: Changed wal_compression from 'pglz' to 'on'
2025-06-27 10:04:52,266 INFO: Reloading PostgreSQL configuration.
server signaled
2025-06-27 10:04:53,335 INFO: PAUSE: no action. I am (postgresql--pg-0), a secondary, and following a leader (postgresql--pg-1)
2025-06-27 10:05:02,440 INFO: PAUSE: no action. I am (postgresql--pg-0), a secondary, and following a leader (postgresql--pg-1)
2025-06-27 10:05:12,802 INFO: PAUSE: no action. I am (postgresql--pg-0)
2025-06-27 10:05:22,723 INFO: PAUSE: no action. I am (postgresql--pg-0)
2025-06-27 10:05:32,720 INFO: PAUSE: no action. I am (postgresql--pg-0)
2025-06-27 10:05:42,720 INFO: PAUSE: no action. I am (postgresql--pg-0)
2025-06-27 10:05:49,079 INFO: establishing a new patroni restapi connection to postgres
2025-06-27 10:05:50,009 INFO: establishing a new patroni heartbeat connection to postgres
2025-06-27 10:05:50,010 WARNING: Retry got exception: connection problems
2025-06-27 10:05:50,010 ERROR: get_postgresql_status
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/patroni/api.py", line 1425, in query
heartbeat_connection.get() # try to open psycopg connection to postgres
File "/usr/local/lib/python3.10/dist-packages/patroni/postgresql/connection.py", line 54, in get
self._connection = psycopg.connect(**self._conn_kwargs)
File "/usr/local/lib/python3.10/dist-packages/patroni/psycopg.py", line 120, in connect
ret = _connect(*args, **kwargs)
File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/patroni/utils.py", line 763, in __call__
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/patroni/api.py", line 1427, in query
raise PostgresConnectionException('connection problems') from exc
patroni.exceptions.PostgresConnectionException: connection problems
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/patroni/api.py", line 1302, in get_postgresql_status
row = self.query(stmt.format(postgresql.wal_name, postgresql.lsn_name,
File "/usr/local/lib/python3.10/dist-packages/patroni/api.py", line 1236, in query
return Retry(delay=1, retry_exceptions=PostgresConnectionException)(self.server.query, sql, *params)
File "/usr/local/lib/python3.10/dist-packages/patroni/utils.py", line 768, in __call__
raise RetryFailedError("Too many retry attempts")
patroni.utils.RetryFailedError: Too many retry attempts
2025-06-27 10:05:50,501 INFO: establishing a new patroni heartbeat connection to postgres
2025-06-27 10:05:50,501 WARNING: Retry got exception: connection problems
2025-06-27 10:05:50,501 ERROR: get_postgresql_status
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/patroni/api.py", line 1425, in query
heartbeat_connection.get() # try to open psycopg connection to postgres
File "/usr/local/lib/python3.10/dist-packages/patroni/postgresql/connection.py", line 54, in get
self._connection = psycopg.connect(**self._conn_kwargs)
File "/usr/local/lib/python3.10/dist-packages/patroni/psycopg.py", line 120, in connect
ret = _connect(*args, **kwargs)
File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
The above exception was the direct cause of the following exception:
2025-06-27 10:04:28,072 INFO: no action. I am (postgresql--pg-0), a secondary, and following a leader (postgresql--pg-1)
2025-06-27 10:04:38,076 INFO: no action. I am (postgresql--pg-0), a secondary, and following a leader (postgresql--pg-1)
2025-06-27 10:04:48,019 INFO: no action. I am (postgresql--pg-0), a secondary, and following a leader (postgresql--pg-1)
2025-06-27 10:04:52,262 INFO: PAUSE: no action. I am (postgresql--pg-0), a secondary, and following a leader (postgresql--pg-1)
2025-06-27 10:04:52,265 INFO: Changed tcp_keepalives_idle from '0' to '900'
2025-06-27 10:04:52,265 INFO: Changed tcp_keepalives_interval from '0' to '100'
2025-06-27 10:04:52,265 INFO: Changed wal_compression from 'pglz' to 'on'
2025-06-27 10:04:52,266 INFO: Reloading PostgreSQL configuration.
server signaled
2025-06-27 10:04:53,335 INFO: PAUSE: no action. I am (postgresql--pg-0), a secondary, and following a leader (postgresql--pg-1)
2025-06-27 10:05:02,440 INFO: PAUSE: no action. I am (postgresql--pg-0), a secondary, and following a leader (postgresql--pg-1)
2025-06-27 10:05:12,802 INFO: PAUSE: no action. I am (postgresql--pg-0)
2025-06-27 10:05:22,723 INFO: PAUSE: no action. I am (postgresql--pg-0)
2025-06-27 10:05:32,720 INFO: PAUSE: no action. I am (postgresql--pg-0)
2025-06-27 10:05:42,720 INFO: PAUSE: no action. I am (postgresql--pg-0)
2025-06-27 10:05:49,079 INFO: establishing a new patroni restapi connection to postgres
2025-06-27 10:05:50,009 INFO: establishing a new patroni heartbeat connection to postgres
2025-06-27 10:05:50,010 WARNING: Retry got exception: connection problems
2025-06-27 10:05:50,010 ERROR: get_postgresql_status
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/patroni/api.py", line 1425, in query
heartbeat_connection.get() # try to open psycopg connection to postgres
File "/usr/local/lib/python3.10/dist-packages/patroni/postgresql/connection.py", line 54, in get
self._connection = psycopg.connect(**self._conn_kwargs)
File "/usr/local/lib/python3.10/dist-packages/patroni/psycopg.py", line 120, in connect
ret = _connect(*args, **kwargs)
File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/patroni/utils.py", line 763, in __call__
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/patroni/api.py", line 1427, in query
raise PostgresConnectionException('connection problems') from exc
patroni.exceptions.PostgresConnectionException: connection problems
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/patroni/api.py", line 1302, in get_postgresql_status
row = self.query(stmt.format(postgresql.wal_name, postgresql.lsn_name,
File "/usr/local/lib/python3.10/dist-packages/patroni/api.py", line 1236, in query
return Retry(delay=1, retry_exceptions=PostgresConnectionException)(self.server.query, sql, *params)
File "/usr/local/lib/python3.10/dist-packages/patroni/utils.py", line 768, in __call__
raise RetryFailedError("Too many retry attempts")
patroni.utils.RetryFailedError: Too many retry attempts
2025-06-27 10:05:50,501 INFO: establishing a new patroni heartbeat connection to postgres
2025-06-27 10:05:50,501 WARNING: Retry got exception: connection problems
2025-06-27 10:05:50,501 ERROR: get_postgresql_status
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/patroni/api.py", line 1425, in query
heartbeat_connection.get() # try to open psycopg connection to postgres
File "/usr/local/lib/python3.10/dist-packages/patroni/postgresql/connection.py", line 54, in get
self._connection = psycopg.connect(**self._conn_kwargs)
File "/usr/local/lib/python3.10/dist-packages/patroni/psycopg.py", line 120, in connect
ret = _connect(*args, **kwargs)
File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
The things that stands out in the logs is this:
2025-06-27 13:03:04.920 UTC,,,86513,,685e9684.151f1,6,,2025-06-27 13:03:00 UTC,,0,FATAL,XX000,"requested timeline 2 is not a child of this server's history","Latest checkpoint is at 0/10000348 on timeline 1, but in the history of the requested timeline, the server forked off from that timeline at 0/80000A0.",,,,,,,,"","startup",,0
And indeed the timeline is different:
postgres@postgresql--pg-1:~$ patronictl list
+ Cluster: postgresql--pg (7520567970673238895) ------------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+------------------+---------------+---------+--------------+----+-----------+
| postgresql--pg-0 | 10.233.72.107 | Replica | start failed | | unknown |
| postgresql--pg-1 | 10.233.73.203 | Leader | running | 1 | |
+------------------+---------------+---------+--------------+----+-----------+
Versions details:
- running with operator version: ghcr.io/zalando/postgres-operator:v1.14.0
- spilo version: ghcr.io/zalando/spilo-16:3.3-p3
I found a workaround, which is not really satisfying: I clear the object store of all the backup data, create a manual backup and reinit the replica, and it runs again smoothly.
(it appears, I'm the first one having this issue, not sure what I'm doing wrong :) )