.env file support with retry mechanism #907
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With this change, pgcopydb will read the connection strings from .env file on retrying connections. This allows users to change the connection strings in .env file while pgcopydb is running. This particular change is useful for users who are using managed identities or other similar solutions, where the connection strings might change during the execution of pgcopydb. On such systems, the passwords are not stored in the environment variables, but managed in the .env file. -------------------- Changes Made * Added `--connection-retry-timeout` parameter to stop retrying after N seconds (60 default) * Added .env file support that will be fetched from ($XDG_CONFIG_HOME OR $HOME/.config)/pgcopydb/.env * following parameters will be read from .env file on init, overriding environment variables, but have lower priority than cli parameters: PGCOPYDB_SOURCE_PGURI PGCOPYDB_TARGET_PGURI PGCOPYDB_TABLE_JOBS PGCOPYDB_INDEX_JOBS PGCOPYDB_RESTORE_JOBS PGCOPYDB_LARGE_OBJECTS_JOBS PGCOPYDB_SPLIT_MAX_PARTS PGCOPYDB_ESTIMATE_TABLE_SIZES PGCOPYDB_SNAPSHOT PGCOPYDB_WAL2JSON_NUMERIC_AS_STRING PGCOPYDB_DROP_IF_EXISTS PGCOPYDB_FAIL_FAST PGCOPYDB_SKIP_VACUUM * Added mechanism to re-read .env file and retry with new connection string in pgsql connections. -------------------- Description * param > .env file > environment variable on init, .env file on retry * ($XDG_CONFIG_HOME OR $HOME/.config)/pgcopydb/.env * Retry only will fetch connection strings (PGCOPYDB_SOURCE_PGURI, PGCOPYDB_TARGET_PGURI) from .env file -------------------- Testing * I have a url with a wrong password in my env variable, but my .env file have the correct one. The clone works without problem. * I run the clone with timeout variable set to 5, with wrong url in both env file and env variable, after 5 seconds of trying to connect, the process stops retrying. * I have a url with a wrong password in my env variable, my .env file also have the wrong password. I run the clone, while it's retrying, I edit the .env file with correct password. In the next try, the clone continues to work as expected.
dimitri
reviewed
Feb 23, 2025
There was a problem hiding this comment.
Thank for your contribution Hanefi!
I left a comment which might be wrong, but on a quick read I was confused. On a higher design note, why using an env file (which looks like something from Docker or other software inspiration) rather than a service file, like Postgres is doing, see https://www.postgresql.org/docs/current/libpq-pgservice.html
We could have our own .ini format (we already have a parser for that format) and parse all our configuration from there, including at connection retry. The filename would probably be pgcopydb.conf rather than .env, too.
Comment on lines
+218
to
+229
| if (env_exists("XDG_CONFIG_HOME")) | ||
| { | ||
| char *configHome; | ||
| get_env_dup("XDG_CONFIG_HOME", &configHome); | ||
| sformat(envFilePath, sizeof(envFilePath), "%s/pgcopydb/.env", configHome); | ||
| } | ||
| else if (env_exists("HOME")) | ||
| { | ||
| char *homeDir; | ||
| get_env_dup("HOME", &homeDir); | ||
| sformat(envFilePath, sizeof(envFilePath), "%s/.config/pgcopydb/.env", homeDir); | ||
| } |
There was a problem hiding this comment.
If we have a XDG_CONFIG_HOME environment variable defined, then we never look at te HOME path, even if the file happens to only exists there. I don't think that's the intended behavior?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With this change, pgcopydb will read the connection strings from .env file on retrying connections. This allows users to change the connection strings in .env file while pgcopydb is running.
This particular change is useful for users who are using managed identities or other similar solutions, where the connection strings might change during the execution of pgcopydb. On such systems, the passwords are not stored in the environment variables, but managed in the .env file.
Changes Made
Added
--connection-retry-timeoutparameter to stop retrying after N seconds (60 default)Added .env file support that will be fetched from ($XDG_CONFIG_HOME OR $HOME/.config)/pgcopydb/.env
following parameters will be read from .env file on init, overriding environment variables, but have lower priority than cli parameters: PGCOPYDB_SOURCE_PGURI PGCOPYDB_TARGET_PGURI PGCOPYDB_TABLE_JOBS PGCOPYDB_INDEX_JOBS PGCOPYDB_RESTORE_JOBS PGCOPYDB_LARGE_OBJECTS_JOBS PGCOPYDB_SPLIT_MAX_PARTS PGCOPYDB_ESTIMATE_TABLE_SIZES PGCOPYDB_SNAPSHOT PGCOPYDB_WAL2JSON_NUMERIC_AS_STRING PGCOPYDB_DROP_IF_EXISTS PGCOPYDB_FAIL_FAST PGCOPYDB_SKIP_VACUUM
Added mechanism to re-read .env file and retry with new connection string in pgsql connections.
Description
param > .env file > environment variable on init, .env file on retry
($XDG_CONFIG_HOME OR $HOME/.config)/pgcopydb/.env
Retry only will fetch connection strings (PGCOPYDB_SOURCE_PGURI, PGCOPYDB_TARGET_PGURI) from .env file
Testing
I have a url with a wrong password in my env variable, but my .env file have the correct one. The clone works without problem.
I run the clone with timeout variable set to 5, with wrong url in both env file and env variable, after 5 seconds of trying to connect, the process stops retrying.
I have a url with a wrong password in my env variable, my .env file also have the wrong password. I run the clone, while it's retrying, I edit the .env file with correct password. In the next try, the clone continues to work as expected.