Add support for remote cred manager with Globus Compute backend

[GKNB]

Add a remote credential manager that synchronizes freshly renewed IAM tokens to a remote filesystem path using Globus Compute.

• New cred manager: IamTokenCredManagerRemoteGlobusCompute
  • Extends the existing IamTokenCredManager
  • When a token is renewed locally, it additionally attempts remote synchronization (if configured)
• New helper: GlobusTokenReplicator
  • Encrypts the token locally with Fernet (symmetric authenticated encryption)
  • Submits a Globus Compute task to decrypt and atomically write the token on the endpoint

+-------------------+ +-----------------------------+
| Harvester (local) | | Remote site (GC endpoint) |
+-------------------+ +-----------------------------+
|
| IssuerBroker.get_access_token
v
[ plaintext token in memory ]
|
| write token locally
v
out_dir/<token_filename>
|
| Fernet.encrypt
v
[ ciphertext token (string) ]
|
| Executor.submit(_remote_write_token, ciphertext, remote_path, remote_key_file)
v
+-------------------+ +-----------------------------+
| GC task queued | -------> | _remote_write_token() |
+-------------------+ | - read remote_key_file |
| - Fernet.decrypt() |
| - write tmp (0600) |
| - os.replace(tmp, final) |
+-----------------------------+
|
v
remote_out_dir/<token_filename>

Notes:

• Remote write is atomic (tmp + os.replace)
• Feature is opt-in via config; default behavior is unchanged

Below are example of how to use this in the panda_harvester.cfg: In pluginConfigs, add/modify the following
{
"module": "pandaharvester.harvestercredmanager.iam_token_cred_manager_globus_compute",
"name": "IamTokenCredManagerRemoteGlobusCompute",
"configs": {
"pilot-pandaserver-token-remote": {
"client_cred_file": "...",
"target_type": "panda",
"panda_token_filename": "panda_token",
"out_dir": "...",
"check_interval": 1800,
"refresh_interval": 3500,
"remote_out_dir": "...",
"globus_compute": {
"endpoint_id": "...",
"local_key_file": "...",
"remote_key_file": "...",
"task_timeout": 60
}
}
}
}

Successful tests:

• On Perlmutter, tested with ATLAS workflow with a local execution
• On Perlmutter and BNL, tested with a test job where a harvester is installed at BNL and job is executed at Perlmutter

Planned follow-ups:

• Merge into local cred manager
• Upgrade this into a module to support multiple backend (SFAPI, S3M) and multiple cred manager

[dougbenjamin]

@tmaeno any ETA on approving this PR.

[mightqxc]

@GKNB @dougbenjamin
Could you please put globus_token_sync.py under pandaharvester/harvestermisc and put
iam_token_cred_manager_globus_compute.py under pandaharvester/harvestercredmanager ?

[GKNB]

@GKNB @dougbenjamin Could you please put globus_token_sync.py under pandaharvester/harvestermisc and put iam_token_cred_manager_globus_compute.py under pandaharvester/harvestercredmanager ?

@mightqxc Sorry I made a mistake when I generate my repo for submitting PR. Now those files are put in correct position, and I also change the name for all class/filename that looks like globus_xxx to globus_compute_xxx to distinguish them from globus toolkit