aravneel update lcs - openzfs mount and user add permissions

[aravneelaws]

Issue: #896 #897

Description of changes:
This PR addresses the following issues:

1. OpenZFS mounting issue in compute nodes
2. .ssh permissions issue
3. Adding users during cluster creation with correct permissions
4. easy-ssh script updated with -u flag for logging as a specific user (non-ubuntu user)
5. automate-cluster-creation script print statement updated for SSM preference if openZFS /home directory exists.

The changes has been tested on cluster with openZFS for /home and without openZFS (with only fsxL).

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[PremiumSpider]

Is this a potential silent error due to (...|| true)
Do we need to retry ? can we at least log if there is issue ?

Example

echo "[STEP] Triggering automount..."
if ! ls -la "$OPENZFS_MOUNT_POINT" >/dev/null 2>&1; then
    echo "[WARN] Could not list mount directory contents"
    # log the issue
fi

[aravneelaws]

@PremiumSpider This does qualify as a silent error due to (..|| true). But we don't need to log anything here because this is just triggering the automount. The ls command is trying to access the space which will mount the FS. We have explicit verification in the next step with touch and delete. I would leave as it is in my opinion.

[PremiumSpider]

Suppose we have 2 nodes, if they are all creating a global static test_file, do we need to worry about race conditions ?

T1: Instance A creates test_file
T2: Instance B creates test_file (same name)
T3: Instance A deletes test_file  
T4: Instance B tries to delete test_file → FAILS (already gone)
T5: Instance B thinks mount is broken, retries entire process

Would something like below help ?

TEST_FILE="$OPENZFS_MOUNT_POINT/test_file_${HOSTNAME}_$$"

[aravneelaws]

That is a good catch. I can implement this here. Thanks for catching this.

[aravneelaws]

@PremiumSpider This has been implemented now.

[aravneelaws]

Tested this as well and looks good.

[PremiumSpider]

Thank you for testing on your end as well.

[PremiumSpider]

Would it help if started adding Header comments such as

# SSH Key Setup Script for SageMaker HyperPod
# 
# Purpose: 
#   - Configures SSH keys for passwordless communication between cluster nodes
#   - Supports multiple users via shared_users.txt CSV file (username,uid,home)
#   - Creates symlinks between FSx Lustre and OpenZFS storage for unified SSH access
#
# Storage Strategy:
#   - Primary SSH keys stored on FSx Lustre (/fsx/username/.ssh)
#   - OpenZFS home directories (/home/username/.ssh) symlinked to FSx Lustre
#   - Ensures SSH keys persist and are accessible from both storage systems
#
# Multi-User Support:
#   - Always configures default 'ubuntu' user first
#   - Processes additional users from ../shared_users.txt if present
#   - Validates user existence before SSH setup
#   - Generates RSA 4096-bit keypairs with proper permissions (600/644/700)
#
# Cluster Behavior:
#   - Safe to run on multiple nodes simultaneously (|| true for key generation)
#   - Reuses existing keys if present, generates new ones if missing
#   - Adds public keys to authorized_keys for passwordless SSH
#

set -exuo pipefail

etc

[aravneelaws]

Agreed. But I am going to keep this for a separate PR to improve code documentation

[PremiumSpider]

Hi I might be missing context, but is this change mainly for when users log into HOME in any of the nodes ?

[aravneelaws]

Correct. When the cluster is deployed only with fsx lustre, there won't be a /home dir and the HOME_DIR is set to /fsx/USER. When fsx for OpenZFS is used as /home dir, we have to explicitly call out this so the users can set the correct preference in SSM to successfully login as a USER.

[aravneelaws]

Common tests that can be added to pipeline include:

1. Checking if filesystem mounting was successful - /fsx/USER and /home/USER (for openZFS) - on all nodes (login/controller/compute).
2. Checking the if .ssh directory exists with correct user permissions.
3. Checking if key-pair and authorized_keys exists within .ssh directory.
4. Checking all these functionality work for all USERs (in addition to the default ubuntu user).

These are the main tests that needs to be performed for the above PR. I have tested these on my end. These can be added to your LCS test pipeline as well.

[PremiumSpider]

Hyperpod team mainly concerned with changes under LifecycleScripts but I've gone forward and reviewed other changes as well. Don't see any high risk changes.