Enable persistent Ray cluster state via external Redis GCS fault tolerance

[kramaranya]

Issue link

RHOAIENG-11115

What changes have been made

Provided Ray cluster head pod persistency through GCS fault tolerance.
Added new config options: enable_gcs_ft, redis_address, redis_password_secret and external_storage_namespace
Added unit tests to cover gcs fault tolerance

Verification steps

• Provision a cluster
• Deploy Redis in your cluster ( you can use this example)
• Deploy KubeRay operator (1.3.0 or newer, RHOAI currently provides an older version)
• Create RayCluster with GCS fault tolerance enabled, for example:

cluster = Cluster(ClusterConfiguration(
        name='raycluster',
        head_cpu_requests='500m',
        head_cpu_limits='500m',
        head_memory_requests=2,
        head_memory_limits=2,
        num_workers=1,
        worker_cpu_requests='250m',
        worker_cpu_limits=1,
        worker_memory_requests=4,
        worker_memory_limits=4,
        enable_gcs_ft=True,
        redis_address="redis:6379",
        redis_password_secret={
            "name": "redis-password-secret",
            "key": "password"
        }
    ))

• Make sure all pods are running
• Create a detached actor
• Check if the actor is visible - ray list actors from RayCluster head pod
• Restart the head pod
• Run ray list actors - you should see previously created actor

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[kryanbeane]

great work! lgtm

[codecov]

Codecov Report

Attention: Patch coverage is 95.65217% with 1 line in your changes missing coverage. Please review.

Project coverage is 92.45%. Comparing base (c311665) to head (b552de7).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/codeflare_sdk/ray/cluster/build_ray_cluster.py	88.88%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #821      +/-   ##
==========================================
+ Coverage   92.40%   92.45%   +0.04%     
==========================================
  Files          23       23              
  Lines        1396     1418      +22     
==========================================
+ Hits         1290     1311      +21     
- Misses        106      107       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kryanbeane]

thanks for adding the tests!

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kryanbeane

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [kryanbeane]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment