Cli update

README.md

@@ -33,6 +33,7 @@ We use pre-commit to make sure the code is consistently formatted. To make sure
 - To run the unit tests, run `pytest -v tests/unit_test.py`
 - Any new test functions/scripts can be added into the `tests` folder
 - NOTE: Functional tests coming soon, will live in `tests/func_test.py`
+- To test CLI, run `codeflare` followed by any command. To see list of commands, simply run `codeflare`
 
 #### Code Coverage
 

pyproject.toml

@@ -29,6 +29,7 @@ codeflare-torchx = "0.6.0.dev0"
 cryptography = "40.0.2"
 executing = "1.2.0"
 pydantic = "< 2"
+click = "8.0.4"
 
 [tool.poetry.group.docs]
 optional = true
@@ -40,3 +41,10 @@ pdoc3 = "0.10.0"
 pytest = "7.4.0"
 coverage = "7.2.7"
 pytest-mock = "3.11.1"
+
+[tool.poetry.scripts]
+codeflare = "codeflare_sdk.cli.codeflare_cli:cli"
+
+[build-system]
+requires = ["poetry_core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"

requirements.txt

@@ -6,3 +6,4 @@ codeflare-torchx==0.6.0.dev0
 pydantic<2 # 2.0+ broke ray[default] see detail: https://github.com/ray-project/ray/pull/37000
 cryptography==40.0.2
 executing==1.2.0
+click==8.0.4

src/codeflare_sdk.egg-info/SOURCES.txt

@@ -19,3 +19,7 @@ src/codeflare_sdk/utils/generate_cert.py
 src/codeflare_sdk/utils/generate_yaml.py
 src/codeflare_sdk/utils/kube_api_helpers.py
 src/codeflare_sdk/utils/pretty_print.py
+src/codeflare_sdk/cli/__init__.py
+src/codeflare_sdk/cli/codeflare_cli.py
+src/codeflare_sdk/cli/commands/create.py
+src/codeflare_sdk/cli/cli_utils.py

src/codeflare_sdk/cli/CodeflareCLI_Design_Doc.md

@@ -0,0 +1,179 @@
+# CodeFlare CLI Design
+
+
+## Context and Scope
+
+
+The primary purpose of the CLI is to serve as an interaction layer between a user and the CodeFlare stack (MCAD, InstaScale, KubeRay) from within the terminal. This addition is required due to the fact that a large set of our target users come from a high-performance computing background and are most familiar and comfortable submitting jobs to a cluster via a CLI.
+
+
+The CLI will utilize the existing CodeFlare SDK. It will allow for similar operations that the SDK provides (such as Ray Cluster and job management) but in the terminal. The CLI adds some additional functions, allows for saved time, simpler workspaces, and automation of certain processes via bash scripts on top of the existing SDK.
+
+
+
+
+## Goals
+
+
+- Provide users the ability to request, monitor and stop the Kubernetes resources associated with the CodeFlare stack within the terminal.
+- Serve as an interaction layer between the data scientist and CodeFlare stack  (MCAD, InstaScale, KubeRay)
+- Allow for a user-friendly workflow  within the terminal
+- Allow for automation and scripting of job/RayCluster management via bash scripts
+
+
+## Non-Goals
+
+
+- Do not want to re-make the functionality that is found in the existing CodeFlare SDK or any of the SDK’s clients for Ray, MCAD, or any other service
+
+
+## Architecture and Design
+
+
+The CodeFlare CLI is an extension to the  CodeFlare SDK package that allows a user to create, monitor, and shut down framework clusters (RayClusters for now) and distributed training jobs on an authenticated Kubernetes cluster from the terminal.
+
+
+The user should have the ability to do the following from within the terminal:
+- Create, view details, view status, submit, delete Ray Clusters via appwrappers
+- Create, view logs, view status, submit, delete jobs
+- List out all jobs
+- List out all ray clusters
+- Login to Kubernetes cluster
+- Logout of Kubernetes cluster
+
+
+To support these operations, additional functions to the SDK may include:
+- Formatted listing ray clusters
+- Formatted listing jobs
+- Getting a job given the name
+
+
+For the majority of functionality, the CLI will utilize the SDK’s already built functionality.
+
+
+### CLI Framework:
+
+
+[Click](https://click.palletsprojects.com/en/8.1.x/) is the chosen CLI framework for the following reasons
+- Simple syntax/layout: Since the CLI commands are very complex, it is important that the CLI framework doesn’t add any unnecessary complexity
+- Supports functional commands instead of objects: This is important because the SDK is designed with various functions, and the CLI being similar improves readability
+- Comes with testing and help generation: Testing library and automatic help generation quickens development process
+- Large community support/documentation: extensive documentation and large community leads to less errors and easier development.
+
+
+### Framework Clusters:
+
+
+When the user invokes the `define raycluster` command, a yaml file with default values is created and put in the user’s current working directory. Users can customize their clusters by adding parameters to the define command and these values will override the defaults when creating the AppWrapper yaml file.
+
+
+Once the appwrapper is defined, the user can create the ray cluster via a create command. When the user invokes the `create raycluster`, they will specify the name of the cluster to submit. The CLI will first check to see whether or not the specified name is already present in the Kubernetes cluster. If it isn’t already present, then it will search the current working directory for a yaml file corresponding to cluster name and apply it to the K8S cluster. If the wait flag is specified, then the CLI will display a loading sign with status updates until the cluster is up.
+
+
+We will try to find a good balance between exposing more parameters and simplifying the process by acting on feedback from CLI users.
+
+
+For `delete raycluster`, the user will invoke the command, and the CLI will shut it down and delete it.
+
+
+### Training Jobs
+
+
+When the user invokes `define job` command, a DDPJobDefiniton object will be created and saved into a file. Users can customize their jobs using parameters to the define command.
+
+
+Once the job is defined, the user can submit the job via a `job submit` command. When the user submits a job, the user will specify the job name. The CLI will then check to see if the job is already on the Kubernetes cluster and if not it will submit the job. The job submitted will be a DDPJob and it will be submitted onto a specified ray cluster.
+
+
+When the user wants to delete a job, they just invoke the job delete command, and the CLI will stop the job and delete it. This can happen at any time assuming there is a job running.
+
+
+### Authentication
+
+
+Users will need to be authenticated into a Kubernetes cluster in order to be able to perform all operations.
+
+
+If the user tries to perform any operation without being logged in, the CLI will prompt them to authenticate. A kubeconfig will have to be valid in the users environment in order to perform any operation.
+
+
+The user will be able to login using a simple `login` command and will have the choice of logging in via server + token. The user can also choose whether or not they want tls-verification. If there is a kubeconfig, the CLI will update it, else it will create one for the user.
+
+
+Alternatively, the user can invoke the login command with their kubeconfig file path, and this will login the user using their kubeconfig file.
+
+
+Users can logout of their cluster using the `logout` command.
+
+
+
+
+### Listing Info
+
+
+Users can list both ray cluster information and job information by invoking respective commands. CLI will list information for each raycluster/job such as requested resources, status, name, and namespace.
+
+
+## Alternatives Considered
+
+
+- Existing CodeFlare CLI
+   - Written in TypeScript and overcomplicated. Did not support
+- Just using SDK
+   - Making a CLI saves a lot of time and is easier for the user in some cases
+- Interactive CLI
+   - Interactive CLIs make it harder for automation via bash scripts
+- Other CLI libraries
+   - **Cliff:** Ugly syntax, less readability, not much functionality.
+   - **Argparse:** Less functionality out of the box. More time spent on unnecessary reimplementation.
+   - **Cement:** Ugly syntax and low community support.
+
+
+## Security Considerations
+
+
+We will rely on Kubernetes default security, where users can not perform any operations on a cluster if they are not authenticated correctly.
+
+
+## Testing and Validation
+The CLI is found within the SDK, so it will be [tested](https://github.com/project-codeflare/codeflare-sdk/blob/main/CodeFlareSDK_Design_Doc.md#testing-and-validation) the same way.
+
+
+## Deployment and Rollout
+- The CLI will be deployed within the CodeFlare SDK so similar [considerations](https://github.com/project-codeflare/codeflare-sdk/blob/main/CodeFlareSDK_Design_Doc.md#deployment-and-rollout) will be taken into account.
+
+
+## Command Usage Examples
+Create ray cluster
+- `codeflare create raycluster [options]`
+
+
+Doing something to a ray cluster:
+- `codeflare {operation} raycluster {cluster_name} [options e.g. --gpu=0]`
+
+
+Create job
+- `codeflare create job [options]`
+
+
+Doing something to a job:
+- `codeflare {operation} job {job_name} [options e.g. cluster-name=”mycluster”]`
+- Namespace and ray cluster name will be required as options
+
+
+Listing out clusters
+- `codeflare list raycluster -n {namespace} OR codeflare list ray-cluster –all`
+
+
+Listing out jobs
+- `codeflare list job -c {cluster_name} -n {namespace}`
+- `codeflare list job -n {namespace}`
+- `codeflare list job --all`
+
+
+Login to kubernetes cluster
+- `codeflare login [options e.g. --configpath={path/to/kubeconfig}]` (if configpath is left blank default value is used)
+
+
+Logout of kubernetes cluster
+- `codeflare logout`

src/codeflare_sdk/cli/cli_utils.py

@@ -0,0 +1,173 @@
+import ast
+import click
+from kubernetes import client, config
+import pickle
+import os
+from ray.job_submission import JobSubmissionClient
+from torchx.runner import get_runner
+from rich.table import Table
+from rich import print
+
+from codeflare_sdk.cluster.cluster import list_clusters_all_namespaces, get_cluster
+from codeflare_sdk.cluster.model import RayCluster
+from codeflare_sdk.cluster.auth import _create_api_client_config, config_check
+from codeflare_sdk.utils.kube_api_helpers import _kube_api_error_handling
+import codeflare_sdk.cluster.auth as sdk_auth
+
+
+class PythonLiteralOption(click.Option):
+    def type_cast_value(self, ctx, value):
+        try:
+            if not value:
+                return None
+            return ast.literal_eval(value)
+        except:
+            raise click.BadParameter(value)
+
+
+class AuthenticationConfig:
+    """
+    Authentication configuration that will be stored in a file once
+    the user logs in using `codeflare login`
+    """
+
+    def __init__(
+        self,
+        token: str,
+        server: str,
+        skip_tls: bool,
+        ca_cert_path: str,
+    ):
+        self.api_client_config = _create_api_client_config(
+            token, server, skip_tls, ca_cert_path
+        )
+        self.server = server
+        self.token = token
+
+    def create_client(self):
+        return client.ApiClient(self.api_client_config)
+
+
+def load_auth():
+    """
+    Loads AuthenticationConfiguration and stores it in global variables
+    which can be used by the SDK for authentication
+    """
+    try:
+        auth_file_path = os.path.expanduser("~/.codeflare/auth")
+        with open(auth_file_path, "rb") as file:
+            auth = pickle.load(file)
+            sdk_auth.api_client = auth.create_client()
+            return auth
+    except (IOError, EOFError):
+        click.echo("No authentication found, trying default kubeconfig")
+    except client.ApiException:
+        click.echo("Invalid authentication, trying default kubeconfig")
+
+
+class PluralAlias(click.Group):
+    def get_command(self, ctx, cmd_name):
+        rv = click.Group.get_command(self, ctx, cmd_name)
+        if rv is not None:
+            return rv
+        for x in self.list_commands(ctx):
+            if x + "s" == cmd_name:
+                return click.Group.get_command(self, ctx, x)
+        return None
+
+    def resolve_command(self, ctx, args):
+        # always return the full command name
+        _, cmd, args = super().resolve_command(ctx, args)
+        return cmd.name, cmd, args
+
+
+def print_jobs(jobs):
+    headers = ["Submission ID", "Job ID", "RayCluster", "Namespace", "Status"]
+    table = Table(show_header=True)
+    for header in headers:
+        table.add_column(header)
+    for job in jobs:
+        table.add_row(*[job[header] for header in headers])
+    print(table)
+
+
+def list_all_kubernetes_jobs(print_to_console=True):
+    k8s_jobs = []
+    runner = get_runner()
+    jobs = runner.list(scheduler="kubernetes_mcad")
+    rayclusters = {
+        raycluster.name for raycluster in list_clusters_all_namespaces(False)
+    }
+    for job in jobs:
+        namespace, name = job.app_id.split(":")
+        status = job.state
+        if name not in rayclusters:
+            k8s_jobs.append(
+                {
+                    "Submission ID": name,
+                    "Job ID": "N/A",
+                    "RayCluster": "N/A",
+                    "Namespace": namespace,
+                    "Status": str(status),
+                    "App Handle": job.app_handle,
+                }
+            )
+    if print_to_console:
+        print_jobs(k8s_jobs)
+    return k8s_jobs
+
+
+def list_all_jobs(print_to_console=True):
+    k8s_jobs = list_all_kubernetes_jobs(False)
+    rc_jobs = list_all_raycluster_jobs(False)
+    all_jobs = rc_jobs + k8s_jobs
+    if print_to_console:
+        print_jobs(all_jobs)
+    return all_jobs
+
+
+def list_raycluster_jobs(cluster: RayCluster, print_to_console=True):
+    rc_jobs = []
+    client = JobSubmissionClient(cluster.dashboard)
+    jobs = client.list_jobs()
+    for job in jobs:
+        job_obj = {
+            "Submission ID": job.submission_id,
+            "Job ID": job.job_id,
+            "RayCluster": cluster.name,
+            "Namespace": cluster.namespace,
+            "Status": str(job.status),
+            "App Handle": "ray://torchx/" + cluster.dashboard + "-" + job.submission_id,
+        }
+        rc_jobs.append(job_obj)
+    if print_to_console:
+        print_jobs(rc_jobs)
+    return rc_jobs
+
+
+def list_all_raycluster_jobs(print_to_console=True):
+    rc_jobs = []
+    clusters = list_clusters_all_namespaces(False)
+    for cluster in clusters:
+        cluster.dashboard = "http://" + cluster.dashboard
+        rc_jobs += list_raycluster_jobs(cluster, False)
+    if print_to_console:
+        print_jobs(rc_jobs)
+    return rc_jobs
+
+
+def get_job_app_handle(job_submission):
+    job = get_job_object(job_submission)
+    return job["App Handle"]
+
+
+def get_job_object(job_submission):
+    all_jobs = list_all_jobs(False)
+    for job in all_jobs:
+        if job["Submission ID"] == job_submission:
+            return job
+    raise (
+        FileNotFoundError(
+            f"Job {job_submission} not found. Try using 'codeflare list --all' to see all jobs"
+        )
+    )