Coldpress on Nerc test

[tssala23]

This is an issue for tracking the transition of coldpress from it's development environment (Barcelona Cluster) to the Nerc test cluster.

Steps:

• Look through Coldpress repo
• Install coldpress on the Test cluster
• Run through example workloads in the Repo
• Identify test workloads (talk with NAIRR engineers to see the workloads that they are running)
• Run identified test workloads with coldpress and evaluate

Any feedback throughout the process will be fed back to Ahmed

[tssala23]

Coldpress operator requires Kueue and Jobset operators to be installed on the cluster first.
These operator both exist in the in the software catalog on openshift and are provided by Red Hat.
The versions installed in Barcelona were the newest tech preview versions provided by the creators. These versions used slightly different API versions that the Red Hat builds.
Operators were replaced on Barcelona and Ahmed has updated the Coldpress repo to use the stable API version. This had initially stopped the set up of the coldpress operator

[tssala23]

Correction made to missing namespace field in template

[tssala23]

Initial install attempt unsuccessful
Readme instruction missing key steps
Readme assumes operators are already installed and setup, however it is very easy to install the operator and not set them up correctly.
Manifest to set up job set operator not in repo yet

[tssala23]

Kueue operator already installed and managed by RHOAI, so no need to install again. Second installation was messing up the operator.
Steps to prep cluster:
Kueue Install -> kueue-create.yaml -> JobSet Install -> Jobset operator create -> kueue-init.yaml
However the test cluster did not need the install Kueue step
Everything running now, including coldpress manager

[tssala23]

Tried to run Coldpress discovery jobs, pods stuck pending, pod taints blocking pods from from being scheduled.

[tssala23]

Ahmed pushed changed to image to tolerate all taints for the time being:
tolerations=[client.V1Toleration(operator="Exists")] if run_params.get("tolerate_all") else None

Future development, detect taints on nodes and set appropriate tolerations?

[tssala23]

This PR asanaullah/coldpress#31 fixes issues highlighted above

[tssala23]

Tried running vllm guide benchmark example
Operator not picking up on resource creation

[tssala23]

Ahmed found bug and pushed new image for the operator pod

[tssala23]

This example still did not work due to taints on the nodes.
Tried a different example:
https://github.com/asanaullah/coldpress/tree/main/examples/deepti_qwen
https://github.com/asanaullah/coldpress/tree/main/examples/pytorch_ddp_training
Pods were not created

[tssala23]

Parser were missing from the cluster, some installed in readme process but not all

[tssala23]

Parsers installed but still running into issue with the taints, tried adding the taints to the parser however they kept getting removed

[tssala23]

CRD for the workload templates had to be updated to include a toleration field, this way when it is added to the parser it does not get removed.
Ahmed has updated the CRDs and new ones have been added to cluster

[tssala23]

Tried to run example workload again
https://github.com/asanaullah/coldpress/tree/main/examples/deepti_qwen
However pod not scheduling as there are not enough GPUs. job is requesting 4 GPUs. There are 2 nodes labelled coldpress nodes, 1 has 3 gpus available the other 4, however it is only trying to schedule on the first one