[Umbrella] Ray Autoscaling tests

[kevin85421]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Currently, there are some autoscaling end-to-end tests for KubeRay in the Ray repository. However, there are some issues with the tests:

• It uses Python.
• To run the tests, you need to build the Ray source code for the test utilities.
• There are some dependencies between the Ray version of the Pods in the K8s cluster and the Ray version in your local environment because it uses kubectl port-forward and interacts with the Ray head via Ray client and ray job submit.
• It only tests the Ray nightly and KubeRay latest stable release. It doesn't test the compatibility between the Ray nightly and KubeRay nightly.

Here, we plan to build new autoscaling end-to-end tests in the KubeRay repository.

• Use Golang instead of Python so that we can leverage the K8s ecosystem in Golang to add new tests easily.
• Test KubeRay nightly with Ray nightly.
• It doesn't require to build Ray.
• It shouldn't have any dependencies between Ray Pods and local Ray version.
• Then, we will replace the existing Ray autoscaling e2e tests with this one.

Progress

• [core][autoscaler] Remove local dependencies on Ray for KubeRay autoscaling e2e tests ray#48566
• [Test][Autoscaler][1/n] Add Ray Autoscaler e2e tests #2168
• Fake GPU tests (@rueian)
  • We don't set GPUs in the Pod's resource request/limit, but we set --num-gpus in rayStartParams to simulate the autoscaling behavior of GPUs.
  • [Test][Autoscaler][2/n] Add Ray Autoscaler e2e tests for GPU workers #2181
• Make the Ray image configurable.
• Fake single-host TPU tests
• Fake multi-host TPU tests
• Self-defined resource tests (example) (@MortalHappiness): [Test][Autoscaling] Add custom resource test #2193
• [Feature] Add e2e Ray v2 Autoscaler Tests with KubeRay #2561
• [Feature] Add E2E Test for Autoscaler IdleTimeOut #2568
• [Feature] Add e2e tests for Autoscaler V2 #2574
• [Feature] Add e2e tests for inconsistency between worker group's replicas and the number of Pods #2575
• [Feature] Add an e2e test for Autoscaler to scale up by manually updating minReplicas #2576
• Replace the old Ray autoscaling e2e tests in the Ray repository.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[Irvingwangjr]

https://kuttl.dev/docs/#pre-requisites
maybe we can consider using this tool?

[kevin85421]

@Irvingwangjr Kuttl looks cool! I read the README. It seems suitable for testing some sample YAMLs, but it doesn't seem to support well the case where we need to execute some commands in the Pods to trigger certain behaviors.

[Irvingwangjr]

@Irvingwangjr Kuttl looks cool! I read the README. It seems suitable for testing some sample YAMLs, but it doesn't seem to support well the case where we need to execute some commands in the Pods to trigger certain behaviors.

yeah, that might be a problem, right now we use TestStep to execute scripts to trigger behaviors.
here is an example, we try to simulate the eviction of ray head, and check whether the RayJob will eventually enter the failed state:

apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
  - script: |
      pod_name=$(kubectl get pod -l "xx.com/ray-pod-name=mlrayjob-head-failed-on-running-head-0" -n=ns -o jsonpath="{.items[0].metadata.name}")

      cmd="kubectl exec -it $pod_name -n=ns -c ray-container -- dd if=/dev/zero of=/tmp/test.txt bs=1M count=100000"

      $cmd &

      cmd_pid=$!

      wait $cmd_pid

      exit_code=$?
      if [ $exit_code -eq 137 ]; then
          echo "the process was killed, exit with return code of 137"
      else
          echo "the process was killed, exit with return code of $exit_code"
      fi

then we assert the RayJob to be in status of failed

apiVersion: kuttl.dev/v1beta1
kind: TestAssert
commands:
- command: kubectl assert exist-enhanced rayjob/mlrayjob-head-failed-on-running -n=ns --field-selector status.phase=Failed

we also use kube-assert here, it might be helpful.

[Irvingwangjr]

https://github.com/open-feature/open-feature-operator
OpenFeature(an CNCF project) adopt this tool, it also provides some examples