[Feature Request]: integration test cases with DSP, CodeFlare and KubeRay in a multi-step pipeline

[yuanchi2807]

Feature description

Create configurible integration test cases with Data Science Pipelines, CodeFlare and KubeRay, with the following characteristics:

• Small enough footprint that can be executed in a test environment;
• Container image and data should be free of intellectual property concerns;
• Easy to increase or decrease KubeRay cluster footprint;
• Easy to go from a single-step to N-step pipelines;

Describe alternatives you've considered

IBM Research has been running pipelines with DSP, CodeFlare, and KubeRay internally but the configuration and images contain proprietary libraries that are not suitable for integration testing purposes.

Anything else?

No response

[yuanchi2807]

Created a feature description in codeflare-sdk repo here (project-codeflare/codeflare-sdk#425)
Not sure which team should own the test case. Please advise.

Reposted here.

Name of Feature or Improvement

Create an integration test case to validate DSP, CodeFlare and KubeRay implementation.

Describe the Solution You Would Like to See

Test environment assumptions:

1. Data Science Pipeline v1.
2. Ray cluster shall consist of no more than 2 worker pods, with 2 CPU cores and less than 6 GB available for each pod.
3. An integration test execution time shall be less than 20 mins in total.
4. S3 storage may be available, if needed.
5. Free of proprietary intellectual property.
6. Public data only.

Proposed test case: Clustering text documents using k-means on scikit-learn education page.

https://scikit-learn.org/stable/auto_examples/text/plot_document_clustering.html

Data Science Pipeline stages:

1. Downloading test data (https://scikit-learn.org/stable/auto_examples/text/plot_document_clustering.html#loading-text-data)
2. Launch Ray cluster with two worker pods.
3. Ray driver launches two Ray actors, deployed to a pod each. The first actor runs TfidfVectorizer, followed by Kmeans clustering and evaluation. The second actor runs HashingVectorizer, followed by Kmeans clustering and evaluation.
4. Ray driver collects evaluation results from the two actors. Then it reports the summaries.
5. Ray cluster is stopped and shutdown.
6. Pipeline run is completed.

Expected test assets:

1. DSP pipeline yaml to deploy and kick off test runs.
2. Test image with Ray and document clustering code.
3. CodeFlare image to deploy the test image.
4. Preconfigured credentials and configmaps in the test environment.

[yuanchi2807]

A prototype following the above solution design can be found at this link.

https://github.com/yuanchi2807/dsp_codeflare_int_testing

Ray application image can be pulled from quay.io/yuanchichang_ibm/integration_testing/dsp_codeflare_int_testing:0.1

The pipeline definition yet_another_ray_integration_test.py is modified from https://github.com/diegolovison/ods-ci/blob/ray_integration/ods_ci/tests/Resources/Files/pipeline-samples/ray_integration.py to point to the custom image and invokes docker_clustering_driver.py through Ray jobs API.

Please feel free to comment.

[gregsheremeta]

Migrated to Jira: https://issues.redhat.com/browse/RHOAIENG-1621