-
Notifications
You must be signed in to change notification settings - Fork 30
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #109 from tsailiming/dsp-example
Add data science pipeline example
- Loading branch information
Showing
3 changed files
with
50 additions
and
0 deletions.
There are no files selected for viewing
49 changes: 49 additions & 0 deletions
49
docs/tools-and-applications/datasciencepipeline/datasciencepipeline.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
# Data Science Pipeline | ||
|
||
## What is it? | ||
|
||
OpenShift AI allows building of machine line workflows with a data science pipeline. From OpenShift AI version 2.9, data science pipelines are based on KubeFlow Pipelines (KFP) version 2.0. | ||
|
||
## What is Kubeflow Pipelines? | ||
Kubeflow Pipelines (KFP) is a platform for building and deploying portable and scalable machine learning (ML) workflows using Docker containers. | ||
|
||
With KFP you can author components and pipelines using the KFP Python SDK, compile pipelines to an intermediate representation YAML, and submit the pipeline to run on a KFP-conformant backend. | ||
|
||
The current version of KFP 2.0 in OpenShift AI uses Argo Workflow as the backend. | ||
|
||
## Why do I see OpenShift Pipeline in this example? | ||
|
||
The example uses OpenShift Pipeline (Tekton) to compile the pipeline into an intermediate representation (IR) YAML and submit it to the Kubeflow Pipeline server (instead of doing it from your Jupyter environment using Elyra, or importing it directly through the Dashboard). | ||
|
||
The Tekton pipeline has 2 main tasks: | ||
* git-clone | ||
* execute-kubeflow-pipeline to compile and submit the pipeline | ||
|
||
## Example | ||
|
||
## Architectural Diagram | ||
|
||
![dsp-arch](img/rhoai-dsp.jpg) | ||
|
||
The demo uses the following components: | ||
|
||
| Component | Descrioption| | ||
|---|---| | ||
| Gitea | To store pipeline source code | ||
| Model Registry | To store model metadata | ||
| OpenShift Pipelines | Using Tekton to build the pipeline | ||
| Data Science Pipeline | To run the pipeline using KFP | ||
| Minio | S3 bucket to store the model | ||
| KServe | To serve the model | ||
|
||
## Prerequisite | ||
|
||
You will need OpenShift 2.15 installed with ModelRegistry set to `Managed`. In 2.15, the model registry feature is currently in Tech Preview. | ||
|
||
### Running the Example | ||
|
||
The sample code is available [here](https://github.com/tsailiming/openshift-ai-dsp). | ||
|
||
|
||
|
||
|
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters