Skip to content

Commit

Permalink
fix spelling issues
Browse files Browse the repository at this point in the history
  • Loading branch information
@strangiato
strangiato committed Jul 30, 2024
1 parent a506962 commit 6250435
Showing 1 changed file with 18 additions and 20 deletions.
38 changes: 18 additions & 20 deletions docs/odh-rhoai/gitops.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ spec:
```
Subscription Options:
1) Operator versions are managed with the `channel` in OLM. Users are able to select a channel that corrisponds to the upgrade lifecycle they wish to follow and OLM will update versions as they are released on that channel. To learn more about the available channels and the release lifecycle, please refer to the official [lifecycle documentation](https://access.redhat.com/support/policy/updates/rhoai-sm/lifecycle)
1) Operator versions are managed with the `channel` in OLM. Users are able to select a channel that corresponds to the upgrade lifecycle they wish to follow and OLM will update versions as they are released on that channel. To learn more about the available channels and the release lifecycle, please refer to the official [lifecycle documentation](https://access.redhat.com/support/policy/updates/rhoai-sm/lifecycle)
2) Platform administrators also have an option to set how upgrades are applied for the operator with the `installPlanApproval`. If set to `Automatic` RHOAI is automatically updated to the latest version that is available on the selected channel. If set to `Manual` administrators will be required to approve all upgrades.

### Component Configuration
Expand Down Expand Up @@ -126,7 +126,7 @@ After the `DataScienceCluster` object is created, the operator will install and

## Administration

### Dashbaord Configs
### Dashboard Configs

The Red Hat OpenShift AI Dashboard has many different configurable options through the UI that can be managed using the `OdhDashboardConfig` config object. A default `OdhDashboardConfig` is created when the Dashboard component is installed

Expand Down Expand Up @@ -236,7 +236,7 @@ spec:
```

OdhDashboardConfig Options:
1) The Dashboard creates a group called `rhods-admins` by default which users can be added to be granted admin priviledges through the Dashboard. Additionally, any user with the cluster-admin role are admins in the Dashboard by default. If you wish to change the group which is used to manage admin access, this option can be updated. It is important to note that this field only impacts a users ability to modify settings in the Dashboard, and will have no impact to a users ability to modify configurations through the Kubernetes objects such as this OdhDashboardConfig object.
1) The Dashboard creates a group called `rhods-admins` by default which users can be added to be granted admin privileges through the Dashboard. Additionally, any user with the cluster-admin role are admins in the Dashboard by default. If you wish to change the group which is used to manage admin access, this option can be updated. It is important to note that this field only impacts a users ability to modify settings in the Dashboard, and will have no impact to a users ability to modify configurations through the Kubernetes objects such as this OdhDashboardConfig object.
2) By default any user that has access to the OpenShift cluster where Red Hat OpenShift AI is installed will have the ability to access the Dashboard. If you wish to restrict who has access to the Dashboard this option can be updated to another group. Like the admin group option, this option only impacts the users ability to access the Dashboard and does not restrict their ability to interact directly with the Kubernetes objects used to deploy AI resources.
3) When a user creates a new Model Server through the Dashboard they are presented with an option to choose a server size which will impact the resources available to the pod created for the Model Server. Administrators have the ability to configure the default options that are available to their users.
4) When creating a new Workbench, users are asked to create storage for their Workbench. The storage will default to the value set here and users will have the option to choose a different amount of storage if their use case requires more or less storage. Admins can choose another default storage size that is presented to users by configuring this option.
Expand Down Expand Up @@ -325,7 +325,7 @@ Notebook Image Options:
3) The notebook image requires several labels for them to appear in the Dashboard, including the `app.kubernetes.io/created-by: byon` label. While traditionally this label is utilized to trace where an object originated from, this label is required for the notebooks to be made available to end users.
4) Multiple image versions can be configured as part of the same Notebook and users have the ability to select which version of the image they wish to use. This is helpful if you release updated versions of the Notebook image and you wish to avoid breaking end user environments with package changes and allow them to upgrade as they wish.
5) When selecting a Notebook image users will be presented with some information about the notebook based on the information presented in this annotation. `opendatahub.io/notebook-python-dependencies` is most commonly used to present information about versions from the most important Python packages that are pre-installed in the Image.
6) Like the the pyton dependencies annotation, the `opendatahub.io/notebook-software` annotation is used to present the end user with information about what software is installed in the Image. Most commonly this field is used to present information such as the Python version, Jupyter versions, or CUDA versions.
6) Like the python dependencies annotation, the `opendatahub.io/notebook-software` annotation is used to present the end user with information about what software is installed in the Image. Most commonly this field is used to present information such as the Python version, Jupyter versions, or CUDA versions.
7) When multiple tags are created on the ImageStream, the `opendatahub.io/workbench-image-recommended` is used to control what version of the image is presented by default to end users. Only one tag should be set to `true` at any give time.
8) Notebook images are generally recommended to be stored in an Image Registry outside of the cluster and referenced in the ImageStream.

Expand Down Expand Up @@ -463,7 +463,7 @@ objects:

### Data Science Projects

Data Science Projects are simply a normal OpenShift Project with an extra label to distinguish them from normal OpenShift projects by the Red Hat OpenShift AI Dashboard. Like OpenShift Projects it is recommended to create a `namespace` object and allow OpenShift to create the corrisponding `project` object:
Data Science Projects are simply a normal OpenShift Project with an extra label to distinguish them from normal OpenShift projects by the Red Hat OpenShift AI Dashboard. Like OpenShift Projects it is recommended to create a `namespace` object and allow OpenShift to create the corresponding `project` object:

```yaml
apiVersion: v1
Expand All @@ -488,9 +488,9 @@ metadata:

### Workbenches

Workbench objects are managed using the Notebook custom resource. The Notebook object contains a fairly complex configuration, with many items that will be autogenerated, and required annotations to display correctly in the Dashboard. The Notebook object essentially acts as a wrapper around a normal pod definition and you will find many simliarities to managing a pod with options such as the image, pvcs, secrets, etc.
Workbench objects are managed using the Notebook custom resource. The Notebook object contains a fairly complex configuration, with many items that will be autogenerated, and required annotations to display correctly in the Dashboard. The Notebook object essentially acts as a wrapper around a normal pod definition and you will find many similarities to managing a pod with options such as the image, pvcs, secrets, etc.

It is highly recommended to thuroughly test any Notebook configurations configured with GitOps.
It is highly recommended to thoroughly test any Notebook configurations configured with GitOps.

```yaml
apiVersion: kubeflow.org/v1
Expand Down Expand Up @@ -682,7 +682,7 @@ Generally, you do not want to include this annotation in your GitOps configurati

### Data Science Connections

A Data Science Connection is a normal Kubernetes Secret object with several annotations, that follow a specific format for the data.
A Data Science Connection is a normal Kubernetes Secret object with several annotations that follow a specific format for the data.

```
kind: Secret
Expand Down Expand Up @@ -767,17 +767,17 @@ spec:
```

1) The Dashboard expects to look for an object called `dspa` and it is not recommended to deploy more than a single DataSciencePipelineApplication object in a single namespace.
2) The externalStorage is a critical configuration for setting up S3 backend storage for Data Science Pipelines. While using the dashboard you are required to configure the connection details. While you can import these details from a data connection, it will create a seperate secret containing the s3 secrets instead of reusing the existing data connection secret.
2) The externalStorage is a critical configuration for setting up S3 backend storage for Data Science Pipelines. While using the dashboard you are required to configure the connection details. While you can import these details from a data connection, it will create a separate secret containing the s3 secrets instead of reusing the existing data connection secret.

Once a Data Science Pipeline instance has been created, users may wish to configure and manage their pipelines via GitOps. It is important to note that Data Science Pipelines is not "gitops friendly". While working with Elyra or a kfp pipeline, users are requried to manually upload a pipeline file to the Dashboard which does not generate a corrisponding Kubernetes object. Additionally, when executing a pipeline run, uses may find a ArgoWorkflow object that is generated for the run, however this object can not be re-used in a gitops application to create a new pipeline run in Data Science Pipelines.
Once a Data Science Pipeline instance has been created, users may wish to configure and manage their pipelines via GitOps. It is important to note that Data Science Pipelines is not "gitops friendly". While working with Elyra or a kfp pipeline, users are required to manually upload a pipeline file to the Dashboard which does not generate a corresponding Kubernetes object. Additionally, when executing a pipeline run, uses may find a ArgoWorkflow object that is generated for the run, however this object can not be re-used in a gitops application to create a new pipeline run in Data Science Pipelines.

As a work around, one common pattern to "gitops-ify" a Data Science Pipeline while using kfp is to instead create a Tekton pipeline that either compiles the pipeline, and uses the kfp skd to upload the pipeline to Data Science Pipelines, or the kfp sdk can automatically trigger a new pipeline run directly from your pipeline code.

### Model Serving

Model Serving in RHOAI has two different flavors, Single Model Serving (KServe) and Multi-Model Serving (ModelMesh). Both model server options utilize the same Kubernetes objects (ServingRuntime and InferenceService), but have different controllers managing them.

As mentioned in the Data Sciece Project section, in order to utilize ModelMesh, a `modelmesh-enabled` label must be applied to the namespace:
As mentioned in the Data Science Project section, in order to utilize ModelMesh, a `modelmesh-enabled` label must be applied to the namespace:

```yaml
apiVersion: v1
Expand All @@ -789,7 +789,7 @@ metadata:
modelmesh-enabled: "true"
```
When creating a model server through the Dashboard, users can select a "Serving Runtime Template" which will create a ServingRuntime instance in their namespace which can be managed via GitOps. The ServingRuntime helps to define different things such as the container defenition, the supported model types, and available ports.
When creating a model server through the Dashboard, users can select a "Serving Runtime Template" which will create a ServingRuntime instance in their namespace which can be managed via GitOps. The ServingRuntime helps to define different things such as the container definition, the supported model types, and available ports.
```yaml
apiVersion: serving.kserve.io/v1alpha1
Expand Down Expand Up @@ -859,9 +859,9 @@ spec:
grpcDataEndpoint: 'port:8001'
```
1) While KServe and ModelMesh share the same object defenition, they have some subtle differences, in particular the annotations that are available on them. `enable-route` is one annotation that is available on a ModelMesh ServingRuntime that is not available on a KServe based Model Server.
1) While KServe and ModelMesh share the same object definition, they have some subtle differences, in particular the annotations that are available on them. `enable-route` is one annotation that is available on a ModelMesh ServingRuntime that is not available on a KServe based Model Server.

The InferenceService is responsible for a defition of the model that will be deployed as well as which ServingRuntime it should use to deploy it.
The InferenceService is responsible for a definition of the model that will be deployed as well as which ServingRuntime it should use to deploy it.

```yaml
apiVersion: serving.kserve.io/v1beta1
Expand Down Expand Up @@ -889,16 +889,14 @@ spec:

1) The runtime must match the name of the ServingRuntime object that you wish to utilize to deploy the model.

One major difference between ModelMesh and KServe is which object is reponsible for creating and managing the pod where the model is deployed.
One major difference between ModelMesh and KServe is which object is responsible for creating and managing the pod where the model is deployed.

With KServe, the ServingRuntime acts as a "pod template" and each InferenceService creates it's own pod to deploy a model. A ServingRuntime can be used by multiple InferenceServices and each InferenceService will create a seperate pod to deploy a model.
With KServe, the ServingRuntime acts as a "pod template" and each InferenceService creates it's own pod to deploy a model. A ServingRuntime can be used by multiple InferenceServices and each InferenceService will create a separate pod to deploy a model.

By contrast, a ServingRuntime creates a pod with ModelMesh, and the InferenceService simply tells the model server pod what models to load and from where. With ModelMesh a single ServingRuntime with multiple InferenceServices will create a single pod to load all of the models.

## ArgoCD Health Checks

Out of the box, ArgoCD and OpenShift GitOps ship with check for a KServe InferenceService which is not compatible with a ModelMesh InferenceService.
Out of the box, ArgoCD and OpenShift GitOps ship with a health check for a KServe InferenceService which is not compatible with a ModelMesh InferenceService. When attempting to deploy a ModelMesh based InferenceService, ArgoCD will report the object as degraded.

Custom health checks can be added to your ArgoCD instance that are compatible with both KServe and ModelMesh as well as other RHOAI objects.

The Red Hat AI Services Practice maintains several custom health checks that you can utilize in your own ArgoCD instance [here](https://github.com/redhat-ai-services/ai-accelerator/tree/main/components/operators/openshift-gitops/instance/components/health-check-openshift-ai).
Custom health checks can be added to your ArgoCD instance that are compatible with both KServe and ModelMesh as well as other RHOAI objects to resolve this issue. The Red Hat AI Services Practice maintains several custom health checks that you can utilize in your own ArgoCD instance [here](https://github.com/redhat-ai-services/ai-accelerator/tree/main/components/operators/openshift-gitops/instance/components/health-check-openshift-ai).

0 comments on commit 6250435

Please sign in to comment.