Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(docs): Recommend use of multi-cloud gateway for object storage #59

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

chore(docs): Recommend use of multi-cloud gateway for object storage #59

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7302aef
chore(docs): Recommend use of multi-cloud gateway for object storage
butler54 May 15, 2024
1ccb219
chore(docs): Fix object storage
butler54 May 15, 2024
546e816
fix: Allow md files to have duplicate headings, as the duplication of…
butler54 May 15, 2024
891eaf9
fix: Allow md files to have duplicate headings, as the duplication of…
butler54 May 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions .github/linters/.markdown-lint.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1 @@
{
"default": true,
"MD003": false,
"MD013": false,
"MD033": false
}
{"default": true, "MD003": false, "MD013": false, "MD033": false, "MD024": false}
59 changes: 59 additions & 0 deletions docs/ADR/ADR_001.md
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this proposal, are we considering the need to support simple deployment and demos as well i.e. Minio is good to show a simple flow including what is happening on storage side for the data, model, etc... ? I'd almost consider that having both minio and Nooba / MCG could make sense. Feel free to comment and elaborate.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I didn't make it as clear as I could. MCG can replicate the functionality of minio - e.g using in cluster storage to build object storage pool.

Some of the stripped back minio manifests are simpler, however, when you have the operator included I believe it's as complex.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My question was simpler: does this still provide us with a simple user interface to access and interact with the storage objects? (typically needed in a demo)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay this is definitely a gap then. Looks the console was removed: https://access.redhat.com/documentation/en-us/red_hat_openshift_data_foundation/4.11/html/4.11_release_notes/removed_functionality

I'll have a bit of a look at options for demos.

Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
x-trestle-template-version: 0.0.1
---

# ADR-001: Use of noobaa / multi-cloud gateway as a consistent object storage interface

## Context and Problem Statement

The data science workflows that are used within OpenShift AI are dependent on object storage, based on the S3 protocol.
While object storage / blob is wide spread, interfaces for provisioning are inconsistent.
In order to automate the provisioning lifecycle of data science projects within the pattern it is essential that the pattern, via argoCD and helm charts, can orchestrate the provisioning lifecycle.
The second question is to ensure that the data scientists don't need to directly understand the underlying storage infrastructure, including for application development.

## Considered Options

### Depend exclusively on external object storage

#### Positives

- No infastructure consumption for storage on cluster.
-

#### Negatives

- Requires credential management and access to s3 in the cloud account with appropriate IAM credentials.
- S3 is not a given in on-premise environment

### Leverage object storage via an internal platform exclusively using minio

#### Positives

- Consistent addressing for object storage in cluster for application development
- Closed loop credential management easing automation

#### Negatives

- Always consumes expensive (block) storage
- Additional vendor required for commercial support
- Minio operator is not currently supported on 4.15
- Multi-cluster setups must be cluster aware

### Leverage Noobaa (Multi cloud gateway) to provide a flexible interface

#### Positives

- Flexible choice between using in cluster and out of cluster storage
- Multi-cluster setups can obfuscate origin of s3 across multiple sites
- Clear support path for downstream development

#### Negatives

- Flexibility provides choice but also increases required parameters, documentation and instrumentation

## Decision Outcome

Use Noobaa / MCG as:

1. It worked out of the box easily
1. There is a clear support path.
Loading