Add lakeFS data lake recipe for GCP & k3d

[AdrianoKF]

Describe changes

This PR adds support for the lakeFS data lake to MLStacks. It adds a new high-level component type data_lake with lakefs as its only flavor.

Everything is up for debate and discussion at this point, I just wanted to put together an end-to-end example to have a starting point for the review.

Note

This design choice is up for discussion. Data lakes are conceptually not covered in the current component types and don't really fit anywhere else (like artifact stores).

Defining a new component type would also impact the ZenML side of things, which hasn't been considered yet.

Feedback in the review is highly appreciated! 🙏🏻

At this point, deployment on a local k3d cluster with MinIO as the storage backend and Google Cloud Platform (with GCS for data storage and Cloud SQL with private VPC connectivity as the database backend) are supported.

The lakeFS instance is exposed via a Kubernetes ingress, initial admin credentials are provisioned automatically on k3d clusters, and all user-relevant options are exposed through the stack output (under the data_lake_configuration key).

Open questions/issues

• Debate the introduction of a new component type (see above)
• gcp-modular: The VPC service networking resource does not destroy cleanly, since it depends on the k8s cluster being removed from the VPC first
• gcp-modular: Open TODO https://github.com/aai-institute/mlstacks/pull/1/files#diff-1e6881032f0907b314f95cf6b5fe4e113b4fdcf4b140d9967c050072787ea7f1R41

Example stack definitions

Below are the stacks used during development/testing:

GCP:

# my-stack.yaml
spec_version: 1
spec_type: stack
name: "quickstart_stack"
provider: gcp
default_region: "europe-west3"
default_tags:
  deployed-by: "mlstacks"
components:
  - my-stack-lakefs.yaml
---
# my-stack-lakefs.yaml
spec_version: 1
spec_type: component
component_type: "data_lake"
component_flavor: "lakefs"
name: "quickstart_lakefs"
provider: gcp
metadata:
  config:
    project_id: "mlstacks-sandbox-1"
  tags:
    deployed-by: "mlstacks"

k3d local:

# my-stack.yaml
spec_version: 1
spec_type: stack
name: "quickstart_stack"
provider: k3d
default_tags:
  deployed-by: "mlstacks"
components:
  - my-stack-minio.yaml
  - my-stack-lakefs.yaml
---
# my-stack-lakefs.yaml
spec_version: 1
spec_type: component
component_type: "data_lake"
component_flavor: "lakefs"
name: "quickstart_lakefs"
provider: k3d
metadata:
  tags:
    deployed-by: "mlstacks"
---
# my-stack-minio.yaml
spec_version: 1
spec_type: component
component_type: "artifact_store"
component_flavor: "minio"
name: "quickstart_minio_bucket"
provider: k3d
metadata:
  tags:
    deployed-by: "mlstacks"

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• If my change requires a change to docs, I have updated the documentation
  accordingly.
• I have added tests to cover my changes.
• I have based my new branch on develop and the open PR is targeting
  develop. If your branch wasn't based on develop read
  Contribution guide on rebasing branch to develop.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to
  change)
• Other (add details above)

[AdrianoKF]

Needs addressing