Add GCP and AWS deployment templates

Add cloudshell instructions

Update

Update GCP templates

WIP new zenml deploy CLI command

Change AWS template URL

Implemented zenml deploy CLI command

Complete zenml stack deploy CLI command

Adapt stack registration to new endpoint structure

Add default repository configuration attr to all container registries

Update template with proper sagemaker permissions

Fix template permissions

Final IAM permission fixes

Fix AWS service connector permissions

Remove unneeded permissions

Fix container registry default repository setting support

Fix the S3 artifact store to use the AWS region in the service connector

Fix linter

Improve zenml stack deploy CLI

Update cloud formation template

Fix cloud formation template

Move cloud stack deployment logic behind the REST API

Re-add local store check

Use HTTPS for the server URL

Fix stack link in zenml stack deploy CLI

Fix docstring errors

Fixed more docstrings

Apply suggestions from code review

Co-authored-by: Barış Can Durak <[email protected]>
Co-authored-by: Michael Schuster <[email protected]>

Add labels to full stacks and use them in event metadata and to detect deployed stacks

Update deployment scripts

Add templates publish github action

Fix bugs

Rename stack labels to start with zenml:

Fix service connector labels to be strings

Fix linter errors

Fix component labels in CLI and add stack labels to CLI

Fix deployed stack detection to account for new labels

Move AWS CF templates to their own subdir

.github/workflows/publish_stack_templates.yml

@@ -0,0 +1,33 @@
+---
+name: Publish Cloud Stack Templates
+on:
+  workflow_call:
+  workflow_dispatch:
+jobs:
+  publish_stack_templates:
+    name: Publish Cloud Stack Templates
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repo
+        uses: actions/[email protected]
+      # Setup AWS CLI
+      - name: Configure AWS credentials
+        uses: aws-actions/configure-aws-credentials@v2
+        with:
+          role-to-assume: arn:aws:iam::715803424590:role/gh-action-role-zenml-stack-cf-templates
+          aws-region: eu-central-1
+      # Publish the CloudFormation template to S3
+      - name: Push CloudFormation template to S3
+        run: |-
+          aws s3 cp infra/aws/aws-ecr-s3-sagemaker.yaml s3://zenml-cf-templates/aws-ecr-s3-sagemaker.yaml
+      # Setup gcloud CLI
+      - uses: google-github-actions/setup-gcloud@v0
+        with:
+          service_account_email: ${{ secrets.GCP_CLOUDBUILD_EMAIL }}
+          service_account_key: ${{ secrets.GCP_CLOUDBUILD_KEY }}
+          project_id: ${{ secrets.GCP_CLOUDBUILD_PROJECT }}
+      # Publish the Deployment Manager function code to GCS
+      - name: Push Deployment Manager function to GCS
+        run: |-
+          zip -j gcp-dm-stack-register.zip infra/gcp/main.py infra/gcp/requirements.txt
+          gsutil cp gcp-dm-stack-register.zip gs://zenml-public-bucket/zenml-gcp-dm-templates/

.github/workflows/release.yml

@@ -121,6 +121,11 @@ jobs:
       - name: Sleep for 4 minutes
         run: sleep 240
         shell: bash
+  publish-stack-templates:
+    if: github.repository == 'zenml-io/zenml'
+    needs: publish-python-package
+    uses: ./.github/workflows/publish_stack_templates.yml
+    secrets: inherit
   # create a tag on the ZenML cloud plugins repo
   create_tag_on_cloud_plugins_repo:
     runs-on: ubuntu-latest

infra/README.md

@@ -0,0 +1,17 @@
+# Assisted ZenML Stack Deployment
+
+These are a set of scripts that can be used to provision infrastructure for **ZenML stacks directly in your browser** in AWS and GCP with minimal user input. The scripts are designed to be run with a single click and will deploy the ZenML stack in your AWS or GCP account.
+
+## Deploy a full ZenML Stack on AWS
+
+Click the button below to deploy a ZenML stack in your AWS account using AWS Cloud Formation. Log in to AWS and follow the instructions in the Cloud Formation console to deploy the stack.
+
+[![Launch Stack](https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png)](https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/create/review?stackName=zenml-stack&templateURL=https://zenml-cf-templates.s3.eu-central-1.amazonaws.com/aws-ecr-s3-sagemaker.yaml)
+
+
+## Deploy a full ZenML Stack on GCP
+
+Click the button below to deploy a ZenML stack on your GCP project using Google Cloud Shell. Google Cloud Shell will open and clone this repository. Follow the instructions in the Cloud Shell terminal to deploy the stack.
+
+[![Open in Cloud Shell](https://gstatic.com/cloudssh/images/open-btn.svg)](https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https://github.com/zenml-io/zenml&cloudshell_working_dir=infra&cloudshell_open_in_editor=gcp-gar-gcs-vertex.yaml,gcp-gar-gcs-vertex-config.yaml&cloudshell_print=gcp-gar-gcs-vertex.txt&cloudshell_git_branch=feature/prd-482-one-click-stacks)
+