chore(release): 0.2.2 (#80)

artefactory · Nov 6, 2023 · 66d3759 · 66d3759
2 parents afa6c4a + 395b256
commit 66d3759
Show file tree

Hide file tree

Showing 18 changed files with 289 additions and 62 deletions.
diff --git a/.github/workflows/deploy_docs.yaml b/.github/workflows/deploy_docs.yaml
@@ -0,0 +1,45 @@
+name: Deploy MkDocs to GitHub Pages
+
+on:
+  push:
+    branches:
+      - main
+
+env:
+    python-version: "3.10"
+
+jobs:
+  deploy-docs:
+    name: Deploy docs
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Set up Python ${{ env.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ env.python-version }}
+
+      - name: Install poetry
+        run: make download-poetry
+
+      - name: Set up pip cache
+        uses: actions/[email protected]
+        with:
+          path: ~/.cache/pypoetry/virtualenvs
+          key: venv-${{ env.python-version }}-${{ hashFiles('pyproject.toml') }}-${{ hashFiles('poetry.lock') }}
+
+      - name: Set Poetry Path
+        run: |
+          echo "$HOME/.poetry/bin" >> $GITHUB_PATH
+
+
+      - name: Install requirements
+        run: |
+          poetry run pip install --upgrade pip
+          poetry install --with docs
+
+      - name: Deploying MkDocs documentation
+        run: |
+          poetry run mkdocs build
+          poetry run mkdocs gh-deploy --force
diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
@@ -6,9 +6,10 @@
 # 2. If it succeeds, the version is updated using Python Semantic Release
 # 3. The release is uploaded to GitHub (same step and GitHub action)
 # 5. Authentication to Google Cloud is achieved using Workload Identity Federation
-# 6. The release is uploaded to Google Cloud Storage
+# 6. The release is uploaded to Google Artifact Registry
 #
 # For more details on setting up Workload Identity Federation for GitHub, visit https://github.com/google-github-actions/auth#setting-up-workload-identity-federation
+# WIF service account must have roles/artifactregistry.write access to the Artifact Registry repository
 
 name: CI and Release on main
 
@@ -46,12 +47,12 @@ jobs:
         workload_identity_provider: '${{ secrets.WIF_PROVIDER }}' # e.g. - projects/123456789/locations/global/workloadIdentityPools/my-pool/providers/my-provider
         service_account: '${{ secrets.WIF_SERVICE_ACCOUNT }}' # e.g. - [email protected]
 
-    - name: Copy release to root
-      run: cp -r dist/*.tar.gz .
+    - name: Install twine
+      run: pip install twine keyrings.google-artifactregistry-auth
 
-    - name: 'Upload Release to Google Cloud Storage'
-      uses: 'google-github-actions/upload-cloud-storage@v1'
-      with:
-        path: '.'
-        destination: vertex-pipelines-deployer
-        glob: '*.tar.gz'
+    - name: Upload dist to Google Artifact Registry
+      run: |
+        python3 -m twine upload \
+          --repository-url https://europe-west1-python.pkg.dev/vertex-deployer-sandbox-3a8a/vertex-deployer \
+          --verbose \
+          dist/*
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -32,8 +32,8 @@ To develop, you will need dev requirements too. Run:
 make install-dev-requirements
 ```
 
-> **Note**
-> `poetry.lock` is not committed deliberately, as recommended by Poetry's doc. You can read more about it [here](https://python-poetry.org/docs/basic-usage/#as-a-library-developer).
+!!! note "About poetry.lock"
+    `poetry.lock` is not committed deliberately, as recommended by Poetry's doc. You can read more about it [here](https://python-poetry.org/docs/basic-usage/#as-a-library-developer).
 
 ### Codestyle
 
@@ -69,6 +69,11 @@ The Release GitHub Action does the following:
 
 The action is triggered by any push to main.
 
-> [!NOTE]
-> The release action will be triggered by any push to `main` only if the 'CI' job in the 'release.yaml' workflow succeeds.
-> Python Semantic Release will take care of version number update, tag creation and release creation.
+!!! tip
+    The release action will be triggered by any push to `main` only if the 'CI' job in the 'release.yaml' workflow succeeds.
+    Python Semantic Release will take care of version number update, tag creation and release creation.
+
+
+When it's done, rebase develop to keep it up to date with main.
+
+And you're done ! 🎉
diff --git a/README.md b/README.md
@@ -3,7 +3,7 @@
     <h1 align="center">Vertex Pipelines Deployer</h1>
     <h3 align="center">Deploy Vertex Pipelines within minutes</h3>
         <p align="center">
-        This tool is a wrapper aound <a href="https://www.kubeflow.org/docs/components/pipelines/v2/hello-world/">kfp</a> and <a href="https://cloud.google.com/python/docs/reference/aiplatform/latest">google-cloud-aiplatform</a> that allows you to check, compile, upload, run and schedule Vertex Pipelines to a Vertex AI Pipelines endpoint in a standardized manner.
+        This tool is a wrapper around <a href="https://www.kubeflow.org/docs/components/pipelines/v2/hello-world/">kfp</a> and <a href="https://cloud.google.com/python/docs/reference/aiplatform/latest">google-cloud-aiplatform</a> that allows you to check, compile, upload, run and schedule Vertex Pipelines in a standardized manner.
         </p>
 </div>
 </br>
@@ -23,48 +23,55 @@
 </div>
 
 
-## Table of Contents
-- [Why this tool?](#why-this-tool)
-- [Prerequisites](#prerequisites)
-- [Installation](#installation)
+## 📚 Table of Contents
+- [Why this tool?](#❓-why-this-tool)
+- [Prerequisites](#📋-prerequisites)
+- [Installation](#📦-installation)
     - [From git repo](#from-git-repo)
     - [From GCS (not available in PyPI yet)](#from-gcs-not-available-in-pypi-yet)
     - [Add to requirements](#add-to-requirements)
-- [Usage](#usage)
-  - [Setup](#setup)
-  - [Folder Structure](#folder-structure)
-  - [CLI: Deploying a Pipeline with `deploy`](#cli-deploying-a-pipeline-with-deploy)
-  - [CLI: Checking Pipelines are valid with `check`](#cli-checking-pipelines-are-valid-with-check)
-  - [CLI: Other commands](#cli-other-commands)
+- [Usage](#🚀-usage)
+  - [Setup](#🛠️-setup)
+  - [Folder Structure](#📁-folder-structure)
+  - [CLI: Deploying a Pipeline with `deploy`](#🚀-cli-deploying-a-pipeline-with-deploy)
+  - [CLI: Checking Pipelines are valid with `check`](#✅-cli-checking-pipelines-are-valid-with-check)
+  - [CLI: Other commands](#🛠️-cli-other-commands)
     - [`create`](#create)
     - [`list`](#list)
-  - [CLI: Options](#cli-options)
+  - [CLI: Options](#🍭-cli-options)
 
 [Full CLI documentation](docs/CLI_REFERENCE.md)
 
-## Why this tool?
+<!-- --8<-- [start:why] -->
+## ❓ Why this tool?
+
 
 Three uses cases:
+
 1. **CI:** check pipeline validity.
 1. **Dev mode:** duickly iterate over your pipelines by compiling and running them in multiple environments (test, dev, staging, etc) without duplicating code or looking for the right kfp / aiplatform snippet.
 2. **CD:** deploy your pipelines to Vertex Pipelines in a standardized manner in your CD with Cloud Build or GitHub Actions.
 
 
 Four commands:
+
 - `check`: check your pipelines (imports, compile, check configs validity against pipeline definition).
 - `deploy`: compile, upload to Artifact Registry, run and schedule your pipelines.
 - `create`: create a new pipeline and config files.
 - `list`: list all pipelines in the `vertex/pipelines` folder.
+<!-- --8<-- [end:why] -->
 
-## Prerequisites
+## 📋 Prerequisites
+<!-- --8<-- [start:prerequisites] -->
 
 - Unix-like environment (Linux, macOS, WSL, etc...)
 - Python 3.8 to 3.10
 - Google Cloud SDK
 - A GCP project with Vertex Pipelines enabled
+<!-- --8<-- [end:prerequisites] -->
 
-## Installation
-
+## 📦 Installation
+<!-- --8<-- [start:installation] -->
 
 ### From git repo
 
@@ -107,32 +114,33 @@ Then add the following line to your `requirements.in` file:
 ```bash
 file:my/path/to/vertex_deployer-$VERSION.tar.gz
 ```
+<!-- --8<-- [end:installation] -->
 
-## Usage
-
-### Setup
+## 🚀 Usage
+<!-- --8<-- [start:setup] -->
+### 🛠️ Setup
 
 1. Setup your GCP environment:
-
 ```bash
 export PROJECT_ID=<gcp_project_id>
 gcloud config set project $PROJECT_ID
 gcloud auth login
 gcloud auth application-default login
 ```
-2. You need the following APIs to be enabled:
-    - Cloud Build API
-    - Artifact Registry API
-    - Cloud Storage API
-    - Vertex AI API
 
+2. You need the following APIs to be enabled:
+- Cloud Build API
+- Artifact Registry API
+- Cloud Storage API
+- Vertex AI API
 ```bash
 gcloud services enable \
     cloudbuild.googleapis.com \
     artifactregistry.googleapis.com \
     storage.googleapis.com \
     aiplatform.googleapis.com
 ```
+
 3. Create an artifact registry repository for your base images (Docker format):
 ```bash
 export GAR_DOCKER_REPO_ID=<your_gar_repo_id_for_images>
@@ -141,20 +149,24 @@ gcloud artifacts repositories create ${GAR_DOCKER_REPO_ID} \
     --location=${GAR_LOCATION} \
     --repository-format=docker
 ```
+
 4. Build and upload your base images to the repository. To do so, please follow Google Cloud Build documentation.
+
 5. Create an artifact registry repository for your pipelines (KFP format):
 ```bash
 export GAR_PIPELINES_REPO_ID=<your_gar_repo_id_for_pipelines>
 gcloud artifacts repositories create ${GAR_PIPELINES_REPO_ID} \
     --location=${GAR_LOCATION} \
     --repository-format=kfp
 ```
+
 6. Create a GCS bucket for Vertex Pipelines staging:
 ```bash
 export GCP_REGION=<your_gcp_region>
 export VERTEX_STAGING_BUCKET_NAME=<your_bucket_name>
 gcloud storage buckets create gs://${VERTEX_STAGING_BUCKET_NAME} --location=${GCP_REGION}
 ```
+
 7. Create a service account for Vertex Pipelines:
 ```bash
 export VERTEX_SERVICE_ACCOUNT_NAME=foobar
@@ -178,7 +190,7 @@ gcloud artifacts repositories add-iam-policy-binding ${GAR_PIPELINES_REPO_ID} \
 
 You can use the deployer CLI (see example below) or import [`VertexPipelineDeployer`](deployer/pipeline_deployer.py) in your code (try it yourself).
 
-### Folder Structure
+### 📁 Folder Structure
 
 You must respect the following folder structure. If you already follow the
 [Vertex Pipelines Starter Kit folder structure](https://github.com/artefactory/vertex-pipeline-starter-kit), it should be pretty smooth to use this tool:
@@ -192,9 +204,11 @@ vertex
    └─ {pipeline_name}.py
 ```
 
-> [!NOTE]
-> You must have at least these files. If you need to share some config elements between pipelines,
-> you can have a `shared` folder in `configs` and import them in your pipeline configs.
+!!! tip "About folder structure"
+    You must have at least these files. If you need to share some config elements between pipelines,
+    you can have a `shared` folder in `configs` and import them in your pipeline configs.
+
+    You can use the [`create`](../usage#create) command to create a new pipeline and config files.
 
 #### Pipelines
 
@@ -226,10 +240,12 @@ They must be located in the `config/{pipeline_name}` folder.
 `.py` files are useful to define complex configs (e.g. a list of dicts) while `.json` / `.toml` files are useful to define simple configs (e.g. a string).
 
 **How to format them?**
+
 - `.json` and `.toml` files must be valid json files containing only one dict of key: value representing parameter values.
 - `.py` files must be valid python files with two important elements:
-    - `parameter_values` to pass arguments to your pipeline
-    - `input_artifacts` if you want to retrieve and create input artifacts to your pipeline.
+
+    * `parameter_values` to pass arguments to your pipeline
+    * `input_artifacts` if you want to retrieve and create input artifacts to your pipeline.
         See [Vertex Documentation](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.PipelineJob) for more information.
 
 **How to name them?**
@@ -252,13 +268,15 @@ VERTEX_STAGING_BUCKET_NAME=YOUR_VERTEX_STAGING_BUCKET_NAME  # GCS Bucket for Ver
 VERTEX_SERVICE_ACCOUNT=YOUR_VERTEX_SERVICE_ACCOUNT  # Vertex Pipelines Service Account
 ```
 
-> **Note**
-> We're using env files and dotenv to load the environment variables.
-> No default value for `--env-file` argument is provided to ensure that you don't accidentally deploy to the wrong project.
-> An [`example.env`](./example/example.env) file is provided in this repo.
-> This also allows you to work with multiple environments thanks to env files (`test.env`, `dev.env`, `prod.env`, etc)
+!!! note "About env files"
+    We're using env files and dotenv to load the environment variables.
+    No default value for `--env-file` argument is provided to ensure that you don't accidentally deploy to the wrong project.
+    An [`example.env`](./example/example.env) file is provided in this repo.
+    This also allows you to work with multiple environments thanks to env files (`test.env`, `dev.env`, `prod.env`, etc)
+<!-- --8<-- [end:setup] -->
 
-### CLI: Deploying a Pipeline with `deploy`
+<!-- --8<-- [start:usage] -->
+### 🚀 CLI: Deploying a Pipeline with `deploy`
 
 Let's say you defines a pipeline in `dummy_pipeline.py` and a config file named `config_test.json`. You can deploy your pipeline using the following command:
 ```bash
@@ -274,7 +292,7 @@ vertex-deployer deploy dummy_pipeline \
     --enable-caching
 ```
 
-### CLI: Checking Pipelines are valid with `check`
+### ✅ CLI: Checking Pipelines are valid with `check`
 
 To check that your pipelines are valid, you can use the `check` command. It uses a pydantic model to:
 - check that your pipeline imports and definition are valid
@@ -292,7 +310,7 @@ vertex-deployer check --all
 ```
 
 
-### CLI: Other commands
+### 🛠️ CLI: Other commands
 
 #### `create`
 
@@ -310,17 +328,24 @@ You can list all pipelines in the `vertex/pipelines` folder using the `list` com
 vertex-deployer list --with-configs
 ```
 
-### CLI: Options
+### 🍭 CLI: Options
 
 ```bash
 vertex-deployer --help
 ```
 
+To see package version:
+```bash
+vertex-deployer --version
+```
+
 To adapt log level, use the `--log-level` option. Default is `INFO`.
 ```bash
 vertex-deployer --log-level DEBUG deploy ...
 ```
 
+<!-- --8<-- [end:usage] -->
+
 ## Repository Structure
 
 ```
@@ -332,7 +357,7 @@ vertex-deployer --log-level DEBUG deploy ...
 │  │  └─ release.yaml
 │  ├─ CODEOWNERS
 │  └─ PULL_REQUEST_TEMPLATE.md
-├─ deployer
+├─ deployer                                     # Source code
 │  ├─ __init__.py
 │  ├─ cli.py
 │  ├─ constants.py
@@ -344,8 +369,10 @@ vertex-deployer --log-level DEBUG deploy ...
 │     ├─ logging.py
 │     ├─ models.py
 │     └─ utils.py
+├─ docs/                                        # Documentation folder (mkdocs)
+├─ templates/                                   # Semantic Release templates
 ├─ tests/
-├─ example
+├─ example                                      # Example folder with dummy pipeline and config
 |   ├─ example.env
 │   └─ vertex
 │      ├─ components
@@ -364,8 +391,12 @@ vertex-deployer --log-level DEBUG deploy ...
 │         └─ dummy_pipeline.py
 ├─ .gitignore
 ├─ .pre-commit-config.yaml
+├─ catalog-info.yaml                            # Roadie integration configuration
+├─ CHANGELOG.md
+├─ CONTRIBUTING.md
 ├─ LICENSE
 ├─ Makefile
+├─ mkdocs.yml                                   # Mkdocs configuration
 ├─ pyproject.toml
 └─ README.md
 ```