From e2b6cb9bf014687d823921f8ecdd96d87ac059b4 Mon Sep 17 00:00:00 2001 From: ZenML GmbH Date: Tue, 12 Nov 2024 12:20:08 +0000 Subject: [PATCH 1/3] Adding the new version to the necessary files. --- README.md | 2 +- RELEASE_NOTES.md | 60 +++++++++++++++++++ examples/quickstart/configs/training_aws.yaml | 2 +- .../quickstart/configs/training_azure.yaml | 2 +- examples/quickstart/configs/training_gcp.yaml | 2 +- examples/quickstart/quickstart.ipynb | 6 +- examples/quickstart/requirements.txt | 2 +- examples/quickstart/requirements_aws.txt | 2 +- examples/quickstart/requirements_azure.txt | 2 +- examples/quickstart/requirements_gcp.txt | 2 +- pyproject.toml | 2 +- src/zenml/VERSION | 2 +- src/zenml/zen_server/deploy/helm/Chart.yaml | 2 +- src/zenml/zen_server/deploy/helm/README.md | 4 +- .../migrations/versions/0.70.0_release.py | 23 +++++++ 15 files changed, 99 insertions(+), 16 deletions(-) create mode 100644 src/zenml/zen_stores/migrations/versions/0.70.0_release.py diff --git a/README.md b/README.md index 77aab72684..25970ddab5 100644 --- a/README.md +++ b/README.md @@ -333,7 +333,7 @@ the Apache License Version 2.0. Projects Showcase

- 🎉 Version 0.68.1 is out. Check out the release notes + 🎉 Version 0.70.0 is out. Check out the release notes here.
🖥️ Download our VS Code Extension here. diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 932ce5dd71..8f6abb8c9c 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,5 +1,65 @@ +# 0.70.0 + +## What's Changed +* Optimizing the CI workflows by @bcdurak in https://github.com/zenml-io/zenml/pull/3145 +* Adding 0.68.0 to the migration tests by @bcdurak in https://github.com/zenml-io/zenml/pull/3144 +* Move step durations to body by @schustmi in https://github.com/zenml-io/zenml/pull/3046 +* Docs on ZenML setup by @strickvl in https://github.com/zenml-io/zenml/pull/3100 +* Remove wrongly set Model.was_created_in_this_run attribute by @schustmi in https://github.com/zenml-io/zenml/pull/3129 +* Allow specifying run tags in pipeline configuration by @schustmi in https://github.com/zenml-io/zenml/pull/3130 +* Fix materializer type compatibility check during loading by @schustmi in https://github.com/zenml-io/zenml/pull/3105 +* [docs] Add icons to headers in docs by @wjayesh in https://github.com/zenml-io/zenml/pull/3149 +* fix icons and remove redundant file by @wjayesh in https://github.com/zenml-io/zenml/pull/3150 +* Merge 0.68.1 release into develop by @schustmi in https://github.com/zenml-io/zenml/pull/3153 +* Allow filtering pipeline runs by stack component by @schustmi in https://github.com/zenml-io/zenml/pull/3142 +* Allow artifact response as step input by @schustmi in https://github.com/zenml-io/zenml/pull/3134 +* Filter component by user name by @schustmi in https://github.com/zenml-io/zenml/pull/3126 +* [docs] Restructure how-to section to make it more readable by @wjayesh in https://github.com/zenml-io/zenml/pull/3147 +* ZenML Pro web login implementation by @stefannica in https://github.com/zenml-io/zenml/pull/3141 +* Scalability improvements: Reduce misc/hydration server requests by @schustmi in https://github.com/zenml-io/zenml/pull/3093 +* Fix in-process main module source loading by @schustmi in https://github.com/zenml-io/zenml/pull/3119 +* Catch assertion in GH library by @schustmi in https://github.com/zenml-io/zenml/pull/3160 +* Enable cache precomputation for run templates by @schustmi in https://github.com/zenml-io/zenml/pull/3156 +* Add LLM and ML engineering books to README by @htahir1 in https://github.com/zenml-io/zenml/pull/3159 +* Add helper method to quickly create run template from pipeline by @schustmi in https://github.com/zenml-io/zenml/pull/3155 +* Add CLI command to export stack requirements by @schustmi in https://github.com/zenml-io/zenml/pull/3158 +* Scalability improvements: Increment artifact version server side by @schustmi in https://github.com/zenml-io/zenml/pull/3095 +* Update OpenAI integration by @safoinme in https://github.com/zenml-io/zenml/pull/3163 +* Remove deprecated torch version constraint by @safoinme in https://github.com/zenml-io/zenml/pull/3166 +* vLLM model deployer by @dudeperf3ct in https://github.com/zenml-io/zenml/pull/3032 +* Don't initialize client during flavor sync by @schustmi in https://github.com/zenml-io/zenml/pull/3168 +* Cleanup materializer temporary directories after step execution by @schustmi in https://github.com/zenml-io/zenml/pull/3162 +* Fix langchain in API docs by @avishniakov in https://github.com/zenml-io/zenml/pull/3171 +* Finetuning guide by @strickvl in https://github.com/zenml-io/zenml/pull/3157 +* Fix mypy issue vllm evidently by @safoinme in https://github.com/zenml-io/zenml/pull/3169 +* Add artifact version batch request by @schustmi in https://github.com/zenml-io/zenml/pull/3164 +* Add missing section links by @strickvl in https://github.com/zenml-io/zenml/pull/3172 +* Fix uvloop mypy by @avishniakov in https://github.com/zenml-io/zenml/pull/3174 +* Multiple output versions for a step outputs by @avishniakov in https://github.com/zenml-io/zenml/pull/3072 +* Simplify Metadata handling by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3096 +* assign value to component_name in preset stack registration by @hirekk in https://github.com/zenml-io/zenml/pull/3178 +* Updating the template versions with `zenml login` by @bcdurak in https://github.com/zenml-io/zenml/pull/3177 +* Better input artifacts typing by @avishniakov in https://github.com/zenml-io/zenml/pull/3099 +* Refactor environment setup and caching by @safoinme in https://github.com/zenml-io/zenml/pull/3077 +* Fix spelling errors by @safoinme in https://github.com/zenml-io/zenml/pull/3181 +* Prevent some race conditions by @schustmi in https://github.com/zenml-io/zenml/pull/3167 +* Update stack deployments with latest features by @stefannica in https://github.com/zenml-io/zenml/pull/3183 +* Terraform best practices by @htahir1 in https://github.com/zenml-io/zenml/pull/3131 +* Fix sagemaker pipeline URLs by @stefannica in https://github.com/zenml-io/zenml/pull/3176 +* Fix lightning orchestrator for multi-step pipelines by @wjayesh in https://github.com/zenml-io/zenml/pull/3170 +* Port bugfixes from #2497 by @avishniakov in https://github.com/zenml-io/zenml/pull/3179 +* Removing the `enable_cache` from the config files by @bcdurak in https://github.com/zenml-io/zenml/pull/3184 +* Don't pass tags to step config by @schustmi in https://github.com/zenml-io/zenml/pull/3186 +* New `log_metadata` function, new `oneof` filtering, additional `run_metadata` filtering by @bcdurak in https://github.com/zenml-io/zenml/pull/3182 + +## New Contributors +* @hirekk made their first contribution in https://github.com/zenml-io/zenml/pull/3178 + +**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.68.1...0.70.0 + + # 0.68.1 Fixes an issue with some partially cached pipelines running on remote orchestrators. diff --git a/examples/quickstart/configs/training_aws.yaml b/examples/quickstart/configs/training_aws.yaml index aaa406ebfa..48e6abd8bf 100644 --- a/examples/quickstart/configs/training_aws.yaml +++ b/examples/quickstart/configs/training_aws.yaml @@ -1,7 +1,7 @@ # Environment configuration settings: docker: - parent_image: "715803424590.dkr.ecr.eu-central-1.amazonaws.com/zenml-public-pipelines:quickstart-0.68.1-py3.11-aws" + parent_image: "715803424590.dkr.ecr.eu-central-1.amazonaws.com/zenml-public-pipelines:quickstart-0.70.0-py3.11-aws" skip_build: True # If you switch this to False remove the parent_image requirements: requirements.txt orchestrator.sagemaker: diff --git a/examples/quickstart/configs/training_azure.yaml b/examples/quickstart/configs/training_azure.yaml index 7a5a583203..b7112016e2 100644 --- a/examples/quickstart/configs/training_azure.yaml +++ b/examples/quickstart/configs/training_azure.yaml @@ -1,7 +1,7 @@ # Environment configuration settings: docker: - parent_image: "zenmldocker/zenml-public-pipelines:quickstart-0.68.1-py3.11-azure" + parent_image: "zenmldocker/zenml-public-pipelines:quickstart-0.70.0-py3.11-azure" skip_build: True requirements: requirements.txt # Uncomment the following lines to specify the accelerator for your azureml orchestrator diff --git a/examples/quickstart/configs/training_gcp.yaml b/examples/quickstart/configs/training_gcp.yaml index 15982e9c24..47b4cce6b2 100644 --- a/examples/quickstart/configs/training_gcp.yaml +++ b/examples/quickstart/configs/training_gcp.yaml @@ -1,7 +1,7 @@ # Environment configuration settings: docker: - parent_image: "zenmldocker/zenml-public-pipelines:quickstart-0.68.1-py3.11-gcp" + parent_image: "zenmldocker/zenml-public-pipelines:quickstart-0.70.0-py3.11-gcp" skip_build: True requirements: requirements.txt # Uncomment the following two lines to specify the accelerator for your vertex orchestrator diff --git a/examples/quickstart/quickstart.ipynb b/examples/quickstart/quickstart.ipynb index 8b3190b81a..067c262afb 100644 --- a/examples/quickstart/quickstart.ipynb +++ b/examples/quickstart/quickstart.ipynb @@ -494,7 +494,7 @@ "# Common imports and setup\n", "if CLOUD_PROVIDER.lower() == \"gcp\":\n", " parent_image = (\n", - " \"zenmldocker/zenml-public-pipelines:quickstart-0.68.1-py3.11-gcp\"\n", + " \"zenmldocker/zenml-public-pipelines:quickstart-0.70.0-py3.11-gcp\"\n", " )\n", " skip_build = True\n", "\n", @@ -503,7 +503,7 @@ " SagemakerOrchestratorSettings,\n", " )\n", "\n", - " parent_image = \"339712793861.dkr.ecr.eu-central-1.amazonaws.com/zenml-public-pipelines:quickstart-0.68.1-py3.11-aws\"\n", + " parent_image = \"339712793861.dkr.ecr.eu-central-1.amazonaws.com/zenml-public-pipelines:quickstart-0.70.0-py3.11-aws\"\n", " skip_build = True # if you switch this to False, you need to remove the parent image\n", "\n", " settings[\"orchestrator.sagemaker\"] = SagemakerOrchestratorSettings(\n", @@ -512,7 +512,7 @@ "\n", "elif CLOUD_PROVIDER.lower() == \"azure\":\n", " parent_image = (\n", - " \"zenmldocker/zenml-public-pipelines:quickstart-0.68.1-py3.11-azure\"\n", + " \"zenmldocker/zenml-public-pipelines:quickstart-0.70.0-py3.11-azure\"\n", " )\n", " skip_build = True\n", "\n", diff --git a/examples/quickstart/requirements.txt b/examples/quickstart/requirements.txt index 87736f9e21..d1f413a694 100644 --- a/examples/quickstart/requirements.txt +++ b/examples/quickstart/requirements.txt @@ -1,4 +1,4 @@ -zenml[server]==0.68.1 +zenml[server]==0.70.0 notebook pyarrow datasets diff --git a/examples/quickstart/requirements_aws.txt b/examples/quickstart/requirements_aws.txt index 72ba104d26..2dbaf5f90a 100644 --- a/examples/quickstart/requirements_aws.txt +++ b/examples/quickstart/requirements_aws.txt @@ -1,4 +1,4 @@ -zenml[server]==0.68.1 +zenml[server]==0.70.0 notebook pyarrow datasets diff --git a/examples/quickstart/requirements_azure.txt b/examples/quickstart/requirements_azure.txt index 816822cbad..c25c0d82c9 100644 --- a/examples/quickstart/requirements_azure.txt +++ b/examples/quickstart/requirements_azure.txt @@ -1,4 +1,4 @@ -zenml[server]==0.68.1 +zenml[server]==0.70.0 notebook pyarrow datasets diff --git a/examples/quickstart/requirements_gcp.txt b/examples/quickstart/requirements_gcp.txt index 92df9e2605..12471da183 100644 --- a/examples/quickstart/requirements_gcp.txt +++ b/examples/quickstart/requirements_gcp.txt @@ -1,4 +1,4 @@ -zenml[server]==0.68.1 +zenml[server]==0.70.0 notebook pyarrow datasets diff --git a/pyproject.toml b/pyproject.toml index 994407ddf7..6da6f36ed2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "zenml" -version = "0.68.1" +version = "0.70.0" packages = [{ include = "zenml", from = "src" }] description = "ZenML: Write production-ready ML code." authors = ["ZenML GmbH "] diff --git a/src/zenml/VERSION b/src/zenml/VERSION index 655291922b..d9bd950076 100644 --- a/src/zenml/VERSION +++ b/src/zenml/VERSION @@ -1 +1 @@ -0.68.1 \ No newline at end of file +0.70.0 \ No newline at end of file diff --git a/src/zenml/zen_server/deploy/helm/Chart.yaml b/src/zenml/zen_server/deploy/helm/Chart.yaml index 118fd16bce..7c899cef38 100644 --- a/src/zenml/zen_server/deploy/helm/Chart.yaml +++ b/src/zenml/zen_server/deploy/helm/Chart.yaml @@ -1,6 +1,6 @@ apiVersion: v2 name: zenml -version: "0.68.1" +version: "0.70.0" description: Open source MLOps framework for portable production ready ML pipelines keywords: - mlops diff --git a/src/zenml/zen_server/deploy/helm/README.md b/src/zenml/zen_server/deploy/helm/README.md index 8641c303c4..569c2fc2a7 100644 --- a/src/zenml/zen_server/deploy/helm/README.md +++ b/src/zenml/zen_server/deploy/helm/README.md @@ -20,8 +20,8 @@ ZenML is an open-source MLOps framework designed to help you create robust, main To install the ZenML chart directly from Amazon ECR, use the following command: ```bash -# example command for version 0.68.1 -helm install my-zenml oci://public.ecr.aws/zenml/zenml --version 0.68.1 +# example command for version 0.70.0 +helm install my-zenml oci://public.ecr.aws/zenml/zenml --version 0.70.0 ``` Note: Ensure you have OCI support enabled in your Helm client and that you are authenticated with Amazon ECR. diff --git a/src/zenml/zen_stores/migrations/versions/0.70.0_release.py b/src/zenml/zen_stores/migrations/versions/0.70.0_release.py new file mode 100644 index 0000000000..b9f977e656 --- /dev/null +++ b/src/zenml/zen_stores/migrations/versions/0.70.0_release.py @@ -0,0 +1,23 @@ +"""Release [0.70.0]. + +Revision ID: 0.70.0 +Revises: 904464ea4041 +Create Date: 2024-11-12 12:20:05.537646 + +""" + +# revision identifiers, used by Alembic. +revision = "0.70.0" +down_revision = "904464ea4041" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + """Upgrade database schema and/or data, creating a new revision.""" + pass + + +def downgrade() -> None: + """Downgrade database schema and/or data back to the previous revision.""" + pass From e5ab382e91567d48a931cc97d1f95ca00ad21ee3 Mon Sep 17 00:00:00 2001 From: Andrei Vishniakov <31008759+avishniakov@users.noreply.github.com> Date: Tue, 12 Nov 2024 13:53:04 +0100 Subject: [PATCH 2/3] updated release notes --- RELEASE_NOTES.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 8f6abb8c9c..aa52ac85ac 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -2,6 +2,23 @@ # 0.70.0 +The **ZenML 0.70.0** release includes a significant number of database schema changes and migrations, which means upgrading to this version will require extra caution. As always, please make sure to make a copy of your production database before upgrading. + +## Key Changes + +* **Artifact Versioning Improvements**: The handling of artifact versions has been improved, including the API improvements like the ability to batch artifact version requests to improve the execution times and more types for the step input/output artifacts, including multiple versions of the same artifact (e.g. model checkpoints), to improve the UX using ZenML UI or while working directly with the API. +* **Scalability Enhancements**: Various scalability improvements have been made, such as reducing unnecessary server requests and incrementing artifact versions server-side. These enhancements are expected to provide significant speed and scale improvements for ZenML users. +* **Metadata management**: Now, all the metadata-creating functions are gathered under one method called `log_metadata`. It is possible to call this method with different inputs to log run metadata for artifact versions, model versions, steps, and runs. +* **The oneof filtering**: This allows to filter entities using a new operator called `oneof`. You can possibly use this with IDs, tags, and all the other attributes like `PipelineRunFilter(tag='oneof:["cats", "dogs"]')`. +* **Documentation Improvements**: The ZenML documentation has been restructured and expanded, including the addition of new sections on [finetuning](https://docs.zenml.io/user-guide/llmops-guide/finetuning-llms) and [LLM/ML engineering](https://docs.zenml.io/user-guide/llmops-guide/evaluation) resources. +* **Bug Fixes**: This release includes several bug fixes, including issues with in-process main module source loading, and more. + +## Caution: Make sure to back up your data before upgrading! +While this release brings many valuable improvements, the database schema changes and migrations pose a potential risk to users. It is strongly recommended that users: + +* **Test the upgrade on a non-production environment**: Before upgrading a production system, test the upgrade process in a non-production environment to identify and address any issues. +* **Back up your data**: Ensure that you have a reliable backup of your ZenML data before attempting the upgrade. + ## What's Changed * Optimizing the CI workflows by @bcdurak in https://github.com/zenml-io/zenml/pull/3145 * Adding 0.68.0 to the migration tests by @bcdurak in https://github.com/zenml-io/zenml/pull/3144 From b481952a7a6fc782a57fb5b38024665769726bf4 Mon Sep 17 00:00:00 2001 From: Andrei Vishniakov <31008759+avishniakov@users.noreply.github.com> Date: Tue, 12 Nov 2024 16:03:38 +0100 Subject: [PATCH 3/3] Update release notes --- RELEASE_NOTES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index aa52ac85ac..2809cc6016 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -9,7 +9,7 @@ The **ZenML 0.70.0** release includes a significant number of database schema ch * **Artifact Versioning Improvements**: The handling of artifact versions has been improved, including the API improvements like the ability to batch artifact version requests to improve the execution times and more types for the step input/output artifacts, including multiple versions of the same artifact (e.g. model checkpoints), to improve the UX using ZenML UI or while working directly with the API. * **Scalability Enhancements**: Various scalability improvements have been made, such as reducing unnecessary server requests and incrementing artifact versions server-side. These enhancements are expected to provide significant speed and scale improvements for ZenML users. * **Metadata management**: Now, all the metadata-creating functions are gathered under one method called `log_metadata`. It is possible to call this method with different inputs to log run metadata for artifact versions, model versions, steps, and runs. -* **The oneof filtering**: This allows to filter entities using a new operator called `oneof`. You can possibly use this with IDs, tags, and all the other attributes like `PipelineRunFilter(tag='oneof:["cats", "dogs"]')`. +* **The oneof filtering**: This allows to filter entities using a new operator called `oneof`. You can use this with IDs (UUID type) or tags (or other string-typed attributes) like this `PipelineRunFilter(tag='oneof:["cats", "dogs"]')`. * **Documentation Improvements**: The ZenML documentation has been restructured and expanded, including the addition of new sections on [finetuning](https://docs.zenml.io/user-guide/llmops-guide/finetuning-llms) and [LLM/ML engineering](https://docs.zenml.io/user-guide/llmops-guide/evaluation) resources. * **Bug Fixes**: This release includes several bug fixes, including issues with in-process main module source loading, and more.