Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pointed docs links to the new docs #110

Merged
merged 2 commits into from
Jun 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ the Apache License Version 2.0.
| 🗺 **[Roadmap]** | See where ZenML is working to build new features. |
| 🙋‍♀️ **[Contribute]** | How to contribute to the ZenML project and code base. |

[ZenML 101]: https://docs.zenml.io/
[ZenML 101]: https://docs.zenml.io/user-guide/starter-guide
[Core Concepts]: https://docs.zenml.io/getting-started/core-concepts
[Our latest release]: https://github.com/zenml-io/zenml/releases
[Vote for Features]: https://zenml.io/discussion
Expand Down
2 changes: 1 addition & 1 deletion classifier-e2e/run_skip_basics.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@
"\u001b[0;34m\u001b[0m \u001b[0mtarget\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"target\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
"\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mAnnotated\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\u001b[0m\n",
"\u001b[0;34m\u001b[0m \u001b[0mClassifierMixin\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
"\u001b[0;34m\u001b[0m \u001b[0mArtifactConfig\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"breast_cancer_classifier\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_model_artifact\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",

Check warning on line 152 in classifier-e2e/run_skip_basics.ipynb

View workflow job for this annotation

GitHub Actions / spell-check

"mis" should be "miss" or "mist".
"\u001b[0;34m\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
"\u001b[0;34m\u001b[0m \u001b[0;34m\"\"\"Configure and train a model on the training dataset.\u001b[0m\n",
"\u001b[0;34m\u001b[0m\n",
Expand Down Expand Up @@ -283,7 +283,7 @@
"\u001b[0;34m\u001b[0m \u001b[0;31m# of one step as the input of the next step.\u001b[0m\u001b[0;34m\u001b[0m\n",
"\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
"\u001b[0;34m\u001b[0m \u001b[0;31m# Execute Feature Engineering Pipeline\u001b[0m\u001b[0;34m\u001b[0m\n",
"\u001b[0;34m\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mtrain_dataset_id\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mtest_dataset_id\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",

Check warning on line 286 in classifier-e2e/run_skip_basics.ipynb

View workflow job for this annotation

GitHub Actions / spell-check

"mis" should be "miss" or "mist".

Check warning on line 286 in classifier-e2e/run_skip_basics.ipynb

View workflow job for this annotation

GitHub Actions / spell-check

"mor" should be "more".
"\u001b[0;34m\u001b[0m \u001b[0mdataset_trn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdataset_tst\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfeature_engineering\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrandom_state\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrandom_state\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
"\u001b[0;34m\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
"\u001b[0;34m\u001b[0m \u001b[0mclient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mClient\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
Expand Down Expand Up @@ -597,7 +597,7 @@
"\u001b[0;34m In this example, the step can be configured to use different input data.\u001b[0m\n",
"\u001b[0;34m See the documentation for more information:\u001b[0m\n",
"\u001b[0;34m\u001b[0m\n",
"\u001b[0;34m https://docs.zenml.io/user-guide/advanced-guide/configure-steps-pipelines\u001b[0m\n",
"\u001b[0;34m https://docs.zenml.io/how-to/build-pipelines\u001b[0m\n",
"\u001b[0;34m\u001b[0m\n",
"\u001b[0;34m Args:\u001b[0m\n",
"\u001b[0;34m model: Trained model.\u001b[0m\n",
Expand Down
6 changes: 3 additions & 3 deletions customer-churn/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ We showcase two solutions to this problem:

## Deploy pipelines to production using orchestrator Pipelines

We will be using ZenML's [Kubeflow](https://docs.zenml.io/stacks-and-components/component-guide/orchestrators/kubeflow) integration to deploy pipelines to production using Kubeflow Pipelines on the cloud.
We will be using ZenML's [Kubeflow](https://docs.zenml.io/stack-components/orchestrators/kubeflow) integration to deploy pipelines to production using Kubeflow Pipelines on the cloud.

Our training pipeline `run_kubeflow_pipeline.py` will be built using the following steps:

Expand Down Expand Up @@ -154,11 +154,11 @@ Seldon Core. The following diagram shows the flow of the whole pipeline:

## Continuous model deployment with Seldon Core

While building the real-world workflow for predicting whether a customer will churn or not, you might not want to train the model once and deploy it to production. Instead, you might want to train the model and deploy it to production when something gets triggered. This is where one of our recent integrations is valuable: [Seldon Core](https://docs.zenml.io/stacks-and-components/component-guide/model-deployers/seldon).
While building the real-world workflow for predicting whether a customer will churn or not, you might not want to train the model once and deploy it to production. Instead, you might want to train the model and deploy it to production when something gets triggered. This is where one of our recent integrations is valuable: [Seldon Core](https://docs.zenml.io/stack-components/model-deployers/seldon).

[Seldon Core](https://github.com/SeldonIO/seldon-core) is a production-grade open-source model serving platform. It packs a wide range of features built around deploying models to REST/GRPC microservices, including monitoring and logging, model explainers, outlier detectors, and various continuous deployment strategies such as A/B testing and canary deployments, and more.

In this project, we build a continuous deployment pipeline that trains a model and then serves it with Seldon Core as the industry-ready model deployment tool of choice. If you are interested in learning more about Seldon Core, you can check out our [docs](https://docs.zenml.io/stacks-and-components/component-guide/model-deployers/seldon). The following diagram shows the flow of the whole pipeline:
In this project, we build a continuous deployment pipeline that trains a model and then serves it with Seldon Core as the industry-ready model deployment tool of choice. If you are interested in learning more about Seldon Core, you can check out our [docs](https://docs.zenml.io/stack-components/model-deployers/seldon). The following diagram shows the flow of the whole pipeline:
![seldondeployment](_assets/seldoncondeploy.gif)


Expand Down
4 changes: 2 additions & 2 deletions customer-satisfaction/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ model and model parameters, and prediction outputs. ZenML helps us to build such
a pipeline in a simple, yet powerful, way.

In this Project, we give special consideration to
the [MLflow integration](https://docs.zenml.io/stacks-and-components/component-guide/model-deployers/mlflow)
the [MLflow integration](https://docs.zenml.io/stack-components/experiment-trackers/mlflow)
of ZenML. In particular, we utilize MLflow tracking to track our metrics and
parameters, and MLflow deployment to deploy our model. We also
use [Streamlit](https://streamlit.io/) to showcase how this model will be used
Expand Down Expand Up @@ -178,7 +178,7 @@ service.predict(...) # Predict on incoming data from the application

While this ZenML Project trains and deploys a model locally, other ZenML
integrations such as
the [Seldon](https://docs.zenml.io/stacks-and-components/component-guide/model-deployers/seldon)
the [Seldon](https://docs.zenml.io/stack-components/model-deployers/seldon)
deployer can also be used in a similar manner to deploy the model in a more
production setting (such as on a Kubernetes cluster). We use MLflow here for the
convenience of its local deployment.
Expand Down
12 changes: 6 additions & 6 deletions end-to-end-computer-vision/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,31 +66,31 @@ providers.
### Follow our guide to set up your credential for GCP

[Set up a GCP service
connector](https://docs.zenml.io/stacks-and-components/auth-management/gcp-service-connector)
connector](https://docs.zenml.io/how-to/auth-management/gcp-service-connector)

### Set up a bucket to persist your training data

### Set up a bucket to use as artifact store within ZenML

[Learn how to set up a GCP artifact store stack component within zenml
here](https://docs.zenml.io/stacks-and-components/component-guide/artifact-stores)
here](https://docs.zenml.io/stack-components/artifact-stores)
### Set up vertex for pipeline orchestration

[Learn how to set up a Vertex orchestrator stack component within zenml
here](https://docs.zenml.io/stacks-and-components/component-guide/orchestrators/vertex)
here](https://docs.zenml.io/stack-components/orchestrators/vertex)
### For training on accelerators like GPUs/TPUs set up Vertex

[Learn how to set up a Vertex step operator stack component within zenml
here](https://docs.zenml.io/stacks-and-components/component-guide/step-operators/vertex)
here](https://docs.zenml.io/stack-components/step-operators/vertex)
### Set up Container Registry

[Learn how to set up a google cloud container registry component within zenml
here](https://docs.zenml.io/stacks-and-components/component-guide/container-registries/gcp)
here](https://docs.zenml.io/stack-components/container-registries/gcp)

## Label Studio

### [Start Label Studio locally](https://labelstud.io/guide/start)
### [Follow these ZenML instructions to set up Label Studio as a stack component](https://docs.zenml.io/stacks-and-components/component-guide/annotators/label-studio)
### [Follow these ZenML instructions to set up Label Studio as a stack component](https://docs.zenml.io/stack-components/annotators/label-studio)
### Create a project within Label Studio and name it `ship_detection_gcp`
### [Set up Label Studio to use external storage](https://labelstud.io/guide/storage)
use the first bucket that you created to data persistence
Expand Down
2 changes: 1 addition & 1 deletion label_studio_annotation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -340,7 +340,7 @@ the IAM rules and the S3 bucket are what you need to delete.

If you want to learn more about annotation in general or about how to use your
own annotation tool in ZenML check out our
[docs](https://docs.zenml.io/stacks-and-components/component-guide/annotators/annotators).
[docs](https://docs.zenml.io/stack-components/annotators/annotators).

To explore other available CLI commands related to the annotator stack
components, check out the [CLI docs annotator
Expand Down
8 changes: 4 additions & 4 deletions llm-agents/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,13 +101,13 @@ artifacts for your own data, you can change values as appropriate.
## Running it on GCP

It is much more ideal to run a pipeline like the agent creation pipeline on a regular schedule. In order to achieve that,
you have to [deploy ZenML](https://docs.zenml.io/user-guide/starter-guide/switch-to-production)
you have to [deploy ZenML](https://docs.zenml.io/user-guide/production-guide/deploying-zenml)
and set up a stack that supports
[our scheduling
feature](https://docs.zenml.io/user-guide/advanced-guide/pipelining-features/schedule-pipeline-runs). If you
feature](https://docs.zenml.io/how-to/build-pipelines/schedule-a-pipeline). If you
wish to deploy the slack bot on GCP Cloud Run as described above, you'll also
need to be using [a Google Cloud Storage Artifact
Store](https://docs.zenml.io/stacks-and-components/component-guide/artifact-stores/gcp). Note that
Store](https://docs.zenml.io/stack-components/artifact-stores/gcp). Note that
certain code artifacts like the `Dockerfile` for this project will also need to
be adapted for your own particular needs and requirements. Please check [our docs](https://docs.zenml.io/user-guide/starter-guide/follow-best-practices)
be adapted for your own particular needs and requirements. Please check [our docs](https://docs.zenml.io/how-to/setting-up-a-project-repository/best-practices)
for more information.
10 changes: 5 additions & 5 deletions llm-agents/run.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -476,13 +476,13 @@
"metadata": {},
"source": [
"#### Stack\n",
"A [stack](https://docs.zenml.io/user-guide/starter-guide/understand-stacks) in ZenML is the combination of tools and infrastructure that your pipelines can run on. When you run ZenML code without configuring a stack, the pipeline will run on the so-called default stack.\n",
"A [stack](https://docs.zenml.io/user-guide/production-guide/understand-stacks) in ZenML is the combination of tools and infrastructure that your pipelines can run on. When you run ZenML code without configuring a stack, the pipeline will run on the so-called default stack.\n",
"\n",
"![image.png](attachment:image.png)\n",
"\n",
"A stack allows ZenML to separate code from infrastructure. This means, that it is easy to switch the environment that the pipeline runs on without making changes to the code. \n",
"\n",
"Let's take a look at the current active stack. You will find that you are running a default stack which comes with a local [orchestrator](https://docs.zenml.io/stacks-and-components/component-guide/orchestrators) and a local [artifact store](https://docs.zenml.io/stacks-and-components/component-guide/artifact-stores). Discover more about stack components [here](https://docs.zenml.io/stacks-and-components/component-guide)."
"Let's take a look at the current active stack. You will find that you are running a default stack which comes with a local [orchestrator](https://docs.zenml.io/stack-components/orchestrators) and a local [artifact store](https://docs.zenml.io/stack-components/artifact-stores). Discover more about stack components [here](https://docs.zenml.io/stack-components)."
]
},
{
Expand Down Expand Up @@ -821,7 +821,7 @@
"\n",
"#### Custom Materializer\n",
"\n",
"You may notice that we have specified an `output_materializer` in the step decorator. A [materializer](https://docs.zenml.io/user-guide/advanced-guide/artifact-management/handle-custom-data-types) in ZenML is what defines how an object is serialized and deserialized while moving between steps. ZenML comes with a host of built-in materializers that include basic types, collections and pydantic objects, along with a default cloudpickle materializer. \n",
"You may notice that we have specified an `output_materializer` in the step decorator. A [materializer](https://docs.zenml.io/how-to/handle-data-artifacts/handle-custom-data-types) in ZenML is what defines how an object is serialized and deserialized while moving between steps. ZenML comes with a host of built-in materializers that include basic types, collections and pydantic objects, along with a default cloudpickle materializer. \n",
"\n",
"You can also define your own custom materializers. Here, we are using a custom materializer (defined in the agent/ directory) for the AgentExecutor type, which is a pydantic object that doesn't have a `.json()` implementation. \n",
"Instead of passing it in the decorator, you can also simply specify the data type a materializer can handle while defining it and ZenML will pick it up automatically and match any relevant output types to it."
Expand All @@ -833,7 +833,7 @@
"source": [
"### Creating a ZenML pipeline\n",
"\n",
"A [pipeline](https://docs.zenml.io/user-guide/starter-guide/create-an-ml-pipeline) is where you chain all the steps together. The order of execution is determined by the relationship between outputs and inputs across steps. You can also choose to add explicit dependencies using `.after()` and `.before()`.\n",
"A [pipeline](https://docs.zenml.io/how-to/build-pipelines) is where you chain all the steps together. The order of execution is determined by the relationship between outputs and inputs across steps. You can also choose to add explicit dependencies using `.after()` and `.before()`.\n",
"\n",
"The output of each step is tracked and versioned in the artifact store that you configure in your ZenML stack (local, by default). ZenML does the heavy-lifting of making sure that the right data is available to the step that needs it."
]
Expand Down Expand Up @@ -1092,7 +1092,7 @@
"\n",
"If you are satisfied with the performance of your agent, you can promote it to production with a simple function call. This will attach the `production` tag to your model and it can then be retrieved and consumed directly or in other pipelines using this tag.\n",
"\n",
"You can also visit the dashboard to see the model in the production stage. Learn more about [Stages and Promotion](https://docs.zenml.io/user-guide/advanced-guide/data-management/model-management#stages-and-promotion) in our docs."
"You can also visit the dashboard to see the model in the production stage. Learn more about [Stages and Promotion](https://docs.zenml.io/how-to/use-the-model-control-plane/promote-a-model) in our docs."
]
},
{
Expand Down
11 changes: 5 additions & 6 deletions llm-complete-guide/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,10 +125,9 @@ the database to evaluate.

The basic RAG pipeline will run using a local stack, but if you want to improve
the speed of the embeddings step you might want to consider using a cloud
orchestrator. Please follow the instructions in [our basic cloud setup
guides](https://docs.zenml.io/user-guide/cloud-guide) (currently available for
[AWS](https://docs.zenml.io/user-guide/cloud-guide/aws-guide) and
[GCP](https://docs.zenml.io/user-guide/cloud-guide/gcp-guide)) to learn how you
orchestrator. Please follow the instructions in [documentation on popular integrations](https://docs.zenml.io/how-to/popular-integrations) (currently available for
[AWS](https://docs.zenml.io/how-to/popular-integrations/aws-guide) and
[GCP](https://docs.zenml.io/how-to/popular-integrations/gcp-guide)) to learn how you
can run the pipelines on a remote stack.

If you run the pipeline using a cloud artifact store, logs from all the steps as
Expand All @@ -142,11 +141,11 @@ you can try out the platform without any cost. Visit the [ZenML Cloud
dashboard](https://cloud.zenml.io/) to get started.

You can also self-host the ZenML dashboard. Instructions are available in our
[documentation](https://docs.zenml.io/deploying-zenml/zenml-self-hosted).
[documentation](https://docs.zenml.io/getting-started/deploying-zenml).

## 📜 Project Structure

The project loosely follows [the recommended ZenML project structure](https://docs.zenml.io/user-guide/starter-guide/follow-best-practices):
The project loosely follows [the recommended ZenML project structure](https://docs.zenml.io/how-to/setting-up-a-project-repository/best-practices):

```
.
Expand Down
2 changes: 1 addition & 1 deletion llm-complete-guide/tests/test_url_scraping_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
"user-guide",
),
(
"https://docs.zenml.io/v/0.56.1/stacks-and-components/component-guide/integration-overview",
"https://docs.zenml.io/v/0.56.1/stack-components/integration-overview",
"stacks-and-components",
),
],
Expand Down
4 changes: 2 additions & 2 deletions llm-finetuning/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ One of the first jobs of somebody entering MLOps is to convert their manual scri
2. Type annotating the steps properly
3. Connecting the steps together in a pipeline
4. Creating the appropriate YAML files to [configure your pipeline](https://docs.zenml.io/user-guide/production-guide/configure-pipeline)
5. Developing a Dockerfile or equivalent to encapsulate [the environment](https://docs.zenml.io/user-guide/advanced-guide/infrastructure-management/containerize-your-pipeline).
5. Developing a Dockerfile or equivalent to encapsulate [the environment](https://docs.zenml.io/how-to/customize-docker-builds).

Frameworks like [ZenML](https://github.com/zenml-io/zenml) go a long way in alleviating this burden by abstracting much of the complexity away. However, recent advancement in Large Language Model based Copilots offer hope that even more repetitive aspects of this task can be automated.

Expand Down Expand Up @@ -154,7 +154,7 @@ This project recently did a [call of volunteers](https://www.linkedin.com/feed/u
- [ ] Create a Dockerfile for the training pipeline with all requirements installed including ZenML, torch, CUDA etc. CUrrently I am having trouble creating this in this [config file](configs/finetune_local.yaml). Probably might make sense to create a docker imag with the right CUDA and requirements including ZenML. See here: https://sdkdocs.zenml.io/0.54.0/integration_code_docs/integrations-aws/#zenml.integrations.aws.flavors.sagemaker_step_operator_flavor.SagemakerStepOperatorSettings

- [ ] Tests trained model on various metrics
- [ ] Create a custom [model deployer](https://docs.zenml.io/stacks-and-components/component-guide/model-deployers) that deploys a huggingface model from the hub to a huggingface inference endpoint. This would involve creating a [custom model deployer](https://docs.zenml.io/stacks-and-components/component-guide/model-deployers/custom) and editing the [deployment pipeline accordingly](pipelines/deployment.py)
- [ ] Create a custom [model deployer](https://docs.zenml.io/stack-components/model-deployers) that deploys a huggingface model from the hub to a huggingface inference endpoint. This would involve creating a [custom model deployer](https://docs.zenml.io/stack-components/model-deployers/custom) and editing the [deployment pipeline accordingly](pipelines/deployment.py)

## :bulb: More Applications

Expand Down
Loading
Loading