Skip to content

Commit

Permalink
[docs] Fix links in the how-to section of docs (#3196) (#3197)
Browse files Browse the repository at this point in the history
* one batch

* second batch
  • Loading branch information
wjayesh authored Nov 14, 2024
1 parent 496a0d5 commit 539b5bf
Show file tree
Hide file tree
Showing 90 changed files with 364 additions and 364 deletions.
6 changes: 3 additions & 3 deletions docs/book/how-to/advanced-topics/control-logging/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ description: Configuring ZenML's default logging behavior

ZenML produces various kinds of logs:

* The [ZenML Server](../../getting-started/deploying-zenml/README.md) produces server logs (like any FastAPI server).
* The [Client or Runner](../configure-python-environments/README.md#client-environment-or-the-runner-environment) environment produces logs, for example after running a pipeline. These are steps that are typically before, after, and during the creation of a pipeline run.
* The [Execution environment](../configure-python-environments/README.md#execution-environments) (on the orchestrator level) produces logs when it executes each step of a pipeline. These are logs that are typically written in your steps using the python `logging` module.
* The [ZenML Server](../../../getting-started/deploying-zenml/README.md) produces server logs (like any FastAPI server).
* The [Client or Runner](../../infrastructure-deployment/configure-python-environments/README.md#client-environment-or-the-runner-environment) environment produces logs, for example after running a pipeline. These are steps that are typically before, after, and during the creation of a pipeline run.
* The [Execution environment](../../infrastructure-deployment/configure-python-environments/README.md#execution-environments) (on the orchestrator level) produces logs when it executes each step of a pipeline. These are logs that are typically written in your steps using the python `logging` module.

This section talks about how users can control logging behavior in these various environments.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ export ZENML_ENABLE_RICH_TRACEBACK=false

This will ensure that you see only the plain text traceback output.

Note that setting this on the [client environment](../configure-python-environments/README.md#client-environment-or-the-runner-environment) (e.g. your local machine which runs the pipeline) will **not automatically disable rich tracebacks on remote pipeline runs**. That means setting this variable locally with only effect pipelines that run locally.
Note that setting this on the [client environment](../../infrastructure-deployment/configure-python-environments/README.md#client-environment-or-the-runner-environment) (e.g. your local machine which runs the pipeline) will **not automatically disable rich tracebacks on remote pipeline runs**. That means setting this variable locally with only effect pipelines that run locally.

If you wish to disable it also for [remote pipeline runs](../../user-guide/production-guide/cloud-orchestration.md), you can set the `ZENML_ENABLE_RICH_TRACEBACK` environment variable in your pipeline runs environment as follows:
If you wish to disable it also for [remote pipeline runs](../../../user-guide/production-guide/cloud-orchestration.md), you can set the `ZENML_ENABLE_RICH_TRACEBACK` environment variable in your pipeline runs environment as follows:

```python
docker_settings = DockerSettings(environment={"ZENML_ENABLE_RICH_TRACEBACK": "false"})
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ def my_step() -> None:

These logs are stored within the respective artifact store of your stack. You can display the logs in the dashboard as follows:

![Displaying step logs on the dashboard](../../.gitbook/assets/zenml\_step\_logs.png)
![Displaying step logs on the dashboard](../../../.gitbook/assets/zenml_step_logs.png)

{% hint style="warning" %}
Note that if you are not connected to a cloud artifact store with a service connector configured then you will not
Expand All @@ -37,7 +37,7 @@ If you do not want to store the logs in your artifact store, you can:
def my_pipeline():
...
```
2. Disable it by using the environmental variable `ZENML_DISABLE_STEP_LOGS_STORAGE` and setting it to `true`. This environmental variable takes precedence over the parameters mentioned above. Note this environmental variable needs to be set on the [execution environment](../configure-python-environments/README.md#execution-environments), i.e., on the orchestrator level:
2. Disable it by using the environmental variable `ZENML_DISABLE_STEP_LOGS_STORAGE` and setting it to `true`. This environmental variable takes precedence over the parameters mentioned above. Note this environmental variable needs to be set on the [execution environment](../../infrastructure-deployment/configure-python-environments/README.md#execution-environments), i.e., on the orchestrator level:

```python
docker_settings = DockerSettings(environment={"ZENML_DISABLE_STEP_LOGS_STORAGE": "true"})
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ export ZENML_LOGGING_VERBOSITY=INFO
Choose from `INFO`, `WARN`, `ERROR`, `CRITICAL`, `DEBUG`. This will set the logs
to whichever level you suggest.

Note that setting this on the [client environment](../configure-python-environments/README.md#client-environment-or-the-runner-environment) (e.g. your local machine which runs the pipeline) will **not automatically set the same logging verbosity for remote pipeline runs**. That means setting this variable locally with only effect pipelines that run locally.
Note that setting this on the [client environment](../../infrastructure-deployment/configure-python-environments/README.md#client-environment-or-the-runner-environment) (e.g. your local machine which runs the pipeline) will **not automatically set the same logging verbosity for remote pipeline runs**. That means setting this variable locally with only effect pipelines that run locally.

If you wish to control for [remote pipeline runs](../../user-guide/production-guide/cloud-orchestration.md), you can set the `ZENML_LOGGING_VERBOSITY` environment variable in your pipeline runs environment as follows:
If you wish to control for [remote pipeline runs](../../../user-guide/production-guide/cloud-orchestration.md), you can set the `ZENML_LOGGING_VERBOSITY` environment variable in your pipeline runs environment as follows:

```python
docker_settings = DockerSettings(environment={"ZENML_LOGGING_VERBOSITY": "DEBUG"})
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,14 @@ These logs are stored within the respective artifact store of your stack. This m
*if the deployed ZenML server has direct access to the underlying artifact store*. There are two cases in which this will be true:

* In case of a local ZenML server (via `zenml login --local`), both local and remote artifact stores may be accessible, depending on configuration of the client.
* In case of a deployed ZenML server, logs for runs on a [local artifact store](../../component-guide/artifact-stores/local.md) will not be accessible. Logs
for runs using a [remote artifact store](../../user-guide/production-guide/remote-storage.md) **may be** accessible, if the artifact store has been configured
with a [service connector](../auth-management/service-connectors-guide.md). Please read [this chapter](../../user-guide/production-guide/remote-storage.md) of
* In case of a deployed ZenML server, logs for runs on a [local artifact store](../../../component-guide/artifact-stores/local.md) will not be accessible. Logs
for runs using a [remote artifact store](../../../user-guide/production-guide/remote-storage.md) **may be** accessible, if the artifact store has been configured
with a [service connector](../../infrastructure-deployment/auth-management/service-connectors-guide.md). Please read [this chapter](../../../user-guide/production-guide/remote-storage.md) of
the production guide to learn how to configure a remote artifact store with a service connector.

If configured correctly, the logs are displayed in the dashboard as follows:

![Displaying step logs on the dashboard](../../.gitbook/assets/zenml\_step\_logs.png)
![Displaying step logs on the dashboard](../../../.gitbook/assets/zenml_step_logs.png)

{% hint style="warning" %}
If you do not want to store the logs for your pipeline (for example due to performance reduction or storage limits),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,16 @@ Follow the tips below while upgrading your server to mitigate data losses, downt
- **Database Backup**: Before upgrading, create a backup of your MySQL database. This allows you to rollback if necessary.
- **Automated Backups**: Consider setting up automatic daily backups of your database for added security. Most managed services like AWS RDS, Google Cloud SQL, and Azure Database for MySQL offer automated backup options.

![Screenshot of backups in AWS RDS](../../.gitbook/assets/aws-rds-backups.png)
![Screenshot of backups in AWS RDS](../../../.gitbook/assets/aws-rds-backups.png)

### Upgrade Strategies

- **Staged Upgrade**: For large organizations or critical systems, consider using two ZenML server instances (old and new) and migrating services one by one to the new version.

![Server Migration Step 1](../../.gitbook/assets/server_migration_1.png)
![Server Migration Step 1](../../../.gitbook/assets/server_migration_1.png)


![Server Migration Step 2](../../.gitbook/assets/server_migration_2.png)
![Server Migration Step 2](../../../.gitbook/assets/server_migration_2.png)

- **Team Coordination**: If multiple teams share a ZenML server instance, coordinate the upgrade timing to minimize disruption.
- **Separate ZenML Servers**: Coordination between teams might be difficult if one team requires new features but the other can't upgrade yet. In such cases, it is recommended to use dedicated ZenML server instances per team or product to allow for more flexible upgrade schedules.
Expand All @@ -48,7 +48,7 @@ Sometimes, you might have to upgrade your code to work with a new version of Zen

- **Local Testing**: It's a good idea to test it locally first after you upgrade (`pip install zenml --upgrade`) and run some old pipelines to check for compatibility issues between the old and new versions.
- **End-to-End Testing**: You can also develop simple end-to-end tests to ensure that the new version works with your pipeline code and your stack. ZenML already has an [extensive test suite](https://github.com/zenml-io/zenml/tree/main/tests) that we use for releases and you can use it as an example.
- **Artifact Compatibility**: Be cautious with pickle-based [materializers](../../how-to/data-artifact-management/handle-data-artifacts/handle-custom-data-types.md), as they can be sensitive to changes in Python versions or libraries. Consider using version-agnostic materialization methods for critical artifacts. You can try to load older artifacts with the new version of ZenML to see if they are compatible. Every artifact has an ID which you can use to load it in the following way:
- **Artifact Compatibility**: Be cautious with pickle-based [materializers](../../../how-to/data-artifact-management/handle-data-artifacts/handle-custom-data-types.md), as they can be sensitive to changes in Python versions or libraries. Consider using version-agnostic materialization methods for critical artifacts. You can try to load older artifacts with the new version of ZenML to see if they are compatible. Every artifact has an ID which you can use to load it in the following way:

```python
from zenml.client import Client
Expand All @@ -59,7 +59,7 @@ loaded_artifact = artifact.load()

### Dependency Management

- **Python Version**: Make sure that the Python version you are using is compatible with the ZenML version you are upgrading to. Check out the [installation guide](../../getting-started/installation.md) to find out which Python version is supported.
- **Python Version**: Make sure that the Python version you are using is compatible with the ZenML version you are upgrading to. Check out the [installation guide](../../../getting-started/installation.md) to find out which Python version is supported.
- **External Dependencies**: Be mindful of external dependencies (e.g. from integrations) that might be incompatible with the new version of ZenML. This could be the case when some older versions are no longer supported or maintained and the ZenML integration is updated to use a newer version. You can find this information in the [release notes](https://github.com/zenml-io/zenml/releases) for the new version of ZenML.

### Handling API Changes
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,7 @@ my_pipeline()
{% endtab %}
{% endtabs %}

Check out [this page](../../how-to/pipeline-development/build-pipelines/schedule-a-pipeline.md) for more information on how to schedule your pipelines.
Check out [this page](../../pipeline-development/build-pipelines/schedule-a-pipeline.md) for more information on how to schedule your pipelines.

## Fetching pipelines after execution

Expand Down Expand Up @@ -353,7 +353,7 @@ loaded_model = model.load()
{% endtab %}
{% endtabs %}

Check out [this page](../../how-to/model-management-metrics/track-metrics-metadata/fetch-metadata-within-steps.md) for more information on how to programmatically fetch information about previous pipeline runs.
Check out [this page](../../../model-management-metrics/track-metrics-metadata/fetch-metadata-within-steps.md) for more information on how to programmatically fetch information about previous pipeline runs.

## Controlling the step execution order

Expand Down Expand Up @@ -385,7 +385,7 @@ def my_pipeline():
{% endtab %}
{% endtabs %}

Check out [this page](../../how-to/pipeline-development/build-pipelines/control-execution-order-of-steps.md) for more information on how to control the step execution order.
Check out [this page](../../../pipeline-development/build-pipelines/control-execution-order-of-steps.md) for more information on how to control the step execution order.

## Defining steps with multiple outputs

Expand Down Expand Up @@ -424,7 +424,7 @@ def my_step() -> Tuple[
{% endtab %}
{% endtabs %}

Check out [this page](../../how-to/pipeline-development/build-pipelines/step-output-typing-and-annotation.md) for more information on how to annotate your step outputs.
Check out [this page](../../../pipeline-development/build-pipelines/step-output-typing-and-annotation.md) for more information on how to annotate your step outputs.

## Accessing run information inside steps

Expand Down Expand Up @@ -457,6 +457,6 @@ def my_step() -> Any: # New: StepContext is no longer an argument of the step
{% endtab %}
{% endtabs %}

Check out [this page](../../how-to/model-management-metrics/track-metrics-metadata/fetch-metadata-within-steps.md) for more information on how to fetch run information inside your steps using `get_step_context()`.
Check out [this page](../../../model-management-metrics/track-metrics-metadata/fetch-metadata-within-steps.md) for more information on how to fetch run information inside your steps using `get_step_context()`.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ is still using `sqlalchemy` v1 and is incompatible with pydantic v2. As a
solution, we have removed the dependencies of the `airflow` integration. Now,
you can use ZenML to create your Airflow pipelines and use a separate
environment to run them with Airflow. You can check the updated docs
[right here](../../component-guide/orchestrators/airflow.md).
[right here](../../../../component-guide/orchestrators/airflow.md).

### AWS

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,11 @@ High-level overview of the changes:

## ZenML takes over the Metadata Store role

ZenML can now run [as a server](../../user-guide/getting-started/core-concepts.md#zenml-server-and-dashboard) that can be accessed via a REST API and also comes with a visual user interface (called the ZenML Dashboard). This server can be deployed in arbitrary environments (local, on-prem, via Docker, on AWS, GCP, Azure etc.) and supports user management, workspace scoping, and more.
ZenML can now run [as a server](../../../../getting-started/core-concepts.md#zenml-server-and-dashboard) that can be accessed via a REST API and also comes with a visual user interface (called the ZenML Dashboard). This server can be deployed in arbitrary environments (local, on-prem, via Docker, on AWS, GCP, Azure etc.) and supports user management, workspace scoping, and more.

The release introduces a series of commands to facilitate managing the lifecycle of the ZenML server and to access the pipeline and pipeline run information:

* `zenml connect / disconnect / down / up / logs / status` can be used to configure your client to connect to a ZenML server, to start a local ZenML Dashboard or to deploy a ZenML server to a cloud environment. For more information on how to use these commands, see [the ZenML deployment documentation](../../user-guide/getting-started/deploying-zenml/deploying-zenml.md).
* `zenml connect / disconnect / down / up / logs / status` can be used to configure your client to connect to a ZenML server, to start a local ZenML Dashboard or to deploy a ZenML server to a cloud environment. For more information on how to use these commands, see [the ZenML deployment documentation](../../../../user-guide/production-guide/deploying-zenml.md).
* `zenml pipeline list / runs / delete` can be used to display information and about and manage your pipelines and pipeline runs.

In ZenML 0.13.2 and earlier versions, information about pipelines and pipeline runs used to be stored in a separate stack component called the Metadata Store. Starting with 0.20.0, the role of the Metadata Store is now taken over by ZenML itself. This means that the Metadata Store is no longer a separate component in the ZenML architecture, but rather a part of the ZenML core, located wherever ZenML is deployed: locally on your machine or running remotely as a server.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ To scale your ZenML server deployed as a service on ECS, you can follow the step
- If you scroll down, you will see the "Service auto scaling - optional" section.
- Here you can enable autoscaling and set the minimum and maximum number of tasks to run for your service and also the ECS service metric to use for scaling.
![Image showing autoscaling settings for a service](../../.gitbook/assets/ecs_autoscaling.png)
![Image showing autoscaling settings for a service](../../../.gitbook/assets/ecs_autoscaling.png)
{% endtab %}
Expand All @@ -60,7 +60,7 @@ To scale your ZenML server deployed on Cloud Run, you can follow the steps below
- Scroll down to the "Revision auto-scaling" section.
- Here you can set the minimum and maximum number of instances to run for your service.
![Image showing autoscaling settings for a service](../../.gitbook/assets/cloudrun_autoscaling.png)
![Image showing autoscaling settings for a service](../../../.gitbook/assets/cloudrun_autoscaling.png)
{% endtab %}
{% tab title="Docker Compose" %}
Expand Down Expand Up @@ -159,7 +159,7 @@ sum by(namespace) (rate(container_cpu_usage_seconds_total{namespace=~"zenml.*"}[

This query would give you the CPU utilization of your server pods in all namespaces that start with `zenml`. The image below shows how this query would look like in Grafana.

![Image showing CPU utilization of ZenML server pods](../../.gitbook/assets/grafana_dashboard.png)
![Image showing CPU utilization of ZenML server pods](../../../.gitbook/assets/grafana_dashboard.png)


{% endtab %}
Expand All @@ -168,15 +168,15 @@ On ECS, you can utilize the [CloudWatch integration](https://docs.aws.amazon.com

In the "Health and metrics" section of your ECS console, you should see metrics pertaining to your ZenML service like CPU utilization and Memory utilization.

![Image showing CPU utilization ECS](../../.gitbook/assets/ecs_cpu_utilization.png)
![Image showing CPU utilization ECS](../../../.gitbook/assets/ecs_cpu_utilization.png)

{% endtab %}
{% tab title="Cloud Run" %}
In Cloud Run, you can utilize the [Cloud Monitoring integration](https://cloud.google.com/run/docs/monitoring) to monitor your ZenML server.

The "Metrics" tab in the Cloud Run console will show you metrics like Container CPU utilization, Container memory utilization, and more.

![Image showing metrics in Cloud Run](../../.gitbook/assets/cloudrun_metrics.png)
![Image showing metrics in Cloud Run](../../../.gitbook/assets/cloudrun_metrics.png)

{% endtab %}
{% endtabs %}
Expand Down
Loading

0 comments on commit 539b5bf

Please sign in to comment.