Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Made changes to enterprisesify the e2e template #3213

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 6 additions & 8 deletions examples/e2e/.copier-answers.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,13 @@
_commit: 2024.10.30
_src_path: gh:zenml-io/template-e2e-batch
data_quality_checks: true
email: [email protected]
full_name: ZenML GmbH
hyperparameters_tuning: true
hyperparameters_tuning: false
metric_compare_promotion: true
notify_on_failures: true
notify_on_failures: false
notify_on_successes: false
open_source_license: apache
product_name: e2e_use_case
project_name: ZenML E2E project
target_environment: staging
open_source_license: none
product_name: supply_chain_forecasting
project_name: Supply Chain Forecasting
target_environment: production
version: 0.0.1
zenml_server_url: ''
15 changes: 0 additions & 15 deletions examples/e2e/LICENSE
Original file line number Diff line number Diff line change
@@ -1,15 +0,0 @@
Apache Software License 2.0

Copyright (c) ZenML GmbH 2024. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
20 changes: 10 additions & 10 deletions examples/e2e/README.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
# ZenML E2E project
# Supply Chain Forecasting

This is a comprehensive supervised ML project built with the
ZenML framework and its integration. The project trains one or more
scikit-learn classification models to make predictions on the tabular
classification datasets provided by the scikit-learn library. The project was
generated from the [E2E Batch ZenML project template](https://github.com/zenml-io/template-e2e-batch)
with the following properties:
- Project name: ZenML E2E project
- Technical Name: e2e_use_case
- Project name: Supply Chain Forecasting
- Technical Name: supply_chain_forecasting
- Version: `0.0.1`
- Licensed with apache to ZenML GmbH<[email protected]>
- Deployment environment: `staging`
- Licensed with none to ZenML GmbH<>
- Deployment environment: `production`

Settings of your project are:
- Hyperparameters and model architecture tuning using configuration from `config.py`
- Trained model promotion to `staging` based on accuracy metric vs currently deployed model
- Fixed model architecture defined in `config.py`
- Trained model promotion to `production` based on accuracy metric vs currently deployed model
- Data drift checks based on Evidently report
- Notifications about failures enabled
- All notifications disabled

> [!NOTE]
> If you are coming from [our production guide](https://docs.zenml.io/user-guide/production-guide),
Expand All @@ -30,7 +30,7 @@ Settings of your project are:

## 👋 Introduction

Welcome to your newly generated "ZenML E2E project" project! This is
Welcome to your newly generated "Supply Chain Forecasting" project! This is
a great way to get hands-on with ZenML using production-like template.
The project contains a collection of standard and custom ZenML steps,
pipelines and other artifacts and useful resources that can serve as a
Expand Down Expand Up @@ -80,7 +80,7 @@ what you can do with ZenML.

## 📦 What's in the box?

The ZenML E2E project project demonstrates how the most important steps of
The Supply Chain Forecasting project demonstrates how the most important steps of
the ML Production Lifecycle can be implemented in a reusable way remaining
agnostic to the underlying infrastructure, and how to integrate them together
into pipelines serving Training and Batch Inference purposes.
Expand Down
19 changes: 2 additions & 17 deletions examples/e2e/configs/deployer_config.yaml
Original file line number Diff line number Diff line change
@@ -1,18 +1,3 @@
# Apache Software License 2.0
#
# Copyright (c) ZenML GmbH 2024. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# environment configuration
Expand All @@ -35,8 +20,8 @@ steps:

# configuration of the Model Control Plane
model:
name: e2e_use_case
version: staging
name: supply_chain_forecasting
version: production

# pipeline level extra configurations
extra:
Expand Down
19 changes: 2 additions & 17 deletions examples/e2e/configs/inference_config.yaml
Original file line number Diff line number Diff line change
@@ -1,18 +1,3 @@
# Apache Software License 2.0
#
# Copyright (c) ZenML GmbH 2024. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# environment configuration
Expand All @@ -35,8 +20,8 @@ steps:

# configuration of the Model Control Plane
model:
name: e2e_use_case
version: staging
name: supply_chain_forecasting
version: production

# pipeline level extra configurations
extra:
Expand Down
116 changes: 40 additions & 76 deletions examples/e2e/configs/train_config.yaml
Original file line number Diff line number Diff line change
@@ -1,18 +1,3 @@
# Apache Software License 2.0
#
# Copyright (c) ZenML GmbH 2024. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# environment configuration
Expand All @@ -31,80 +16,59 @@ settings:
steps:
model_trainer:
parameters:
name: e2e_use_case
name: supply_chain_forecasting
promote_with_metric_compare:
parameters:
mlflow_model_name: e2e_use_case
mlflow_model_name: supply_chain_forecasting
notify_on_success:
parameters:
notify_on_success: False

# configuration of the Model Control Plane
model:
name: e2e_use_case
license: apache
description: e2e_use_case E2E Batch Use Case
audience: All ZenML users
name: supply_chain_forecasting
license: Proprietary
description: |
Advanced forecasting model designed to optimize supply chain operations for jet engine manufacturing.
The model predicts demand, lead times, and inventory requirements for critical jet engine components
and raw materials, helping maintain optimal production schedules while minimizing inventory costs.
audience: |
- Supply Chain Managers
- Procurement Teams
- Production Planners
- Aerospace Manufacturing Operations
use_cases: |
The ZenML E2E project project demonstrates how the most important steps of
the ML Production Lifecycle can be implemented in a reusable way remaining
agnostic to the underlying infrastructure, and shows how to integrate them together
into pipelines for Training and Batch Inference purposes.
ethics: No impact.
- Predicts demand for critical jet engine components up to 12 months in advance
- Forecasts lead times for specialized materials and parts
- Optimizes inventory levels for high-value components
- Identifies potential supply chain bottlenecks
- Supports just-in-time manufacturing practices
- Helps reduce working capital tied up in inventory
limitations: |
- Requires minimum 3 years of historical data for optimal performance
- May need recalibration during major market disruptions
- Limited accuracy for newly introduced components
ethics: |
Low to Medium Risk:
- No personal data processing
- Critical for aviation safety - requires regular validation
- Environmental impact considerations in inventory optimization
tags:
- e2e
- batch
- sklearn
- from template
- ZenML delivered
- supply_chain
- c2
- minimal risk

# pipeline level extra configurations
extra:
notify_on_failure: True
# pipeline level parameters
parameters:
target_env: staging
# This set contains all the model configurations that you want
# to evaluate during hyperparameter tuning stage.
model_search_space:
random_forest:
model_package: sklearn.ensemble
model_class: RandomForestClassifier
search_grid:
criterion:
- gini
- entropy
max_depth:
- 2
- 4
- 6
- 8
- 10
- 12
min_samples_leaf:
range:
start: 1
end: 10
n_estimators:
range:
start: 50
end: 500
step: 25
decision_tree:
model_package: sklearn.tree
model_class: DecisionTreeClassifier
search_grid:
criterion:
- gini
- entropy
max_depth:
- 2
- 4
- 6
- 8
- 10
- 12
min_samples_leaf:
range:
start: 1
end: 10
target_env: production
# This model configuration will be used for the training stage.
model_configuration:
model_package: sklearn.tree
model_class: DecisionTreeClassifier
params:
criterion: gini
max_depth: 5
min_samples_leaf: 3
1 change: 1 addition & 0 deletions examples/e2e/constants.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
DATA_CLASSIFICATION = "c2"
23 changes: 4 additions & 19 deletions examples/e2e/pipelines/__init__.py
Original file line number Diff line number Diff line change
@@ -1,21 +1,6 @@
# Apache Software License 2.0
#
# Copyright (c) ZenML GmbH 2024. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#


from .batch_inference import e2e_use_case_batch_inference
from .training import e2e_use_case_training
from .deployment import e2e_use_case_deployment
from .batch_inference import supply_chain_forecast_batch_inference
from .deployment import supply_chain_forecast_deployment
from .training import supply_chain_forecast_training
35 changes: 11 additions & 24 deletions examples/e2e/pipelines/batch_inference.py
Original file line number Diff line number Diff line change
@@ -1,20 +1,11 @@
# Apache Software License 2.0
#
# Copyright (c) ZenML GmbH 2024. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#

from zenml import get_pipeline_context, pipeline
from zenml.integrations.evidently.metrics import EvidentlyMetricConfig
from zenml.integrations.evidently.steps import evidently_report_step
from zenml.logger import get_logger

from constants import DATA_CLASSIFICATION
from steps import (
data_loader,
drift_quality_gate,
Expand All @@ -24,16 +15,11 @@
notify_on_success,
)

from zenml import get_pipeline_context, pipeline
from zenml.integrations.evidently.metrics import EvidentlyMetricConfig
from zenml.integrations.evidently.steps import evidently_report_step
from zenml.logger import get_logger

logger = get_logger(__name__)


@pipeline(on_failure=notify_on_failure)
def e2e_use_case_batch_inference():
@pipeline(on_failure=notify_on_failure, tags=[DATA_CLASSIFICATION])
def supply_chain_forecast_batch_inference():
"""
Model batch inference pipeline.

Expand All @@ -46,7 +32,8 @@ def e2e_use_case_batch_inference():
model = get_pipeline_context().model
########## ETL stage ##########
df_inference, target, _ = data_loader(
random_state=model.get_artifact("random_state"), is_inference=True
random_state=model.get_artifact("random_state"),
is_inference=True
)
df_inference = inference_data_preprocessor(
dataset_inf=df_inference,
Expand Down
Loading
Loading