Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MLflow app creation crashes because of protobuf version #82

Open
ConsciousML opened this issue Mar 28, 2023 · 1 comment
Open

MLflow app creation crashes because of protobuf version #82

ConsciousML opened this issue Mar 28, 2023 · 1 comment

Comments

@ConsciousML
Copy link

Hi there,

Describe the bug
When running make one-click-mlflow, an error appears while creating the app-engine-flex:

Error: Error waiting to create FlexibleAppVersion: Error waiting for Creating FlexibleAppVersion: Error code 9, message: An internal error occurred while processing task /app-engine-flex/flex_await_healthy/flex_await_healthy>2023-03-28T10:26:33.695Z25166.wd.0: Traceback (most recent call last):
│   File "/usr/local/bin/mlflow", line 5, in <module>
│     from mlflow.cli import cli
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/__init__.py", line 32, in <module>
│     import mlflow.tracking._model_registry.fluent
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/tracking/__init__.py", line 8, in <module>
│     from mlflow.tracking.client import MlflowClient
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/tracking/client.py", line 8, in <module>
│     from mlflow.entities import ViewType
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/entities/__init__.py", line 6, in <module>
│     from mlflow.entities.experiment import Experiment
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/entities/experiment.py", line 2, in <module>
│     from mlflow.entities.experiment_tag import ExperimentTag
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/entities/experiment_tag.py", line 2, in <module>
│     from mlflow.protos.service_pb2 import ExperimentTag as ProtoExperimentTag
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/protos/service_pb2.py", line 18, in <module>
│     from .scalapb import scalapb_pb2 as scalapb_dot_scalapb__pb2
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/protos/scalapb/scalapb_pb2.py", line 35, in <module>
│     serialized_options=None, file=DESCRIPTOR)
│   File "/usr/local/lib/python3.7/dist-packages/google/protobuf/descriptor.py", line 561, in __new__
│     _message.Message._CheckCalledFromGeneratedFile()
│ TypeError: Descriptors cannot not be created directly.
│ If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
│ If you cannot immediately regenerate your protos, some other possible workarounds are:
│  1. Downgrade the protobuf package to 3.20.x or lower.
│  2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
│ 
│ More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
│ Traceback (most recent call last):
│   File "/usr/local/bin/mlflow", line 5, in <module>
│     from mlflow.cli import cli
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/__init__.py", line 32, in <module>
│     import mlflow.tracking._model_registry.fluent
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/tracking/__init__.py", line 8, in <module>
│     from mlflow.tracking.client import MlflowClient
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/tracking/client.py", line 8, in <module>
│     from mlflow.entities import ViewType
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/entities/__init__.py", line 6, in <module>
│     from mlflow.entities.experiment import Experiment
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/entities/experiment.py", line 2, in <module>
│     from mlflow.entities.experiment_tag import ExperimentTag
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/entities/experiment_tag.py", line 2, in <module>
│     from mlflow.protos.service_pb2 import ExperimentTag as ProtoExperimentTag
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/protos/service_pb2.py", line 18, in <module>
│     from .scalapb import scalapb_pb2 as scalapb_dot_scalapb__pb2
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/protos/scalapb/scalapb_pb2.py", line 35, in <module>
│     serialized_options=None, file=DESCRIPTOR)
│   File "/usr/local/lib/python3.7/dist-packages/google/protobuf/descriptor.py", line 561, in __new__
│     _message.Message._CheckCalledFromGeneratedFile()
│ TypeError: Descriptors cannot not be created directly.
│ If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
│ If you cannot immediately regenerate your protos, some other possible workarounds are:
│  1. Downgrade the protobuf package to 3.20.x or lower.
│  2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
│ 
│ More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
│ 
│ 
│   with module.mlflow.module.server.google_app_engine_flexible_app_version.mlflow_app,
│   on modules/mlflow/server/main.tf line 112, in resource "google_app_engine_flexible_app_version" "mlflow_app":
│  112: resource "google_app_engine_flexible_app_version" "mlflow_app" {
│ 
╵

To Reproduce
Steps to reproduce the behavior:

terraform version
Terraform v1.4.2
on linux_amd64

  1. Install Terraform v1.4.2
  2. run make one-click-mlflow
  3. See error

Expected behavior
The Terraform script should create the MLflow app without error

Desktop (please complete the following information):

  • OS: Ubuntu
  • Version: 22.04
@ConsciousML
Copy link
Author

ConsciousML commented Mar 28, 2023

It works after forcing the protobuf version to the tracking_server\requirements.txt:

protobuf==3.20.0

Should I PR the change ?

@ConsciousML ConsciousML mentioned this issue Mar 28, 2023
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant