workflow is longer than maximum allowed size. compressed size 1049974 > maxSize 1048576Tried

[pantlavanya]

Describe the bug

Hi All, I am facing the below error while running a pipeline, this pipeline is 1001k in size. It say the workflow is longer than maximum allowed size. Here is the same issue kubeflow/pipelines#4170 where it say there is a limit of pipeline size? Do we have any fix for this ? cc

time="2023-06-29T17:51:38.732Z"` level=warning msg="Failed to dehydrate: workflow is longer than maximum allowed size. compressed size 1049974 > maxSize 1048576Tried to offload but encountered error: offload node status is not supported" namespace=dev workflow=ml-project-rxqzr
time="2023-06-29T17:51:38.732Z" level=info msg="Updated phase Running -> Error" namespace=dev workflow=ml-project-rxqzr
time="2023-06-29T17:51:38.732Z" level=info msg="Updated message -> workflow is longer than maximum allowed size. compressed size 1049974 > maxSize 1048576Tried to offload but encountered error: offload node status is not supported" namespace=dev workflow=ml-project-rxqzr
time="2023-06-29T17:51:38.732Z" level=info msg="Marking workflow completed" namespace=dev workflow=ml-project-rxqzr
time="2023-06-29T17:51:38.737Z" level=info msg="cleaning up pod" action=deletePod key=dev/ml-project-rxqzr-1340600742-agent/deletePod
time="2023-06-29T17:51:38.985Z" level=warning msg="Error updating workflow: rpc error: code = ResourceExhausted desc = trying to send message larger than max (2430673 vs. 2097152) " namespace=dev workflow=ml-project-rxqzr

Environment

• Kubernetes version :- v1.22.17-eks-48e63af
• Kubeflow version :- v1.6.1-aws-b1.0.0
• KFP Client version :- 1.8.x

[jsitu777]

Hi @pantlavanya have you tried the method to reduce the pipeline size by deleting the component_spec annotation and if the issue still persist?

[sachdevayash1910]

Could this work?
https://argoproj.github.io/argo-workflows/offloading-large-workflows/#:~:text=Argo%20stores%20workflows%20as%20Kubernetes,This%20can%20be%20over%201MB.
I did try to enable this in my kubeflow cluster but I am getting some issues with the postgres connection for some reason:

time="2023-09-11T21:33:26.580Z" level=info msg="Persistence configuration enabled" time="2023-09-11T21:33:26.580Z" level=info msg="Creating DB session" time="2023-09-11T21:33:26.590Z" level=info msg="Get secrets 200" time="2023-09-11T21:33:26.594Z" level=info msg="Get secrets 200" time="2023-09-11T21:33:26.607Z" level=fatal msg="Failed to update config: EOF"

But I am curious if this is even a viable option