Inject run parameters via pipelines to notebooks

[dilverse]

Hi I am trying to build kubeflow pipelines using Elyria and I would like to inject Run Parameters during kubeflow run something like below

Is it possible to inject such parameters via the Elyria pipelines into to the notebooks, papermill does inject parameters into notebook by adding parameters tag at the notebook cell level let me know how this can be done

[ptitzler]

The only supported means to perform customization for generic nodes is through the use of environment variables, which notebooks and scripts can evaluate. However environment variables values can currently only be set prior to pipeline submission through the Elyra pipeline editor, but not the elyra-pipeline CLI or through the Kubeflow Central Dashboard (like in your example screen shot). That said we do have an open requirement to support pipeline [run] parameters #1903

[dilverse]

Got it. Could I make it work with Kubeflow based nodes/components?
If so it would be helpful if you can share a sample code

[ptitzler]

Could I make it work with Kubeflow based nodes/components?

Unfortunately not through environment variables because the pipeline editor currently does not expose them as a settable node property. The reasoning is that in a way using environment variables to pass data (in the general sense) to a component would be a bad practice because the fact that they impact the component behavior isn't explicitly captured in the component specification.

Could you elaborate bit on your scenario? Is your goal to submit/export the pipeline once from Elyra and then re-run it (as required) through the run wizard in the Kubeflow Central dashboard?

[dilverse]

My objective is to export the pipeline to kubeflow from Elyra and if I had access to the run parameters then the notebooks could become dynamic for instance if I want to do incremental re-training of the model on new training data (which I can specify in the run parameter) even using Katib for hyperparameter tuning where the data scientist can set number of layers, learning rate, optimizer....
Let me know if this makes sense or you need more details, thanks for the help

[ptitzler]

It's a perfectly valid scenario! In essence you are using Elyra to generate and upload the DAG for you to KFP (top of the image) and then you use the KFP Central Dashboard or CLI to run the DAG several times with different parameters (bottom of the image), without the need to make changes to the pipeline in the Elyra Pipeline editor:

[lresende]

I started doing the support for this a long time ago, but was interrupted and moved away to other priorities, in any case see my old branch as it might inspire you on where the code needs to be changed --> https://github.com/lresende/elyra/commits/dev-parameter

NOT A CONTRIBUTION