Spin off pipeline inspection to separate package

[astrojuanlu]

Description

Kedro-Viz has done a lot of work to statically derive the structure of the pipeline, now without even having all the imports in place (see discussion in kedro-org/kedro-viz#1742, kedro-org/kedro-viz#1966)

The idea here is to split that functionality as a separate Python package that the Kedro Viz backend would depend on.

Context

There's growing evidence that this functionality could be useful for other use cases, for example to facilitate translating Kedro pipelines into other formats. Plugin authors probably know better, but I'm sure every translator plugin (think kedro-vertexai, kedro-mlrun, kedro-databricks) needs some form of pipeline inspection.¹

There's a proof of concept of how that could look like in https://github.com/AlpAribal/kedro-inspect, which was created as part of this research https://github.com/kedro-org/kedro/wiki/Synthesis-of-research-related-to-deployment-of-Kedro-to-modern-MLOps-platforms .

Possible Implementation

There's a strong indication that this process could use OpenLineage, specifically the concept of Static Lineage (admittedly not very well documented). Some earlier thoughts in #4054

Possible Alternatives

Use a more ad-hoc format, more similar to whatever Kedro Viz is currently using, maybe even the output of --save-file (although at the moment it's not very clear what's expected there, see kedro-org/kedro-viz#1681).

Since there has been reluctance in the past towards spinning off packages, another solution could be that the functionality stays in Kedro Viz, and plugins depend on it. With the amount of dependencies Kedro Viz has, I really hope this isn't the preferred solution.

Another solution is to have that in pypi.org/p/kedro. I don't even think it would be too bad, since we're talking about exporting or serializing kedro.pipelines.pipeline.Pipeline objects in the end.

Very likely there are other possible solutions here, ideas welcome.

Since this is in the Kedro-Viz tracker, cc @merelcht for visibility.

Checklist

• Include labels so that we can categorise your feature request

Footnotes

1. This is a mere hypothesis. I think it would be good to sweep the different translator plugins and see if there is overlapping code among them cc @DimedS ↩

[astrojuanlu]

(It seems like I'm solutionising here, please don't take it as such - it's more of a brain dump after an exceedingly long yak shaving session)

[astrojuanlu]

Earlier, related, idea at kedro-org/kedro-viz#1857.

[datajoely]

I ❤️ Love this - so excited to see it come up here

[astrojuanlu]

Yeah it's connected to the deployment piece in a sense! Moving this here for visibility since it's mostly a Kedro "core" workstream.

[astrojuanlu]

Today a user asked how to access the pipeline definition "outside the normal kedro run life cycle" with this use case in mind:

generate documentation via a mermaid diagram that I can store in a readme file. Similar to kedro viz (but with some subtle key features)

https://kedro.hall.community/catalog-and-pipeline-definitions-nLRzztxNvv5Y

[datajoely]

Taps sign (#3094) we needed this yesterday!