Improve Node Grouping in Kedro Deployment

[DimedS]

Overview

Part of #4317. Users have expressed the need to merge multiple Kedro nodes into a single task on deployment platforms for better clarity and efficiency. Current plugins offer limited support for this, often requiring manual grouping, which complicates deployment and reduces performance.

User Insights and Challenges

• "Combining nodes into single tasks improves overview, but we currently have to manually group them in Databricks."
• "We can convert a single node to a Kubeflow Component, but deploying 400 nodes as separate containers adds complexity."
• "Running each Kedro node in a separate container could make a small node execute in one or two seconds, but Argo’s longer pod startup time would make this inefficient."

Problem Statement

How can we design a flexible and efficient node grouping mechanism - using tags, namespaces, pipelines, or other methods - to maximise usefulness for users and streamline the deployment process?

Proposed Solution

• Centralised Grouping Functionality: Instead of developing node grouping features separately for each plugin, centralise this functionality within the Kedro framework. This approach would standardise and simplify node grouping, making it easier to implement and maintain across different deployment platforms.

[datajoely]

This is the most important problem for me. It's also tightly coupled with dependency management - the minute we make it easier to isolate different parts of the pipeline to be run on different containers you get into dependency isolation questions.

[datajoely]

Users today also tend towards tags because namespaces are a pain to use