Python API for creating DAG?

[skshetry]

I thought we had an issue #5178, but looks like it is more for displaying dag rather than creating it.

With dvc.yaml workflow file, the declared definition are static in nature, while runtime being isolated in cmd. This allows us to provide other data management features such as checkout, pull, push etc. without executing.

By providing a python API, I can think of 2 kinds of APIs:

1. Similar to dvc.yaml file, providing an API that statically explains the stage or workflow. Examples: doit, nox

@nox.session
def tests(session):
    session.install('pytest')
    session.run('pytest')

@nox.session
def lint(session):
    session.install('flake8')
    session.run('flake8', '--import-order-style', 'google')

With this kind of API, they are not that different from dvc.yaml file, in that they just explain stage definition through python api. But, as they are static in nature, we'll still be able to provide other data management commands.

What about dvc.lock though? Maybe, run-cache should be used instead here, as having a python API and a lock file (that needs to be added to git) does not make much sense.

2. Dynamic API: dynamic dag through code. Might have different behaviour when it's not executing or not. As it's dynamic, dvc might not always be able to determine dependencies and outputs. Eg: Airflow, dagster, prefect, kedro pipelines, etc.

How will viewer and other similar tools understand the DAG?

So, regarding api, we need to find a way to statically determine metadata for dvc without executing it, while providing scalability and flexibility when executing it dynamically. But, this way it seems we'll have to clearly separate what should be determined statically and what should be dynamic data, which might not be as clear in the future.

# example separation, inspired by https://typer.tiangolo.com/tutorial/parameter-types/file/
import devee
from devee import task, dependency, output, param
from devee import context

@task
def func(
  # static definition here
  input: Path = dependency("dataset.csv"),
  output: Path = output("model.pickle"),
  split: int = param("param.split"),
  metric: Path = output("metrics.json", cache=False)
):
  context.log("yippee") # dynamic stuff here

if __name__ == "__main__":
  devee.run()

Same problem here though, regarding dvc.lock. As dvc's metadata are static, we'll be able to provide other data management features as well.

How should function composition work? In dvc.yaml file, there is no support for map/reduce/join/accumulate, but with Python API, it seems an important feature. Another important feature would be a dependency between stages (functions with python api). How to detect those dependencies?

[dberenbaum]

Thanks, @skshetry! Looks like a lot of design decisions to be made (maybe a good enhancement proposal candidate?).

Before we can decide on the design, can we focus on what issues a Python API solves:

• Duplicating dvc.yaml in Python? Honestly, I'm not sure that's worthwhile on its own.
• Automatic detection of dependencies and outputs?
• Enabling object dependencies and outputs that don't need to be written to files?

[skshetry]

Duplicating dvc.yaml in Python?

We have already introduced foreach (aka for loops) and parametrization (aka fstring). There's a discussion about introducing map-reduce and nested-foreach within dvc.yaml. The question could be asked another way around: is it worth adding them in dvc.yaml if we can provide a good python API?

The Python API are usually very granular, so they can capture much more scenarios (known and unknown ones) and semantics than the supposedly high-level dvc.yaml file. With python, we have a complete programming language to our side, we can easily provide generalized and specialized APIs to satisfy user needs. In dvc.yaml, this is very much limited (or, we are going to add too many keywords and schemas to support them which makes it hard to read/edit/write.

Automatic detection of dependencies and outputs?

I am not sure if it's possible to detect automatically. We might need to ask for metadata from users. If we want to provide data management related features, it'd better be static.

Enabling object dependencies and outputs that don't need to be written to files?

Yeah, this is a great feature and will allow us to have better integrations to user's tools. dvclive could simply be an API (see above context.log example), plots could be visualized as it runs, progress and metrics could be monitored live. We could pass parameters directly to user's code from params.yaml. We can do more stuff in a granular way. API will unlock a great potential for integrations too.

maybe a good enhancement proposal candidate?

I am just dumping my thoughts here. A lot of things seem to be odd with how dvc works right now, so I am not sure how existing stuffs would work with it (for example lock file). The API is the most visible product out of this, so we might need competing API proposals. Internal working changes need their own proposals (runcache, hashing functions and matching with runcache, scheduler and runner, executors, etc).

[dberenbaum]

The Python API are usually very granular, so they can capture much more scenarios (known and unknown ones) and semantics than the supposedly high-level dvc.yaml file.

Yes, I can imagine a bunch of use cases it could solve. For example, with experiments and checkpoints, it would be great to be able to reproduce experiments with all parameter values that were tried, or to run or a fixed number of checkpoints. Having a Python API for the DAG might make this much easier to do in an automated way.

[dberenbaum]

How to provide more flexible DAG definitions is a broad discussion for which a Python API is one possible solution. I'm not sure whether to put this into a separate discussion or maybe rename this one, but I'll start with a new thread here.

For example, see #3633 (comment), which proposes adding more advanced filters and custom filters to dvc.yaml (similar to Ansible, which allows for any jinja2 filters). This approach might start to get confusing, especially for users not used to templating or working with tools like Travis or Ansible. In order to do advanced filtering, users might need to know both jinja2 and dvc syntax. It also still might not solve every use case.

A Python API would provide more flexibility, but it makes DAGs less readable and might make editing and debugging more difficult.

[daavoo]

This library was recently shared on discord (https://discord.com/channels/485586884165107732/485586884165107734/869509789791715338):

https://github.com/zincware/py-track

[asmith26]

I like the idea of being about to execute/evaluate normal python code in dvc.yaml (e.g. dvc.py) - more thoughts/details in #6512 (comment)

[marcosfelt]

I just wanted to jump in and suggest why a python API to DAGs/pipelines would be useful for me.

1. Identifying changes at the function level: I often use a modular architecture where functionality for different experiments is located in the same python file. For example, in a project I am working on currently, I have single-task and multi-task implementations of the same underlying model in a python file. Currently, if I make a change to the single-task implementation, DVC will assume that the multi-task implementation has changed as well. Therefore, DVC will say the downstream steps that use the multi-task implementation need to be re-run, even though nothing has changed about the multi-task implementation. Being able to specify steps at the function level (as is done in things like Dagster and Airflow) would solve this problem.
2. Specifying pipelines in python: Currently, I am having to write everything twice: first in python and then in dvc.yaml. This is particularly painful for looping. Currently, I write python files that generate yaml files, which I reference in dvc.yaml. It wold be great if I could just write everything in python and be done.

[uwaisiqbal]

I'm currently working on a project and trying out dvc. I've used kedro in the past and I really liked how the pipelining and orchestration of the dag could be done in Python. I think it would be super useful to have some detection of dependencies and outputs. Inspiration could be taken from variable typing to create data typing. In my workflow, I currently use typer to wrap my Python scripts. I'm thinking whether there could be an easy way to create a wrapper for function arguments that declares them as dependencies, outputs, params etc. If this could be done, then the dag could automatically be generated from looking at the function calls, dependencies and outputs.

For example, at the moment I have this in my Python script:

import typer 
from pathlib import Path

def create_dataset(cuad_path: Path, citrain_path: Path, output_path: Path):
   # do some cool stuff

if __name__ == '__main__':
    typer.run(create_dataset)

and then I have to create the declaration in the .yaml file

create-contracts-dataset:
    cmd: python3 src/create_contract_dataset.py some/path some/other/path output/path
    deps:
    - some/path
    - some/other/path
    outs:
    - output/path

If there was a was to explicitly declare the data type in Python I could write something like:

from pathlib import Path
import typer
from shiny_cool_lib import InputPath, OutputPath

def create_dataset(cuad_path: InputPath = Path('some/path'), citrain_path: InputPath = Path('some/other/path'), output_path: OutputPath = Path('output/path')):
   # do some cool stuff

if __name__ == '__main__':
    typer.run(create_dataset)

Behind the scenes, the shiny_cool_lib would just scoop up the data type declarations across the pipeline stages and dynamically generate the yaml and the dag with dependencies, outputs etc all declared alongside the code. It would save having to write things twice and allow dynamic dag generation