local UI for data and pipeline management & visualization

[vojavocni]

I really think a web UI for dvc would be a big step towards much wider adoption of the tool. The simple DAG view of dvc pipelines (kinda like what apache airflow has), with the ability to edit the stages, reproduce them and visualize their status would make things much nicer.

An example of a airflow DAG:

Disregarding everything else on the airflow UI, what I was thinking is a DAG representation of pipelines that is similar to the one on the image, but also interactive - you can click on a stage and edit the commands or dependencies as you see fit, reproduce stages with a click, add a new stage in the UI just by creating an edge between the stages and specifying dependencies manually, etc.

I would really like to hear your opinions on this, I feel like it would be a big deal.

[vojavocni]

Also, I would also work on it myself, though I have to admit I'm not a UI guy. :)

[shcheklein]

Related #1074

[shcheklein]

@vojavocni hi! that sounds great, do you want this UI to be run "locally", on top of your local repo? Or some SaaS that connects to Github?

[vojavocni]

@shcheklein I would like it to be run "locally", as it would ideally allow modifying the actual .dvc files, or creating new ones. Something like a jupyter notebook. Then the user can decide weather he wants to push those to the repo.

[shcheklein]

@vojavocni yes, makes sense! Okay, the next question would be - should we integrate with some IDE or build an independent tool? Do you have any technology in mind already (Electron or browser based - like run local server, open it in browser, etc).

[vojavocni]

@shcheklein I was primarily thinking about running a local server and using the tool in a browser. It seems most universal to me. I haven't really thought about an IDE, tbh. From my personal perspective, I would prefer a browser based tool, but I'd also like to hear other people if they tune in here :).

[shcheklein]

@vojavocni I still have certain doubts about running it as a server to be honest, but we can try to hack something quick if you feel that browser is better.

Why do I have doubts.

Web UI dashboard makes sense for Airflow, because Airflow primary goal is to run DAGs on a schedule, remotely, Kinda production ETL use case. While DVC is more like Git + makefiles. DVC pipelines potentially could be used as a replacement for Airflow, indeed, but it's not the major use case as far as I understand at the moment (may be I'm wrong? do you use DVC in that way?).

When you use DVC locally on the hand for experimentation, the closer you "blend" with regular environments - IDEs, Jupiter, CLI - the better. It decreases complexity in adoption, it means that you don't switch between different environment, for example, to edit files and run something, etc.

Also, VS Code is really powerful this days - it can cover even remote execution case nicely.

[vojavocni]

@shcheklein Hmm, yes, I see your point. Now that you mention it, and since I'm using it, having some integration with VS Code might be a very good idea. Since I know VS Code supports notebooks, I guess it's doable to create an interactive UI inside it for dvc?

[shcheklein]

@vojavocni yep, since it's a regular JS (TypeScript), I think it should be possible to implement this. But first step can be as simple as some support for .dvc (show actual data they represent), support pull/push on those. Recognize dvc.yaml (see DVC 1.0 pre-release) and even w/o visualization allow better stages management, etc. If you are interested we can brainstorm the list of features.

[vojavocni]

@shcheklein Sure, I'm definitely in for starting with simple features like pull/push and maybe reproduce. Initially, some kind of visualisation (a nested list, for example) would be good. This can be easily turned into a DAG for eye candy. I will think about this as I use dvc, and post if any interesting features comes to mind.

EDIT: Some of the features I realized would definitely be useful is efficiently renaming things from the UI, where all dependencies would be updated accordingly. To be more general, editing the stages without breaking the pipeline using a simple and intuitive interface.

[shcheklein]

@vojavocni yep, I agree. I would also probably start with just trying to do what dvc list effectively does - show "virtual" files that are DVC-tracked in the list of files. I think that would be a good starting point and also would enable next features after that.

Also, I would recommend to take a look at the DVC pre-release 1.0a - it introduced a single DVC-file. Which means that pipelines are effectively saved into a single file - so it's easier to edit the dependencies now. I think it would be great to have visualize and edit helper still though.

[snehalvartak]

@shcheklein Would definitely like to see DVC integrated with VS code as this is something our team is adopting and having an UI would be a really great add on!

[PuneethaPai]

@shcheklein can we elaborate on how the VS code plugin should look like. What features should it support while integrating with DVC. Some ideas I was getting are:

• Interactive Pipeline Visualization when you open dvc.yaml.
  • Not sure in what all way it needs to be interactive:
    • Stage: Edit stage commands, update dependencies, link stages
    • Status: Check dependencies status
    • Data: Push pull stage/all outputs
    • Pipeline: Trigger pipeline / execute specific stages
    • Params: Update hyper params and trigger pipeline
    • Checkout: List all previous versions of a dependency, checkout that particular version, and update pipeline

All these are exploding in my mind 🤯 😄.

But I am not able to concertize these. Need teams help in detailing (roadmap) and explore possible implementation approach.

[shcheklein]

@PuneethaPai we've started building some parts of this. I can invite you to the document with the roadmap if you want to participate. But this is mostly JS, we can definitely use some help if you are open to do JS.

[PuneethaPai]

Sure @shcheklein. I am rusty with JS, but would love to work on it. Pls add me to the team.

Thanks

[shcheklein]

@PuneethaPai done! ping on Discord if you want to be involved and help with some things there.

[INF800]

Hey @shcheklein I am open to work with JS. I know it is late but is there any way I can contribute? Love this project!

[shcheklein]

@rakesh4real sure! please contact me directly on Discord please, let's see what would be the best way to involve you! (dvc.org/chat - find @ivan)

[vwoloszyn]

I believe that a UI for data and pipeline management will be helpful :)

[uvnikgupta]

I stumbled upon this discussion as I was looking for a similar solution. Is there is update on this?