DVC and HPC Clusters / Slurm

[PythonFZ]

I'm currently in the progress of investigating the usage of DVC at our University for the fields of not only machine learning but also simulation science in general. We are using HPC Clusters for most of our simulations, often based on Slurm Workload Manager.

On most clusters it is not possible to have a process running continuously on the head node which leads to some issues.

My original idea was to use dvc <options> srun <srun options> script.py for e.g. a python script.
The srun command stays open until script.py finishes.
The issue here is, that I would have to run this command on the head node which is not possible.

The next option is, to use sbatch to go "inside" a computing Node and then use the standard dvc <options> script.py inside that node. But in that case I can not run multiple experiments in parallel, because after the first one starts the rw.lock prohibits other nodes to run the dvc command. I tried dvc exp run --temp and was suprised that that command also writes the rw.lock.

The final idea I have would be to make a copy of the repository for each sbatch and then merge them together afterwards. But I would like to avoid that as much as possible, because locally making a copy of a repository and then merging it back together will cause a lot of duplicated files and possibly other issues.

There are two scenarios that I could envision to solve this:

1. Do not write the rw.lock when using dvc exp run --temp. Could this be similar to how dvc exp run --run-all works? This would allow the queue a run with slurm and when finished investigate with dvc exp show.
2. Use dvc exp run --run-all on the head Node which uses sbatch and will close shortly after the runs are queued and have an API command that I can trigger manually when the simulations are either successfully finished or failed.

Is any of this already possible? Are there other options that you would suggest fit better? I think this would bring some major benefits for the usage of DVC on HPC Clusters that rely on slurm or other workload managers without keeping a process open on the head node for potentially months.

[dberenbaum]

Hi @PythonFZ! We are working on more robust queuing functionality to allow for less locking and more granular control, like adding experiments while others are running. See #5615 for work related to this. The prerequisites are in progress, so this is a current priority. Please keep track of progress in that issue 🙏 !

cc @pmrowla @karajan1001

[hfawaz]

Is this still being worked on ?

[dberenbaum]

You should be able to run dvc exp run --temp multiple times now, and the planned additional queuing support is available so that you can do things like keep adding to a running queue. Maybe we can close this issue unless you have specific requirements that are still not met?

[hfawaz]

So in the same repository I can run several times the dvc exp run --temp and have the metrics in the same repository assuming the different instances of my dvc exp run --temp do not write the same metrics output ?

[PythonFZ]

@hfawaz You can look at the metrics as described in the Introduction to Experiments https://dvc.org/doc/start/experiment-management/experiments.

If you are using SLURM and want to combine it with DVC I developed a small CLI tool that will allow you to easily combine both. Unfortunatley you still have to run some process on the login Node. With the DVC queuing it will run a Celery worker in the background.
I'm usingon Dask Jobqueue which is more targeted towards HPC computing in my opinion. The main difference is, that with celery you need to define your slurm command through the dvc.yaml and with my approach you don't have to change your dvc.yaml depending on the cluster. I haven't published the package yet, but it will be Apache licensed and work natively with DVC relying only on the DVC CLI. Feel free to conact me, if you are interested in the pre-release version fabian.zills(at)web.de.

@dberenbaum from my perspective this issue is resolved.

[hfawaz]

Okay thanks I will have a look.

[dberenbaum]

So in the same repository I can run several times the dvc exp run --temp and have the metrics in the same repository assuming the different instances of my dvc exp run --temp do not write the same metrics output ?

@hfawaz Each experiment will run in its own copy of the repo in a separate temp directory. Therefore, if you make sure to write out the metrics to a path that is relative to the script or other temp directory location, then everything should work.

[hfawaz]

Okay perfect thanks

[pared]

@hfawaz Were you able to make your use case work?

[hfawaz]

I am having hard time using the --set-param so still was not able to test this.
All of my configs are in json so I have to migrate to yaml in order to use Hydra I guess.

For now each kubernetes jobs is cloning the repo and launching its own job.

[dberenbaum]

@hfawaz Do you want to use Hydra composition? You only need YAML files if you want to use that feature. --set-param can work with JSON configs. Sorry for any confusion.

[hfawaz]

Yeah I have some templates in json and sometimes I need to update a lot the params.json used by dvc.yaml

Is there something that automatically translates my new json to a list of --set-param ?

FYI I tested manually the --set-param it works with json so thanks !

No worries about confusion I just lack the necessary knowledge of Hydra the language used in --set-param

[PythonFZ]

I made a related Issue with some feature requests that could help HPC users #9235