dvc exp push: pushing/pulling cache fails if params.yaml not in workspace

[dberenbaum]

Bug Report

Description

If there is no params.yaml in the workspace and interpolation is used in dvc.yaml, dvc exp push will fail to push the cache for the experiments.

Reproduce

Fork and clone https://github.com/dberenbaum/params-test and then run:

dvc exp run
rm params.yaml
dvc exp push origin

You will see an error like ERROR: failed to push cache: failed to parse 'stages.params_test.cmd' in 'dvc.yaml': Could not find 'params'.

Expected

Since params.yaml is present in the experiment, it should be possible to push it. Can we interpolate using the state of the experiment rather than the workspace?

[dberenbaum]

It seems very loosely related to #9754 in that we aren't respecting what's in the revision but instead are using the workspace.

[daavoo]

For the record in case someone encounters this in the context of using Hydra integration:

I missed the Hydra part before. That might be easier to handle as a special case in DVC.
And even though it still adds some friction, assuming the Hydra stuff is present in the workspace, you could do dvc exp run --dry as a fallback.
That would create the params.yaml in the workspace and allow to run dvc commands that would previously fail at the interpolation stage

[dberenbaum]

This can also happen during pull, and not only on the first run, since parameters may change frequently (esp with hydra), and any missing parameter in the workspace causes a failure. See https://discord.com/channels/485586884165107732/485596304961962003/1137288948914339860.

[daavoo]

This can also happen during pull, and not only on the first run, since parameters may change frequently (esp with hydra), and any missing parameter in the workspace causes a failure. See https://discord.com/channels/485586884165107732/485596304961962003/1137288948914339860.

Should we compose_and_dump by default on dvc exp commands?

[dberenbaum]

This can also happen during pull, and not only on the first run, since parameters may change frequently (esp with hydra), and any missing parameter in the workspace causes a failure. See https://discord.com/channels/485586884165107732/485596304961962003/1137288948914339860.

Should we compose_and_dump by default on dvc exp commands?

Do you mean in the workspace? I don't think it would solve the issue mentioned in discord since it seems like whether the params exist depends on the config in the specific experiment, and it may also mess up the user's workspace.

[Danila89]

Are there any updates on this issue? Garbage collection is also affected: dvc gc -a -c --remote remote-name --all-experiments --force fails with similar error

[dberenbaum]

@Danila89 I don't have any updates unfortunately. Have you tried adding params.yaml to your workspace? I think it should work once you have a params.yaml file with the expected parameters.

[Danila89]

@Danila89 I don't have any updates unfortunately. Have you tried adding params.yaml to your workspace? I think it should work once you have a params.yaml file with the expected parameters.

It will, but it is inconvenient. Different experiments have different parameters at params.yaml (I mean not just different values, but different parameter names as well). And for dvc exp pull or dvc gc I have to get the proper params.yaml first. Using garbage collection when many experiments are deleted becomes literally impossible(

[dberenbaum]

Upon further investigation, the cache is pushed properly, so this is mostly a UI issue. Brancher always yields the workspace, so an invalid workspace state will throw an error even if a command like dvc exp push only relies on non-workspace revs:

dvc/dvc/repo/brancher.py

Lines 76 to 80 in 7c61463

	yield "workspace"
	revs = revs.copy() if revs else []
	if "workspace" in revs:
	revs.remove("workspace")

Can we yield workspace only if it's included in revs @iterative/dvc?

Reproducible example

set -ux

# Clone repo and setup remotes
REMOTE=/private/tmp/dvcremote
rm -rf params-test params-test-remote
rm -rf $REMOTE
mkdir $REMOTE
git clone -q [email protected]:dberenbaum/params-test.git params-test-remote
git clone -q params-test-remote params-test
cd params-test
dvc remote modify --local default url $REMOTE

# Run experiment with different params and then drop params.yaml
dvc exp run -q -S params=bar
rm params.yaml

# Push experiments and check what's been pushed
dvc exp push origin
dvc exp list origin
tree $REMOTE

# Revert to original params
git restore .

# Run gc and check if garbage collected
dvc exp remove -q --all
dvc gc -f -c -w --all-experiments
tree $REMOTE

Here is the output:

+ REMOTE=/private/tmp/dvcremote
+ rm -rf params-test params-test-remote
+ rm -rf /private/tmp/dvcremote
+ mkdir /private/tmp/dvcremote
+ git clone -q [email protected]:dberenbaum/params-test.git params-test-remote
+ git clone -q params-test-remote params-test
+ cd params-test
+ dvc remote modify --local default url /private/tmp/dvcremote
+ dvc exp run -q -S params=bar
+ rm params.yaml
+ dvc exp push origin
ERROR: failed to collect 'workspace' - failed to parse 'stages.params_test.cmd' in 'dvc.yaml': Could not find 'params'
Pushed experiment grand-eyas to Git remote 'origin'.
1 file uploaded
+ dvc exp list origin
main:
        grand-eyas
+ tree /private/tmp/dvcremote
/private/tmp/dvcremote
└── files
    └── md5
        └── 03
            └── 6683c57ba049f38cd3dd4c20dd119e

3 directories, 1 file
+ git restore .
+ dvc exp remove -q --all
+ dvc gc -f -c -w --all-experiments
WARNING: This will remove all cache except items used in the workspace and all experiments of the current repo.
Removed 1 objects from repo cache.
No unused 'local' cache to remove.
No unused 'legacy' cache to remove.
Removed 1 objects from remote.
No unused cache to remove from remote.
+ tree /private/tmp/dvcremote
/private/tmp/dvcremote
└── files
    └── md5
        └── 03

3 directories, 0 files

@Danila89 tldr This is not likely the same as the issue you are seeing, and dvc is using the correct params.yaml associated with each individual experiment. If you are seeing a similar error during dvc gc -a -c --remote remote-name --all-experiments --force, it likely means that one of your branches (since -a specifies all branches) is invalid and missing params.yaml entirely.

[Danila89]

@dberenbaum thank you for the update!
It seems that I don't follow the best practise of working with dvc+hydra integration. Could you please clarify:

1. Should I have params.yaml in every branch in my repo (even if there are no experiments in the branch)?
2. How should I handle modification of hydra config (some parameters added, some parameters deleted)? What version of the params.yaml (if any) should I keep in each git branch?

In my project I have:

1. Master branch without params.yaml
2. Feature branches that are used for staff not directly related to running the experiments. These branched don't have params.yaml
3. Experiment branches. These branches are created using dvc exp branch and they contain params.yaml
   The experiment workflow is the following:
4. From master of some feature branch I run dvc exp run
5. dvc exp branch
6. dvc exp push
7. git checkout experiment-branch -f
8. git push origin experiment-branch

I would appreciate any advice on what to change in my current workflow to get rid of this invalid params.yaml problem.
I seems that for garbage collection I can just drop -a option and that works for me, but dvc exp pull is still not available for me from master branch.

[dberenbaum]

I seems that for garbage collection I can just drop -a option and that works for me

👍 If you don't intend to keep params.yaml on your branches, this is the way to go.

dvc exp pull is still not available for me from master branch.

Can you show the output for this or maybe how to reproduce it? I'm able to run dvc exp pull even when there is no params.yaml in either the branch or the workspace.

[Danila89]

Sorry, it happens that the problem reproduces just with DVC 2 and vanishes in DVC 3.
It outputs the error message like this ERROR: failed to collect 'workspace' - failed to parse 'stages.main.cmd' in 'dvc.yaml': Could not find 'parameters.parameter_name', but it does not prevent normal work of dvc pull/push. Probably this issue could be closed, thank you very much!

[Danila89]

Sorry, I've checked once again, with dvc 3.33.3 - dvc gc still fails without params.yaml in the active branch.
To reproduce: git clone https://github.com/Danila89/dvc_empty.git && cd dvc_empty && dvc gc --all-experiments --force

[dberenbaum]

@Danila89 It's happening because dvc gc always includes the workspace right now, and your workspace is technically in an "invalid" state in that dvc can't read it. We can add a --no-workspace option so that you can gc only the experiments and not the workspace. Would that work for you?

[Danila89]

@dberenbaum Yes it will, thanks in advance!