Duplicate exps and exp naming

[dberenbaum]

4. How should queued experiments be named and should they match the final experiment name?

This is also related to the bug where generated final exp names for checkpoint exps can't be used with queue kill and queue remove (the queued exp isn't associated with the separate generated exp name).

We probably need to just revisit exp automatic naming, the simplest solution here would be to just generate random exp names when they are queued instead of using the current method of naming exps based on the pipeline result. However, this means we would lose the "prevent duplicate exps" behavior which we have now (but in previous discussions it's been raised that this may not be desirable behavior in the first place).

Basically, if a user runs exp run twice with the same set of parameters, is it ok for us to generate two separate experiments in git (even though they have the same result)? On the DVC side we would still be using run-cache, so the subsequent runs wouldn't be fully reproduced/duplicated. But exp show/queue status/etc would all show two entries.

If having duplicate exps is acceptable then we can just auto generate names right away, so queued exp names will always match the final ones.

Originally posted by @pmrowla in #7592 (comment)

[dberenbaum]

The same set of parameters will still try to generate a duplicate experiment but then fail at the end, right? I think this is the worst option and it always seems inconsistent and surprising to me (sometimes I still end up with duplicated experiments). Either we should fail to queue/start the experiment (with an exception for checkpoints), or we should allow duplicates.

[karajan1001]

This is also related to the bug where generated final exp names for checkpoint exps can't be used with queue kill and queue remove (the queued exp isn't associated with the separate generated exp name).

Yes, I also noticed this, and I think we need a better way to handle checkpoint exps. On the queue side, they are regarded as a single experiment when the worker runs them. But on the dvc exp show map, they are regarded as a sequence of independents and can be operated individually.

We probably need to just revisit exp automatic naming, the simplest solution here would be to just generate random exp names when they are queued instead of using the current method of naming exps based on the pipeline result. However, this means we would lose the "prevent duplicate exps" behavior which we have now (but in previous discussions it's been raised that this may not be desirable behavior in the first place).

Not quite understand why generate random exp names when they are queued would cause name duplication?

1. The exp-1234 style of name does not need to be associated with the parameters.
2. We can detect the duplication of parameter and for example add a serial number to it.

Basically, if a user runs exp run twice with the same set of parameters, is it ok for us to generate two separate experiments in git (even though they have the same result)?

I think it is OK.

Either we should fail to queue/start the experiment (with an exception for checkpoints), or we should allow duplicates.

We can check the naming duplication in previous before the experiment is started.

[pmrowla]

Not quite understand why generate random exp names when they are queued would cause name duplication?

Duplication of the final experiment result/workspace, not duplication of names (so exp-123 could have the same params/deps/outputs as exp-456)

[pmrowla]

Right now, if you run dvc exp run twice in a row without changing params, you will only end up with one experiment in exp show (since we generate names according to pipeline output state, to avoid duplicates). Generating the names before running will make it so you have two rows in exp show with different names/timestamps but otherwise identical contents