Support safely resubmitting workflows if an earlier version of that workflow is partially running

[ihodes]

This can fail due to race conditions now: e.g. a cleanup node to rm a failed .bam write gets killed, then you have a broken product (truncated .bam) if you kill before resubmitting. If you don't kill, then you might submit and have a .bam cleaned up between the new node being marked completed due to the truncated/broken .bam existing, thus making the new workflow fail as the .bam is rmed out from under it.

[smondet]

When a workflow is partiallly running "equivalence" is fine; the problem here is partially "killed" or "failed" workflows that are still in the process of “cleaning-up”.

[ihodes]

Exactly, though I think that if a "cleanup" (on_failure) node was implemented as a conditional normal (depends_on) node, it'd still be a problem. Nevertheless, somewhat tangential to the issue.