Implement restart capability

[XiaohuaZhangLLNL]

It will be useful that maestro can restart the previous workflow runs.

[FrankD412]

So the next question on this ticket is -- what do you consider to be a restart?

There are two logical interpretations of the request:

1. Clear the existing directory and start fresh
2. Scan the DAG and attempt to restart that way. This interpretation opens us up to a few things things:
   • If a step in the workflow was running and no longer is do we attempt to restart the step (if a restart is specified)?
   • If a step has no restart specified, do we mark it as failed? The other option is to restart it from scratch without any sort of guarantee that it will complete.
   • If a step previously failed, do we reattempt?

I ask mostly because there needs to be a level of specificity to this functionality. @jsemler -- You had requested this informally before. Thoughts?

[FrankD412]

I've been thinking about the implications of this issue. Here's the proposed command lines:

• maestro restart <study_path> -- restart a study from scratch.
  • This would simply be like re-executing the study but the outstanding issue is restarting the study exactly the same way. This requires some metadata:
    • A way to reconstruct all dependencies as previously specified. This will be difficult for PathDependency instances because that's either placing them in a tarball or hdf5 file and recording a hash of the contents or finding a way to verify that a directory's contents hasn't changed (and then what do we do if it does?).
    • The Study instance used to construct the ExecutionGraph. The importance of the exact instance is subtle, but construction of the ExecutionGraph can change from version to version of Maestro. While the goal is to keep construction of studies as stable as possible, new use cases and improvements can change how construction behaves.
• maestro restart --failed <study_path> -- restart the study from the failed steps. (Resume from failure point #51)
  • Dependencies are the crucial one here. We have to re-run the failed steps with the same dependencies otherwise the integrity of the study is compromised.
    • GitDependencies can rely on commit history and hashes. PathDependencies still have the same issues as above.

Another idea that can tie into this would be the notion of rerunning a study (something like maestro restart --rerun <study path>). That would re-pull dependencies and execute from scratch in a new space. This command could be a good way to repeat studies using the same execution plan, but with new dependencies to limit variability due to changes in study construction. @jsemler and @XiaohuaZhangLLNL -- Any thoughts?

[FrankD412]

This conversation is going to become more relevant again as we discuss the features around adding new parameters to studies. We should probably review and discuss this again @jsemler @jwhite242