Improve robustness against OpenMMException (e.g. CUDA_ERROR_ILLEGAL_ADDRESS)

[jchodera]

cc #924 (comment)

We should implement some level of protection against these failures. One option is to wrap the whole simulation in a try...except block. That might be the most conservative, safest option, since we don't know how many CUDA contexts need to be invalidated if this occurs.

[mikemhenry]

Retry logic will go here:
https://github.com/choderalab/perses/blob/main/perses/app/setup_relative_calculation.py#L758

[mikemhenry]

When using a different code entry point than the yaml, PointMutationExecutor the user will need to add the restart logic (for now, until we fix the api). I'll update the documentation but it will be something like:

retry_attempt = 0
MAX_ATTEMPTS = 5
while retry_attempt < MAX_ATTEMPTS:
    try:
        hss.extend(n_cycles)
    except OpenMMException as err:
        _logger.error(f"OpenMMException! {err}")
        retry_attempt += 1
        _logger.error(f"retry attempt {retry_attempt}/{MAX_ATTEMPTS}")
else:
    _logger.error(f"Failed to retry simulation in {MAX_ATTEMPTS} attempts")
    _logger.error(f"Will try one last time and not catch the exception")
    hss.extend(n_cycles)

also some changes to the minimizer may help as well with these errors:
choderalab/openmmtools#557