Any restart option?

[turbosonics]

Let's say I have SevenNet training stopped before it reaches the proper termination condition like epoch number or wall time limit.

In this case, should I need to fine-tune the model using exactly same conditions from the latest model to continue this training?

Or are there any keywords that just continue/restart the training from latest epoch?

[YutackPark]

There is no exact keyword for restart, but you can do the same by continuing from the latest checkpoint. Set reset_optimizer and reset_scheduler as Fasle. It will smoothly restart from the given checkpoint without any artifacts.

You can also change an optimizer, learning rate, and scheduler as you want but in this case, corresponding reset_* keys should be True.

If sevennet finds a log.csv under working directory, it will append logs to the same file. But there might exist an overlap between the previous run and now if it is not continued from the exact last epoch, where the training stopped.

[turbosonics]

I was using ReduceLROnPlateau schedulder with factor of 0.5 and patience of 50, and beginning LR was 0.002 but becomes 0.001 before terminated by wall time limit. So, in this case, I need to restart with beginning LR of 0.001 but reset_* keys should be false.

One other question though, what is the default metric for criteria for ReduceLROnPlateau, and how can I set metric for ReduceLROnPlateau scheduler? Like force loss, energy loss, or mae of energy...

[YutackPark]

I was using ReduceLROnPlateau schedulder with factor of 0.5 and patience of 50, and beginning LR was 0.002 but becomes 0.001 before terminated by wall time limit. So, in this case, I need to restart with beginning LR of 0.001 but reset_* keys should be false.

Yes, and you don't need to change LR value of input.yaml. If reset_optimizer is false, it will simply ignore optimizer and optim_param inputs. Instread it loads those information from the checkpoint.

One other question though, what is the default metric for criteria for ReduceLROnPlateau, and how can I set metric for ReduceLROnPlateau scheduler? Like force loss, energy loss, or mae of energy...

Default is TotalLoss of validation set. Other options can be set if it is enrolled as error_record. You can change the default via setting best_metric: {metric you want} under train: of your input file.

The {metric you want} is can be set as {target}_{metric name}, based on error_record. For example,

error_record:
      - ['Energy', 'MAE']
      - ['Force', 'MAE']
      - ['Energy', 'Loss']
      - ['Force', 'Loss']
      - ['TotalLoss', 'None']

The valid choice for best_metric will be one of "Energy_MAE", "Force_MAE", "Energy_Loss", "Force_Loss", "TotalLoss". (Total loss is special)
Note that SevenNet will also write checkpoint_best.pth based on the best_metric. Although, the behavior of ReduceLROnPlateau and SevenNet writing the checkpoint_best.pth might not exactly same as former has threshold policy.