Integrate initializing from checkpoint

[hvgazula]

Currently, the model needs to be explicitly initialized from a checkpoint. Otherwise, training starts from scratch.

aha-

Resumption works always (if called as follows)

 bem = Segmentation.init_with_checkpoints(
            unet,
            model_args=dict(batchnorm=True),
            checkpoint_filepath="000-toy",
        )

 print("training")
    _ = bem.fit(
        dataset_train=dataset_train,
        dataset_validate=dataset_eval,
        epochs=n_epochs,
        callbacks=callbacks,
    )

Resumption fails (starts from scratch)

 bem = Segmentation(
            unet,
            model_args=dict(batchnorm=True),
            multi_gpu=True,
            checkpoint_filepath="000-toy",
        )

 print("training")
    _ = bem.fit(
        dataset_train=dataset_train,
        dataset_validate=dataset_eval,
        epochs=n_epochs,
        callbacks=callbacks,
    )

Solution:

• init_from_checkpoint isn't intuitive 🤔 (being a class method). Call it at estimator instantiation time.

[hvgazula]

For more info #331

[hvgazula]

It looks like resumption only works if there's been a SIGINT, otherwise it's starting from scratch.

[hvgazula]

Would it make sense to add init_from_checkpoints(...) in init or fitofSegmentation()`? 🤔