Add option for preemptible instances on GCP

[tomwhite]

Here's a simple fix for #255.

As it stands, if preemptible instances are enabled, all instances in the cluster - including the scheduler - are preemptible. It would be better if the scheduler instance was always a standard (non-preemptible) instance. Suggestions for how to do that would be very welcome.

[quasiben]

Thank @tomwhite ! This looks great. Left one small comment. I think it would also be good if there was a test for preemptible instances. Lastly, will GCP reject the submission if preemptible is True and automaticRestart is True?

[quasiben]

What do you think about making the default preemptible=False rather than None ? I think keeping the massive dictionary cleaner is a good goal

[tomwhite]

Fixed

[tomwhite]

Thanks for the review @quasiben.

I haven't added a test yet - what would you recommend?

Also, any thoughts on how to make the scheduler non-preemptible?

will GCP reject the submission if preemptible is True and automaticRestart is True?

Yes, it did when I tried.

[quasiben]

I haven't added a test yet - what would you recommend?

Naively, a test where preemptible=True -- thinking about this now, it's probably not worth it given how long it already takes to run the test suite. Last question according to google docs I think preemptible instances should just work, is that correct ?

[tomwhite]

I've updated with a change so that scheduler instances are never preemptible. This is important, since losing the scheduler means the whole cluster is unusable.

Last question according to google docs I think preemptible instances should just work, is that correct ?

Are you asking about GPUs on preemptible instances? I haven't tried them, but that doc suggests they would work the same.

[quasiben]

Sorry, yes. I was asking about GPU instances. The last change where schedulers are not preemptible is a very good idea. Merging now.

Thanks @tomwhite !