Support Amazon S3 API for storing instances and templates

[dormidon]

Hi!
Thanks a lot for great and well-written service!

It seems that the service is used to simplify deploy into Nomad. And deploy process should be as robust as possible. Since the application stores its data on local file system it can't have any robustness and fault-tolerance guaranties.

What do you think about using Amazon S3 API to store templates and instances data? It can greatly improve Cluster Broccoli availability.

How do you think there is place for this functionality in the service?
If yes, I will submit correspondent PR.

[FRosner]

Hi @dormidon!

The application is mainly used if you have some recurring patterns of your nomad applications that you want to make available to end users. If you, for example, need to run multiple versions of the same job that differ only in some parameters and you want people to adjust those parameters themselves, then Broccoli is useful.

About robustness I see your point. We currently store the instances on a network filesystem which is available across the whole nomad cluster. If Broccoli needs to be scheduled somewhere else, this is possible. I think S3 would be similar.

If you wanted to support S3, you would have to write your own implementation of an InstanceStorage, making it configurable and injecting it by providing a Guice module. Switching between different storage implementations is possible through the configuration.

I am not sure about the performance as the instance service is currently not caching anything and this might be a problem when using S3 but we can find out. Would you be interested in adding an S3 storage back-end?

[FRosner]

Feel free to poke me in Gitter for a more responsive communication.

[FRosner]

Ah and one side note, Broccoli can only be deployed as a singleton at the moment, because the login mechanism is stateful and the cache only local. We would need to make some adjustments to fully support that.

[dormidon]

Hi @FRosner!
Thank you for quick response.

Unfortunately, we don't have network filesystem and don't have any plans about it. And I suppose a lot of users don't have too.
But we need to decompose application logic from persistent state for robustness and availability.
Meanwhile, S3 seems to be very common solution and has a lot of Java or Scala client libraries.

I've already written a bit for S3 support (thanks to code structure and clearness). You have mentioned that I need to implement InstanceStorage and provide correspondent configuration settings. But it seems TemplateSource should be implemented on top of S3 too, shouldn't it?

Can you explain your worry about S3 performance?
I think the following.
Typically we have tens or hundreds of templates and instances, S3 API implementations is very fast typically. And it seems that even naive implementation should not have performance issues. Or I'm missing something?

Yep, I want to add S3 storage back-end. And would you be interested in merging it into your project? If so, I will make correspondent PR.

[FRosner]

Hi @dormidon,

Unfortunately, we don't have network filesystem and don't have any plans about it. And I suppose a lot of users don't have too.

That's why I think adding an S3 storage is a great idea!

But we need to decompose application logic from persistent state for robustness and availability.

True that's a good way to do it. I believe that it is mostly done, at least for the instance storage implementations. There is a common interface and you can inject a different storage implementation by providing a corresponding Guice module.

But it seems TemplateSource should be implemented on top of S3 too, shouldn't it?

Right now, templates are read only. One way to deploy them would be to store them in consul and use the template stanza to write them inside the container. Then you can make Nomad send SIGUSR2 to Broccoli which will trigger a reload of the templates.

Of course we can also add an S3 storage for the templates if you'd like. The logic would be the same but with TemplateSource.

About the performance I don't know. I just never actually used anything else than the filesystem storage but I don't see any reason for it to be slow with S3, you're right. Keep in mind that with the current implementation you cannot have two Broccoli instances use the same instance storage! This is because we read the instances once on startup and then only interact with the persistence layer when writing updates.

I'm happy to accept a contribution for making Broccoli work with S3! I was anyway going to release version 0.7.0 in a couple of days so we can aim for 0.8.0 if you like 👍