Best tools for automated deployment

[edwardchalstrey1]

One of the tasks I'm looking to achieve is #764 - but the questions below are also relevant to the current Azure setup being used to host https://colouringlondon.org/

There are a few questions which I think it would be good to have a discussions on, which will be a bit easier for me to comment on once I'm more familiar with the current setup on the Azure VMs for colouring-london.

My thoughts at the moment are that there should probably be a docker container that does everything in the setup, rather than the current vagrant script - my understanding is the vagrant/virtualbox is useful for running things on a local machine with a bit more interactivity than a container would have, but do we want/need that? How do you use it currently @tomalrussell @matkoniecz @mz8i ?

It looks like @matkoniecz is having problems with getting npm playing nicely with vagrant (see #762 ) - I also had issues when I got to the npm install part of the setup and run on my vagrant vm. Could we use a base image for docker that already has things like npm and postgres installed to simplify the setup?

If we did it this, the 2nd step of #764 could be relatively simple, creating a terraform script which:

1. Sets up an appropriate VM
2. Does whatever is needed to get the data in the right place
3. Installs Docker
4. Runs the container

This container could also be run on a laptop with a flag that indicates to load the test OSM polygons, which will make testing changes relatively simple (just re-build the container with the updated repo).

Thoughts?

[tomalrussell]

Background notes: I used to use vagrant a lot for local web development, with the main advantage that it can automatically configure a whole VM environment to match production quite closely. Currently I have less need for multiple full virtual environments and have a dev setup directly on my laptop (postgres and node installed in the main OS).

I think we have three related use cases:

1. dev environment
2. initial site deployment and cloud resource management
3. code deployment and release to staging and production

With a side note on a) data loading and b) database schema migrations.

For a dev environment, ease and reproducibility of setup are important, as well as having a short time between making a change and seeing its effect (code accessible to host OS editor, hot module reload in guest). Docker would make sense for this, especially if that's the more generally familiar tech option. Suggestions: mount code directory as a volume, keep app and database in separate containers (or connect to database hosted outside the container).

Initial deployment should get as far as "empty application running" or "server stuff running ready for first code deploy". Terraform seems like the right tool for initial site setup and cloud resource management - vm/s, storage buckets, updating DNS. Can handle managed database service, container service, etc. if we decide we want any of those things.

Data management is a separate concern - for a development database there are migrations and an initial data import to run (e.g. with OSM polygons) and then potentially lots of up/down migrations to run and test. For a production database, there are similar data import steps to set up building footprints, but after that almost all interaction with the data itself should be via the API, or for archive/backup.

Deploying updates to the code happens much more frequently, and it would be good to have this more automated and reproducible. We want a way to encapsulate the code and deploy reliably to staging and production while preserving cache/logs/archive/backups/database. We also sometimes want to run database migrations (schema changes, new tables).

[matkoniecz]

Right now I use a VM - that sadly was setup in a way that I have not managed to reproduce (!). Which is absurd for obvious reasons, and why I tried to improve setup documentation with a varying success.

I tried to upgrade Vagrant script to (a) work (b) match production, but as seen it run into some issues. Including bizarre ones.

I actually considered to try using Docker before sinking more time into Vagrant.

[edwardchalstrey1]

Thanks @matkoniecz & @tomalrussell - sorry I missed this before as I was looking at the simplified setup steps on the README, but I notice there are instructions in the docs for a dev env and prod env. I'll aim to test these instructions (see #697), but if/once they work it should be in principle straightforward to dockerise