Dockerize ttnhabbridge

[MedadRufus]

Its quite fiddly to deploy TTNhabbridge on remote servers due to the installation requirements of Java etc. I have been using Docker to easily set things up much more recently. In a single command(docker-compose up) I can install all dependencies, including java, and start running ttnhabbridge. I think it will be a good addition to the repo.

My dockerfile looks like this at the moment:

FROM gradle:4.2.1-jdk8-alpine
WORKDIR /ttnhabbridge

# Copy over the files
ADD --chown=gradle:gradle . /ttnhabbridge

# add access rights to run stuff
USER root
RUN chown -R gradle /ttnhabbridge
USER gradle

# Get gradle to build
RUN ./gradlew assemble

# Extract the tar executable
WORKDIR /ttnhabbridge/ttnhabbridge/build/distributions
RUN tar -xvf ttnhabbridge.tar

# Copy our yaml settings file to the executable location. Ensure there is a `ttnhabbridge.yaml` config file in the root of the project.
WORKDIR /ttnhabbridge
COPY ttnhabbridge.yaml ttnhabbridge/build/distributions/ttnhabbridge/

# Enter the executable folder and run the program
WORKDIR /ttnhabbridge/ttnhabbridge/build/distributions/ttnhabbridge
CMD ./ttnhabbridge.sh

My docker-compose.yml looks like this:

version: '3.8'

services:
  web:
    build: .
    ports:
      - "5000:5000"

Things todo:

• Put in docker files into repo.
• Put in readme instructions how to deploy with docker.

[bertrik]

In general I like the idea, I don't have much experience with docker myself, so please go ahead :)

[LukePrior]

I would also recommend setting up GitHub actions to automatically build the docker container when the repo is updated.

We currently do this for many of the Project Horus applications: https://github.com/orgs/projecthorus/packages

You can see some examples of how snh configured it with Github Actions here: https://github.com/projecthorus/tawhiri/blob/master/.github/workflows/container.yml, https://github.com/projecthorus/tawhiri-downloader-container/blob/main/.github/workflows/container.yml, https://github.com/projecthorus/wenet/blob/master/.github/workflows/container.yml

This would make it very easy for newcomers to install and update.

[MedadRufus]

agree. Pushing the latest ttnhabbridge version to a docker registry will be quite useful. The TTNhabbridge currently has a github action to test a successful build of the project but no docker integration.

github action yml: https://github.com/bertrik/ttnhabbridge/blob/1632f137ae0c0e12ecf54d7703255e8945800bbc/.github/workflows/gradle.yml

[bertrik]

My docker knowledge is not so good, but I'm wondering, is this really making things easier, or just introducing different problems.

The current situation is that someone has to install the JDK, run the gradle command and then they have a zip or tarball that they can extract and run.

What do you have with Docker?
Doesn't ​it result in a huge image, about 100 MB if I understand correctly (the Linux runtime environment + JRE)
Now instead of building the Java application with the JDK, you have to install the docker binaries, add local user as a docker user, download a huge image, and you still need to do the docker incantations to start it, along with complications of getting the configuration file into the container and getting the log files out of it.
Is it actually easier to run with docker?

And what would you expect to be present in the archive/releases?
The docker image file as built? or just a Dockerfile with instructions to build the docker image?

[MedadRufus]

Is it actually easier to run with docker?

The main reason I use the docker setup is once I setup the pipeline(install docker, docker-compose), its only one command(docker-compose up) to run the application. Easy. With the original java pipeline, just to run the application, I have to build, then extract the executable file, and then run the program(3+ steps, including moving between folders). Its quite slow and error prone. The docker file contains all the steps that I would use in the java pipeline and will execute all of them in a single command. In addition, for first time setup, I find that its much easier to install the docker tool chain than the Java tool chain(openjdk etc).

But from using the docker route, I have run in to problems extracting the log files from the image; Its quite easy to wipe them out if not careful with docker(docker-compose up --build will wipe it out). This is something I don't have a good solution for.

Passing the config.yaml file into the docker container is done by placing it in the repo's root directory and specifying in the docker file to copy this file over to the right location inside the container(COPY ttnhabbridge.yaml ttnhabbridge/build/distributions/ttnhabbridge/ in the dockerfile example).

[MedadRufus]

Docker seems much better suited for stateless applications. Typically, many docker based applications send out logs and other data to be stored to dedicated databases outside the container.

[MedadRufus]

Also another benefit I found with docker is it will auto restart the application if it somehow is stopped. This could happen if the server is restarted. On startup, docker will fire up the application.

[bertrik]

How I can see this work:

• We create a Dockerfile that tells docker how it can build a docker image. The image is based on a Linux system with a JRE (not JDK). Input is an YAML file with configuration, output is logdata.
• We create a docker-compose.yaml file. This specifies the YAML file as an environment variable and binds the log directory. Alternatively we could just redirect log4j logging to stdout and catch it like that.
• On github we add an action that builds the Java application using gradle, builds the docker file according to the Dockerfile and publishes it (on github also?) as 'latest' version or 'master' something like that.

A user wanting to use it, then has to do the following:

• Install Docker
• Get the docker-compose file, e.g. clone the git repo and enter a 'docker' directory
• Edit an example YAML file with their own TheThingsNetwork application/keys/settings etc
• Run docker-compose up in the 'docker' directory
• Watch the application log file or stdout for things happening in the java application

This way, there is a uniform docker image that can be used by all users, which is customized with their own yaml file.

[LukePrior]

I would recommend a setup similar to what radiosonde_auto_rx uses with an automatically built Docker image using GitHub actions that can accept an external configuration file as an argument that would contain all the keys and settings.

This way the user just needs to download and edit a default configuration file then they can pull the latest docker image and link it to that file. The same method can be used for saving logs to a specified directory.

You can see some info on it here: https://github.com/projecthorus/radiosonde_auto_rx/wiki/Docker

[bertrik]

I played a bit with it this weekend. Steps to run are now:

• Get the source code from git, for a typical user now only the 'docker' directory is relevant
• The 'docker' directory has a docker-compose.yaml file, a default config file and a log directory
• With "docker-compose pull" you can download the docker image from the github docker repo, about 50-60 MB
• You configure the application by editing the application config file, editing keys, payload encoding, etc.
• You start the application using "docker-compose up". Logs are written to stdout and into the log/ directory

This image was created by me by hand by running the gradle script and pushing the resulting image to the github docker repo.
This could be automated further by means of a gitlab action script.

[MedadRufus]

Many thanks @bertrik, I too was working on dockerising ttnhabbridge last weekend. I was working on integrating the MQTT broker for helium in as well. So my objective was to fire up both ttnhabbridge, and an MQTT broker, and get them talking, all in one command:

docker-compose up

I have got it working on my fork here: https://github.com/ImperialSpaceSociety/ttnhabbridge/blob/master/docker-compose.yml

[MedadRufus]

however, I think it would be best to keep ttnhabbridge seperate, and make a seperate repo that hosts a docker-compose.yml file for ttnhabbridge+MQTT broker.
All this repo contains is the config yaml file for ttnhabbridge, and config files, containing MQTT broker credentials.

The user will be able to fire up ttnhabbridege, as well as the MQTT broker easily, and provision the passwords in config files.

[MedadRufus]

I need to figure out how to synchronize my repo with yours. And contribute to bertrik/ttnhabbridge. The only difference is I have a custom packet format for my balloon flights, which is not in bertrik/ttnhabbridge.