Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add info about Executor Builder App to the installation guide #371

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add info about Executor Builder App to the installation guide #371

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1fb6ecb
Add info about Executor Builder App to the installation guide
koettert Nov 5, 2024
3fbe232
revised and added the prebuilt image part
wybert Nov 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 86 additions & 27 deletions docs/geospatial-install.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,32 +1,99 @@
# Setup Custom Executor Docker Image with the Geospatial Extension for KNIME Business Hub

Running a workflow on KNIME Business Hub requires an Execution Context (EC). An EC runs one or multiple pods. A pod holds Docker containers in turn, which run Docker images in the end. In general, one can simply use the official Docker images from the public [KNIME Artifact Registry](https://registry.hub.knime.com/) to create an EC.

## TL;DR

To run workflows with Geospatial Analytics extension nodes on KNIME Business Hub, you'll need,
1. a Docker image that includes this extension. You can either use our prebuilt Docker image or create your own—automatically using the Executor Image Builder or manually building it yourself.
2. create an execution context


## Introduction

Running a workflow on KNIME Business Hub requires an Execution Context (EC). Inside that EC runs one or multiple pods - that hold Docker containers - which run Docker images. In general, one can simply use the official Docker images from the public [KNIME Artifact Registry](https://registry.hub.knime.com/) or [Docker Hub](https://hub.docker.com/r/knime/knime-full) to create an EC. A list of all available default KNIME docker images can be found [here](https://docs.knime.com/latest/business_hub_admin_guide/index.html#docker-executor-images).

However, the official Docker images hold KNIME Extensions only; no Community (trusted or experimental) or Partner Extensions included. Since the "Geospatial Analytics Extension for KNIME" is a Trusted Community Extension, it needs to be integrated into the Docker image, which requires to enhance the official image and build your own custom Docker image.

## Requirements
### Options
This can be achieved in three ways.

1. **Prebuilt**

- One can use prebuilt docker image with geospatial extension installed from [Docker Hub here](https://hub.docker.com/repository/docker/happybeetles/knime-execution/general). The docker file used for building these Docker image is from [here](https://github.com/spatial-data-lab/knime_docker).


2. **Automatic**
- One can use the [_"Executor Image Builder"_](https://hub.knime.com/knime/spaces/KNIME%20Business%20Hub%20Admin%20Workflows/Executor%20Image%20Builder~V8b8qdFAdrBOziiD/current-state) Data App that is part of the Administrator Workflows for KNIME Business Hub can be found [here](https://hub.knime.com/knime/spaces/KNIME%20Business%20Hub%20Admin%20Workflows/~6vfL9nZbxwU3yNbU/).
- This currently only works for online installations. For airgap installations, please refer to the manual steps.
- See below, for more details.
3. **Manual**
- One can use the default KNIME docker images (mentioned above), and build on top of it. This requires to create a Dockerfile that pulls the default image and adds the extension while building the custom image.
- See below, for more details.

### Docker Image Registry

If you are using the first way, you don't have to setup the image registry in the KOTS Admin Console.

In other two ways, one needs to configure a docker V2 registry to be able to provide the custom image to the KNIME Businss Hub. Here, one can either use the [embedded registry](https://docs.knime.com/latest/business_hub_installation_guide/index.html#embedded-docker-registry-configuration) of the KNIME Business Hub itself, or use an [external registry](https://docs.docker.com/get-started/docker-concepts/the-basics/what-is-a-registry/) (like ECR, ACR, etc.). The latter link also explains what a docker image registry is in general.

In case, you are not using the embedded registry but an external one, you can configure your KNIME Business Hub to use that within the KOTS Admin Console unless you use public-access images from Docker Hub.

![registry-settings](./imgs/registry-settings.png)

Please also refer to the official documentation on [Connecting to an External Registry](https://docs.replicated.com/vendor/packaging-private-images).

## Prebuilt Image

If you choose this option, no action is required to create the image; it is automatically built and published by GitHub Actions.

## Automatic Image Creation

This is the most automatic way to provide a docker image that includes the Geospatial Extension to the KNIME Business Hub. This will allow you to
- create the Dockerfile,
- build an executor docker image,
- push it to a registry, and eventually
- make it available when creating or editing an execution context.

Our [documentation](https://docs.knime.com/latest/business_hub_admin_guide/index.html#build-dockerfile-with-data-app) describes the application step by step.

### Requirements
Our [documentation](https://docs.knime.com/latest/business_hub_admin_guide/index.html#add-extension-docker-executor-images) describes the necessary requirements. Among them, one needs
- to enable the Execution Image Builder within the ```KOTS Admin Console``` > ```Config``` > ```Execution Image Builder``` > check ```Enable Execution Image Builder```
- global admin priviliges
- global admin application password

## Manual Image Creation
If it is desired or necessary to not to go with the Data App (e.g. due to an airgap installation or technical preference), one can do the following steps manually:
- create the Dockerfile,
- build an executor docker image,
- push it to a registry, and eventually
- make it available when creating or editing an execution context

Besides the below steps, our [documentation](https://docs.knime.com/latest/business_hub_admin_guide/index.html#build-image-manually) also describes steps around manual docker image creation.

### Requirements
To be able to build your custom Docker image it is required to run [Docker](https://www.docker.com/), more precisely the [Docker Engine](https://docs.docker.com/engine/install/).
Please make yourself familiar with the install instructions according to your operating system.
In general, you can build the image locally. It is not required that the image is built somewhere on the KNIME Business Hub.

## Dockerfile
### Dockerfile
The basis of a Docker image is a Dockerfile. It defines
- the source image (in this case the official Docker image from KNIME; check out all available images [here](https://docs.knime.com/latest/business_hub_admin_guide/index.html#docker-executor-images)),
- the source image (in this case the offical Docker image from KNIME; check out all available images [here](https://docs.knime.com/latest/business_hub_admin_guide/index.html#docker-executor-images)),
- the libraries/packages to be installed, and subsequently
- which extensions should be installed into the executor.

Make sure to save that file with the name ```Dockerfile``` and add no file extension.

Within the Dockerfile, we define two variables at the bottom that are important while installing the extension(s):

```KNIME_UPDATE_SITES``` - the URL to the necessary Update Site(s)
```KNIME_UPDATE_SITES``` the URL to the necessary Update Site(s)

```KNIME_FEATURES``` - the Feature ID of the extension to be installed
```KNIME_FEATURES``` the Feature ID of the extension to be installed

Check out [this video](https://www.youtube.com/watch?v=-dO79Id3VAo&t=143s) from the KNIMETV channel that helps you to find the right update site URL and the Feature ID.
Check out [this video](https://www.youtube.com/watch?v=-dO79Id3VAo&t=143s) from the KNIMETV channel that helps you to find the right update site URL and the feature ID.

# Define the base image
FROM registry.hub.knime.com/knime/knime-full:r-5.2.1-369
FROM knime/knime-full:r-5.3.2-564

# Change to root user to be able to install system packages
USER root
Expand All @@ -52,30 +119,22 @@ Check out [this video](https://www.youtube.com/watch?v=-dO79Id3VAo&t=143s) from
# Execute extension installation script
RUN ./install-extensions.sh

## Build and push the Docker image
Now, the Docker image can be built from the Dockerfile and made available to the KNIME Business Hub with the following commands. Please make sure to adapt the following lines accordingly, when running them in the CLI of your laptop.
### Build and push the Docker image
Now, the Docker image can be build from the Dockerfile and made available to the KNIME Business Hub with the following commands.

# 1) Build the Docker image
# Build the Docker image
docker build . -f .\Dockerfile -t knime-full-geo:5.2.1 --no-cache

# 2) Retag the image to make it readable for KNIME Business Hub
docker tag knime-full-geo:5.2.1 registry.<base-url>/knime-full-geo:5.2.1

# 3) Login to the embedded registry of the KNIME Business Hub (if configured)
# Authenticate with the registry:
# You will need to provide a password. In case you do not have one you can manage this from the KOTS Admin Console > Config > Embedded Registry.
docker login --username <username> registry.<base-url>

# 4) Push the image to registry
docker push registry.<base-url>/knime-full-geo:5.2.1
# Retag the image to make it readable for KNIME Business Hub
docker tag knime-full-geo:5.2.1 registry.<hub-url>/knime-full-geo:5.2.1

In case, you are not using the embedded registry but an external one, you can configure your KNIME Business Hub to use that within the KOTS Admin Console. Using an external docker registry, requires adapting the steps above accordingly.
# Login to the embedded registry of the KNIME Business Hub (if configured)
docker login --username <username> registry.<hub-url>

![registry-settings](./imgs/registry-settings.png)

Please also refer to the official documentation on [Connecting to an External Registry](https://docs.replicated.com/vendor/packaging-private-images).
# Push the image to registry
docker push registry.<hub-url>/knime-full-geo:5.2.1

## Check if the Docker image is available on the registry
### Check if the Docker image is available on the registry
Once the upload is finished you can double check if that image can be found on the registry by searching the Docker registry API. Therefore, it is basically enough to use your web browser.

# Check which image names are available
Expand All @@ -94,4 +153,4 @@ Once the upload is finished you can double check if that image can be found on t
## Creating an Execution Context
Now you can apply the Docker image and create an EC from it by passing the registry string under the "Docker image" section while creating an EC. Please read the [KNIME docs](https://docs.knime.com/latest/business_hub_user_guide/index.html#execution_contexts) to find out more about how to setup a new execution context.

![create-execution-context](./imgs/create-execution-context.png)
![create-execution-context](./imgs/create-execution-context.png)