diff --git a/README.md b/README.md index 0d25053..588ab20 100644 --- a/README.md +++ b/README.md @@ -1,38 +1,59 @@ # Helm Ansible Template Exporter -## Description +Helm Ansible Template Exporter is a set of tools aimed at helping K8S application developers transition from Helm to +Ansible Role Based Operators. This project is separated into two main tools: -This tool automates conversion from Helm Chart to Ansible Playbook Role. Helm Charts are defined utilizing Go -templates, while Ansible Playbook Roles are defined using Yaml and Jinja2 templates. Due to the fact that Go and Jinja2 -are completely separate languages, some facets of Helm charts are difficult or impossible to convert directly to -Ansible Playbook roles. This tool attempts to automatically convert a Helm chart into an Ansible Playbook Role, but -some aspects of the conversion process must be performed manually by hand after utilizing this tool. +1) Helm To Ansible Exporter +2) [Ansible Operator Builder](./hack/README.md) -## Current Capabilities +The Ansible Operator Builder is targeted towards developer usage, and is located in the [hack directory](./hack). + +## Helm To Ansible Exporter + +This tool automates conversion from [Helm Chart](https://helm.sh/) to +[Ansible Playbook Role](https://docs.ansible.com/ansible/latest/user_guide/playbooks_reuse_roles.html). Helm Charts are +defined utilizing [Go Template Language](https://golang.org/pkg/text/template/), and Ansible Playbook Roles are +defined using Yaml and [Jinja2 Template Language](https://jinja.palletsprojects.com). Due to the fact that Go Template +Language and Jinja2 Template Language are completely separate languages, some facets of Helm charts are difficult or +impossible to convert automatically. This tool performs a best attempt conversion of a Helm chart into an Ansible +Playbook Role, but some aspects of the conversion process must be performed manually by hand after utilizing this tool. +Additionally, when ambiguities in conversion arise, this tool makes a best attempt to emit warnings where a manual +conversion or inspection is needed after running the tool. + +The output of this tool is an Ansible Role which can be installed using +[Ansible Playbook](https://docs.ansible.com/ansible/latest/cli/ansible-playbook.html), or built into an operator using +the Ansible Operator Builder. + +### Helm To Ansible Exporter Current Capabilities The current offering does the following: -1) Creates a role in the workspace directory using ansible-galaxy. -2) Raw copies templates into the generated Ansible Playbook Role templates directory, renaming each template with a - ".j2" extension. -3) Merges values.yml (or values.yaml) into the generated Ansible Playbook Role defaults/main.yml file. -4) Searches the generated Ansible Playbook Role's defaults/main.yml file for self references (i.e., references to - .Values.) and comments them out. Ansible Playbook is incapable of expressing self references in defaults/main.yml, - a clear technology gap between Ansible Playbook and Helm charts. A "WARN" message is output indicating that a manual - change is required to defaults/main.yml on the appropriate lines. -5) Convert Branch syntax for "if", "for" and "for key/value" in each template to utilize proper Jinja2 syntax. This - includes a heuristic which attempts to determine if conditionals are checking for definition v.s. boolean evaluation. -6) Convert boolean composition ordering. Go Templating utilizes "and " format. On the other - hand, Jinja2 utilizes " and " formatting. helmExport handles this conversion automatically. -7) Template functions invocations are converted to Jinja2 Ansible Filter invocations. This requires converting direct - function invocations to piped invocations, as well as inserting the appropriate parentheses and commas for argument - lists. -8) Removes references to ".Values." in the generated Ansible Playbook's Roles' templates. Ansible Playbook can directly - reference the values in defaults/main.yml, so this reference isn't needed. -9) If the "generateFilters" flag is set to true, some stub Ansible Filter implementations are installed for use in - converting Go template functions to Ansible Playbook Filters. - -## Known Limitations - -### Go/Sprig Template Function Limitations +1) Creates a role in the workspace directory using ansible-galaxy. +2) Raw copies templates into the generated Ansible Playbook Role templates directory, renaming each template with a + ".j2" extension. +3) Merges values.yml (or values.yaml) into the generated Ansible Playbook Role defaults/main.yml file. +4) Searches the generated Ansible Playbook Role's defaults/main.yml file for self references (i.e., references to + .Values.) and comments them out. Ansible Playbook is incapable of expressing self references in defaults/main.yml, + a clear technology gap between Ansible Playbook and Helm charts. A "WARN" message is output indicating that a + manual change is required to defaults/main.yml on the appropriate lines. +5) Convert Branch syntax for `if`, `range ` and `range ` in each template to utilize proper Jinja2 syntax. + This includes a heuristic which attempts to determine if conditionals are checking for definition v.s. boolean + evaluation. +6) Convert boolean composition ordering. Go Templating utilizes "and " format. On the other + hand, Jinja2 utilizes " and " formatting. helmExport handles this conversion automatically. +7) Template functions invocations are converted to Jinja2 Ansible Filter invocations. This requires converting direct + function invocations to piped invocations, as well as inserting the appropriate parentheses and commas for argument + lists. +8) Removes references to ".Values." in the generated Ansible Playbook's Roles' templates. Ansible Playbook can directly + reference the values in defaults/main.yml, so this reference isn't needed. +9) If the "generateFilters" flag is set to true, some stub Ansible Filter implementations are installed for use in + converting Go template functions to Ansible Playbook Filters. +10) If the "emitKeysSnakeCase" flag is set to true (default), all Ansible keys are converted to snake_case. This option + is especially useful if you plan to generate a K8S operator, since the operator-sdk converts Custom Resource + variables to snake_case. This tool directly invokes the ToSnake provided by operator-sdk in an attempt to exactly + match the conversion functionality. + +### Helm To Ansible Exporter Known Limitations + +#### Go/Sprig Template Function Limitations [Ansible Playbook Filter(s)](https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html) are one candidate to replace Go/Sprig Template Functions. However, they are not a direct 1:1 mapping, so a few extra steps must @@ -46,13 +67,13 @@ into Ansible. For instances where there does not seem to be a replacement, you "generateFilters" command line argument, some example filters will be installed into your generated Ansible Playbook Role. -### "template" and "include" are not supported +#### "template" and "include" are not supported Go Templates provide "template" and "include" in order to support dynamic template creation. Ansible has no direct replacement, and although there are some similar constructs, helmExport currently doesn't support any conversion. Instead, consider using Ansible defaults as a replacement. -## Building +### Building The Exporter To build the code, use the following command: @@ -65,15 +86,15 @@ To clean the code base, use the following command: make clean ``` -## Running the Code +### Running the Exporter -### Runtime Dependencies +#### Runtime Dependencies Helm Ansible Template Exporter requires [ansible-galaxy](https://galaxy.ansible.com/) to initialize exported roles. -Additionally, ansible-galaxy must be included in the $PATH. Additionally, if you utilize the "generateFilters" flag, -then a Go compiler must be installed. +Additionally, ansible-galaxy must be included in the $PATH. If you utilize the "generateFilters" flag, then a Go +compiler must be installed. -### Run Instructions +#### Run Instructions ```shell script make run @@ -83,5 +104,62 @@ This will run the current implementation of code. Alternatively: ```shell script -./helmExport export nginx --helm-chart=./example --workspace=./workspace --generateFilters=true +./helmExport export nginx --helm-chart=./example --workspace=./workspace --generateFilters=true --emitKeysSnakeCase=true +``` + +### Testing the Ansible Playbook Role + +Ansible Operators are deployed using the +[`k8s` Ansible Module](https://docs.ansible.com/ansible/latest/modules/k8s_module.html). The `k8s` Ansible Module +allows developers to leverage existing Kubernetes resource YAML files, and express lifecycle management in native +Ansible. One benefit of using Ansible in conjunction with existing Kubernetes resource files is the ability to use +Jinja2 templating to customize deployments. + +#### Installing the k8s Ansible Module + +The k8s Ansible Module requires Ansible. To install on Fedora/CentOS: + +```shell script +sudo dnf install ansible +``` + +The OpenShift Restclient Python package is also required. Install using pip: + +```shell script +pip3 install openshift ``` + +Lastly, the Ansible Kubernetes collection is required. Install using ansible-galaxy: + +```shell script +ansible-galaxy collection install community.kubernetes +``` + +#### Testing the Ansible Playbook Role Using the k8s Ansible Module + +1. Start a K8S cluster. For example, using [minikube](https://minikube.sigs.k8s.io/docs/): + +```shell script +minikube start +``` + +2. Create an Ansible Playbook called `workspace/role/playbook.yaml`. + +```yaml +--- +- hosts: localhost + roles: + - Foo +``` + +3. Debug any issues not covered by the Helm Ansible Template Exporter by running this diagnostic command: + +```shell script +ansible-playbook playbook.yaml -vvv +``` + +## Known Test Environments + +This tool has been tested using: +* Fedora 31 +* macOS Catalina Version 10.15.4 diff --git a/hack/README.md b/hack/README.md index 272ca07..8194b1f 100644 --- a/hack/README.md +++ b/hack/README.md @@ -1,156 +1,116 @@ -### Steps to build an operator from an existing helm charts +## Ansible Operator Builder -##### ./hack/build-operator.sh -h -~~~ +Transitioning a Helm Chart to an Ansible Playbook Role is useful since many users prefer Ansible to Go Templating. +However, a transition to Ansible Playbook Role does not address Day 2 facets of +[Operational Lifecycle Management (OLM)](https://github.com/operator-framework/operator-lifecycle-manager). +The operator-framework [operator-sdk](https://github.com/operator-framework/operator-sdk) includes functionality to +develop Lifecycle Management using an Ansible Playbook Role. + +[`build-operator.sh`](./hack/build-operator.sh) is a tool that takes an exported Ansible Role and builds an Ansible +Operator. + +### Build An Operator From An Existing Helm Chart + +build-operator comes with a number of command line options described below: + +``` usage: build-operator.sh -r -b -d -e -c | -h -e | --export : Export helmcharts and create Ansible operator -b | --build : Build Operator image and push it to quay.io -d | --deploy: Deploy this operator to existing cluster -r | --run : Option to run the operator outside the cluster -c | --delete : Clean the cluster by deleteing the operator -~~~ -#####Setting up environment variables - Step 1: update and source ./env.sh to set environment variables or - set following variables in command line. - * Note: If you override any variables via CLI, first source env.sh, which will unset some env variables. - Example: - ``` -source env.sh -export role=foo -export workspace=./workspace -export helm_chart=path to your helmchart -#Change this to your namespace -export quay_namespace="YOUR_NAMESPACE" -export INSTALL_OPERATOR_SDK=1 - -``` - Note : To unset all exported variables run this command. - ` unset $(awk -F'[ =]+' '/^export/{print $2}' ./env.sh)` - -##### Change following variables. -* `role` : should be your helm chart name, which becomes Ansible role. -* `worskpace` : Location where the operator and exported templates will be created. -* `helm_chart`: Local path to the helm charts to export. -* INSTALL_OPERATOR_SDK=1 #if you want the script to install operator sdk. - - ###### Variables required by operator-sdk -*`quay_namespace`: Your quay namespace to build and push the operator image -* `kind` : Kind of the CR to be created -* `apiVersion` : Version of the CR to be created. - -Example -``` -apiVersion: "foo.example.com/v1alpha1" -kind: "Foo" -metadata: - name: "{{ meta.name }}" ``` -Once you are satisfied with above variables, source the file(if you not overriding via cli) -` -source ./env.sh -` -#####Running build-operator script to create an operator -1. ``./hack/build-operator.sh --export`` -The `--export` option will export the existing helm charts into an operator, by creating a scaffold using operator sdk and -parsing helm templates and converting them to Ansible templates. -The output will create two folder under the workspace folder as set via workspace env variable. -``` -{workspace}- - - {Role} # This is exported helm files generated but this tool for you reference. - Operator sdk will copy the contents into {role}-operator folder and create an operator. - You can delete this folder or keep it for your reference. - - {Operator} #This is the final operator folder structure. - The Anisible file structure within the operator will be same as mentioned in main README file for the outputed exported ansible role. - ``` - -The converted operator is not yet usable since exporting tool is unable to convert all the templates fields. -For more details , read main [README.md](../README.md) file. -### Testing Exported K8s Ansible template -Now you have an operator created , Lets see how to test the exported Ansible templates -and make changes to the templates and make it work. - - Step 1 : Have a Cluster up and running , example using minikube -`minikube start` - - Step 2 Create a playbook.yaml (Which you can delete after testing and fixing Ansible templates) and copy it under roles folder - `Create an Ansible playbook playbook.yml in the top-level directory which includes role example Foo:` -``` ---- -- hosts: localhost - roles: - - Foo -``` -Getting started with the k8s Ansible modules -Since we are interested in using Ansible for the lifecycle management of our application on Kubernetes, -it is beneficial for a developer to get a good grasp of the k8s Ansible module. -This Ansible module allows a developer to either leverage their existing Kubernetes resource files (written in YaML) or express the lifecycle management in native Ansible. -One of the biggest benefits of using Ansible in conjunction with existing Kubernetes resource files is the ability to -use Jinja templating so that you can customize deployments with the simplicity of a few variables in Ansible. +#### Configuring Environment Variables +build-operator.sh is configured through setting a number of environment variables in [`env.sh`](env.sh). -The easiest way to get started is to install the modules on your local machine and test them using a playbook. -Installing the k8s Ansible modules +1. Change following variables: + * `role`: The Helm Chart name, which becomes Ansible Role Name. + * `worskpace`: An arbitrary directory location to export the target Operator and Ansible Playbook. + * `helm_chart`: File Path location to the original helm charts. + * `quay_namespace`: Your quay.io namespace. + * `kind`: The Kind of the Custom Resource(CR) to create. + * `apiVersion`: Version of the CR to create. + * `INSTALL_OPERATOR_SDK=1`: If you want the script to install operator-sdk. -To install the k8s Ansible modules, one must first install Ansible 2.9+. On Fedora/Centos: +2. Source ./env.sh to set environment variables or set following variables in command line. -`$ sudo dnf install ansible` +```shell script +source env.sh +``` -In addition to Ansible, a user must install the OpenShift Restclient Python package. This can be installed from pip: +*Note*: To unset all exported variables, run this command: -`$ pip3 install openshift` +```shell script +unset $(awk -F'[ =]+' '/^export/{print $2}' ./env.sh) +``` -Finally, a user must install the Ansible Kubernetes collection from ansible-galaxy: +#### Running build-operator script to create an operator -`$ ansible-galaxy collection install community.kubernetes` +Export the Helm Chart as an Ansible Playbook -Run `$ansible-playbook playbook.yaml -vvv` +```shell script +./hack/build-operator.sh --export +``` -Debug the output and fix all the changes required to make this template working as mentioned in README. +The `--export` option will generate an Ansible Playbook Role and corresponding Operator implementation in the +`workspace` directory. The `workspace` directory will contain two folders: +* `role`: The Ansible Playbook Role, which is exported using Helm Template Ansible Exporter. This directory + can optionally be deleted, since its contents are copied into `workspace/{role}-operator`. +* `role-operator`: The generated operator, which can be deployed to a K8S cluster. + +The converted operator in `workspace/role-operator` may not yet be usable since Helm Template Ansible Exporter cannot +automate 100% of the conversion process. See the "Helm To Ansible Exporter Known Limitations" for more information. -2. Edit all the templates and change the name of the resource to `name: {{ meta.name }}` and any namespace mentioned to -`namespace: {{ meta.namespace }}`, This will give the k8s unique name based off its custom resource name and namespace. - -3. ``./hack/build-operator.sh --build`` -This will build the operator image and deploy to a quay repository and update the operator.yaml with the uploaded -image name. -Under the hood, the script will be running following commands -``` -$ operator-sdk build quay.io/example/foo-operator:v0.0.1 -$ docker push quay.io/example/foo-operator:v0.0.1 +### Building the Ansible Operator + +Invoke `build-operator.sh` using the build argument: + +```shell script +./hack/build-operator.sh --build ``` -4. ``./hack/build-operator.sh --deploy`` -This is will deploy the operator to running cluster -The above script will be running similar commands +The `--build` argument builds the Docker container image, uploads the built image to the quay repository, and update +`operator.yaml` with the uploaded image name. + +### Deploying the Ansible Operator + +Invoke `build-operator.sh` uisng the deploy argument: + +```shell script +./hack/build-operator.sh --deploy ``` -sed -i 's|{{REPLACE_IMAGE}}|quay.io/example/foo-operator:v0.0.1|g' deploy/operator.yaml -$ kubectl create -f deploy/crds/foo.example.com_foos_crd.yaml # if CRD doesn't exist already -$ kubectl create -f deploy/service_account.yaml -$ kubectl create -f deploy/role.yaml -$ kubectl create -f deploy/role_binding.yaml -$ kubectl create -f deploy/operator.yaml - ``` + +The `--deploy` argument deploys the operator to the K8S cluster. -Now you can now override the Ansible variables ( roles/deafault/main.yaml) from your CR file which is found under deploy/crds/foo.example.com_foos_cr.yaml -All the variables you wish to override should be under the spec: metadata of the this CR -Then apply the CR -` kubectl apply -f deploy/crds/foo.example.com_foos_cr.yaml ` -Applying the CR will make the operator to deploy all kubernetes resources from the ansible templates. +You can now override the Ansible defaults variables `roles/defaults/main.yaml` from your CR file +`deploy/crds/foo.example.com_foos_cr.yaml`. The variables that can be overridden are in the `spec` metadata of the CR. +After making appropriate edits to the defaults, you can apply the CR using: -For more information read this [operator sdk ansible developers guide](https://github.com/operator-framework/operator-sdk/blob/master/doc/ansible/dev/developer_guide.md) +```shell script +kubectl apply -f deploy/crds/foo.example.com_foos_cr.yaml +``` +Applying the CR will cause the operator to deploy all K8S resources from the Ansible templates. For more information +consult the [operator-sdk Ansible Developer Guide](https://github.com/operator-framework/operator-sdk/blob/master/doc/ansible/dev/developer_guide.md). -5. Debugging Operator and checking if its running fine -* checking if operator is running - * kubectl get pods -n default #namespace can be default or the namespace that was deployed to - * kubectl logs -f {the operator pod} -n default -c operator - * kubectl logs -f {the operator pod} -n default -c ansible +### Debugging the Running Ansible Operator - -6. ``./hack/build-operator.sh --delete `` -This will delete the deployed operator from the cluster. +Use commands similar to the following in order to debug the running Ansible Operator: - +```shell script +kubectl get pods -n default +kubectl logs -f -n default -c operator +kubectl logs -f -n default -c ansible +``` + +### Delete the Ansible Operator from the K8S Cluster +To delete the Ansible Operator from the K8S cluster, run the following: +```shell script +./hack/build-operator.sh --delete +``` +This will delete the deployed operator from the cluster.