About

Repository for provisioning K3s container orchestration tool, basing on always free infrastructure resources from Oracle Infrastructure Cloud.

Repository

Structure

General overview of the repository structure. Not all files/directories are listed, only these that are specific to the tools in the repository.

.
├── .github                 # GitHub config files
│   ├── workflows           # GitHub Actions config files
│   └── renovate.json       # Renovate config
├── .spacelift              # Spacelift config files
│   └── workflow.yml        # Spacelift workflow tool config file
├── certificates            # Certificates
├── components              # Terraform root modules
├── machine-images          # Source files for machine images
├── modules                 # Terraform modules
│   ├── <module-0>          # Source files for Terraform <module-0>
│   └── ...                 # Other modules
├── stacks                  # Atmos stacks
├── toolbox                 # Toolbox config files
│   ├── rootfs              # Atmos config file dir
│   ├── .gitconfig
│   ├── .mise.toml          # Mise config file
│   └── Dockerfile
├── .pre-commit-config.yaml # Pre-commit config file
├── .releaserc.yaml         # Semantic-release config file
├── tflint.hcl              # (temporary, until static analysis is migrated) TFLint config file
├── .trivyignore.yaml       # Trivy config file
├── README.md
├── vendor.yaml             # Atmos vendor config

Setup

mtweeman/hajle-silesia_provisioning-toolbox Docker image is a preferred way to distribute the tools used in this repository. It's designed to bring consistency for local and remote usage by being cross-platform (macOS, Linux, WSL), multi architecture (linux/amd64, linux/arm64), version controlled and reusable.

Run once:

docker run --rm mtweeman/hajle-silesia_provisioning-toolbox:latest init | bash

Run every time new version was released and updated in the workflows files (for consistency with remote workflows):

docker pull mtweeman/hajle-silesia_provisioning-toolbox:latest

Run on daily basis, preferably as a second terminal, next to the terminal used for git commands:

hajle-silesia_provisioning-toolbox

Example: static analysis with hooks managed by pre-commit:

pre-commit --all-files --hook-stage manual

Following CLI tools are contained within the image:

Name	Description
mise	Tool version manager
k3s	Container orchestration
oci	Oracle Cloud Infrastructure cloud provider
terraform	Infrastructure provisioning, static analysis
atmos	Cloud architecture framework for native Terraform
tflint	Static analysis
trivy	Static analysis
pre-commit	Managing pre-commit hooks
helm	Container orchestration package manager
packer	Machine images provisioning

Dependency updates

Renovate is used as a tool for automated dependency updates. Although it handles many dependencies out of the box, there are many that are not supported yet. These have to be taken care of separately via config file. Verify periodically all dependencies against Renovate latest documentation/config file, to see if dependency support is added/separate handling is still needed. See Renovate console for scanning details.

Static analysis

Dedicated Docker image is a preferred way to run static analysis as it brings consistency for local and remote usage.

Run once, to install hooks in the repository:

pre-commit install

Following static analysis tools are contained within the image with pre-commit hooks serving as an execution tool with following configuration.

Name	Description
pre-commit for Terraform	Hooks manager
Terraform fmt	Canonical format check
Terraform validate	Configuration files validation
TFLint	Linter
Trivy	Security vulnerabilities check

Deployment

Spacelift is used as a tool for orchestration of infrastructure provisioning. It's configured to work with Atmos as described here. See Spacelift console for configuration details. Custom workflow tool is defined here due to Terraform FOSS version constraints. Additional information:

• Spacelift components
• Spacelift admin stack component
• Spacelift spaces component
• Spacelift as TACOS

Version management and package publishing

semantic-release is used as a tool for automated version management and package publishing. See configuration here.

Installation: TODO

1. Oracle Cloud Infrastructure account (free or paid)
2. Deploy OCI KMS

Authentication

Authentication method: API Key Authentication

Architecture

Immutable infrastructure

To avoid configuration drift and shorten deployment time for newly spun instances, immutable infrastructure is a preferred solution for provisioning machine images. HashiCorp Packer is used as a tool for building them.

Secrets management

HashiCorp Vault is used as a secrets management solution for the cluster, deployed as an external service, preferably prior to cluster spinning. Contrary to being installed as a cluster service, it prevents chicken-and-egg situation where it needs to use some sensitive data during provisioning, not yet available at that time. External Secrets Operator automatically generates cluster secrets from the data stored within the vault.

DNS

Hostnames are used for network resources to allow wildcard certificate usage with name defined as described here. Certificate is used in the vault setup. Otherwise, certificate would have to be updated each time private IP address of network resource (VPC, subnet, instance, load balancer, etc.) change.

Access to cluster

terraform init
terraform workspace list
terraform workspace select default
mkdir -p ~/.kube
terraform output -raw kubeconfig > ~/.kube/config-google-cloud
echo 'export KUBECONFIG=~/.kube/config-google-cloud' >> ~/.bashrc
kubectl get nodes