Skip to content

Commit

Permalink
feat: add external workers (#20)
Browse files Browse the repository at this point in the history
* feat: add external workers

This adds a new input `enable_external_workers` that replaces the old
`create_compute_address_for_mqtt` that was incomplete.

If this flag is enabled, we create two additional IPs and a DNS zone to
be able to resolve the same mqtt endpoint address from both inside and
outside of the cluster.

It also create a new artifact repo and enable public read access on it.
We use the public repo as the exported launcher image variable.

I moved subnetwork in network module.

* fixup! feat: add external workers

* fixup! feat: add external workers
  • Loading branch information
eliecharra authored Dec 17, 2024
1 parent 8941848 commit ac18ce9
Show file tree
Hide file tree
Showing 18 changed files with 287 additions and 98 deletions.
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ The module creates:
- a compute subnetwork for the GKE cluster
- Artifact repository
- a Google Artifact Registry repository for storing Docker images
- a PUBLIC Google Artifact Registry repository for storing Docker images for workers (if external workers are enabled)
- Database resources
- a Postgres Cloud SQL instance
- Storage resources
Expand All @@ -41,7 +42,7 @@ The module creates:
### Inputs

| Name | Description | Type | Default | Required |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------- | --------------------- | -------- |
|---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------- |-----------------------| -------- |
| region | The region in which the resources will be created. | string | - | yes |
| project | The ID of the project in which the resources will be created. | string | - | yes |
| website_domain | The domain under which the Spacelift instance will be hosted. This is used for the CORS rules of one of the buckets. | string | - | yes |
Expand All @@ -55,6 +56,7 @@ The module creates:
| ip_cidr_range | The IP CIDR range for the subnetwork used by the GKE cluster | string | 10.0.0.0/16 | no |
| secondary_ip_range_for_services | The secondary IP range for the subnetwork used by the GKE cluster. This range is used for services | string | 192.168.16.0/22 | no |
| secondary_ip_range_for_pods | The secondary IP range for the subnetwork used by the GKE cluster. This range is used for pods | string | 192.168.0.0/20 | no |
| enable_external_workers | Switch this to true if you want to run workers from outside of the GCP infrastructure. | string | false | no |

### Outputs

Expand All @@ -72,8 +74,8 @@ The module creates:
| gke_cluster_name | Name of the GKE cluster. |
| gke_public_v4_address | Public IPv4 address of the GKE cluster. |
| gke_public_v6_address | Public IPv6 address of the GKE cluster. |
| mqtt_ipv4_address | IPv4 address of the MQTT service. It's null if create_compute_address_for_mqtt is false. It's only useful in case the workerpool is outside the GKE cluster. |
| mqtt_ipv6_address | IPv6 address of the MQTT service. It's null if create_compute_address_for_mqtt is false. It's only useful in case the workerpool is outside the GKE cluster. |
| mqtt_ipv4_address | IPv4 address of the MQTT service. It's null if enable_external_workers is false. It's only useful in case the workerpool is outside the GKE cluster. |
| mqtt_ipv6_address | IPv6 address of the MQTT service. It's null if enable_external_workers is false. It's only useful in case the workerpool is outside the GKE cluster. |
| artifact_repository_url | URL of the Docker artifact repository. |
| db_name | Name of the database. |
| db_root_password | Database root password |
Expand Down
26 changes: 22 additions & 4 deletions main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,20 @@ module "artifacts" {
source = "./modules/artifacts"
depends_on = [module.iam]
seed = random_id.seed.hex

enable_external_workers = var.enable_external_workers
}

module "network" {
source = "./modules/network"
depends_on = [module.iam]
seed = random_id.seed.hex

enable_external_workers = var.enable_external_workers
ip_cidr_range = var.ip_cidr_range
secondary_ip_range_for_pods = var.secondary_ip_range_for_pods
secondary_ip_range_for_services = var.secondary_ip_range_for_services
region = var.region
}

module "gke" {
Expand All @@ -29,15 +37,14 @@ module "gke" {
app_service_account_name = var.app_service_account_name
backend_service_account_id = module.iam.backend_service_account_id
compute_network_id = module.network.network_id
subnetwork = module.network.subnetwork
pods_ip_range_name = module.network.pods_ip_range_name
services_ip_range_name = module.network.services_ip_range_name
compute_network_name = module.network.network_name
create_compute_address_for_mqtt = var.create_compute_address_for_mqtt
gke_service_account_email = module.iam.gke_service_account_email
ip_cidr_range = var.ip_cidr_range
k8s_namespace = var.k8s_namespace
project = var.project
region = var.region
secondary_ip_range_for_pods = var.secondary_ip_range_for_pods
secondary_ip_range_for_services = var.secondary_ip_range_for_services
}

module "db" {
Expand All @@ -62,3 +69,14 @@ module "storage" {
project = var.project
region = var.region
}

module "dns" {
source = "./modules/dns"
seed = random_id.seed.hex

enable_external_workers = var.enable_external_workers
website_domain = var.website_domain
compute_network_id = module.network.network_id
gke_public_v4_address = module.network.gke_public_v4_address
gke_public_v6_address = module.network.gke_public_v6_address
}
22 changes: 22 additions & 0 deletions modules/artifacts/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,25 @@ resource "google_artifact_registry_repository" "spacelift" {
}
}
}

resource "google_artifact_registry_repository" "spacelift-public" {
count = var.enable_external_workers ? 1 : 0
repository_id = "spacelift-public-${var.seed}"
format = "DOCKER"
description = "This repository contains the public images for Spacelift"

cleanup_policies {
id = "keep-10-images"
action = "KEEP"
most_recent_versions {
keep_count = 10
}
}
}

resource "google_artifact_registry_repository_iam_binding" "public" {
count = var.enable_external_workers ? 1 : 0
members = ["allUsers"]
repository = google_artifact_registry_repository.spacelift-public[0].id
role = "roles/artifactregistry.reader"
}
15 changes: 13 additions & 2 deletions modules/artifacts/outputs.tf
Original file line number Diff line number Diff line change
@@ -1,9 +1,20 @@
locals {
repository_domain = "${google_artifact_registry_repository.spacelift.location}-docker.pkg.dev"
repository_url = "${local.repository_domain}/${google_artifact_registry_repository.spacelift.project}/${google_artifact_registry_repository.spacelift.repository_id}"
public_repository_url = var.enable_external_workers ? "${local.repository_domain}/${google_artifact_registry_repository.spacelift-public[0].project}/${google_artifact_registry_repository.spacelift-public[0].repository_id}" : local.repository_url
}

output "repository_domain" {
value = "${google_artifact_registry_repository.spacelift.location}-docker.pkg.dev/"
value = local.repository_domain
description = "The domain of the Docker repository"
}

output "repository_url" {
value = "${google_artifact_registry_repository.spacelift.location}-docker.pkg.dev/${google_artifact_registry_repository.spacelift.project}/${google_artifact_registry_repository.spacelift.repository_id}"
value = local.repository_url
description = "The URL of the Docker repository"
}

output "launcher_repository_url" {
value = local.public_repository_url
description = "The URL of the public Docker repository"
}
4 changes: 4 additions & 0 deletions modules/artifacts/variables.tf
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
variable "seed" {
type = string
}

variable "enable_external_workers" {
type = bool
}
51 changes: 51 additions & 0 deletions modules/dns/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
locals {
dns_name = join(".", slice(split(".", var.website_domain), length(split(".", var.website_domain))-2, length(split(".", var.website_domain))))
count = var.enable_external_workers ? 1 : 0
}

resource "google_dns_managed_zone" "main" {
count = local.count
name = "${replace(local.dns_name, ".", "-")}-${var.seed}"
dns_name = "${local.dns_name}."

visibility = "private"

private_visibility_config {
networks {
network_url = var.compute_network_id
}
}
}

resource "google_dns_record_set" "CNAME_mqtt" {
count = local.count
managed_zone = google_dns_managed_zone.main[0].name

name = "${var.mqtt_subdomain}.${var.website_domain}."
type = "CNAME"
ttl = 300

rrdatas = [var.mqtt_service_alias]
}

resource "google_dns_record_set" "A_website_domain" {
count = local.count
managed_zone = google_dns_managed_zone.main[0].name

name = "${var.website_domain}."
type = "A"
ttl = 300

rrdatas = [var.gke_public_v4_address]
}

resource "google_dns_record_set" "AAAA_website_domain" {
count = local.count
managed_zone = google_dns_managed_zone.main[0].name

name = "${var.website_domain}."
type = "AAAA"
ttl = 300

rrdatas = [var.gke_public_v6_address]
}
4 changes: 4 additions & 0 deletions modules/dns/outputs.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
output "mqtt_endpoint" {
value = var.enable_external_workers ? trimsuffix(google_dns_record_set.CNAME_mqtt[0].name, ".") : var.mqtt_service_alias
description = "Address of the MQTT endpoint."
}
35 changes: 35 additions & 0 deletions modules/dns/variables.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
variable "seed" {
type = string
}

variable "enable_external_workers" {
type = bool
}

variable "website_domain" {
type = string
description = "Domain name for the Spacelift frontend without protocol (e.g. spacelift.mycompany.com)."
}

variable "compute_network_id" {
type = string
description = "The ID of the network to create the GKE cluster in"
}

variable "gke_public_v4_address" {
type = string
}

variable "gke_public_v6_address" {
type = string
}

variable "mqtt_subdomain" {
type = string
default = "mqtt"
}

variable "mqtt_service_alias" {
type = string
default = "spacelift-mqtt.spacelift.svc.cluster.local."
}
6 changes: 3 additions & 3 deletions modules/gke/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ resource "google_container_cluster" "spacelift" {
}

network = var.compute_network_id
subnetwork = google_compute_subnetwork.default.self_link
subnetwork = var.subnetwork.self_link

private_cluster_config {
# This prevents nodes to be assigned any external IP that can be routed on the internet
Expand All @@ -24,8 +24,8 @@ resource "google_container_cluster" "spacelift" {

ip_allocation_policy {
stack_type = "IPV4_IPV6"
services_secondary_range_name = google_compute_subnetwork.default.secondary_ip_range[0].range_name
cluster_secondary_range_name = google_compute_subnetwork.default.secondary_ip_range[1].range_name
services_secondary_range_name = var.services_ip_range_name
cluster_secondary_range_name = var.pods_ip_range_name
}

# Set `deletion_protection` to `true` will ensure that one cannot
Expand Down
17 changes: 0 additions & 17 deletions modules/gke/mqtt.tf

This file was deleted.

25 changes: 2 additions & 23 deletions modules/gke/network.tf
Original file line number Diff line number Diff line change
@@ -1,24 +1,3 @@
resource "google_compute_subnetwork" "default" {
name = "spacelift-gke-cluster-subnetwork"

ip_cidr_range = var.ip_cidr_range
region = var.region

stack_type = "IPV4_IPV6"
ipv6_access_type = "EXTERNAL"

network = var.compute_network_id
secondary_ip_range {
range_name = "services-range"
ip_cidr_range = var.secondary_ip_range_for_services
}

secondary_ip_range {
range_name = "pod-ranges"
ip_cidr_range = var.secondary_ip_range_for_pods
}
}

# This module create a Cloud router & Cloud NAT is used to allow outbound traffic from k8s pods
# Only SNAT is allowed here, this should not be used for routing incoming traffic to pods.
module "gke-router" {
Expand All @@ -34,10 +13,10 @@ module "gke-router" {
source_subnetwork_ip_ranges_to_nat = "LIST_OF_SUBNETWORKS"
subnetworks = [
{
name = google_compute_subnetwork.default.id
name = var.subnetwork.id
source_ip_ranges_to_nat = ["PRIMARY_IP_RANGE", "LIST_OF_SECONDARY_IP_RANGES"]
secondary_ip_range_names = [
google_compute_subnetwork.default.secondary_ip_range[1].range_name,
var.pods_ip_range_name,
]
}
]
Expand Down
20 changes: 0 additions & 20 deletions modules/gke/outputs.tf
Original file line number Diff line number Diff line change
@@ -1,24 +1,4 @@
output "gke_subnetwork_id" {
value = google_compute_subnetwork.default.id
description = "The ID of the subnetwork that the GKE cluster was created in"
}

output "gke_subnetwork_name" {
value = google_compute_subnetwork.default.name
description = "The name of the subnetwork that the GKE cluster was created in"
}

output "gke_cluster_name" {
value = google_container_cluster.spacelift.name
description = "The name of the GKE cluster"
}

output "mqtt_ipv4_address" {
value = var.create_compute_address_for_mqtt ? google_compute_address.gke-mqtt-v4[0].address : null
description = "The IPv4 address of the MQTT service"
}

output "mqtt_ipv6_address" {
value = var.create_compute_address_for_mqtt ? google_compute_address.gke-mqtt-v6[0].address : null
description = "The IPv6 address of the MQTT service"
}
27 changes: 11 additions & 16 deletions modules/gke/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -23,29 +23,24 @@ variable "compute_network_id" {
description = "The ID of the network to create the GKE cluster in"
}

variable "compute_network_name" {
type = string
description = "The name of the network to create the GKE cluster in"
variable "subnetwork" {
type = object({
id = string
self_link = string
})
}

variable "create_compute_address_for_mqtt" {
type = bool
description = "Whether to create a compute address for MQTT. It is meant to be used by Service of type LoadBalancer from the GKE cluster to expose the embedded MQTT server to the world. This is only required if you want to run worker outside of the GKE cluster."
}

variable "ip_cidr_range" {
type = string
description = "The IP CIDR range for the GKE cluster"
variable "services_ip_range_name" {
type = string
}

variable "secondary_ip_range_for_services" {
type = string
description = "The secondary IP range for services"
variable "pods_ip_range_name" {
type = string
}

variable "secondary_ip_range_for_pods" {
variable "compute_network_name" {
type = string
description = "The secondary IP range for pods"
description = "The name of the network to create the GKE cluster in"
}

variable "gke_service_account_email" {
Expand Down
Loading

0 comments on commit ac18ce9

Please sign in to comment.