diff --git a/README.md b/README.md index 66b3b3b..a078a35 100644 --- a/README.md +++ b/README.md @@ -10,18 +10,19 @@ The operator: * manages the creation and updating of all the Kubernetes resources required to run CoreMedia Content Cloud. Care has been taken to have sensible defaults for all parameters wherever possible. -* can create a fresh installation from scratch, creating MariaDB and MongoDB database servers, and initialize the necessary database schemas and secrets. +* can create a fresh installation from scratch, creating MariaDB and MongoDB database servers, and initialize the necessary database schemas and secrets, suitable for a development environment. For production, persistence should be provided, for example by using cloud services, or using other operators to provision databases. * can deploy against existing databases, using pre-existing secrets provided. -* can use a custom resource definition or a config map to supply the values. +* can use a custom resource definition or a config map to supply the values. This makes it possible to use the operator even on clusters where you cannot install cluster-wide resources. * deploys the CoreMedia Content Cloud components step by step. This ensures that components that require other components are only started when the dependencies have been initialized successfully. * imports test users and contents initially. -* builds ingresses automatically from CAE site mappings. -* creates random passwords for all components and configures them to use them (MariaDB, MongoDB, and UAPI/Corba). * run additional jobs, for example to re-import content into a running installation. +* configures the live CAE deployment with the desired number of replicas. +* builds ingresses automatically from CAE site mappings. +* creates random passwords for all components and configures the components to use them (MariaDB, MongoDB, and UAPI/Corba). +* configures Solr clustering by specifying the number of replicas to create. Planned features include: -* Creating a scalable delivery stage automatically by simply providing the number of Replication Live Servers and minimum and maximum number of Content Application Engines. -* Configure Solr clustering by specifying the number of replicas to create. +* Creating a scalable delivery stage automatically by simply providing the number of Replication Live Servers. * Support for Traefik ingress controller and its resource types (in addition to the [kubernetes/ingress-nginx](https://github.com/kubernetes/ingress-nginx)). * Admission webhook that verifies consistency of the custom resource, and can migrate between CRD versions. @@ -29,10 +30,10 @@ Planned features include: * [Helm chart cmcc-operator](charts/cmcc-operator) to install the operator * [Helm chart cmcc](charts/cmcc) to create a CoreMedia Content Cloud deployment with the operator -* [CoreMediaContentClouds custom resource documentation](docs/custom-resource.md) +* [Configuring provisioning through the CoreMediaContentClouds custom resource](docs/custom-resource.md): complete description of all options * [Installing the Operator](#preparing-your-cluster-and-installing-the-operator) -* [Using the Operator to create a CoreMedia installation](#using-the-operator) -* [Customizing the CMCC Operator](docs/customizing-the-operator.md) information for developers +* [Using the Operator to create a CoreMedia installation](#using-the-operator): quick start +* [Customizing the CMCC Operator](docs/customizing-the-operator.md): information for developers * [ghcr.io/t-systems-mms/cmcc-operator/cmcc-operator](https://github.com/T-Systems-MMS/cmcc-operator/pkgs/container/cmcc-operator%2Fcmcc-operator) Docker Image ## Preparing Your Cluster and Installing the Operator @@ -103,7 +104,7 @@ kubectl apply -f https://raw.githubusercontent.com/T-Systems-MMS/cmcc-operator/m ### Using the Custom Resource Definition -You need to add the [Custom Resource Definition](k8s/cmcc-crd.yaml) `CoreMediaContentClouds` (or `cmcc` for short) to the cluster, and create a number of object for the operator: a ClusterRole, a ClusterRoleMapping, a ServiceAccount, and a Deployment for the operator. An example can be found in [`k8s/cmcc-operator.yaml`](k8s/cmcc-operator.yaml). +You need to add the [Custom Resource Definition](k8s/cmcc-crd.yaml) `CoreMediaContentClouds` (or `cmcc` for short) to the cluster, and create a number of objects for the operator: a ClusterRole, a ClusterRoleMapping, a ServiceAccount, and a Deployment for the operator. An example can be found in [`k8s/cmcc-operator.yaml`](k8s/cmcc-operator.yaml). ### Using a Config Map diff --git a/build.gradle b/build.gradle index 198ed68..f61b027 100644 --- a/build.gradle +++ b/build.gradle @@ -27,7 +27,7 @@ compileJava { dependencies { implementation 'org.springframework.boot:spring-boot-starter-actuator' implementation 'org.springframework.boot:spring-boot-starter-web' - implementation 'io.javaoperatorsdk:operator-framework-spring-boot-starter:2.0.1' + implementation 'io.javaoperatorsdk:operator-framework-spring-boot-starter:2.2.0' implementation 'org.bouncycastle:bcpkix-jdk15on:1.70' // required for the fabric8 k8s client to grok k3d certificates implementation 'org.springframework.boot:spring-boot-starter' @@ -38,7 +38,7 @@ dependencies { annotationProcessor 'org.projectlombok:lombok' annotationProcessor 'io.fabric8:crd-generator-apt:5.12.1' testImplementation 'org.springframework.boot:spring-boot-starter-test' - testImplementation 'io.javaoperatorsdk:operator-framework-spring-boot-starter-test:2.0.1' + testImplementation 'io.javaoperatorsdk:operator-framework-spring-boot-starter-test:2.2.0' } tasks.named('test') { diff --git a/charts/cmcc-operator/templates/crd.yaml b/charts/cmcc-operator/templates/crd.yaml index 5e72af3..413742b 100644 --- a/charts/cmcc-operator/templates/crd.yaml +++ b/charts/cmcc-operator/templates/crd.yaml @@ -315,14 +315,23 @@ spec: description: Create default components for the delivery stage properties: rls: + anyOf: + - type: integer + - type: string description: Number of RLS to create - type: integer + x-kubernetes-int-or-string: true minCae: + anyOf: + - type: integer + - type: string description: Minimum number of CAEs per RLS - type: integer + x-kubernetes-int-or-string: true maxCae: + anyOf: + - type: integer + - type: string description: Maximum number of CAEs per RLS - type: integer + x-kubernetes-int-or-string: true type: object management: description: Create default components for the management stage diff --git a/k8s/cmcc-crd.yaml b/k8s/cmcc-crd.yaml index 23b9e9e..f740f1c 100644 --- a/k8s/cmcc-crd.yaml +++ b/k8s/cmcc-crd.yaml @@ -314,14 +314,23 @@ spec: description: Create default components for the delivery stage properties: rls: + anyOf: + - type: integer + - type: string description: Number of RLS to create - type: integer + x-kubernetes-int-or-string: true minCae: + anyOf: + - type: integer + - type: string description: Minimum number of CAEs per RLS - type: integer + x-kubernetes-int-or-string: true maxCae: + anyOf: + - type: integer + - type: string description: Maximum number of CAEs per RLS - type: integer + x-kubernetes-int-or-string: true type: object management: description: Create default components for the management stage diff --git a/src/main/java/com/tsystemsmms/cmcc/cmccoperator/components/AbstractComponent.java b/src/main/java/com/tsystemsmms/cmcc/cmccoperator/components/AbstractComponent.java index 5115c30..5b52cb8 100644 --- a/src/main/java/com/tsystemsmms/cmcc/cmccoperator/components/AbstractComponent.java +++ b/src/main/java/com/tsystemsmms/cmcc/cmccoperator/components/AbstractComponent.java @@ -10,15 +10,12 @@ package com.tsystemsmms.cmcc.cmccoperator.components; -import com.tsystemsmms.cmcc.cmccoperator.customresource.ConfigMapCustomResource; import com.tsystemsmms.cmcc.cmccoperator.crds.*; import com.tsystemsmms.cmcc.cmccoperator.customresource.CustomResource; import com.tsystemsmms.cmcc.cmccoperator.targetstate.TargetState; import com.tsystemsmms.cmcc.cmccoperator.utils.EnvVarSet; import io.fabric8.kubernetes.api.model.*; -import io.fabric8.kubernetes.api.model.apps.StatefulSet; -import io.fabric8.kubernetes.api.model.apps.StatefulSetBuilder; -import io.fabric8.kubernetes.api.model.apps.StatefulSetSpecBuilder; +import io.fabric8.kubernetes.api.model.apps.*; import io.fabric8.kubernetes.client.KubernetesClient; import lombok.Getter; import lombok.Setter; @@ -236,6 +233,35 @@ public StatefulSet buildStatefulSet() { .build(); } + /** + * Create the StatefulSet for reconciliation. + * + * @return the created StatefulSet. + */ + public Deployment buildDeployment(int replicas) { + return new DeploymentBuilder() + .withMetadata(getResourceMetadata()) + .withSpec(new DeploymentSpecBuilder() + .withReplicas(replicas) + .withSelector(new LabelSelectorBuilder() + .withMatchLabels(getSelectorLabels()) + .build()) + .withTemplate(new PodTemplateSpecBuilder() + .withMetadata(new ObjectMetaBuilder() + .withLabels(getSelectorLabels()) + .build()) + .withSpec(new PodSpecBuilder() + .withContainers(buildContainers()) + .withInitContainers(getInitContainers()) + .withSecurityContext(getPodSecurityContext()) + .withTerminationGracePeriodSeconds(getTerminationGracePeriodSeconds()) + .withVolumes(getVolumes()) + .build()) + .build()) + .build()) + .build(); + } + public long getTerminationGracePeriodSeconds() { return 5L; } diff --git a/src/main/java/com/tsystemsmms/cmcc/cmccoperator/components/corba/CAEComponent.java b/src/main/java/com/tsystemsmms/cmcc/cmccoperator/components/corba/CAEComponent.java index a982e6d..43d48a2 100644 --- a/src/main/java/com/tsystemsmms/cmcc/cmccoperator/components/corba/CAEComponent.java +++ b/src/main/java/com/tsystemsmms/cmcc/cmccoperator/components/corba/CAEComponent.java @@ -10,23 +10,25 @@ package com.tsystemsmms.cmcc.cmccoperator.components.corba; +import com.fasterxml.jackson.core.type.TypeReference; +import com.tsystemsmms.cmcc.cmccoperator.components.Component; import com.tsystemsmms.cmcc.cmccoperator.components.HasMongoDBClient; import com.tsystemsmms.cmcc.cmccoperator.components.HasService; import com.tsystemsmms.cmcc.cmccoperator.components.HasSolrClient; import com.tsystemsmms.cmcc.cmccoperator.crds.ClientSecretRef; import com.tsystemsmms.cmcc.cmccoperator.crds.ComponentSpec; +import com.tsystemsmms.cmcc.cmccoperator.crds.Milestone; import com.tsystemsmms.cmcc.cmccoperator.crds.SiteMapping; import com.tsystemsmms.cmcc.cmccoperator.targetstate.CustomResourceConfigError; import com.tsystemsmms.cmcc.cmccoperator.targetstate.TargetState; import com.tsystemsmms.cmcc.cmccoperator.utils.EnvVarSet; import io.fabric8.kubernetes.api.model.*; +import io.fabric8.kubernetes.api.model.apps.Deployment; import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.dsl.RollableScalableResource; import lombok.extern.slf4j.Slf4j; -import java.util.HashMap; -import java.util.LinkedList; -import java.util.List; -import java.util.Map; +import java.util.*; import static com.tsystemsmms.cmcc.cmccoperator.utils.Utils.concatOptional; @@ -37,9 +39,12 @@ public class CAEComponent extends CorbaComponent implements HasMongoDBClient, Ha public static final String KIND_PREVIEW = "preview"; public static final String SOLR_COLLECTION_LIVE = "live"; public static final String SOLR_COLLECTION_PREVIEW = "preview"; + public static final String EXTRA_REPLICAS = "replicas"; String servletPathPattern; + private int replicas = 1; + public CAEComponent(KubernetesClient kubernetesClient, TargetState targetState, ComponentSpec componentSpec) { super(kubernetesClient, targetState, componentSpec, "cae-preview"); @@ -64,8 +69,19 @@ public CAEComponent(KubernetesClient kubernetesClient, TargetState targetState, UAPI_CLIENT_SECRET_REF_KIND, "webserver" )); servletPathPattern = String.join("|", getDefaults().getServletNames()); + if (componentSpec.getExtra().containsKey(EXTRA_REPLICAS)) + replicas = Integer.parseInt(componentSpec.getExtra().get(EXTRA_REPLICAS)); } + @Override + public Component updateComponentSpec(ComponentSpec newCs) { + super.updateComponentSpec(newCs); + if (newCs.getExtra().containsKey(EXTRA_REPLICAS)) + replicas = Integer.parseInt(newCs.getExtra().get(EXTRA_REPLICAS)); + return this; + } + + @Override public void requestRequiredResources() { super.requestRequiredResources(); @@ -76,7 +92,7 @@ public void requestRequiredResources() { @Override public List buildResources() { List resources = new LinkedList<>(); - resources.add(buildStatefulSet()); + resources.add(buildDeployment(replicas)); resources.add(buildService()); resources.add(buildPvc()); return resources; @@ -191,4 +207,14 @@ public List getVolumeMounts() { return volumeMounts; } + + @Override + public Optional isReady() { + if (Milestone.compareTo(getCmcc().getStatus().getMilestone(), getComponentSpec().getMilestone()) < 0) + return Optional.empty(); + RollableScalableResource deployment = + getKubernetesClient().apps().deployments().inNamespace(getNamespace()).withName(getTargetState().getResourceNameFor(this)); + return Optional.of(deployment.isReady()); + } + } diff --git a/src/main/java/com/tsystemsmms/cmcc/cmccoperator/crds/WithOptions.java b/src/main/java/com/tsystemsmms/cmcc/cmccoperator/crds/WithOptions.java index 3b7e41d..2e9b3cf 100644 --- a/src/main/java/com/tsystemsmms/cmcc/cmccoperator/crds/WithOptions.java +++ b/src/main/java/com/tsystemsmms/cmcc/cmccoperator/crds/WithOptions.java @@ -11,6 +11,7 @@ package com.tsystemsmms.cmcc.cmccoperator.crds; import com.fasterxml.jackson.annotation.JsonPropertyDescription; +import io.fabric8.kubernetes.api.model.IntOrString; import lombok.Data; @Data @@ -30,10 +31,10 @@ public class WithOptions { @Data public static class WithDelivery { @JsonPropertyDescription("Number of RLS to create") - int rls = 0; + IntOrString rls = new IntOrString(0); @JsonPropertyDescription("Minimum number of CAEs per RLS") - int minCae = 0; + IntOrString minCae = new IntOrString(0); @JsonPropertyDescription("Maximum number of CAEs per RLS") - int maxCae = 0; + IntOrString maxCae = new IntOrString(0); } } diff --git a/src/main/java/com/tsystemsmms/cmcc/cmccoperator/targetstate/DefaultTargetState.java b/src/main/java/com/tsystemsmms/cmcc/cmccoperator/targetstate/DefaultTargetState.java index 70224ae..208a8ba 100644 --- a/src/main/java/com/tsystemsmms/cmcc/cmccoperator/targetstate/DefaultTargetState.java +++ b/src/main/java/com/tsystemsmms/cmcc/cmccoperator/targetstate/DefaultTargetState.java @@ -10,6 +10,8 @@ package com.tsystemsmms.cmcc.cmccoperator.targetstate; +import com.tsystemsmms.cmcc.cmccoperator.components.corba.CAEComponent; +import com.tsystemsmms.cmcc.cmccoperator.crds.WithOptions; import com.tsystemsmms.cmcc.cmccoperator.customresource.CustomResource; import com.tsystemsmms.cmcc.cmccoperator.components.ComponentSpecBuilder; import com.tsystemsmms.cmcc.cmccoperator.components.generic.MongoDBComponent; @@ -25,8 +27,11 @@ import org.springframework.beans.factory.BeanFactory; import java.util.List; +import java.util.Map; import java.util.Optional; +import static com.tsystemsmms.cmcc.cmccoperator.utils.Utils.getInt; + /** * Create the runtime config based on the CRD data. */ @@ -85,14 +90,17 @@ public void convergeDefaultComponents() { )); } - if (cmcc.getSpec().getWith().getDelivery().getRls() != 0 - || cmcc.getSpec().getWith().getDelivery().getMinCae() > 1 - || cmcc.getSpec().getWith().getDelivery().getMaxCae() > cmcc.getSpec().getWith().getDelivery().getMinCae()) { + WithOptions.WithDelivery delivery = cmcc.getSpec().getWith().getDelivery(); + if (getInt(delivery.getRls()) != 0 + || getInt(delivery.getMaxCae()) > getInt(delivery.getMinCae())) { throw new RuntimeException("Unable to configure RLS and HPA, not implemented yet"); } - if (cmcc.getSpec().getWith().getDelivery().getMinCae() == 1) { + if (getInt(delivery.getMinCae()) > 0) { + Map liveCaeExtra = Map.of( + CAEComponent.EXTRA_REPLICAS, String.valueOf(getInt(delivery.getMinCae())) + ); componentCollection.addAll(List.of( - ComponentSpecBuilder.ofType("cae").withKind("live").build() + ComponentSpecBuilder.ofType("cae").withKind("live").withExtra(liveCaeExtra).build() )); } } diff --git a/src/main/java/com/tsystemsmms/cmcc/cmccoperator/utils/Utils.java b/src/main/java/com/tsystemsmms/cmcc/cmccoperator/utils/Utils.java index 0c3d0db..280002e 100644 --- a/src/main/java/com/tsystemsmms/cmcc/cmccoperator/utils/Utils.java +++ b/src/main/java/com/tsystemsmms/cmcc/cmccoperator/utils/Utils.java @@ -13,6 +13,7 @@ import com.fasterxml.jackson.databind.ObjectMapper; import io.fabric8.kubernetes.api.model.EnvVar; import io.fabric8.kubernetes.api.model.EnvVarSource; +import io.fabric8.kubernetes.api.model.IntOrString; import io.fabric8.kubernetes.api.model.SecretKeySelector; import lombok.SneakyThrows; @@ -172,4 +173,16 @@ public static boolean booleanOf(Object o, boolean def) { } return l != 0 || s.equals("on") || s.equals("true") || s.equals("yes"); } + + /** + * Returns the integer value from a IntOrString, converting a string to integer if necessary. + * + * @param v value + * @return integer value + */ + public static int getInt(IntOrString v) { + if (v.getIntVal() != null) + return v.getIntVal(); + return Integer.parseInt(v.getStrVal()); + } }