ClusterControl for Conductor - Orchestrating database deployment using ClusterControl and Netflix Conducor
Facets of database automation and lifecycle management in hybrid and multi-cloud environments using ClusterControl and Netflix Conductor.
Part I - Procuring infrastructure from VMware, AWS, GCP, Azure, Vultr, DigitalOcean, etc
Part II - Database automation by ClusterControl
The glu, i.e., the orchestration of the different parts is accomplished using Conductor (Orchestrator)
- Setting up prerequisites
- Building Conductor tasks
- Setting up Netflix Conductor
- Orchestrate database deployment on VMware
Apache Maven is required to build software from this repository. Here are instructions on how to setup Maven.
- Download the latest maven from https://maven.apache.org/download.cgi and extract it to your machine.
- Install JDK 8 and set JAVA_HOME to the directory where JDK is installed.
export JAVA_HOME=<jdk-install-dir>
- Update PATH environment variable to include the maven and jdk "bin" directories.
export PATH=<maven-bin-dir>:$JAVA_HOME/bin:$PATH
VMware client SDK for Java is required to build and run software from this repository. Here are instructions on how to set it up.
git clone https://github.com/vmware/vsphere-automation-sdk-java.git
cd vsphere-automation-sdk-java
mvn initialize
mvn -DskipTests clean installClusterControl's client API SDK is also required to build and run software from this repository. Here are instructions on how to set it up.
git clone https://github.com/severalnines/clustercontrol-client-sdk.git
cd clustercontrol-client-sdk/java
mvn -DskipTests clean installIn order to build the software in this repsitory, please run the following commands.
git clone https://github.com/severalnines/clustercontrol-conductor.git
cd clustercontrol-conductor
mvn -DskipTests clean packageInstructions on how to setup Netflix Conductor from sources are avaiable at the link below. (Building and Running Netflix Conductor from sources]
Once you have the software from this repository built and Netflix Conductor running, you can start creating Conductor workflows consisting of tasks. For the time being, we will demonstrate a workflow consiting of two tasks namely.
- Obtain a virtual machine (VM) from VMware vSphere
- Setup a MySQL database on the VM from the previous step
Here's the workflow to accomplish the overall goal of deploying a database on the host procured from VMware.
{
"name": "create_database_cluster",
"description": "A workflow to procure infrastructure form vmware and then deploy a specified DB cluster on that infra.",
"version": 1,
"schemaVersion": 2,
"tasks": [
{
"name": "procure_infra_task",
"taskReferenceName": "procure_infra_task_ref",
"inputParameters": {
"cluster_name": "${workflow.input.cluster_name}",
"infra_provider": "${workflow.input.infra_provider}",
"infra_providers": "${workflow.input.infra_providers}",
"plan": "${workflow.input.plan}",
"nodes": "${workflow.input.nodes}",
"vcpus": "${workflow.input.vcpus}",
"ram": "${workflow.input.ram}",
"volume_size": "${workflow.input.volume_size}",
"tags": "${workflow.input.cluster_name}"
},
"type": "SIMPLE"
},
{
"name": "deploy_db_cluster_task",
"taskReferenceName": "deploy_db_cluster_task_ref",
"inputParameters": {
"plan": "${workflow.input.plan}",
"nodes": "${workflow.input.nodes}",
"cluster_name": "${workflow.input.cluster_name}",
"cluster_type": "${workflow.input.cluster_type}",
"vendor": "${workflow.input.vendor}",
"version": "${workflow.input.version}",
"port": "${workflow.input.port}",
"admin_user": "${workflow.input.admin_user}",
"admin_pw": "${workflow.input.admin_pw}",
"hosts": "${procure_infra_task_ref.output.hosts}"
},
"type": "SIMPLE"
}
],
"inputParameters": [],
"outputParameters": {
"cluster_id": "${deploy_db_cluster_task_ref.output.cluster_id}",
"controller_id": "${deploy_db_cluster_task_ref.output.controller_id}"
},
"failureWorkflow": "deploy_database_issues",
"restartable": true,
"workflowStatusListenerEnabled": true,
"ownerEmail": "[email protected]",
"timeoutPolicy": "ALERT_ONLY",
"timeoutSeconds": 0,
"variables": {},
"inputTemplate": {
"plan": "Startup_1c_2gb",
"nodes": 1,
"vcpus": 1,
"ram": 2,
"volume_size": 40,
"infra_provider": "vmware",
"infra_providers": [
{
"provider": "vmware"
}
],
"cluster_name": "conductor-cluster",
"cluster_type": "replication",
"vendor": "Percona",
"version": "8.0",
"port": 3306,
"admin_user": "root",
"admin_pw": "u5iw9sTv$qU2$"
}
}Here's the definition for the task that will procure a VM from VMware vSphere.
{
"name": "procure_infra_task",
"description": "The task to procure infrastructure",
"retryCount": 0,
"inputKeys": [
"cluster_name",
"infra_provider",
"infra_providers",
"plan",
"nodes",
"vcpus",
"ram",
"volume_size",
"tags"
],
"outputKeys": [
"hosts"
],
"timeoutSeconds": 0,
"timeoutPolicy": "TIME_OUT_WF",
"retryLogic": "FIXED",
"retryDelaySeconds": 60,
"responseTimeoutSeconds": 3600,
"inputTemplate": {
"cluster_name": "mysql-replication",
"infra_provider": "vmware",
"infra_providers": [
{
"provider": "vmware"
}
],
"plan": "Startup_1c_2gb",
"nodes": 1,
"vcpu": 1,
"ram": 2,
"volume_size": 40,
"tags": "mysql-replication"
},
"rateLimitPerFrequency": 0,
"rateLimitFrequencyInSeconds": 1,
"ownerEmail": "[email protected]",
"backoffScaleFactor": 1
}Here's the definition for the task that will deploy a MySQL database on the VM procured from VMware.
{
"name": "deploy_db_cluster_task",
"description": "The task to deploy database cluster on infrastructure",
"retryCount": 0,
"inputKeys": [
"plan",
"nodes",
"cluster_name",
"cluster_type",
"vendor",
"version",
"port",
"admin_user",
"admin_pw",
"hosts"
],
"outputKeys": [
"cluster_id",
"controller_id"
],
"timeoutSeconds": 0,
"timeoutPolicy": "TIME_OUT_WF",
"retryLogic": "FIXED",
"retryDelaySeconds": 60,
"responseTimeoutSeconds": 3600,
"inputTemplate": {
"plan": "startup_1c_2gb_40gb",
"nodes": 1,
"cluster_name": "mysql-replication-percona-8-conductor-test",
"cluster_type": "replicaiton",
"vendor": "Percona",
"version": "8.0",
"port": 3306,
"admin_user": "root",
"admin_pw": "u5iw9sTv$qU2$",
"hosts": [
{
"hostname": "10.0.0.4"
}
]
},
"rateLimitPerFrequency": 0,
"rateLimitFrequencyInSeconds": 1,
"ownerEmail": "[email protected]",
"backoffScaleFactor": 1
}Now run the application developed here that will in essence be the backing software to accomplish the tasks in the Conductor workflow defined above.
First, setup the appropriate secrets as environment variables: For VMware:
export VMWARE_API_SERVER="10.63.1.215"
export VMWARE_API_USER="[email protected]"
export VMWARE_API_PASSWORD="XXXXYYYYZZZZZ"For ClusterControl:
export CC_URL="https://10.63.1.217:9501/v2"
export CC_API_USER="the-cc-user"
export CC_API_USER_PW="LLLMMMNNN"Finally, start the application in the same terminal:
java -cp ./cc-conductor-package/target/cc-conductor-package-1.0.0-jar-with-dependencies.jar com.severalnines.clustercontrol.conductor.tasks.MainKick off the conductor workflow with the following input.
{
"infra_provider": "vmware",
"plan": "startup_1c_2gb_40gb",
"nodes": 1,
"vcpus": 1,
"ram": 2,
"volume_size": 40,
"cluster_name": "mysql-replication",
"cluster_type": "replication",
"vendor": "percona",
"version": "8.0",
"port": 3306,
"admin_user": "root",
"admin_pw": "u5iw9sTv$qU2$",
"infra_providers": [
{
"provider": "vmware"
}
]
}