Elastic Cloud Kubernetes

SquareOps Technologies Your DevOps Partner for Accelerating cloud journey.

This ECK module is a Kubernetes operator for Elasticsearch and Kibana that simplifies the deployment, management, and scaling of Elasticsearch and Kibana clusters in Kubernetes environments. The ECK module allows you to easily create and configure Elasticsearch and Kibana clusters, and provides customization options such as persistent volume claim templates and storage classes. Additionally, the ECK module provides security features such as encryption and authentication for Elasticsearch and Kibana clusters. With the ECK module, you can manage Elasticsearch and Kibana clusters in a scalable and efficient manner, while also ensuring the security of your data.

Important Notes:

This module is compatible with EKS, AKS & GKE which is great news for users deploying the module on an AWS, Azure & GCP cloud. Review the module's documentation, meet specific configuration requirements, and test thoroughly after deployment to ensure everything works as expected.

Supported Versions Table:

Resources	Helm Chart Version	K8s supported version (EKS, AKS & GKE)
Elastic-Operator	2.7.0	1.23,1.24,1.25,1.26,1.27
ECK	7.17.3	1.23,1.24,1.25,1.26,1.27
Elastalert2	2.9.0	1.23,1.24,1.25,1.26,1.27

Usage Example

module "aws" {
  source = "squareops/elastic/kubernetes//modules/resources/aws"
  cluster_name     = ""
}

module "eck" {
  source       = "squareops/elastic/kubernetes"
  version      = "3.4.2"
  namespace = ""
  eck_config = {
    provider_type        = "aws"
    hostname             = "eck.dev.skaf.xxxxxx"
    eck_values           = file("./helm/eck.yaml")
    operator_values      = file("./helm/operator.yaml")
    master_node_sc       = "gp2"
    data_hot_node_sc     = "gp2"
    data_warm_node_sc    = "gp2"
    master_node_size     = "20Gi"
    data_hot_node_size   = "50Gi"
    data_warm_node_size  = "50Gi"
    kibana_node_count    = 1
    master_node_count    = 1
    data_hot_node_count  = 2
    data_warm_node_count = 2
    role_arn             = module.aws.role_arn
  }
  exporter_enabled   = true
  elastalert_enabled = true
  elastalert_config = {
    slack_webhook_url = ""
    elastalert_values = file("./helm/elastAlert.yaml")
  }
  # Multiple Indices
  application_index_enabled       = true
  aws_index_enabled               = false
  database_mysql_index_enabled    = true
  database_mongodb_index_enabled  = false
  database_redis_index_enabled    = false
  database_rabbitmq_index_enabled = false
  database_postgres_index_enabled = false
  application_index_name          = "application"
  database_index_name             = "database"
  application_input_type_key      = "kubernetes.namespace"
  application_input_type_value    = "robotshop"
  mongodb_input_type_key          = "kubernetes.namespace"
  mongodb_input_type_value        = "mongodb"
  mysql_input_type_key            = "kubernetes.namespace"
  mysql_input_type_value          = "mysql"
  redis_input_type_key            = "kubernetes.namespace"
  redis_input_type_value          = "redis"
  rabbitmq_input_type_key         = "kubernetes.namespace"
  rabbitmq_input_type_value       = "rabbitmq"
  postgres_input_type_key         = "kubernetes.namespace"
  postgres_input_type_value       = "postgres"
  custom_index_enabled            = true
  custom_index_name               = "mongo"
  custom_input_type_key           = "kubernetes.namespace"
  custom_input_type_value         = "ingress-nginx"
  # Note: If you enabled "aws" index, you won't be able to visualize AWS modules kibana dashboards.
  aws_input_type_key   = "input.type"
  aws_input_type_value = "aws-s3"
  # Filebeat Modules
  aws_modules_enabled              = true
  ingress_nginx_controller_enabled = true
  mongodb_enabled                  = true
  mysql_enabled                    = true
  postgresql_enabled               = false
  filebeat_role_arn                = module.aws.filebeat_role_arn
  aws_cloudtrail_enabled           = true
  cloudtrail_bucket_arn            = ""
  cloudtrail_bucket_prefix         = ""
  aws_elb_enabled                  = false
  elb_bucket_arn                   = ""
  elb_bucket_prefix                = ""
  aws_vpc_flow_logs_enabled        = true
  vpc_flowlogs_bucket_arn          = ""
  vpc_flowlogs_bucket_prefix       = ""
  aws_s3access_enabled             = true
  s3access_bucket_arn              = ""
  s3access_bucket_prefix           = ""
}

• Refer AWS examples for more details.
• Refer Azure examples for more details.
• Refer GCP examples for more details.

IAM Permissions

The required IAM permissions to create resources from this module can be found here

Elast Alert

Elastic Alert is an open-source tool that enables real-time monitoring and detection of changes in Elasticsearch data. It is designed to work with Elasticsearch clusters and is part of the Elastic Stack. Elastic Alert allows you to define rules and thresholds to trigger alerts based on specific conditions in your Elasticsearch data.

Using Elast Alert, you can monitor your Elasticsearch data in real-time and receive alerts when certain conditions are met. For example, you might use Elast Alert to monitor your application logs for a certain number of errors in a given time period or to monitor for changes in system performance.

Elast Alert is highly configurable and can be customized to meet a wide range of monitoring use cases. It includes support for various alerting channels, such as email, Slack, PagerDuty, and more. Additionally, Elast Alert can be extended with custom actions, allowing you to execute custom scripts or webhook integrations when alerts are triggered.

Requirements

No requirements.

Providers

Name	Version
helm	n/a
kubernetes	n/a
null	n/a
time	n/a

Modules

No modules.

Resources

Name	Type
helm_release.eck_operator	resource
helm_release.elastalert	resource
helm_release.elastic_stack	resource
helm_release.elasticsearch_exporter	resource
kubernetes_namespace.elastic_system	resource
null_resource.es_aws_secret	resource
null_resource.es_secret	resource
time_sleep.wait_60_sec	resource
kubernetes_secret.eck_secret	data source

Inputs

Name	Description	Type	Default	Required
application_index_enabled	Application index enabling	bool	false	no
application_index_name	The index name for the application logs	string	""	no
application_input_type_key	The key used to identify the application input type	string	""	no
application_input_type_value	The value associated with the application input type	string	""	no
aws_cloudtrail_enabled	Enable or disable AWS CloudTrail	bool	false	no
aws_cloudwatch_logs_enabled	Enable or disable AWS CloudWatch Logs	bool	false	no
aws_elb_enabled	Enable or disable AWS Elastic Load Balancer	bool	false	no
aws_index_enabled	AWS services index enabling	bool	false	no
aws_input_type_key	The key used to identify the AWS input type	string	""	no
aws_input_type_value	The value associated with the AWS input type	string	""	no
aws_modules_enabled	Enable or disable AWS Modules	bool	false	no
aws_s3access_enabled	Enable or disable AWS S3 Access	bool	false	no
aws_vpc_flow_logs_enabled	Enable or disable AWS VPC Flow Logs	bool	false	no
chart_version	Version of Helm chart to be used for deploying the ECK stack.	string	"2.9.0"	no
cloudtrail_bucket_arn	The ARN of the CloudWatch S3 bucket	string	""	no
cloudtrail_bucket_prefix	The prefix for objects in the CloudWatch S3 bucket	string	""	no
cluster_name	Name of the EKS cluster to which the ECK stack should be deployed.	string	""	no
custom_index_enabled	Custom index enabling	bool	false	no
custom_index_name	Custom index name	string	""	no
custom_input_type_key	Custom index key name	string	""	no
custom_input_type_value	Custom index value name	string	""	no
database_index_name	The index name for the database logs	string	""	no
database_mongodb_index_enabled	Database mongodb index enabling	bool	false	no
database_mysql_index_enabled	MYSQL index enabling	bool	false	no
database_postgres_index_enabled	Database postgres index enabling	bool	false	no
database_rabbitmq_index_enabled	Rabbitmq index enabling	bool	false	no
database_redis_index_enabled	Redis index enabling	bool	false	no
eck_config	Configurations for deploying the Elastic Cloud on Kubernetes (ECK) stack.	any	{ "data_hot_node_count": 1, "data_hot_node_sc": "gp2", "data_hot_node_size": "20Gi", "data_warm_node_count": 1, "data_warm_node_sc": "gp2", "data_warm_node_size": "20Gi", "eck_values": "", "eckpassword": "", "eckuser": "elastic", "hostname": "", "kibana_node_count": 1, "master_node_count": 3, "master_node_sc": "gp2", "master_node_size": "10Gi", "namespace": "elastic-system" }	no
eck_version	Version of ECK to be deployed on Kubernetes.	string	"7.17.3"	no
elastalert_config	Configurations for deploying the Elastalert tool, which is an alerting system for Elasticsearch.	map(any)	{ "elastalert_values": "", "slack_webhook_url": "" }	no
elastalert_enabled	Whether the Elastalert tool should be deployed along with the ECK stack or not.	bool	false	no
elb_bucket_arn	The ARN of the ELB S3 bucket	string	""	no
elb_bucket_prefix	The prefix for objects in the ELB S3 bucket	string	""	no
exporter_enabled	Whether the ECK exporter should be deployed along with the ECK stack or not.	bool	true	no
filebeat_role_arn	AWS filebeat role arn for authentication aws modules	string	""	no
ingress_nginx_controller_enabled	Enable or disable Ingress Nginx Controller	bool	false	no
mongodb_enabled	Whether the mongodb filebeat module should be deployed along with the ECK stack or not.	bool	false	no
mongodb_input_type_key	The key used to identify the database input type	string	""	no
mongodb_input_type_value	The value associated with the MongoDB input type	string	""	no
mysql_enabled	Whether the mysql filebeat module should be deployed along with the ECK stack or not.	bool	false	no
mysql_input_type_key	The key used to identify the database input type	string	""	no
mysql_input_type_value	The value associated with the MySQL input type	string	""	no
namespace	Name of the Kubernetes namespace where the ECK deployment will be deployed.	string	"elastic-system"	no
postgres_input_type_key	The value associated with the Postgres input type	string	""	no
postgres_input_type_value	The value associated with the postgres input type	string	""	no
postgresql_enabled	Whether the postgresql filebeat module should be deployed along with the ECK stack or not.	bool	false	no
provider_type	Choose what type of provider you want (aws, gcp)	string	""	no
rabbitmq_input_type_key	The key used to identify the database input type	string	""	no
rabbitmq_input_type_value	The value associated with the RabbitMQ input type	string	""	no
redis_input_type_key	The key used to identify the database input type	string	""	no
redis_input_type_value	The value associated with the Redis input type	string	""	no
role_arn	The s3 bucket role arn for the aws bucket provider	string	""	no
s3access_bucket_arn	The ARN of the S3 Access S3 bucket	string	""	no
s3access_bucket_prefix	The prefix for objects in the S3 Access S3 bucket	string	""	no
vpc_flowlogs_bucket_arn	The ARN of the VPC Flow Logs S3 bucket	string	""	no
vpc_flowlogs_bucket_prefix	The prefix for objects in the VPC Flow Logs S3 bucket	string	""	no

Outputs

Name	Description
eck	ECK_Info

Contribution & Issue Reporting

To report an issue with a project:

1. Check the repository's issue tracker on GitHub
2. Search to see if the issue has already been reported
3. If you can't find an answer to your question in the documentation or issue tracker, you can ask a question by creating a new issue. Be sure to provide enough context and details so others can understand your problem.

License

Apache License, Version 2.0, January 2004 (http://www.apache.org/licenses/).

Support Us

To support a GitHub project by liking it, you can follow these steps:

1. Visit the repository: Navigate to the GitHub repository.

2. Click the "Star" button: On the repository page, you'll see a "Star" button in the upper right corner. Clicking on it will star the repository, indicating your support for the project.

3. Optionally, you can also leave a comment on the repository or open an issue to give feedback or suggest changes.

Starring a repository on GitHub is a simple way to show your support and appreciation for the project. It also helps to increase the visibility of the project and make it more discoverable to others.

Who we are

We believe that the key to success in the digital age is the ability to deliver value quickly and reliably. That’s why we offer a comprehensive range of DevOps & Cloud services designed to help your organization optimize its systems & Processes for speed and agility.

1. We are an AWS Advanced consulting partner which reflects our deep expertise in AWS Cloud and helping 100+ clients over the last 5 years.
2. Expertise in Kubernetes and overall container solution helps companies expedite their journey by 10X.
3. Infrastructure Automation is a key component to the success of our Clients and our Expertise helps deliver the same in the shortest time.
4. DevSecOps as a service to implement security within the overall DevOps process and helping companies deploy securely and at speed.
5. Platform engineering which supports scalable,Cost efficient infrastructure that supports rapid development, testing, and deployment.
6. 24*7 SRE service to help you Monitor the state of your infrastructure and eradicate any issue within the SLA.

We provide support on all of our projects, no matter how small or large they may be.

To find more information about our company, visit squareops.com, follow us on Linkedin, or fill out a job application. If you have any questions or would like assistance with your cloud strategy and implementation, please don't hesitate to contact us.