Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

elasticsearch cloud deploy for 8.x #123

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

elasticsearch cloud deploy for 8.x #123

wants to merge 12 commits into from

Conversation

shaigbdb
Copy link
Contributor

I've checked out the current master branch to elasticsearch-7.x .

In this PR:

  • 8.x revision for GCP, AWS
  • yml definitions, ES api calls conform to 8.x
  • data nodes are now hot-content (h+c) rather than data (d). This helps add tiering to the cluster.
  • heap dumps are disabled - those usually just exploded disks out of nowhere.
  • conformed to new GCP API filter syntax for unattached volume check
  • improved packer gcp image naming
  • GCP nodes now on ubuntu 20.04
  • grafana installation removed. I've left in optional elasticsearch monitoring in case someone still wants it, but didn't check whether it works
  • added encryption keys for security, saved objects, reports
  • moved es password in Kibana to keystore
  • AWS deployment conforms to new terraform
  • data-voters only in GCP for now
  • Azure untouched and probably not working

known issues:

  • 1-master clusters don't get bootstrapped well (but 1-master + 2 data voters do, and 3-master clusters work well). I suspect that was an existing issue.

FYI @LiorF-BDBQ @synhershko

@shaigbdb shaigbdb requested a review from LiorF-BDBQ February 22, 2024 16:49
assets/scripts/client.sh Outdated Show resolved Hide resolved
Comment on lines +56 to +58
cat <<'EOF' >>/etc/elasticsearch/elasticsearch.yml
xpack.security.http.ssl.enabled: false
EOF
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this makes sense, I think we should configure it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll open a separate issue for this, but I understand this is not the focus of the current task - @synhershko
Either way in 8.x the flag must be either false or true.

assets/scripts/common/config-es.sh Outdated Show resolved Hide resolved
assets/scripts/data.sh Outdated Show resolved Hide resolved
assets/scripts/gcp/config-es-discovery.sh Show resolved Hide resolved
assets/scripts/singlenode.sh Outdated Show resolved Hide resolved
packer/README.md Outdated Show resolved Hide resolved
packer/elasticsearch8-node.packer.json Outdated Show resolved Hide resolved
terraform-aws/vpc.tf Outdated Show resolved Hide resolved
terraform-gcp/main.tf Outdated Show resolved Hide resolved
shaigbdb added 7 commits March 6, 2024 10:09
# Conflicts:
#	terraform-gcp/main.tf
…dpoint ports, adding packer manifest, adding debug bootstrap mode, fixing redundant provider, adding certs renewal
# Conflicts:
#	.gitignore
#	assets/scripts/bootstrap.sh
#	assets/scripts/common/config-clients.sh
#	assets/scripts/common/config-cluster.sh
#	assets/scripts/common/config-es.sh
#	assets/scripts/gcp/autoattach-disk.sh
#	packer/README.md
#	packer/elasticsearch8-node.packer.json
#	packer/kibana8-node.packer.json
#	templates/aws_user_data.sh
#	templates/gcp_user_data.sh
#	terraform-aws/main.tf
#	terraform-aws/variables.tf
#	terraform-aws/vpc.tf
#	terraform-gcp/main.tf
#	terraform-gcp/variables.tf
# Conflicts:
#	templates/gcp_user_data.sh
Comment on lines 1 to 6
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/aws" {
version = "5.37.0"
hashes = [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to keep this file? Lets add a required_providers block with all providers we use here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants