-
Notifications
You must be signed in to change notification settings - Fork 106
Elasticsearch 7.x.0 service and private s3 bucket setup on 'space'
cf create-service aws-elasticsearch <service_plan_name> <fec_instance_name>
ex: cf create-service aws-elasticsearch es-medium-ha fec-api-elasticsearch
Ref: https://cloud.gov/docs/services/aws-elasticsearch/#managing-backups
(1)Create private s3 bucket (It will store the ES snapshot files)
cf create-service s3 basic <s3_bucket_instance_name>
ex: cf create-service s3 basic fec-s3-snapshot
(2)Get S3 bucket name
a)Create a service key to access S3 credentials
cf create-service-key <service_instance_name> <service_key_name>
ex: cf create-service-key fec-s3-snapshot fec-s3-snapshot-key
b)Get s3 credentials
cf service-key <service_instance_name> <service_key_name>
ex: cf service-key fec-s3-snapshot fec-s3-snapshot-key
c)Install jq(Lightweight command-line JSON processor) to get s3 bucket name (keep in same terminal)
brew list
brew install jq
s3_credentials=$(cf service-key fec-s3-snapshot fec-s3-snapshot-key | tail -n +3)
check env s3_credentials value: echo "${s3_credentials}"
s3_bucket=$(echo "${s3_credentials}" | jq -r '.bucket')
check env s3_bucket value: echo "${s3_bucket}"
d)Connect to new S3 bucket and upload the CORS policy to private s3 bucket
Create a cors.json on local and copy below rules into cors.json
{"CORSRules": [
{
"AllowedOrigins": ["*"],
"AllowedHeaders": ["*"],
"AllowedMethods": ["HEAD", "GET"],
"ExposeHeaders": ["ETag"]
}
]
}
upload to s3 bucket on space
['dev' space]:
aws s3api put-bucket-cors --bucket cg-55a0b774-46d0-41ca-bdbd-39c931854856 --cors-configuration file://cors.json --profile s3-dev-snapshot
['stage' space]:
aws s3api put-bucket-cors --bucket cg-52c16ed3-879b-4163-951a-50b1c8b79f8b --cors-configuration file://cors.json --profile s3-stage-snapshot
['feature' space]:
aws s3api put-bucket-cors --bucket cg-1a47b3a6-7351-4414-9ad8-cbe8a8949569 --cors-configuration file://cors.json --profile s3-feature-snapshot
['prod' space]:
(1)Assign the new s3 bucket to Elasticsearch instance (keep in the same terminal)
a)run: export s3_bucket=<bucket_name>
on dev: export s3_bucket=cg-55a0b774-46d0-41ca-bdbd-39c931854856
on stage: export s3_bucket=cg-52c16ed3-879b-4163-951a-50b1c8b79f8b
on prod: export s3_bucket=
on feature: export s3_bucket=cg-1a47b3a6-7351-4414-9ad8-cbe8a8949569
b)run: cf create-service-key <fec_es_instance_name> <fec_es_instance_name_key> -c '{"bucket":"${s3_bucket}"}'
ex: cf create-service-key fec-api-elasticsearch fec-api-elasticsearch-key -c '{"bucket":"${s3_bucket}"}'
(2)Get service key for fec es instance
cf service-key <fec_es_instance_name> <fec_es_instance_name_key>
ex: cf service-key fec-api-elasticsearch fec-api-elasticsearch-key
(3)Get Elasticsearch service snapshotRoleARN from step (2)
(4)Add/Update 'ES_SNAPSHOT_ROLE_ARN' env variable on
follow this instruction: Switch out cf environment variables
follow this:Elasticsearch 7.x.0 management instruction
(1)Re-name s3 service
cf rename-service <org_instance_name> <new_instance_name>
ex(maybe need type commend manually):
cf rename-service fec-s3-api fec-s3-dev
(2)Rebuild the latest api to circleci
this will bind-service api, celery-beat, celery-worker to fec-api-search56
(3)unbind app from ES74 (manually)
cf unbind-service api <fec_es_instance_name>
ex: cf unbind-service api fec-api-elasticsearch
ex: cf unbind-service celery-beat fec-api-elasticsearch
ex: cf unbind-service celery-worker fec-api-elasticsearch
(4)Delete ES7 service
cf delete-service-key <fec_es_instance_name> <fec_es_instance_key>
ex: cf delete-service-key fec-api-elasticsearch fec-api-elasticsearch-key
ex: cf delete-service fec-api-elasticsearch