From c5225d124a32e8d7e65c0da497a815e8a7faaca8 Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Sun, 23 Apr 2023 15:10:48 +0000 Subject: [PATCH 01/24] 108: Updated the readme.md files from [distributed], [per-project] and [gke with the correct Firestore create command parameter --- terraform/cloud-functions/distributed/README.md | 2 +- terraform/cloud-functions/per-project/README.md | 2 +- terraform/gke/README.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/terraform/cloud-functions/distributed/README.md b/terraform/cloud-functions/distributed/README.md index f112c9b9..93c540ed 100644 --- a/terraform/cloud-functions/distributed/README.md +++ b/terraform/cloud-functions/distributed/README.md @@ -174,7 +174,7 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives. if your project does not have yet. ```sh - gcloud firestore databases create --region="${AUTOSCALER_APP_ENGINE_LOCATION}" + gcloud firestore databases create --location="${AUTOSCALER_APP_ENGINE_LOCATION}" ``` In case you want to use Cloud Spanner, no action is needed at this point. diff --git a/terraform/cloud-functions/per-project/README.md b/terraform/cloud-functions/per-project/README.md index 9bf50eb4..53551207 100644 --- a/terraform/cloud-functions/per-project/README.md +++ b/terraform/cloud-functions/per-project/README.md @@ -169,7 +169,7 @@ In this section you prepare your project for deployment. if your project does not have yet. ```sh - gcloud firestore databases create --region="${APP_ENGINE_LOCATION}" + gcloud firestore databases create --location="${APP_ENGINE_LOCATION}" ``` In case you want to use Cloud Spanner, no action is needed at this point. diff --git a/terraform/gke/README.md b/terraform/gke/README.md index 8cbe1a97..1e7c2b49 100644 --- a/terraform/gke/README.md +++ b/terraform/gke/README.md @@ -232,7 +232,7 @@ In this section you prepare your project for deployment. 4. Create a database to store the state of the Autoscaler: ```sh - gcloud firestore databases create --region="${APP_ENGINE_LOCATION}" + gcloud firestore databases create --location="${APP_ENGINE_LOCATION}" ``` You will also need to make a minor modification to the Autoscaler From bfc396c109a9e0bd00966ffc6ba857ca0bf8ae74 Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Thu, 4 May 2023 10:37:13 +0000 Subject: [PATCH 02/24] fix: 108 - Fix for Firestore database create cmd. --- terraform/cloud-functions/distributed/README.md | 9 ++++++--- terraform/cloud-functions/per-project/README.md | 15 +++++++++------ terraform/gke/README.md | 7 +++++-- 3 files changed, 20 insertions(+), 11 deletions(-) diff --git a/terraform/cloud-functions/distributed/README.md b/terraform/cloud-functions/distributed/README.md index 93c540ed..ba252686 100644 --- a/terraform/cloud-functions/distributed/README.md +++ b/terraform/cloud-functions/distributed/README.md @@ -137,14 +137,16 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives. gcloud config set project "${AUTOSCALER_PROJECT_ID}" ``` -4. Choose the [region and zone][region-and-zone] and - [App Engine Location][app-engine-location] where the Autoscaler +4. Choose the [region and zone][region-and-zone], + [App Engine Location][app-engine-location] and + [Firestore Location][firestore-location] where the Autoscaler infrastructure will be located. ```sh export AUTOSCALER_REGION=us-central1 export AUTOSCALER_ZONE=us-central1-c export AUTOSCALER_APP_ENGINE_LOCATION=us-central + export FIRESTORE_DB_LOCATION=nam5 ``` 5. Enable the required Cloud APIs : @@ -174,7 +176,7 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives. if your project does not have yet. ```sh - gcloud firestore databases create --location="${AUTOSCALER_APP_ENGINE_LOCATION}" + gcloud firestore databases create --location="${FIRESTORE_DB_LOCATION}" ``` In case you want to use Cloud Spanner, no action is needed at this point. @@ -409,6 +411,7 @@ its configuration issues independently. [enable-billing]: https://cloud.google.com/billing/docs/how-to/modify-project [region-and-zone]: https://cloud.google.com/compute/docs/regions-zones#locations [app-engine-location]: https://cloud.google.com/appengine/docs/locations +[firestore-location]: https://cloud.google.com/appengine/docs/standard/locations#cloud-firestore-location [cloud-console]: https://console.cloud.google.com [cloud-shell]: https://console.cloud.google.com/?cloudshell=true&_ga=2.43377068.820133692.1587377411-71235912.1585654570&_gac=1.118947195.1584696876.Cj0KCQjw09HzBRDrARIsAG60GP9u6OBk_qQ02rkzBXpwpMd6YZ30A2D4gSl2Wwte1CqPW_sY6mH_xbIaAmIgEALw_wcB [provider-issue]: https://github.com/hashicorp/terraform-provider-google/issues/6782 diff --git a/terraform/cloud-functions/per-project/README.md b/terraform/cloud-functions/per-project/README.md index 53551207..81802621 100644 --- a/terraform/cloud-functions/per-project/README.md +++ b/terraform/cloud-functions/per-project/README.md @@ -127,17 +127,19 @@ In this section you prepare your project for deployment. ```sh export PROJECT_ID= - gcloud config set project "${PROJECT_ID}" - ``` - -4. Choose the [region and zone][region-and-zone] and - [App Engine Location][app-engine-location] where the Autoscaler + gcloud config set project "${PROJE, + T_ID}" + `, +4. Choose the [region and zone][region-and-zone], + [App Engine Location][app-engine-location] and + [Firestore Location][firestore-location] where the Autoscaler infrastructure will be located. ```sh export REGION=us-central1 export ZONE=us-central1-c export APP_ENGINE_LOCATION=us-central + export FIRESTORE_DB_LOCATION=nam5 ``` 5. Enable the required Cloud APIs @@ -169,7 +171,7 @@ In this section you prepare your project for deployment. if your project does not have yet. ```sh - gcloud firestore databases create --location="${APP_ENGINE_LOCATION}" + gcloud firestore databases create --location="${FIRESTORE_DB_LOCATION}" ``` In case you want to use Cloud Spanner, no action is needed at this point. @@ -330,6 +332,7 @@ page to [configure your Autoscaler](../README.md#configuration). [cloud-shell]: https://console.cloud.google.com/?cloudshell=true [region-and-zone]: https://cloud.google.com/compute/docs/regions-zones#locations [app-engine-location]: https://cloud.google.com/appengine/docs/locations +[firestore-location]: https://cloud.google.com/appengine/docs/standard/locations#cloud-firestore-location [terraform-import]: https://www.terraform.io/docs/import/index.html [terraform-import-usage]: https://www.terraform.io/docs/import/usage.html [terraform-spanner-instance]: https://www.terraform.io/docs/providers/google/r/spanner_instance.html diff --git a/terraform/gke/README.md b/terraform/gke/README.md index 8cbd7f9b..53c74838 100644 --- a/terraform/gke/README.md +++ b/terraform/gke/README.md @@ -208,11 +208,13 @@ In this section you prepare your project for deployment. ## Using Firestore for Autoscaler state 1. To use Firestore for the Autoscaler state, choose the - [App Engine Location][app-engine-location] where the Autoscaler + [App Engine Location][app-engine-location] and + [Firestore Location][firestore-location] where the Autoscaler infrastructure will be created, for example: ```sh export APP_ENGINE_LOCATION=us-central + export FIRESTORE_DB_LOCATION=nam5 ``` 2. Enable the additional APIs: @@ -232,7 +234,7 @@ In this section you prepare your project for deployment. 4. Create a database to store the state of the Autoscaler: ```sh - gcloud firestore databases create --location="${APP_ENGINE_LOCATION}" + gcloud firestore databases create --location="${FIRESTORE_DB_LOCATION}" ``` You will also need to make a minor modification to the Autoscaler @@ -547,6 +549,7 @@ following the instructions above. [cloud-console-logging]: https://console.cloud.google.com/logs/query [cloud-shell]: https://console.cloud.google.com/?cloudshell=true [app-engine-location]: https://cloud.google.com/appengine/docs/locations +[firestore-location]: https://cloud.google.com/appengine/docs/standard/locations#cloud-firestore-location [terraform-import]: https://www.terraform.io/docs/import/index.html [terraform-import-usage]: https://www.terraform.io/docs/import/usage.html [terraform-spanner-instance]: https://www.terraform.io/docs/providers/google/r/spanner_instance.html From 3a18be9880e1105e6db9df1f5fbdefa4cc586950 Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Thu, 4 May 2023 15:29:44 +0000 Subject: [PATCH 03/24] fix: 108 updated formatting --- terraform/cloud-functions/distributed/README.md | 2 +- terraform/cloud-functions/per-project/README.md | 6 +++--- terraform/gke/README.md | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/terraform/cloud-functions/distributed/README.md b/terraform/cloud-functions/distributed/README.md index ba252686..0d484c82 100644 --- a/terraform/cloud-functions/distributed/README.md +++ b/terraform/cloud-functions/distributed/README.md @@ -146,7 +146,7 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives. export AUTOSCALER_REGION=us-central1 export AUTOSCALER_ZONE=us-central1-c export AUTOSCALER_APP_ENGINE_LOCATION=us-central - export FIRESTORE_DB_LOCATION=nam5 + export FIRESTORE_DB_LOCATION=nam5 ``` 5. Enable the required Cloud APIs : diff --git a/terraform/cloud-functions/per-project/README.md b/terraform/cloud-functions/per-project/README.md index 81802621..3373a060 100644 --- a/terraform/cloud-functions/per-project/README.md +++ b/terraform/cloud-functions/per-project/README.md @@ -127,9 +127,9 @@ In this section you prepare your project for deployment. ```sh export PROJECT_ID= - gcloud config set project "${PROJE, - T_ID}" - `, + gcloud config set project "${PROJECT_ID}" + ``` + 4. Choose the [region and zone][region-and-zone], [App Engine Location][app-engine-location] and [Firestore Location][firestore-location] where the Autoscaler diff --git a/terraform/gke/README.md b/terraform/gke/README.md index 53c74838..32fef41b 100644 --- a/terraform/gke/README.md +++ b/terraform/gke/README.md @@ -208,7 +208,7 @@ In this section you prepare your project for deployment. ## Using Firestore for Autoscaler state 1. To use Firestore for the Autoscaler state, choose the - [App Engine Location][app-engine-location] and + [App Engine Location][app-engine-location] and [Firestore Location][firestore-location] where the Autoscaler infrastructure will be created, for example: From a388d40f218795c04dfea3059d33d64af9dad1a5 Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Sat, 6 May 2023 14:25:39 +0000 Subject: [PATCH 04/24] updated files formatting to resolve build issues. --- .mdl.json | 2 +- .mdl.rb | 2 +- terraform/cloud-functions/per-project/README.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.mdl.json b/.mdl.json index 1aac20fd..00cab7a3 100644 --- a/.mdl.json +++ b/.mdl.json @@ -5,7 +5,7 @@ "MD002": false, "MD004": { "style": "asterisk" }, "MD007": { "indent": 4 }, - "MD013": { "code_blocks": false, "tables": false }, + "MD013": { "ignore_code_blocks": true, "tables": false }, "MD029": { "style": "ordered" }, "MD030": { "ul_single": 3, "ul_multi": 3, "ol_single": 2, "ol_multi": 2 } } diff --git a/.mdl.rb b/.mdl.rb index 15d24358..d8dc3fb6 100644 --- a/.mdl.rb +++ b/.mdl.rb @@ -15,7 +15,7 @@ rule 'MD007', :indent => 4 # Allow table and code lines to be longer than 80 chars -rule 'MD013', :code_blocks => false, :tables => false +rule 'MD013', :ignore_code_blocks => true, :tables => false # Ordered list item prefixes rule 'MD029', :style => :ordered diff --git a/terraform/cloud-functions/per-project/README.md b/terraform/cloud-functions/per-project/README.md index 3373a060..ff0ed01e 100644 --- a/terraform/cloud-functions/per-project/README.md +++ b/terraform/cloud-functions/per-project/README.md @@ -129,7 +129,7 @@ In this section you prepare your project for deployment. export PROJECT_ID= gcloud config set project "${PROJECT_ID}" ``` - + 4. Choose the [region and zone][region-and-zone], [App Engine Location][app-engine-location] and [Firestore Location][firestore-location] where the Autoscaler From 8331e3cc746858eff3a0ca5ed5d257e79a29f9f3 Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Wed, 10 May 2023 03:42:03 +0000 Subject: [PATCH 05/24] fix: 108 updated files according to rc's. --- terraform/cloud-functions/distributed/README.md | 9 +++------ terraform/cloud-functions/per-project/README.md | 9 +++------ terraform/gke/README.md | 7 ++----- 3 files changed, 8 insertions(+), 17 deletions(-) diff --git a/terraform/cloud-functions/distributed/README.md b/terraform/cloud-functions/distributed/README.md index 0d484c82..4061d60f 100644 --- a/terraform/cloud-functions/distributed/README.md +++ b/terraform/cloud-functions/distributed/README.md @@ -137,16 +137,14 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives. gcloud config set project "${AUTOSCALER_PROJECT_ID}" ``` -4. Choose the [region and zone][region-and-zone], - [App Engine Location][app-engine-location] and - [Firestore Location][firestore-location] where the Autoscaler +4. Choose the [region and zone][region-and-zone] and + [App Engine Location][app-engine-location] where the Autoscaler infrastructure will be located. ```sh export AUTOSCALER_REGION=us-central1 export AUTOSCALER_ZONE=us-central1-c export AUTOSCALER_APP_ENGINE_LOCATION=us-central - export FIRESTORE_DB_LOCATION=nam5 ``` 5. Enable the required Cloud APIs : @@ -176,7 +174,7 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives. if your project does not have yet. ```sh - gcloud firestore databases create --location="${FIRESTORE_DB_LOCATION}" + gcloud firestore databases update --type=firestore-native ``` In case you want to use Cloud Spanner, no action is needed at this point. @@ -411,7 +409,6 @@ its configuration issues independently. [enable-billing]: https://cloud.google.com/billing/docs/how-to/modify-project [region-and-zone]: https://cloud.google.com/compute/docs/regions-zones#locations [app-engine-location]: https://cloud.google.com/appengine/docs/locations -[firestore-location]: https://cloud.google.com/appengine/docs/standard/locations#cloud-firestore-location [cloud-console]: https://console.cloud.google.com [cloud-shell]: https://console.cloud.google.com/?cloudshell=true&_ga=2.43377068.820133692.1587377411-71235912.1585654570&_gac=1.118947195.1584696876.Cj0KCQjw09HzBRDrARIsAG60GP9u6OBk_qQ02rkzBXpwpMd6YZ30A2D4gSl2Wwte1CqPW_sY6mH_xbIaAmIgEALw_wcB [provider-issue]: https://github.com/hashicorp/terraform-provider-google/issues/6782 diff --git a/terraform/cloud-functions/per-project/README.md b/terraform/cloud-functions/per-project/README.md index ff0ed01e..d35e3b1d 100644 --- a/terraform/cloud-functions/per-project/README.md +++ b/terraform/cloud-functions/per-project/README.md @@ -130,16 +130,14 @@ In this section you prepare your project for deployment. gcloud config set project "${PROJECT_ID}" ``` -4. Choose the [region and zone][region-and-zone], - [App Engine Location][app-engine-location] and - [Firestore Location][firestore-location] where the Autoscaler +4. Choose the [region and zone][region-and-zone] and + [App Engine Location][app-engine-location] where the Autoscaler infrastructure will be located. ```sh export REGION=us-central1 export ZONE=us-central1-c export APP_ENGINE_LOCATION=us-central - export FIRESTORE_DB_LOCATION=nam5 ``` 5. Enable the required Cloud APIs @@ -171,7 +169,7 @@ In this section you prepare your project for deployment. if your project does not have yet. ```sh - gcloud firestore databases create --location="${FIRESTORE_DB_LOCATION}" + gcloud firestore databases update --type=firestore-native ``` In case you want to use Cloud Spanner, no action is needed at this point. @@ -332,7 +330,6 @@ page to [configure your Autoscaler](../README.md#configuration). [cloud-shell]: https://console.cloud.google.com/?cloudshell=true [region-and-zone]: https://cloud.google.com/compute/docs/regions-zones#locations [app-engine-location]: https://cloud.google.com/appengine/docs/locations -[firestore-location]: https://cloud.google.com/appengine/docs/standard/locations#cloud-firestore-location [terraform-import]: https://www.terraform.io/docs/import/index.html [terraform-import-usage]: https://www.terraform.io/docs/import/usage.html [terraform-spanner-instance]: https://www.terraform.io/docs/providers/google/r/spanner_instance.html diff --git a/terraform/gke/README.md b/terraform/gke/README.md index 32fef41b..be5e2db4 100644 --- a/terraform/gke/README.md +++ b/terraform/gke/README.md @@ -208,13 +208,11 @@ In this section you prepare your project for deployment. ## Using Firestore for Autoscaler state 1. To use Firestore for the Autoscaler state, choose the - [App Engine Location][app-engine-location] and - [Firestore Location][firestore-location] where the Autoscaler + [App Engine Location][app-engine-location] where the Autoscaler infrastructure will be created, for example: ```sh export APP_ENGINE_LOCATION=us-central - export FIRESTORE_DB_LOCATION=nam5 ``` 2. Enable the additional APIs: @@ -234,7 +232,7 @@ In this section you prepare your project for deployment. 4. Create a database to store the state of the Autoscaler: ```sh - gcloud firestore databases create --location="${FIRESTORE_DB_LOCATION}" + gcloud firestore databases update --type=firestore-native ``` You will also need to make a minor modification to the Autoscaler @@ -549,7 +547,6 @@ following the instructions above. [cloud-console-logging]: https://console.cloud.google.com/logs/query [cloud-shell]: https://console.cloud.google.com/?cloudshell=true [app-engine-location]: https://cloud.google.com/appengine/docs/locations -[firestore-location]: https://cloud.google.com/appengine/docs/standard/locations#cloud-firestore-location [terraform-import]: https://www.terraform.io/docs/import/index.html [terraform-import-usage]: https://www.terraform.io/docs/import/usage.html [terraform-spanner-instance]: https://www.terraform.io/docs/providers/google/r/spanner_instance.html From 4fd5a58ebcf66b75db69326db5354cd93859b814 Mon Sep 17 00:00:00 2001 From: David Cueva Tello Date: Wed, 10 May 2023 12:39:02 +0200 Subject: [PATCH 06/24] Explained Firestore update command --- terraform/cloud-functions/distributed/README.md | 8 ++++---- terraform/cloud-functions/per-project/README.md | 7 +++---- terraform/gke/README.md | 4 +++- 3 files changed, 10 insertions(+), 9 deletions(-) diff --git a/terraform/cloud-functions/distributed/README.md b/terraform/cloud-functions/distributed/README.md index 4061d60f..43c536fc 100644 --- a/terraform/cloud-functions/distributed/README.md +++ b/terraform/cloud-functions/distributed/README.md @@ -167,11 +167,10 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives. gcloud app create --region="${AUTOSCALER_APP_ENGINE_LOCATION}" ``` -7. Create a database to store the state of the Autoscaler. - State can be stored in either Firestore or Cloud Spanner. +7. The Autoscaler state can be stored in either Firestore or Cloud Spanner. - In case you want to use Firestore, create a new instance - if your project does not have yet. + In case you want to use Firestore, update the database created with the + Google App Engine app to use [Firestore native mode][firestore-native]. ```sh gcloud firestore databases update --type=firestore-native @@ -413,3 +412,4 @@ its configuration issues independently. [cloud-shell]: https://console.cloud.google.com/?cloudshell=true&_ga=2.43377068.820133692.1587377411-71235912.1585654570&_gac=1.118947195.1584696876.Cj0KCQjw09HzBRDrARIsAG60GP9u6OBk_qQ02rkzBXpwpMd6YZ30A2D4gSl2Wwte1CqPW_sY6mH_xbIaAmIgEALw_wcB [provider-issue]: https://github.com/hashicorp/terraform-provider-google/issues/6782 [logs-viewer]: https://pantheon.corp.google.com/logs/query +[firestore-native]: https://cloud.google.com/datastore/docs/firestore-or-datastore#in_native_mode \ No newline at end of file diff --git a/terraform/cloud-functions/per-project/README.md b/terraform/cloud-functions/per-project/README.md index d35e3b1d..58493afb 100644 --- a/terraform/cloud-functions/per-project/README.md +++ b/terraform/cloud-functions/per-project/README.md @@ -162,11 +162,10 @@ In this section you prepare your project for deployment. gcloud app create --region="${APP_ENGINE_LOCATION}" ``` -7. Create a database to store the state of the Autoscaler. - State can be stored in either Firestore or Cloud Spanner. +7. The Autoscaler state can be stored in either Firestore or Cloud Spanner. - In case you want to use Firestore, create a new instance - if your project does not have yet. + In case you want to use Firestore, update the database created with the + Google App Engine app to use Firestore native mode. ```sh gcloud firestore databases update --type=firestore-native diff --git a/terraform/gke/README.md b/terraform/gke/README.md index be5e2db4..e3d1249a 100644 --- a/terraform/gke/README.md +++ b/terraform/gke/README.md @@ -229,7 +229,8 @@ In this section you prepare your project for deployment. gcloud app create --region="${APP_ENGINE_LOCATION}" ``` -4. Create a database to store the state of the Autoscaler: +4. To store the state of the Autoscaler, update the database created with the + Google App Engine app to use [Firestore native mode][firestore-native]. ```sh gcloud firestore databases update --type=firestore-native @@ -553,3 +554,4 @@ following the instructions above. [terraform-spanner-db]: https://www.terraform.io/docs/providers/google/r/spanner_database.html [provider-issue]: https://github.com/hashicorp/terraform-provider-google/issues/6782 [organizational-policy]: https://cloud.google.com/resource-manager/docs/organization-policy/overview +[firestore-native]: https://cloud.google.com/datastore/docs/firestore-or-datastore#in_native_mode \ No newline at end of file From ae1b6e60cc9a4284490db64669e5952c8c9a43b4 Mon Sep 17 00:00:00 2001 From: David Cueva Tello Date: Wed, 10 May 2023 12:46:53 +0200 Subject: [PATCH 07/24] Fixed linter issue --- terraform/cloud-functions/distributed/README.md | 2 +- terraform/cloud-functions/per-project/README.md | 3 ++- terraform/gke/README.md | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/terraform/cloud-functions/distributed/README.md b/terraform/cloud-functions/distributed/README.md index 43c536fc..545cb176 100644 --- a/terraform/cloud-functions/distributed/README.md +++ b/terraform/cloud-functions/distributed/README.md @@ -412,4 +412,4 @@ its configuration issues independently. [cloud-shell]: https://console.cloud.google.com/?cloudshell=true&_ga=2.43377068.820133692.1587377411-71235912.1585654570&_gac=1.118947195.1584696876.Cj0KCQjw09HzBRDrARIsAG60GP9u6OBk_qQ02rkzBXpwpMd6YZ30A2D4gSl2Wwte1CqPW_sY6mH_xbIaAmIgEALw_wcB [provider-issue]: https://github.com/hashicorp/terraform-provider-google/issues/6782 [logs-viewer]: https://pantheon.corp.google.com/logs/query -[firestore-native]: https://cloud.google.com/datastore/docs/firestore-or-datastore#in_native_mode \ No newline at end of file +[firestore-native]: https://cloud.google.com/datastore/docs/firestore-or-datastore#in_native_mode diff --git a/terraform/cloud-functions/per-project/README.md b/terraform/cloud-functions/per-project/README.md index 58493afb..db8336ce 100644 --- a/terraform/cloud-functions/per-project/README.md +++ b/terraform/cloud-functions/per-project/README.md @@ -165,7 +165,7 @@ In this section you prepare your project for deployment. 7. The Autoscaler state can be stored in either Firestore or Cloud Spanner. In case you want to use Firestore, update the database created with the - Google App Engine app to use Firestore native mode. + Google App Engine app to use [Firestore native mode][firestore-native]. ```sh gcloud firestore databases update --type=firestore-native @@ -334,3 +334,4 @@ page to [configure your Autoscaler](../README.md#configuration). [terraform-spanner-instance]: https://www.terraform.io/docs/providers/google/r/spanner_instance.html [terraform-spanner-db]: https://www.terraform.io/docs/providers/google/r/spanner_database.html [provider-issue]: https://github.com/hashicorp/terraform-provider-google/issues/6782 +[firestore-native]: https://cloud.google.com/datastore/docs/firestore-or-datastore#in_native_mode diff --git a/terraform/gke/README.md b/terraform/gke/README.md index e3d1249a..33cb3961 100644 --- a/terraform/gke/README.md +++ b/terraform/gke/README.md @@ -554,4 +554,4 @@ following the instructions above. [terraform-spanner-db]: https://www.terraform.io/docs/providers/google/r/spanner_database.html [provider-issue]: https://github.com/hashicorp/terraform-provider-google/issues/6782 [organizational-policy]: https://cloud.google.com/resource-manager/docs/organization-policy/overview -[firestore-native]: https://cloud.google.com/datastore/docs/firestore-or-datastore#in_native_mode \ No newline at end of file +[firestore-native]: https://cloud.google.com/datastore/docs/firestore-or-datastore#in_native_mode From c2c6ae4d948749f3a6f33ae92e24cb2acc2447ac Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Fri, 12 May 2023 10:33:59 +0000 Subject: [PATCH 08/24] enhancement#93: Initial commit. --- terraform/gke/README.md | 11 +++++++++ terraform/modules/firestore/main.tf | 29 ++++++++++++++++++++++++ terraform/modules/firestore/variables.tf | 17 +++++++++----- 3 files changed, 51 insertions(+), 6 deletions(-) diff --git a/terraform/gke/README.md b/terraform/gke/README.md index 33cb3961..6d890c08 100644 --- a/terraform/gke/README.md +++ b/terraform/gke/README.md @@ -207,6 +207,17 @@ In this section you prepare your project for deployment. ## Using Firestore for Autoscaler state +1. If you want to create a new Firestore database for storing the state of the + Autoscaler, set the following variable. + + ```sh + export TF_VAR_terraform_firestore_create=true + ``` + + On the other hand if you do not want to create the Firestore database using + terraform,then continue with the below steps to manually create the Firestore + database using gcloud commands. + 1. To use Firestore for the Autoscaler state, choose the [App Engine Location][app-engine-location] where the Autoscaler infrastructure will be created, for example: diff --git a/terraform/modules/firestore/main.tf b/terraform/modules/firestore/main.tf index 895c597e..16dd887a 100644 --- a/terraform/modules/firestore/main.tf +++ b/terraform/modules/firestore/main.tf @@ -21,9 +21,38 @@ * is not being used to hold state. */ +terraform { + required_providers { + google = { + source = "hashicorp/google" + version = "~> 4.63.0" + } + } +} + +provider "google" { + project = var.project_id + region = var.region + zone = var.zone +} + resource "google_project_iam_member" "scaler_sa_firestore" { project = var.project_id role = "roles/datastore.user" member = "serviceAccount:${var.scaler_sa_email}" } + +resource "google_project_service" "firestore" { + project = var.project_id + service = "firestore.googleapis.com" +} + +resource "google_firestore_database" "database" { + project = var.project_id + name = "(default)" + location_id = var.location + type = var.firestore_database_type + + depends_on = [google_project_service.firestore] +} diff --git a/terraform/modules/firestore/variables.tf b/terraform/modules/firestore/variables.tf index c3cb4c19..b574695e 100644 --- a/terraform/modules/firestore/variables.tf +++ b/terraform/modules/firestore/variables.tf @@ -18,15 +18,20 @@ variable "project_id" { type = string } -variable "poller_sa_email" { - type = string +variable "region" { + description = "The region where the infrastructure will be deployed." + type = string + default = "us-central1" } -variable "scaler_sa_email" { +variable "location" { + description = "The location within the region where the infrastructure will be deployed." type = string + default = "nam5" } -variable "region" { - type = string - default = "us-central1" +variable "firestore_database_type" { + description = "The database type. TYPE must be one of: datastore-mode, firestore-native" + type = string + default = "FIRESTORE_NATIVE" } From 7865d12bec35892a4534d61a8e099aa4622cf51a Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Sat, 13 May 2023 00:29:50 +0000 Subject: [PATCH 09/24] updated poller SA variables. --- terraform/modules/firestore/variables.tf | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/terraform/modules/firestore/variables.tf b/terraform/modules/firestore/variables.tf index b574695e..8edd693a 100644 --- a/terraform/modules/firestore/variables.tf +++ b/terraform/modules/firestore/variables.tf @@ -35,3 +35,11 @@ variable "firestore_database_type" { type = string default = "FIRESTORE_NATIVE" } + +variable "poller_sa_email" { + type = string +} + +variable "scaler_sa_email" { + type = string +} From 050a4a2ed4a4be47516376f621b0b676a478aa40 Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Mon, 15 May 2023 01:57:56 +0000 Subject: [PATCH 10/24] deleted unused variable --- terraform/gke/main.tf | 4 ++-- terraform/modules/firestore/main.tf | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/terraform/gke/main.tf b/terraform/gke/main.tf index f9726aa2..9a4cfe99 100644 --- a/terraform/gke/main.tf +++ b/terraform/gke/main.tf @@ -62,8 +62,8 @@ module "firestore" { source = "../modules/firestore" project_id = var.project_id - poller_sa_email = module.autoscaler-base.poller_sa_email - scaler_sa_email = module.autoscaler-base.scaler_sa_email + #poller_sa_email = module.autoscaler-base.poller_sa_email + #scaler_sa_email = module.autoscaler-base.scaler_sa_email } module "spanner" { diff --git a/terraform/modules/firestore/main.tf b/terraform/modules/firestore/main.tf index 16dd887a..804ba6d0 100644 --- a/terraform/modules/firestore/main.tf +++ b/terraform/modules/firestore/main.tf @@ -33,7 +33,7 @@ terraform { provider "google" { project = var.project_id region = var.region - zone = var.zone + #zone = var.zone } resource "google_project_iam_member" "scaler_sa_firestore" { From d002716d7a485a7b95417835c1d66adfd2a7c3d6 Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Mon, 15 May 2023 09:55:14 +0000 Subject: [PATCH 11/24] Updated the flow of setting up firestore by providing an option to set it up by using Terraform or manually. --- terraform/gke/README.md | 6 +++++- terraform/gke/main.tf | 4 ++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/terraform/gke/README.md b/terraform/gke/README.md index 6d890c08..980715f2 100644 --- a/terraform/gke/README.md +++ b/terraform/gke/README.md @@ -207,6 +207,8 @@ In this section you prepare your project for deployment. ## Using Firestore for Autoscaler state +### Setting up Firestore for storing Autoscaler state using terraform. + 1. If you want to create a new Firestore database for storing the state of the Autoscaler, set the following variable. @@ -217,7 +219,9 @@ In this section you prepare your project for deployment. On the other hand if you do not want to create the Firestore database using terraform,then continue with the below steps to manually create the Firestore database using gcloud commands. - + +### Manually setting up Firestore for storing Autoscaler state + 1. To use Firestore for the Autoscaler state, choose the [App Engine Location][app-engine-location] where the Autoscaler infrastructure will be created, for example: diff --git a/terraform/gke/main.tf b/terraform/gke/main.tf index 9a4cfe99..f9726aa2 100644 --- a/terraform/gke/main.tf +++ b/terraform/gke/main.tf @@ -62,8 +62,8 @@ module "firestore" { source = "../modules/firestore" project_id = var.project_id - #poller_sa_email = module.autoscaler-base.poller_sa_email - #scaler_sa_email = module.autoscaler-base.scaler_sa_email + poller_sa_email = module.autoscaler-base.poller_sa_email + scaler_sa_email = module.autoscaler-base.scaler_sa_email } module "spanner" { From fc0bf7f97d0ac74513116a98170d45b612642376 Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Mon, 15 May 2023 14:34:58 +0000 Subject: [PATCH 12/24] updated file --- terraform/modules/firestore/main.tf | 19 ++++--------------- 1 file changed, 4 insertions(+), 15 deletions(-) diff --git a/terraform/modules/firestore/main.tf b/terraform/modules/firestore/main.tf index 804ba6d0..16bb59f3 100644 --- a/terraform/modules/firestore/main.tf +++ b/terraform/modules/firestore/main.tf @@ -21,21 +21,6 @@ * is not being used to hold state. */ -terraform { - required_providers { - google = { - source = "hashicorp/google" - version = "~> 4.63.0" - } - } -} - -provider "google" { - project = var.project_id - region = var.region - #zone = var.zone -} - resource "google_project_iam_member" "scaler_sa_firestore" { project = var.project_id @@ -44,11 +29,15 @@ resource "google_project_iam_member" "scaler_sa_firestore" { } resource "google_project_service" "firestore" { + count = var.terraform_firestore_create ? 1 : 0 + project = var.project_id service = "firestore.googleapis.com" } resource "google_firestore_database" "database" { + count = var.terraform_firestore_create ? 1 : 0 + project = var.project_id name = "(default)" location_id = var.location From dbfdc365ac9f77ca742b7982afd154529b385e51 Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Mon, 15 May 2023 14:50:44 +0000 Subject: [PATCH 13/24] updated missing variables --- terraform/modules/firestore/variables.tf | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/terraform/modules/firestore/variables.tf b/terraform/modules/firestore/variables.tf index 8edd693a..532c90ed 100644 --- a/terraform/modules/firestore/variables.tf +++ b/terraform/modules/firestore/variables.tf @@ -43,3 +43,8 @@ variable "poller_sa_email" { variable "scaler_sa_email" { type = string } + +variable "terraform_firestore_create" { + type = bool + default = false +} \ No newline at end of file From 28675828f5b0cbf0544ad92dd5ba41aefcf489bc Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Tue, 16 May 2023 02:44:57 +0000 Subject: [PATCH 14/24] commented condition. --- terraform/modules/firestore/main.tf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/terraform/modules/firestore/main.tf b/terraform/modules/firestore/main.tf index 16bb59f3..ca80ba02 100644 --- a/terraform/modules/firestore/main.tf +++ b/terraform/modules/firestore/main.tf @@ -29,14 +29,14 @@ resource "google_project_iam_member" "scaler_sa_firestore" { } resource "google_project_service" "firestore" { - count = var.terraform_firestore_create ? 1 : 0 + #count = var.terraform_firestore_create ? 1 : 0 project = var.project_id service = "firestore.googleapis.com" } resource "google_firestore_database" "database" { - count = var.terraform_firestore_create ? 1 : 0 + #count = var.terraform_firestore_create ? 1 : 0 project = var.project_id name = "(default)" From 5e377163ae9ecc3f8d896ab72a18c632fa59ee5e Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Tue, 16 May 2023 14:20:13 +0000 Subject: [PATCH 15/24] updated condition for creating firestore --- terraform/gke/main.tf | 7 ++++--- terraform/gke/variables.tf | 5 +++++ terraform/modules/firestore/main.tf | 4 ++-- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/terraform/gke/main.tf b/terraform/gke/main.tf index f9726aa2..3888e706 100644 --- a/terraform/gke/main.tf +++ b/terraform/gke/main.tf @@ -61,9 +61,10 @@ module "autoscaler-cluster" { module "firestore" { source = "../modules/firestore" - project_id = var.project_id - poller_sa_email = module.autoscaler-base.poller_sa_email - scaler_sa_email = module.autoscaler-base.scaler_sa_email + project_id = var.project_id + terraform_firestore_create = var.terraform_firestore_create + poller_sa_email = module.autoscaler-base.poller_sa_email + scaler_sa_email = module.autoscaler-base.scaler_sa_email } module "spanner" { diff --git a/terraform/gke/variables.tf b/terraform/gke/variables.tf index e2bd341e..12252743 100644 --- a/terraform/gke/variables.tf +++ b/terraform/gke/variables.tf @@ -61,3 +61,8 @@ variable "terraform_dashboard" { type = bool default = true } + +variable "terraform_firestore_create" { + type = bool + default = false +} \ No newline at end of file diff --git a/terraform/modules/firestore/main.tf b/terraform/modules/firestore/main.tf index ca80ba02..16bb59f3 100644 --- a/terraform/modules/firestore/main.tf +++ b/terraform/modules/firestore/main.tf @@ -29,14 +29,14 @@ resource "google_project_iam_member" "scaler_sa_firestore" { } resource "google_project_service" "firestore" { - #count = var.terraform_firestore_create ? 1 : 0 + count = var.terraform_firestore_create ? 1 : 0 project = var.project_id service = "firestore.googleapis.com" } resource "google_firestore_database" "database" { - #count = var.terraform_firestore_create ? 1 : 0 + count = var.terraform_firestore_create ? 1 : 0 project = var.project_id name = "(default)" From ddd91088b42c3f575d0e58eeb85120261d22064c Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Tue, 16 May 2023 16:20:23 +0000 Subject: [PATCH 16/24] updated the readme steps, included the firestore variable to be set --- .../cloud-functions/distributed/README.md | 8 ++++---- .../cloud-functions/per-project/README.md | 18 ++++++++++++------ terraform/gke/README.md | 10 ++++++++++ 3 files changed, 26 insertions(+), 10 deletions(-) diff --git a/terraform/cloud-functions/distributed/README.md b/terraform/cloud-functions/distributed/README.md index 545cb176..dfd91753 100644 --- a/terraform/cloud-functions/distributed/README.md +++ b/terraform/cloud-functions/distributed/README.md @@ -169,11 +169,11 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives. 7. The Autoscaler state can be stored in either Firestore or Cloud Spanner. - In case you want to use Firestore, update the database created with the - Google App Engine app to use [Firestore native mode][firestore-native]. - + In case you want to use Firestore, set the value of the following variable + to true. + ```sh - gcloud firestore databases update --type=firestore-native + export TF_VAR_terraform_firestore_create=true ``` In case you want to use Cloud Spanner, no action is needed at this point. diff --git a/terraform/cloud-functions/per-project/README.md b/terraform/cloud-functions/per-project/README.md index db8336ce..87363514 100644 --- a/terraform/cloud-functions/per-project/README.md +++ b/terraform/cloud-functions/per-project/README.md @@ -203,9 +203,15 @@ In this section you prepare your project for deployment. For more information on how to make your Spanner instance to be managed by Terraform, see [Importing your Spanner instances](#importing-your-spanner-instances) -3. If you chose to store the state in Firestore, skip this step. If you want - to store the state in Cloud Spanner and you don't have a Spanner - instance yet for that, then set the following variable so that Terraform +3. If you chose to store the state in Firestore, set the value of the following variable + to true and continue to step 5. + + ```sh + export TF_VAR_terraform_firestore_create=true + ``` + +4. Alternatively, If you want to store the state in Cloud Spanner and you don't have + a Spanner instance yet for that, then set the following variable so that Terraform creates an instance for you named `autoscale-test-state`: ```sh @@ -244,20 +250,20 @@ In this section you prepare your project for deployment. For more information on how to make your existing Spanner instance to be managed by Terraform, see [Importing your Spanner instances](../per-project/README.md#importing-your-spanner-instances) -4. Change directory into the Terraform per-project directory and initialize it. +5. Change directory into the Terraform per-project directory and initialize it. ```sh cd "${AUTOSCALER_DIR}" terraform init ``` -5. Import the existing AppEngine application into Terraform state +6. Import the existing AppEngine application into Terraform state ```sh terraform import module.scheduler.google_app_engine_application.app "${PROJECT_ID}" ``` -6. Create the Autoscaler infrastructure. Answer `yes` when prompted, after +7. Create the Autoscaler infrastructure. Answer `yes` when prompted, after reviewing the resources that Terraform intends to create. ```sh diff --git a/terraform/gke/README.md b/terraform/gke/README.md index 980715f2..c099c62a 100644 --- a/terraform/gke/README.md +++ b/terraform/gke/README.md @@ -216,6 +216,16 @@ In this section you prepare your project for deployment. export TF_VAR_terraform_firestore_create=true ``` +2. Enable the additional APIs: + + ```sh + gcloud services enable \ + appengine.googleapis.com \ + firestore.googleapis.com + ``` + +3. Next, continue to [Deploying the Autoscaler](#deploying-the-autoscaler) + On the other hand if you do not want to create the Firestore database using terraform,then continue with the below steps to manually create the Firestore database using gcloud commands. From a9213e72009a28dbe4facd10a6b2e454c8cba326 Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Wed, 17 May 2023 14:32:09 +0000 Subject: [PATCH 17/24] updated cloud function files --- .../cloud-functions/distributed/README.md | 23 +++---------------- .../cloud-functions/per-project/README.md | 22 +----------------- 2 files changed, 4 insertions(+), 41 deletions(-) diff --git a/terraform/cloud-functions/distributed/README.md b/terraform/cloud-functions/distributed/README.md index dfd91753..c04ae02d 100644 --- a/terraform/cloud-functions/distributed/README.md +++ b/terraform/cloud-functions/distributed/README.md @@ -161,13 +161,7 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives. cloudresourcemanager.googleapis.com ``` -6. Create a Google App Engine app, to enable the APIs for Cloud Scheduler and Firestore. - - ```sh - gcloud app create --region="${AUTOSCALER_APP_ENGINE_LOCATION}" - ``` - -7. The Autoscaler state can be stored in either Firestore or Cloud Spanner. +6. The Autoscaler state can be stored in either Firestore or Cloud Spanner. In case you want to use Firestore, set the value of the following variable to true. @@ -180,14 +174,13 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives. ### Deploying the Autoscaler -1. Set the project ID, region, zone and App Engine location in the +1. Set the project ID, region and zone in the corresponding Terraform environment variables ```sh export TF_VAR_project_id="${AUTOSCALER_PROJECT_ID}" export TF_VAR_region="${AUTOSCALER_REGION}" export TF_VAR_zone="${AUTOSCALER_ZONE}" - export TF_VAR_location="${AUTOSCALER_APP_ENGINE_LOCATION}" ``` 2. Change directory into the Terraform scaler-project directory and initialize @@ -229,14 +222,12 @@ topic and function in the project where the Spanner instances live. gcloud config set project "${APP_PROJECT_ID}" ``` -4. Choose the [region and zone][region-and-zone] and - [App Engine Location][app-engine-location] where the Application project +4. Choose the [region and zone][region-and-zone] where the Application project will be located. ```sh export APP_REGION=us-central1 export APP_ZONE=us-central1-c - export APP_APP_ENGINE_LOCATION=us-central ``` 5. Use the following command to enable the Cloud APIs: @@ -252,13 +243,6 @@ topic and function in the project where the Spanner instances live. cloudbuild.googleapis.com ``` -6. Create an App to enable Cloud Scheduler, but do not create a Firestore - database: - - ```sh - gcloud app create --region="${APP_APP_ENGINE_LOCATION}" - ``` - ### Deploy the Application infrastructure 1. Set the project ID, region, zone and App Engine location in the @@ -268,7 +252,6 @@ topic and function in the project where the Spanner instances live. export TF_VAR_project_id="${APP_PROJECT_ID}" export TF_VAR_region="${APP_REGION}" export TF_VAR_zone="${APP_ZONE}" - export TF_VAR_location="${APP_APP_ENGINE_LOCATION}" ``` 2. If you want to create a new Spanner instance for testing the Autoscaler, set diff --git a/terraform/cloud-functions/per-project/README.md b/terraform/cloud-functions/per-project/README.md index 87363514..3018cccc 100644 --- a/terraform/cloud-functions/per-project/README.md +++ b/terraform/cloud-functions/per-project/README.md @@ -130,14 +130,12 @@ In this section you prepare your project for deployment. gcloud config set project "${PROJECT_ID}" ``` -4. Choose the [region and zone][region-and-zone] and - [App Engine Location][app-engine-location] where the Autoscaler +4. Choose the [region and zone][region-and-zone] where the Autoscaler infrastructure will be located. ```sh export REGION=us-central1 export ZONE=us-central1-c - export APP_ENGINE_LOCATION=us-central ``` 5. Enable the required Cloud APIs @@ -155,24 +153,6 @@ In this section you prepare your project for deployment. cloudresourcemanager.googleapis.com ``` -6. Create a Google App Engine app, to enable the APIs - for Cloud Scheduler and Firestore - - ```sh - gcloud app create --region="${APP_ENGINE_LOCATION}" - ``` - -7. The Autoscaler state can be stored in either Firestore or Cloud Spanner. - - In case you want to use Firestore, update the database created with the - Google App Engine app to use [Firestore native mode][firestore-native]. - - ```sh - gcloud firestore databases update --type=firestore-native - ``` - - In case you want to use Cloud Spanner, no action is needed at this point. - ## Deploying the Autoscaler 1. Set the project ID, region and zone in the corresponding Terraform From f234ca311d6353c12db183e71107597b8f9fbf7a Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Wed, 17 May 2023 15:56:10 +0000 Subject: [PATCH 18/24] updated formatting --- terraform/cloud-functions/per-project/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/terraform/cloud-functions/per-project/README.md b/terraform/cloud-functions/per-project/README.md index 3018cccc..4a3c539c 100644 --- a/terraform/cloud-functions/per-project/README.md +++ b/terraform/cloud-functions/per-project/README.md @@ -183,9 +183,9 @@ In this section you prepare your project for deployment. For more information on how to make your Spanner instance to be managed by Terraform, see [Importing your Spanner instances](#importing-your-spanner-instances) -3. If you chose to store the state in Firestore, set the value of the following variable - to true and continue to step 5. - +3. If you chose to store the state in Firestore, set the value of the following + variable to true and continue to step 5. + ```sh export TF_VAR_terraform_firestore_create=true ``` @@ -209,7 +209,7 @@ In this section you prepare your project for deployment. ``` If you already have a Spanner instance where state must be stored, - only set the the name of your instance: + only set the the name of your instance. ```sh export TF_VAR_state_spanner_name= From 9f1432d802c4466fb06792d7a32c29d233b48f61 Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Wed, 17 May 2023 16:04:06 +0000 Subject: [PATCH 19/24] Updated files to remove formatting errors --- terraform/cloud-functions/distributed/README.md | 2 +- terraform/gke/README.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/terraform/cloud-functions/distributed/README.md b/terraform/cloud-functions/distributed/README.md index c04ae02d..8e83cecf 100644 --- a/terraform/cloud-functions/distributed/README.md +++ b/terraform/cloud-functions/distributed/README.md @@ -165,7 +165,7 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives. In case you want to use Firestore, set the value of the following variable to true. - + ```sh export TF_VAR_terraform_firestore_create=true ``` diff --git a/terraform/gke/README.md b/terraform/gke/README.md index c099c62a..f5b178f2 100644 --- a/terraform/gke/README.md +++ b/terraform/gke/README.md @@ -207,14 +207,14 @@ In this section you prepare your project for deployment. ## Using Firestore for Autoscaler state -### Setting up Firestore for storing Autoscaler state using terraform. +### Setting up Firestore for storing Autoscaler state using terraform -1. If you want to create a new Firestore database for storing the state of the +1. If you want to create a new Firestore database for storing the state of the Autoscaler, set the following variable. ```sh export TF_VAR_terraform_firestore_create=true - ``` + ``` 2. Enable the additional APIs: From 33ac88e546210f4c0b888b263f071ef17298d36e Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Thu, 18 May 2023 01:19:00 +0000 Subject: [PATCH 20/24] updated formatting --- terraform/gke/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/terraform/gke/README.md b/terraform/gke/README.md index f5b178f2..58fb11d6 100644 --- a/terraform/gke/README.md +++ b/terraform/gke/README.md @@ -209,7 +209,7 @@ In this section you prepare your project for deployment. ### Setting up Firestore for storing Autoscaler state using terraform -1. If you want to create a new Firestore database for storing the state of the +1. If you want to create a new Firestore database for storing the state of the Autoscaler, set the following variable. ```sh From 4ed3dd6b093ac3a51cc08d8294bd572aa3b06fd9 Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Thu, 18 May 2023 01:26:24 +0000 Subject: [PATCH 21/24] updated text for resolving formatting errors --- terraform/cloud-functions/per-project/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/terraform/cloud-functions/per-project/README.md b/terraform/cloud-functions/per-project/README.md index 4a3c539c..69898caa 100644 --- a/terraform/cloud-functions/per-project/README.md +++ b/terraform/cloud-functions/per-project/README.md @@ -190,9 +190,9 @@ In this section you prepare your project for deployment. export TF_VAR_terraform_firestore_create=true ``` -4. Alternatively, If you want to store the state in Cloud Spanner and you don't have - a Spanner instance yet for that, then set the following variable so that Terraform - creates an instance for you named `autoscale-test-state`: +4. Alternatively, if you want to store the state in Cloud Spanner and you don't + have a Spanner instance yet for that, then set the following variable so that + Terraform creates an instance for you named `autoscale-test-state`. ```sh export TF_VAR_terraform_spanner_state=true From 58f3710d76be372b409297a5dc8f026b649d1a83 Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Thu, 6 Jul 2023 03:08:36 +0000 Subject: [PATCH 22/24] updated files as per the comments. --- .../cloud-functions/distributed/README.md | 9 +-- terraform/gke/README.md | 60 +++---------------- terraform/modules/firestore/main.tf | 11 +--- terraform/modules/firestore/variables.tf | 6 -- 4 files changed, 13 insertions(+), 73 deletions(-) diff --git a/terraform/cloud-functions/distributed/README.md b/terraform/cloud-functions/distributed/README.md index 8e83cecf..6356a48f 100644 --- a/terraform/cloud-functions/distributed/README.md +++ b/terraform/cloud-functions/distributed/README.md @@ -163,14 +163,7 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives. 6. The Autoscaler state can be stored in either Firestore or Cloud Spanner. - In case you want to use Firestore, set the value of the following variable - to true. - - ```sh - export TF_VAR_terraform_firestore_create=true - ``` - - In case you want to use Cloud Spanner, no action is needed at this point. + Firestore database will be created by default, In case you want to use Cloud Spanner, no action is needed at this point. ### Deploying the Autoscaler diff --git a/terraform/gke/README.md b/terraform/gke/README.md index 58fb11d6..3a8cb353 100644 --- a/terraform/gke/README.md +++ b/terraform/gke/README.md @@ -184,9 +184,7 @@ In this section you prepare your project for deployment. export TF_VAR_terraform_spanner_test=true ``` - On the other hand, if you do not want to create a new Spanner instance - because you already have an instance for the Autoscaler to monitor, set the - name name of your instance in the following variable + On the other hand, if you do not want to create a new Spanner instance because you already have an instance for the Autoscaler to monitor, set thevname of your instance in the following variable ```sh export TF_VAR_spanner_name= @@ -209,61 +207,21 @@ In this section you prepare your project for deployment. ### Setting up Firestore for storing Autoscaler state using terraform -1. If you want to create a new Firestore database for storing the state of the - Autoscaler, set the following variable. +1. A Firestore database will be created by default for storing the state of the + Autoscaler in the [Firestore native mode][firestore-native]. - ```sh - export TF_VAR_terraform_firestore_create=true - ``` - -2. Enable the additional APIs: +2. Enable the additional APIs necessary for using Firestore: ```sh - gcloud services enable \ + gcloud services enable \ appengine.googleapis.com \ firestore.googleapis.com ``` 3. Next, continue to [Deploying the Autoscaler](#deploying-the-autoscaler) - On the other hand if you do not want to create the Firestore database using - terraform,then continue with the below steps to manually create the Firestore - database using gcloud commands. - -### Manually setting up Firestore for storing Autoscaler state - -1. To use Firestore for the Autoscaler state, choose the - [App Engine Location][app-engine-location] where the Autoscaler - infrastructure will be created, for example: - - ```sh - export APP_ENGINE_LOCATION=us-central - ``` - -2. Enable the additional APIs: - - ```sh - gcloud services enable \ - appengine.googleapis.com \ - firestore.googleapis.com - ``` - -3. Create a Google App Engine app to enable the API for Firestore: - - ```sh - gcloud app create --region="${APP_ENGINE_LOCATION}" - ``` - -4. To store the state of the Autoscaler, update the database created with the - Google App Engine app to use [Firestore native mode][firestore-native]. - - ```sh - gcloud firestore databases update --type=firestore-native - ``` - - You will also need to make a minor modification to the Autoscaler - configuration. The required steps to do this are later in these - instructions. +4. You will also need to make a minor modification to the Autoscaler + configuration. The required steps to do this are later in these instructions. 5. Next, continue to [Deploying the Autoscaler](#deploying-the-autoscaler) @@ -466,7 +424,7 @@ similar process. the schema of the configuration, see the [Poller configuration] [autoscaler-config-params] section. -9. If you have chosen to use Firestore to hold the Autoscaler state as described +9. If you wish to use Firestore to hold the Autoscaler state as described above, edit the above files, and remove the following lines: ```yaml @@ -526,7 +484,7 @@ following the instructions above. ### If the Poller fails to run successfully -1. If you have chosen to use Firestore for Autoscaler state and you see the +1. If you have chosen to use Firestore (default) for storing Autoscaler state and you see the following error in the logs: ```sh diff --git a/terraform/modules/firestore/main.tf b/terraform/modules/firestore/main.tf index 16bb59f3..c1c15583 100644 --- a/terraform/modules/firestore/main.tf +++ b/terraform/modules/firestore/main.tf @@ -15,8 +15,8 @@ */ /* - * While the Firestore database is created using the gcloud CLI, the - * Terraform-created service account used by the Scaler needs read/write + * The Firestore database is created using this terraform module and the + * terraform-created service account used by the Scaler needs read/write * permissions to the instance in the appropriate project if Spanner * is not being used to hold state. */ @@ -29,19 +29,14 @@ resource "google_project_iam_member" "scaler_sa_firestore" { } resource "google_project_service" "firestore" { - count = var.terraform_firestore_create ? 1 : 0 - project = var.project_id service = "firestore.googleapis.com" } resource "google_firestore_database" "database" { - count = var.terraform_firestore_create ? 1 : 0 - project = var.project_id name = "(default)" location_id = var.location - type = var.firestore_database_type - + type = "FIRESTORE_NATIVE" depends_on = [google_project_service.firestore] } diff --git a/terraform/modules/firestore/variables.tf b/terraform/modules/firestore/variables.tf index 532c90ed..eb674d01 100644 --- a/terraform/modules/firestore/variables.tf +++ b/terraform/modules/firestore/variables.tf @@ -30,12 +30,6 @@ variable "location" { default = "nam5" } -variable "firestore_database_type" { - description = "The database type. TYPE must be one of: datastore-mode, firestore-native" - type = string - default = "FIRESTORE_NATIVE" -} - variable "poller_sa_email" { type = string } From b673ec2a4ed906c779c0053a8d66e55dd563217f Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Thu, 6 Jul 2023 03:49:24 +0000 Subject: [PATCH 23/24] updated formatting. --- terraform/cloud-functions/distributed/README.md | 3 ++- terraform/gke/README.md | 8 +++++--- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/terraform/cloud-functions/distributed/README.md b/terraform/cloud-functions/distributed/README.md index 6356a48f..dcce3806 100644 --- a/terraform/cloud-functions/distributed/README.md +++ b/terraform/cloud-functions/distributed/README.md @@ -163,7 +163,8 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives. 6. The Autoscaler state can be stored in either Firestore or Cloud Spanner. - Firestore database will be created by default, In case you want to use Cloud Spanner, no action is needed at this point. + Firestore database will be created by default, In case you want to use Cloud Spanner, + no action is needed at this point. ### Deploying the Autoscaler diff --git a/terraform/gke/README.md b/terraform/gke/README.md index 3a8cb353..1bc78f69 100644 --- a/terraform/gke/README.md +++ b/terraform/gke/README.md @@ -184,7 +184,9 @@ In this section you prepare your project for deployment. export TF_VAR_terraform_spanner_test=true ``` - On the other hand, if you do not want to create a new Spanner instance because you already have an instance for the Autoscaler to monitor, set thevname of your instance in the following variable + On the other hand, if you do not want to create a new Spanner instance because + you already have an instance for the Autoscaler to monitor, set the name of your + instance in the following variable ```sh export TF_VAR_spanner_name= @@ -484,8 +486,8 @@ following the instructions above. ### If the Poller fails to run successfully -1. If you have chosen to use Firestore (default) for storing Autoscaler state and you see the - following error in the logs: +1. If you have chosen to use Firestore (default) for storing Autoscaler state and you + see the following error in the logs: ```sh Error: 5 NOT_FOUND: Database not found: projects//instances/autoscale-test/databases/spanner-autoscaler-state From d6566aee2873192f2b5a294500b9eeff70b70c20 Mon Sep 17 00:00:00 2001 From: Akshay Kalbhor Date: Fri, 7 Jul 2023 05:51:32 +0000 Subject: [PATCH 24/24] updated line length to resolve markdown failures. --- terraform/cloud-functions/distributed/README.md | 4 ++-- terraform/gke/README.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/terraform/cloud-functions/distributed/README.md b/terraform/cloud-functions/distributed/README.md index dcce3806..f82c3064 100644 --- a/terraform/cloud-functions/distributed/README.md +++ b/terraform/cloud-functions/distributed/README.md @@ -163,8 +163,8 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives. 6. The Autoscaler state can be stored in either Firestore or Cloud Spanner. - Firestore database will be created by default, In case you want to use Cloud Spanner, - no action is needed at this point. + Firestore database will be created by default, In case you want to use Cloud + Spanner, no action is needed at this point. ### Deploying the Autoscaler diff --git a/terraform/gke/README.md b/terraform/gke/README.md index 1bc78f69..86358f67 100644 --- a/terraform/gke/README.md +++ b/terraform/gke/README.md @@ -486,8 +486,8 @@ following the instructions above. ### If the Poller fails to run successfully -1. If you have chosen to use Firestore (default) for storing Autoscaler state and you - see the following error in the logs: +1. If you have chosen to use Firestore (default) for storing Autoscaler state + and you see the following error in the logs: ```sh Error: 5 NOT_FOUND: Database not found: projects//instances/autoscale-test/databases/spanner-autoscaler-state