diff --git a/lab2/README.md b/lab2/README.md index a4fec22..f68cb27 100644 --- a/lab2/README.md +++ b/lab2/README.md @@ -1,12 +1,12 @@ # Lab 2: Create Google Cloud Infrastrcuture Components - + In this lab, you are going to: * Create a VPC * Create two external static IP addresses -* Create a GKE cluster +* Create a GKE cluster in the VPC created in the first step * Provision managed Anthos Service Mesh on the GKE cluster * Create a CloudSQL PostgreSQL database instance - + Create a VPC: ```bash export VPC_NETWORK="redis-vpc-network" @@ -14,10 +14,10 @@ export SUBNETWORK=$VPC_NETWORK gcloud compute networks create $VPC_NETWORK \ --subnet-mode=auto ``` - + On success, you should view your VPC network as follows: ![VPC Network](./img/Redis_VPC_Network.png) - + Reserve external static IP addresses: ```bash @@ -29,9 +29,16 @@ gcloud compute addresses create redis-client-host-ip --region us-central1 export REDIS_CLIENT_HOST_IP="$(gcloud compute addresses describe redis-client-host-ip --region=us-central1 --format='value(address)')" ``` +Make sure above static IP addresses are acquired by printing the values to the console. + +``` +echo $REDIS_API_GATEWAY_IP +echo $REDIS_CLIENT_HOST_IP +``` + On success, you should see the newly created reserved public IP addresses as shown below: ![Reserved IPs](./img/reserved_ips.png) - + Create a GKE cluster: ```bash export PROJECT_ID=$(gcloud info --format='value(config.project)') @@ -50,9 +57,10 @@ gcloud container clusters create $CLUSTER_NAME \ --labels="mesh_id=proj-${PROJECT_NUMBER}" ``` +The GKE cluster creation can take anywhere between 5 to 10 minutes. On success, you should see your newly created GKE cluster like below: ![GKE](./img/GKE_Cluster.png) - + Provision Anthos Service Mesh: Enable Anthos Service Mesh on your project's Fleet: ```bash @@ -67,7 +75,7 @@ gcloud container fleet memberships register $CLUSTER_NAME-membership \ ``` On success, you can verify the GKE cluster's fleet membership in Google Cloud Console: ![ASM Fleet Membership](./img/ASM_Fleet_Membership_Reg.png) - + Provision managed Anthos Service Mesh on the cluster using the Fleet API: ```bash gcloud container fleet mesh update \ @@ -126,7 +134,7 @@ spec: status: phase: Active ``` - + Create the Source DB - Cloud SQL for PostgreSQL: Note: **database-flags=cloudsql.logical_decoding=on** enables logical replication workflows and change data capture (CDC) workflows which is required by RDI. Create PostgreSQL instance: @@ -143,13 +151,15 @@ gcloud sql instances create $POSTGRESQL_INSTANCE \ --root-password=postgres \ --database-flags=cloudsql.logical_decoding=on ``` +The above command may take anywhere from 5 to 10mins to finish and create a PostgresSQL instance for you. + On success, you can see your CloudSQL PostgreSQL database in Google Cloud console like the following: ![Cloud SQL](./img/CloudSQL.png) Capture the `Public IP address` for later use in the lab in an environment variable: ```bash export POSTGRESQL_INSTANCE_IP=$(gcloud sql instances describe $POSTGRESQL_INSTANCE --format=json | jq -r '.ipAddresses[] | select(.type == "PRIMARY") | .ipAddress') ``` - + By default, PostgreSQL database superuser (postgres) does not have permission to create a replication slot which is required by RDI. Run the following commands to grant the permission: ```bash cat < alter_postgres_replication.sql @@ -165,6 +175,5 @@ On success, you should see the following output: Connecting to database with SQL user [postgres].Password: ALTER ROLE ``` - -[<< Previous Lab (1) <<](../lab1/README.md) | [>> Next Lab (3) >>](../lab3/README.md) +[<< Previous Lab (1) <<](../lab1/README.md) | [>> Next Lab (3) >>](../lab3/README.md) diff --git a/lab3/README.md b/lab3/README.md index 1eea050..7435dcd 100644 --- a/lab3/README.md +++ b/lab3/README.md @@ -1,9 +1,9 @@ # Lab 3: Create a Redis Enterprise Cloud subscription on Google Cloud - + In this lab, you are going to: * Create a Redis Enterprise Cloud subscription * Collect Redis Enterprise Database connection information - + Create a Redis Cloud subscription: * Follow this [link](https://docs.redis.com/latest/rc/rc-quickstart/#create-an-account) through step 6. * In step 4, choose Google Cloud. Then come back here to continue on section 3 below to initialize two environments for this workshop. @@ -27,6 +27,15 @@ export REDIS_TARGET_DB_HOST= export REDIS_TARGET_DB_PORT= export REDIS_TARGET_DB_PASSWORD= ``` - -[<< Previous Lab (2) <<](../lab2/README.md) | [>> Next Lab (4) >>](../lab4/README.md) +To double check all the above environment variables are set correctly, print the values to the console. +``` +echo $REDIS_URI +echo $REDIS_INSIGHT_PORT +echo $REDIS_TARGET_DB_HOST +echo $REDIS_TARGET_DB_PORT +echo $REDIS_TARGET_DB_PASSWORD +``` + + +[<< Previous Lab (2) <<](../lab2/README.md) | [>> Next Lab (4) >>](../lab4/README.md) diff --git a/lab5/README.md b/lab5/README.md index 9579f56..c6021ce 100644 --- a/lab5/README.md +++ b/lab5/README.md @@ -1,11 +1,11 @@ # Lab 5: Set up Redis Data Integration (RDI) - + In this lab, you are going to: * Set up Redis Data Integration (RDI) * Create & Deploy two RDI jobs to replicate data from a CloudSQL PostgreSQL database to Redis Enterprise - + ![RDI - CloudSQL](./img/RDI_Ingest_cloudsql.png) - + Deploy a Redis Enterprise cluster: ```bash kubectl create namespace redis @@ -25,24 +25,24 @@ spec: nodes: 3 EOF -kubectl apply -f rec.yaml -n redis +kubectl apply -f rec.yaml -n redis ``` It will take between 6 and 8 minutes to complete. You can run the following command to see the progress: ```bash watch kubectl get all ``` - + Then, retrieve the password for the Redis Enterprise Cluster's default uesr: demo@redis.com: ```bash export REC_PWD=$(kubectl get secrets -n redis rec -o jsonpath="{.data.password}" | base64 --decode) ``` - + Note: You can open another Google Cloud Shell to grab the $REC_PWD and display its value in the shell for later use: ``` export REC_PWD=$(kubectl get secrets -n redis rec -o jsonpath="{.data.password}" | base64 --decode) echo $REC_PWD ``` - + Install Redis Gears: ```bash kubectl exec -it rec-0 -n redis -- curl -s https://redismodules.s3.amazonaws.com/redisgears/redisgears_python.Linux-ubuntu18.04-x86_64.1.2.6.zip -o /tmp/redis-gears.zip @@ -55,7 +55,16 @@ Defaulted container "redis-enterprise-node" out of: redis-enterprise-node, boots Defaulted container "redis-enterprise-node" out of: redis-enterprise-node, bootstrapper {"action_uid":"e0a88d27-4c52-4e9e-b1f1-6095baa4d184","author":"RedisLabs","capabilities":["types","crdb","failover_migrate","persistence_aof","persistence_rdb","clustering","backup_restore","reshard_rebalance","eviction_expiry","intershard_tls","intershard_tls_pass","ipv6"],"command_line_args":"Plugin gears_python CreateVenv 1","config_command":"RG.CONFIGSET","crdb":{},"dependencies":{"gears_python":{"sha256":"5206dfc7199e66c6cfe7a9443c5705e72ceccaccc02d229607e844337e00ce7f","url":"http://redismodules.s3.amazonaws.com/redisgears/redisgears-python.Linux-ubuntu18.04-x86_64.1.2.6.tgz"}},"description":"Dynamic execution framework for your Redis data","display_name":"RedisGears","email":"meir@redislabs.com","homepage":"http://redisgears.io","is_bundled":false,"license":"Redis Source Available License Agreement","min_redis_pack_version":"6.0.12","min_redis_version":"6.0.0","module_name":"rg","semantic_version":"1.2.6","sha256":"ca9c81c7c0e523a5ea5cf41c95ea53abcd6b90094be2f0901814dd5fdbc135d6","uid":"d97a561c5e94e78d60c5b2dfa48a427a","version":10206} ``` - +Instead if you see an error like this, +``` +kubectl exec -it rec-0 -n redis -- curl -k -s -u "demo@redis.com:${REC_PWD}" -F "module=@/tmp/redis-gears.zip" https://localhost:9443/v2/modules +Defaulted container "redis-enterprise-node" out of: redis-enterprise-node, bootstrapper +Defaulted container "redis-enterprise-node" out of: redis-enterprise-node, bootstrapper +{"description":"Cannot perform operation while cluster is unstable","error_code":"cluster_unstable"} +``` +your cluster is not ready yet. Wait for another 5 minutes or so and re-run above `Install Redis Gears` command. + + Install RDI CLI container: ```bash kubectl config set-context --current --namespace=default @@ -88,19 +97,19 @@ spec: EOF kubectl apply -f /tmp/redis-di-cli-pod.yml ``` - + Create a new RDI database: ```bash -kubectl exec -it -n default pod/redis-di-cli -- redis-di create --cluster-host localhost +kubectl exec -it -n default pod/redis-di-cli -- redis-di create --cluster-host localhost ``` - + Use the following input and answer the rest of the prompt: ```bash Host/IP of Redis Enterprise Cluster: rec.redis.svc.cluster.local Redis Enterprise Cluster username: demo@redis.com Redis Enterprise Cluster Password: grab password from $REC_PWD Everything else take the default valuest -Password for the new RDI Database: redis +Password for the new RDI Database: redis ``` On success, you should see similar output like below: ``` @@ -119,8 +128,8 @@ Setting up RDI Engine on port 12001 Successfully configured RDI database on port 12001 Default Context created successfully ``` - -Edit config.yaml: + +Now make sure you are in the `REDIS_REPO` directory and edit the `config.yaml` file: Update the value of the following fields in the `connections:target:` section: ``` host: @@ -128,32 +137,32 @@ port: user: default password: ``` - + Create a ConfigMap for Redis Data Integration: ```bash kubectl create configmap redis-di-config --from-file=config.yaml -n default ``` -You might need to wait for 30 seconds or more for the configmap to be ready for next step. - +You might need to wait for 30 seconds or more for the configmap to be ready for next step. + Deploy the RDI configuration: ```bash kubectl exec -n default -it pod/redis-di-cli -- redis-di deploy ``` When prompted for password (RDI Database Password []:), enter `redis` and hit return. - - -Edit application.properties: + + +Make sure you are in the `REDIS_REPO` directory and edit the `application.properties`: Update the value of the following fields with the CloudSQL PostgreSQL's public IP address. You can run `echo $POSTGRESQL_INSTANCE_IP` to display the value of the IP address. ``` debezium.source.database.hostname= ``` - + Create a ConfigMap for Debezium Server: ```bash kubectl create configmap debezium-config --from-file=application.properties -n default ``` -You might need to wait for 30 seconds or more for the configmap to be ready for next step. - +You might need to wait for 30 seconds or more for the configmap to be ready for next step. + Create the Debezium Server Pod: ```bash cat << EOF > /tmp/debezium-server-pod.yml @@ -185,7 +194,7 @@ spec: EOF kubectl apply -f /tmp/debezium-server-pod.yml ``` - + Create a ConfigMap for the two RDI jobs for replicating order information from CloudSQL to Redis: ```bash kubectl create configmap redis-di-jobs --from-file=./rdi_jobs @@ -200,14 +209,14 @@ You should similar output if the jobs are successfully created: ``` INFO - Reading orders.yaml job INFO - Reading orderProducts.yaml job -RDI Database Password []: +RDI Database Password []: WARNING - Property 'json_update_strategy' will be deprecated in future releases. Use 'on_update' job-level property to define the json update strategy. Deploying settings to 10.96.0.22:12001 INFO - Connected to target database INFO - RedisJSON is installed on the target Redis DB Done ``` - + Check if the job has been created: ```bash kubectl exec -it -n default pod/redis-di-cli -- redis-di list-jobs @@ -223,7 +232,7 @@ Ingest Jobs | orderProducts | | | | orderProducts | Yes | No | No | +---------------+--------+----+--------+---------------+-----------------+--------+-----+ ``` - + Verify the job status in RDI: ```bash kubectl exec -n default -it pod/redis-di-cli -- redis-di status @@ -237,12 +246,12 @@ started Engine State Sync mode: cdc -Last data retrieved (source): 07/22/2023 23:26:56.000000 -Last data updated (target): 07/22/2023 23:26:57.075254 +Last data retrieved (source): 07/22/2023 23:26:56.000000 +Last data updated (target): 07/22/2023 23:26:57.075254 Last snapshot: Number of processes: 4 - Start: 07/22/2023 21:26:12.722103 - End: 07/22/2023 21:30:34.350942 + Start: 07/22/2023 21:26:12.722103 + End: 07/22/2023 21:30:34.350942 Connections +--------+-------+--------------------------------------------------------+-------+----------+---------+----------+-----------+ @@ -276,5 +285,5 @@ Performance Statistics per Batch (batch size: 2000) Last run(s) duration (ms): [4] Average run duration (ms): 2.00 ``` - + [<< Previous Lab (4) <<](../lab4/README.md) | [>> Next Lab (6) >>](../lab6/README.md) diff --git a/lab8/README.md b/lab8/README.md index 0abf342..1805521 100644 --- a/lab8/README.md +++ b/lab8/README.md @@ -1,8 +1,8 @@ # Lab 8: Document Question Answering with Langchain, VertexAI and Redis - + In this lab, you are going to: * Exercise a colab to develop a document-based Question & Answering app with Langchain, VertexAI and Redis - + Access the [colab](https://colab.research.google.com/github/gmflau/google-dev-day-workshop/blob/main/lab8/VertexAI_LangChain_Redis.ipynb) for this lab - -[<< Previous Lab (7) <<](../lab7/README.md) + +[<< Previous Lab (7) <<](../lab7/README.md) | [>> Next Lab (9) >>](../lab9/README.md) diff --git a/lab9/README.md b/lab9/README.md new file mode 100644 index 0000000..bc698ba --- /dev/null +++ b/lab9/README.md @@ -0,0 +1,26 @@ +# Lab 9: Cleanup + +In this lab, you are going to: +* Cleanup all resources so that you do not incur un-intended charges from Google Cloud. + +You can delete the runtime of your Colab environment from Lab8, by simply clicking on `Disconnect and delete runtime` option. + +![Colab Shutdown](./img/lab9-img0.png) + +In GCP, if yo delete the project, all the cloud resources you have provisioned, are all shutdown. So, the clean up is as simple as deleting a project. But before you do so, make sure your project Billing is not set. +Go to `Billing` on your project. +![Project Billing](./img/lab9-img1.png) + +If you are using a vanilla GCP account with free credit, the billing is not started on it. You can confirm this by looking at the `Free trial credit` panel. The fact that `ACTIVATE` button is enabled, means your billing has not started. Do NOT ACTIVATE it, if you have credits still remaining. But you are good because Billing has not started yet. +![Free Trial](./img/lab9-img2.png) + +Now you can go ahead and delete the Project, by selecting it and hitting `DELETE` button. + +![Project Shutdown](./img/lab9-img3.png) + +Project may be flagged for deletiong at a later date, but do not worry. All of your cloud resources are shutdown. +![Project Pending Shutdown](./img/lab9-img4.png) + +You are all good. + +[<< Previous Lab (8) <<](../lab8/README.md) diff --git a/lab9/img/lab9-img0.png b/lab9/img/lab9-img0.png new file mode 100644 index 0000000..fc539e8 Binary files /dev/null and b/lab9/img/lab9-img0.png differ diff --git a/lab9/img/lab9-img1.png b/lab9/img/lab9-img1.png new file mode 100644 index 0000000..b3b9d56 Binary files /dev/null and b/lab9/img/lab9-img1.png differ diff --git a/lab9/img/lab9-img2.png b/lab9/img/lab9-img2.png new file mode 100644 index 0000000..36ff3e7 Binary files /dev/null and b/lab9/img/lab9-img2.png differ diff --git a/lab9/img/lab9-img3.png b/lab9/img/lab9-img3.png new file mode 100644 index 0000000..2b87726 Binary files /dev/null and b/lab9/img/lab9-img3.png differ diff --git a/lab9/img/lab9-img4.png b/lab9/img/lab9-img4.png new file mode 100644 index 0000000..27440c3 Binary files /dev/null and b/lab9/img/lab9-img4.png differ