Skip to content

Latest commit

 

History

History
327 lines (253 loc) · 19 KB

platform-guide.md

File metadata and controls

327 lines (253 loc) · 19 KB
Raw

Platform Guide

This document details the operations where Google Cloud Platform (GCP) is involved.

The instructions in all parts of this document work for Linux, OS X, and Windows, with the following pointers:

  • Replace ./gradlew to gradlew.bat if you are using Windows.
  • All the commands are assumed to be run from the root main project (not the ops project) folder, unless otherwise specified.

Deploying to a staging server

Note: This document does not have preference over either GAE standard or flexible environment. It is your duty to decide on the environment you will be using based on your needs.

  1. The deployment uses Google Cloud SDK, which requires Python 3 (recommended) or Python 2.7. Install it if you have not done so.

  2. Create your own project on GCP.
    This instruction will use teammates-john as the project identifier.
    The eventual URL of the app will be like this: https://teammates-john.appspot.com.

  3. Enable the following APIs in your project:

  4. Create the SQL instance on GCP, and enable Private IP.

  5. Authorize your Google account to be used by the Google Cloud SDK if you have not done so.

    gcloud auth login

    Follow the steps until you see You are now logged in as [...] on the console.

  6. Modify configuration files.

    1. src/main/resources/build.properties
      Edit the file as instructed in its comments. In particular:
    2. src/main/appengine/app.yaml
      • Modify the vpc_access_connector field to specify your project's VPC connector.
      • Modify if necessary, e.g. to change App Engine instance type (standard env), to set static resources cache expiration time (standard env), to set required CPU/memory resources (flexible env), to configure liveness check (flexible env), or to set automatic scaling policies.
    3. src/web/environments/config.ts
      • Modify if necessary, e.g. to change the version number displayed to user. Note that this modification needs to be done before building the front-end files.

  7. Ensure that the front-end files have been built.

  8. (Optional but recommended) If you are deploying the application only for testing purpose, it is highly recommended to not deploy the cron jobs as they will necessitate the application to have 100% uptime (thereby unnecessarily charging instance cost). You can achieve this in two different ways:

    • Use a deployment command that does not deploy the cron job. Details can be found in the next section.
    • Remove all cron job entries in src/main/appengine/cron.yaml before running the deployment command.

  9. Deploy the application to your staging server.

    • Run the following command:

      # Creates the application on GCP
gcloud app create

# Deploy to standard env
./gradlew appengineDeployAll

# Deploy to flex env
./gradlew appengineDeployAll -Pflex

    • Wait until you see all the following messages or similar on the console:

      • Deployed service [default] to [https://8-0-0-dot-teammates-john.appspot.com]
      • Cron jobs have been updated.
      • Indexes are being rebuilt. This may take a moment.
      • Task queues have been updated.

    • You can also deploy individual configurations independently as follows:

      • Application: ./gradlew appengineDeploy (or ./gradlew appengineDeploy -Pflex)
      • Cron job configuration: ./gradlew appengineDeployCron
      • Datastore indexes: ./gradlew appengineDeployIndex
      • Task queue configuration: ./gradlew appengineDeployQueue

  10. (Optional) Set the version you deployed as the "default":

    • Go to App Engine dashboard: https://console.cloud.google.com/appengine?project=teammates-john.
    • Click Versions on the left bar.
    • Tick the checkbox next to the deployed version and select Migrate Traffic. Wait for a few minutes.
    • If you do not wish to set the deployed version as the default, you can access the deployed app using https://{version}-dot-teammates-john.appspot.com, e.g https://8-0-0-dot-teammates-john.appspot.com.

Setting up Authentication

Two forms of authentication are supported: Google OAuth 2.0 and Firebase. You are free to decide which one to use based on your needs.

Setting up OAuth 2.0 Client

  1. Go to Google Cloud APIs & Services Credentials console.
  2. Click Create credentials and then select OAuth client ID.
  3. Choose Web Application and give the client a name (the exact name does not matter).
  4. Under Authorised redirect URIs, add the following URLs:
    • Your app URL + /oauth2callback?ngsw-bypass=true, e.g. https://teammates-john.appspot.com/oauth2callback?ngsw-bypass=true.
    • If you want to test this in your dev server, you also need to add http://localhost:8080/oauth2callback?ngsw-bypass=true.
    • Note that the redirect URIs are exact and only work for the URIs specified, without wildcards, version number specifier, etc. If you want to allow redirect for specific version (e.g. https://8-0-0-dot-teammates-john.appspot.com), you need to add the entry https://8-0-0-dot-teammates-john.appspot.com/oauth2callback?ngsw-bypass=true to the list of URIs.
  5. Click Create. You will be shown the client ID and client secret; save both information for later.

Setting up Firebase Authentication

  1. Go to Firebase console.
  2. Create a Firebase project:
    1. Click Add project.
    2. Enter a project name, e.g. teammates-john, and click Continue. You can also choose to use your TEAMMATES application's project name.
    3. Optionally, check Enable Google Analytics for this project, and click Continue.
    4. Click Continue upon successful project creation.
  3. Set up Firebase Authentication:
    1. Click on the Authentication tile.
    2. Under the Sign-in method tab, enable the sign-in methods to be supported, e.g. Google and Email/Password.
      1. Click Google.
        1. Check Enable.
        2. Select a project support email.
        3. Click Save.
      2. Click Add new provider, and click Email/Password.
        1. Check Enable for Email/Password.
        2. Check Enable for Email link (passwordless sign-in).
        3. Click Save.
    3. Under the Settings tab, add your application domain (e.g. teammates-john.appspot.com) in the list of Authorised domains.
  4. Register your web app with Firebase:
    1. Go to Project settings (the gear icon at the sidebar next to Project Overview).
    2. Under the General tab, in the Your apps section, click on the Web icon (looks like </>).
    3. Enter an app nickname, e.g. teammates-john, and click Register app.
    4. Copy the firebaseConfig and paste it into environment.ts.
    5. Click Continue to console.
  5. Set up Firebase Service Account:
    1. In Project settings, under the Service accounts tab, click Generate new private key.
    2. Copy the generated file to src/main/resources/firebase-credentials.json in TEAMMATES.

Setting up Google Cloud SQL

Beginning with V9, Cloud SQL is the main database that is being used.

To create the SQL instance:

  1. Go to https://console.cloud.google.com/sql/instances and enable the Compute Engine API if prompted.
  2. Create an instance, and choose PostgreSQL, with the following configurations:
    1. Instance ID: any name of your choice
    2. Password: any password of your choice
    3. Cloud SQL Edition: Enterprise, as staging will not require Enterprise Plus features.
    4. (Optional but recommended) Preset for Edition: Sandbox, to save on hosting costs.
    5. Region: the region of your GAE application.
    6. (Optional but recommended): Click Show Configuration Options and configure the smallest machine to save on hosting costs.
  3. Click Save to create the instance.

To connect the SQL instance with the staging or production environment, we will need to set-up private IP:

  1. Select your instance, and click Connections on the left-side bar.
  2. Click the Networking tab, and enable Private IP.
  3. Select the (default) VPC Network, and click Set Up Connection:
    1. Enable the Serverless VPC Access API
    2. Under Allocate an IP range, click Use an automatically allocated IP range.
    3. Click Continue and Create Connection
  4. Continue to connect the staging or production environment to the VPC in the setting up Google VPC section

(Optional) To connect your favorite database tool to query the staging environment's SQL database:

Do note that certain educational institute networks may block connections with GCP. If blocked, do try again with an alternate Wifi connection.

  1. Select your instance, and click Connections on the left-side bar.
  2. Click the Networking tab, and make sure Public IP is enabled.
  3. Add your own public IP under Authorized networks to whitelist your IP.
  4. Click Save on the bottom of the page.
  5. Create a connection on your database tool:
    1. Host being the SQL instance's Public IP.
    2. Database should be default postgres
    3. Port should be default 5432
    4. Username should be default postgres
    5. Password should your password that you've set when creating the SQL instance.

(Optional, but highly recommended for production) Create a lower-privileged user to be used for the application. The default 'postgres' user has all privileges turned on. We should create a 'production_user' with only CRUD abilities for production.

  1. On your database tool, connect to your SQL instance.
  2. Replace the password with your preferred password. In the following code, we will assume that the user created is called 'production_user'. Run the following script:
-- Creates the user
CREATE USER production_user WITH
	PASSWORD 'password'
	NOCREATEDB;

-- Grants CRUD for all existing tables 
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO production_user;

-- Grants CRUD for all future tables
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO production_user;
  1. To verify that the user has successfully been granted privileges, we can run the following too.
SELECT * from information_schema.role_table_grants where grantee='production_user';

Setting up Google Cloud Storage

Some features that require blob/binary data storage (as opposed to structured data storage), such as profile pictures, use Google Cloud Storage.

By default, when you create a Google App Engine instance, you will get a bucket named {your-app-id}.appspot.com. If you do not wish to use this default bucket, you are free to create other buckets.

Setting up Google VPC Connector

Serverless VPC access allows our different services to communicate via internal private IP address, instead of public IP addressing. By default, you should already have a VPC network created called (default).

Serverless VPC access requires connectors to be created and configured:

  1. Enable Serverless VPC Access API for your project.
  2. Go to https://console.cloud.google.com/networking/connectors/list and select Create Connector.
  3. Create a connector with the following configuration:
    • Name: any name of your choice
    • Region: the region of your GAE application
    • Network: default (this should be the default created VPC created automatically)
    • Subnet: Custom IP range
    • IP range: 10.8.0.0
    • All other settings can be modified as necessary.
  4. In app.yaml, add the following lines: 
    vpc_access_connector:
  name: projects/{projectId}/locations/{projectRegion}/connectors/{name}
  5. If needed, re-deploy the application after the above change.

Setting up Solr

We are using Apache Solr to support full-text searches, such as looking up for students/instructors by name or by course.

To enable usage of Solr, we deploy a single Solr instance within Google Compute Engine and build a VPC connector to allow connection from the Google App Engine application.

The steps to create the Solr instance are as follows:

  1. Go to Google Compute Engine console and click Create Instance.
  2. Create a VM with the following configuration:
    • Name: any name of your choice
    • Region, zone: the region of your GAE application
    • Machine configuration: as necessary. To get the cheapest possible machine, select General Purpose > N1 > g1-small.
      • There are cheaper machine types such as f1-micro, however they do not have the amount of memory required for Solr to run. g1-small is the cheapest possible machine type that is able to comfortably run Solr.
    • Boot disk: as necessary. To get the cheapest possible disk, use 10 GB Standard persistent disk.
      • Note that the instruction will assume that the OS used is Debian.
    • Firewall: Disallow both HTTP and HTTPS traffic
    • All other settings can remain as per default.
  3. Click Create and wait until the VM is booted. Note down the internal IP address.
  4. SSH into the VM.
  5. Run all the commands inside the setup file followed by the configuration file, in order.
    • While doing the above, note down your username inside the VM and the directory in which Solr is installed.
    • The next step will assume username of john and Solr installation directory of /home/john/solr-8.11.1.
  6. Add the following custom metadata to the VM:
    • Key: startup-script
    • Value: 
      #! /bin/bash
sudo -u john /home/john/solr-8.11.1/bin/solr start
  7. Stop and restart the VM.
  8. In build.properties, set the value of app.search.service.host to the internal IP address plus the port number (should be 8983 unless you specifically use other port) plus /solr, e.g. http://10.128.0.1:8983/solr.

After the above operation, you will have a running VM with a Solr instance running in it, and have configured your application to connect to it via internal IP address. This is not sufficient as the VM instance is not accessible by public web. However, that is not the intended outcome either; you only want the VM to be accessible by your deployed application and nothing else.

Note: If you are deploying to GAE flexible environment, this step is not required and can be skipped.

To fix that, you need to build a VPC connector. The steps to create the VPC connector can be found at the VPC section

Running client scripts

In order to run client scripts against a production environment, you need to authorize your account for Application Default Credentials.

gcloud auth application-default login

Follow the steps until you see Credentials saved to file: [...]. printed on the console.

Setting up Gmail API Credentials

Gmail API can be used to access Gmail accounts. You may need Gmail API credentials for testing against production server, particularly if you are testing that the emails are sent by the system.

  1. Enable Gmail API for your project.
  2. Create an OAuth 2.0 client to be used by the tests. The step is very similar to creating OAuth 2.0 client for user authentication in the production system, with the following differences:
    • The client application type will be Desktop app.
  3. After creating the client, download the JSON file corresponding to the client setup. You will need this file for later.

Datastore backup and recovery

Google Cloud Datastore provides managed export/import as a means of backup and recovery.

Backup

The backup has been configured to run monthly in the system by setting app.enable.datastore.backup=true in src/main/resources/build.properties. However, a few prerequisites as described here need to be done first.

If you wish to do backup outside the scheduled period, you can simply trigger the backup cron job manually from the GCP console.

Recovery

In order to import a snapshot into the Datastore:

  1. Ensure that all the prerequisites described here are done.

  2. Authenticate and set your project ID (e.g. if your project ID is teammates-john):

    gcloud auth login
gcloud config set project teammates-john

  3. Locate the backup snapshot which you want to use.

    1. Go to your Cloud Storage backup folder: https://console.cloud.google.com/storage/browser/teammates-john-backup/datastore-backups/?project=teammates-john.
      • The above URL is constructed based on the preset values in build.template.properties. In your usage, replace the appropriate values accordingly.
    2. Note down the backup folder name (this is in the form of timestamp) which you want to use. The snapshot URL will be in the form of gs://teammates-john-backup/datastore-backups/{folder}/{folder}.overall_export_metadata.

  4. Run the following command to import the snapshot:

    gcloud datastore import {snapshotUrl}

    e.g.:

    gcloud datastore import gs://teammates-john-backup/datastore-backups/2018-11-28T18:43:38.595Z/2018-11-28T18:43:38.595Z.overall_export_metadata