Security and Maintenance

Note: This page is linked to from the Application Vulnerability Management - Process Improvement Proposal.

Team Rotation

We rotate the responsibility for handling maintenance for the Terra UI codebase amongst teams.

Responsibilities

• Handling security updates. See Dependabot Security Updates below.
• General maintenance requests that are not tied to a specific project
• Triaging integration test failures
  • The maintaining team is responsible for looking at test failures and assigning them appropriately.
  • If the failure is flaky due to shared resources (including the testing infrastructure) the maintaining team would take ownership of the fix.
• Rotating service account keys quarterly
• Adjusting prod deploy times for daylight savings time changes bi-annually

Schedule

• Q1: Batch - Workflows
• Q2: Cloud Integrations
• Q3: Workspaces
• Q4: Interactive Analysis

Service Account Key Rotation

Every three months, the service account key used by CircleCI to deploy Terra UI to prod needs to be rotated for security purposes, by the maintaining team. The steps below describe the process:

Create new key in Google Cloud Console

• Having logged in with your @firecloud.org account, go to Google Cloud Console's CircleCI Service Account Keys page of the IAM & Admin section for project BVDP Saturn Prod. If you receive a permissions error, ask on #terra-ui Slack channel to be granted Editor role on Google project BVDP Saturn Prod.
• Click on the ADD KEY button.
• Select Create new key.
• Make sure JSON is selected as key type, and click on CREATE.
• Make sure the .json file was downloaded onto your computer.

Update Circle CI's GCP_SA_KEY environment Variable

• Create a new browser tab, and go to CircleCI's Environment Variables page. If you can't view this page, ask on #terra-ui Slack channel to be added to GitHub team SaturnDeployers.
• Delete the environment variable named GCP_SA_KEY by clicking on X to the right of it.
• Click on the button Add Environment Variable and fill out Name and Value on the dialog that pops up.
  • Name should be GCP_SA_KEY
  • Value is supposed to be the text content of the .json file you had downloaded.
• Manually redeploy to prod to verify the new SA key works by following the instructionshere.
• Observe on the saturn-ui-prod-deploy CircleCI page that the job has succeeded.
• Go to https://app.terra.bio/#workspaces; click on the hamburger menu on the upper left; click on the link to the right of Built on: on the buttom; verify that the latest GitHub commit is on the new tab that pops up.

Disable the old deployment key

• Now you can disable the old key(s). It's not possible to disable a SA key on the GCP console. You will need to use gcloud on your local system (e.g. Terminal on MacOS). After making sure you're authenticated with your @firecloud.org account on your local system, run gcloud iam service-accounts keys disable <service_account_key_name> [email protected] --project bvdp-saturn-prod. Substitute into that command, the key obtained from Key column on the GCP console page for SA keys. More details on disabling service account keys can be found here. If you get ERROR: (gcloud.iam.service-accounts.keys) Invalid choice: 'disable'., try upgrading your SDK via gcloud components update. Verify on the GCP console page that the key's Status is now Disabled (as opposed to Active).
• To be sure, rerun the prod deploy job and verify that everything still works.
• Assuming everything is still working, you can now safely delete the old key using the trash can icon in the UI.

Prepare for the next rotation

• Set a Slack reminder on #terra-ui channel for the next maintaining team by typing something like /remind :alert_red_light: Reminder to rotate the service account key for prod deploys by following these (https://github.com/DataBiosphere/terra-ui/wiki/Security-and-Maintenance#service-account-key-rotation) instructions in 85 days.

Update vault with the new key

• Add the new .json key file (that you downloaded at the beginning) to Vault (e.g. vault write secret/dsp/service-accts/bvdp-saturn-prod-sa-2021-12-14.json @/Users/<your username>/Downloads/<new sa file name>.json).

Dependabot Security Updates

In order to see Dependabot alerts, your account must be added by a maintainer to the list of people who get notified.

A lot of the Dependabot alerts can be safely dismissed. These are all valid reasons for dismissing the alerts:

• The alert is for a library version that is only used by integration tests, development tools or a library that is only used in the packaging of our code (for example, packaging CSS), and thus not exposed to users. To figure this out, use yarn why -R LIBRARY_NAME. For instance, if you see a root node name in the yarn why output that reads terra-integration-tests@workspace:integration-tests, we know that the library is used by integration-tests. yarn why -R --json LIBRARY_NAME | jq '.value' can also be helpful in seeing the root packages at a glance without the entire tree.
• The alert is incorrect. By examining the yarn.lock file and using yarn why LIBRARY_NAME it is clear that we actually don't use the library version that the alert is warning about.

If the library version does need to be upgraded because it is exposed to users in Terra's UI, these are the recommended steps:

• If the library is one of our direct dependencies, update the version in package.json and generate an updated yarn.lock file as detailed below.
• If the library is a transitive dependency, delete entries in yarn.lock file that reference the library with the security update .
• Run yarn install to generate the new content in yarn.lock. Verify the library versions satisfy the security update.
• Run yarn dedupe to remove duplicated content from the yarn.lock file.
• Create a PR with the modified yarn.lock file.
• Extra points for updating our direct dependencies (package.json files) that introduce the transitive dependency to their most recent versions 🥇 😃

Compliance timeframe for addressing security alerts can be found in Terra Vulnerability Management Plan.

Adjusting Prod Deploy Time for Daylight Savings Time Changes

Prod deploy time has been chosen as 11am Eastern Time. See this PR for start of DST and this PR for end of DST.

Cleaning up old non-prod App Engine versions

Terra UI is a Goole App Engine application. Every time CircleCI deploys Terra UI, it creates a new version. App Engine allows a maximum of 210 versions of any app. Therefore, we must periodically clean up old versions.

Note: There is an automated script to handle this for prod, so these instructions only apply to non-prod: currently dev, alpha, staging, and perf.

For each [bvdp-saturn-dev, bvdp-saturn-alpha, bvdp-saturn-staging, bvdp-saturn-perf],

1. Go to the App Engine versions page in cloud console, e.g.: https://console.cloud.google.com/appengine/versions?project=bvdp-saturn-dev&serviceId=default
2. Recommended: At the bottom of the list, select the maximum rows per page that you can.
3. Select all versions. You can switch between pages if necessary. The UI will remember selections on previous pages as you go.
4. Choose a date within the past week (I like to leave a few days buffer just in case we need to rollback) and unselect any versions deployed after that date. This includes the current version that shows 100% traffic allocation.
5. Alternately, with an empty selection, clicking one and shift clicking another will select all versions in between.
6. Important, dev-only step: Unselect all versions that begin with "pr-". These are the versions that are deployed when a PR is opened. As of this writing, there are 11 of these.
7. Delete the selected versions.

You're done! Thank you for your service!

Adding PR Sites