Security and Maintenance

Please refer to Terra UI Maintenance regarding the maintenance rotation and the related responsibilities.

Service Account Key Rotation

Every three months, the service account key used by CircleCI to deploy Terra UI to prod needs to be rotated for security purposes, by the maintaining team. The steps below describe the process:

Create new key in Google Cloud Console

• Having logged in with your @firecloud.org account, go to Google Cloud Console's CircleCI Service Account Keys page of the IAM & Admin section for project BVDP Saturn Prod. If you receive a permissions error, ask on #terra-ui Slack channel to be granted Editor role on Google project BVDP Saturn Prod.
• Click on the ADD KEY button.
• Select Create new key.
• Make sure JSON is selected as key type, and click on CREATE.
• Make sure the .json file was downloaded onto your computer.

Update Circle CI's GCP_SA_KEY environment Variable

• Create a new browser tab, and go to CircleCI's Environment Variables page. If you can't view this page, ask on #terra-ui Slack channel to be added to GitHub team SaturnDeployers.
• Delete the environment variable named GCP_SA_KEY by clicking on X to the right of it.
• Click on the button Add Environment Variable and fill out Name and Value on the dialog that pops up.
  • Name should be GCP_SA_KEY
  • Value is supposed to be the text content of the .json file you had downloaded.
• Manually redeploy to prod to verify the new SA key works by following the instructionshere.
• Observe on the saturn-ui-prod-deploy CircleCI page that the job has succeeded.
• Go to https://app.terra.bio/#workspaces; click on the hamburger menu on the upper left; click on the link to the right of Built on: on the buttom; verify that the latest GitHub commit is on the new tab that pops up.

Disable the old deployment key

• Now you can disable the old key(s). It's not possible to disable a SA key on the GCP console. You will need to use gcloud on your local system (e.g. Terminal on MacOS). After making sure you're authenticated with your @firecloud.org account on your local system, run gcloud iam service-accounts keys disable <service_account_key_name> [email protected] --project bvdp-saturn-prod. Substitute into that command, the key obtained from Key column on the GCP console page for SA keys. More details on disabling service account keys can be found here. If you get ERROR: (gcloud.iam.service-accounts.keys) Invalid choice: 'disable'., try upgrading your SDK via gcloud components update. Verify on the GCP console page that the key's Status is now Disabled (as opposed to Active).
• To be sure, rerun the prod deploy job and verify that everything still works.
• Finally, delete the old key using the trash can icon in the UI. This is an important step to make sure that the key being retired can’t accidentally be re-enabled.

Prepare for the next rotation

• Set a Slack reminder on #terra-ui channel for the next maintaining team by typing something like /remind :alert_red_light: Reminder to rotate the service account key for prod deploys by following these (https://github.com/DataBiosphere/terra-ui/wiki/Security-and-Maintenance#service-account-key-rotation) instructions in 85 days.

Update Vault with the new key

• Add the new .json key file (that you downloaded at the beginning) to Vault (e.g. vault write secret/dsp/service-accts/bvdp-saturn-prod-sa-2021-12-14.json @/Users/<your username>/Downloads/<new sa file name>.json).

Dependabot Security Updates

In order to see the Dependabot alerts, your GitHub handle must be added to the GitHub team TerraUISecurityAlertAccess. You may also want to check your personal notifications for any filtering.

A lot of the Dependabot alerts can be safely dismissed. These are all valid reasons for dismissing the alerts:

• The alert is for a library version that is only used by integration tests, development tools or a library that is only used in the packaging of our code (for example, packaging CSS), and thus not exposed to users. To figure this out, use yarn why -R LIBRARY_NAME. For instance, if you see a root node name in the yarn why output that reads terra-integration-tests@workspace:integration-tests, we know that the library is used by integration-tests. yarn why -R --json LIBRARY_NAME | jq '.value' can also be helpful in seeing the root packages at a glance without the entire tree.
• The alert is incorrect.
  • The alert is based on integration-tests/yarn.lock (as opposed to yarn.lock at the root of the repo) and the package is not listed as a direct dependency on integration-tests/package.json. Dependabot has a bug where it may continue alerting based on files that no longer exists in a repo. This has happened for this repo with integration-tests/yarn.lock that had been removed.
  • By examining the yarn.lock file and using yarn why LIBRARY_NAME it is clear that we actually don't use the library version that the alert is warning about.

If the library version does need to be upgraded because it is exposed to users in Terra's UI, these are the recommended steps:

• If the library is one of our direct dependencies, update the version in package.json and generate an updated yarn.lock file as detailed below.
• If the library is a transitive dependency, delete entries in yarn.lock file that reference the library with the security update .
• Run yarn install to generate the new content in yarn.lock. Verify the library versions satisfy the security update.
• Run yarn dedupe to remove duplicated content from the yarn.lock file.
• Create a PR with the modified yarn.lock file.
• Extra points for updating our direct dependencies (package.json files) that introduce the transitive dependency to their most recent versions 🥇 😃

Compliance timeframe for addressing security alerts can be found in Terra Vulnerability Management Plan.

Adjusting Prod Deploy Time for Daylight Savings Time Changes

Prod deploy time has been chosen as 11am Eastern Time. See this PR for start of DST and this PR for end of DST.

Cleaning up old non-prod App Engine versions

Terra UI is a Google App Engine application. Every time CircleCI deploys Terra UI, it creates a new version. App Engine allows a maximum of 210 versions of any app. Therefore, we must periodically clean up old versions.

In prod, there is an automated script to handle the cleanup. In non-prod (dev, alpha, staging, and perf) there is a manual script which should be run instead:

1. Ensure you have jq installed beforehand
2. Ensure that you are logged in to the correct gcloud account that has app engine deletion permission
3. git clone the Terra UI repository or download the script from https://github.com/DataBiosphere/terra-ui/tree/dev/scripts
4. Run the delete-old-app-engine-versions.sh script and specify the environment to delete, which will prompt you to confirm deletion of old app engine resources in that environment

You're done! Thank you for your service!

Adding PR Sites