From 131fa8a17048dc1103069d27f0830dd90dd85164 Mon Sep 17 00:00:00 2001 From: Ryan Lovett Date: Fri, 25 Oct 2024 13:04:56 -0700 Subject: [PATCH 1/4] Add CI/CD doc to navigation. --- docs/_quarto.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/_quarto.yml b/docs/_quarto.yml index f8759eee9..eb09e7c28 100644 --- a/docs/_quarto.yml +++ b/docs/_quarto.yml @@ -35,6 +35,7 @@ website: - admins/storage.qmd - admins/cluster-config.qmd - admins/credentials.qmd + - admins/cicd-github-actions.qmd - section: "Common Administrator Tasks" contents: - tasks/documentation.qmd From 29739088010c3904f6810465ebb85d268a709ccf Mon Sep 17 00:00:00 2001 From: Ryan Lovett Date: Fri, 25 Oct 2024 13:05:19 -0700 Subject: [PATCH 2/4] Fix headings. Use callouts, term lists. Fix a11y. In-page headings should start with level 2 since the title is considered 1. Replace some syntactic sugar with semantic syntax. Also replace use of "here" links for accessibility. --- docs/admins/cicd-github-actions.qmd | 114 ++++++++++++---------------- 1 file changed, 48 insertions(+), 66 deletions(-) diff --git a/docs/admins/cicd-github-actions.qmd b/docs/admins/cicd-github-actions.qmd index 350b17a15..dfced3c0c 100644 --- a/docs/admins/cicd-github-actions.qmd +++ b/docs/admins/cicd-github-actions.qmd @@ -2,102 +2,87 @@ title: Datahub CI/CD --- -# Datahub CI/CD overview +## Datahub CI/CD overview Datahub's continuous integration and deployment system uses both [Github Actions](https://github.com/features/actions) and [workflows](https://docs.github.com/en/actions/writing-workflows). These workflows are stored in the Datahub repo in the -[.github/workflows/ directory](https://github.com/berkeley-dsep-infra/datahub/tree/staging/.github/workflows). +[.github/workflows/](https://github.com/berkeley-dsep-infra/datahub/tree/staging/.github/workflows) directory. The basic order of operations is as follows: 1. PR is created in the datahub repo. -2. The labeler workflow applies labels based on the [file type and/or location](https://github.com/berkeley-dsep-infra/datahub/blob/staging/.github/labeler.yml). -3. On PR merge to staging, if the labels match any hub, support or node placeholder deployments those specific systems are deployed. -4. On PR merge to prod, only hubs are deployed (again based on labels). +1. The labeler workflow applies labels based on the [file type and/or location](https://github.com/berkeley-dsep-infra/datahub/blob/staging/.github/labeler.yml). +1. On PR merge to staging, if the labels match any hub, support or node placeholder deployments those specific systems are deployed. +1. On PR merge to prod, only hubs are deployed (again based on labels). The hubs are deployed via [hubploy](https://github.com/berkeley-dsep-infra/hubploy), which is our custom wrapper for `gcloud`, `sops` and `helm`. -# Github Actions architecture +## Github Actions architecture -## Secrets and variables +### Secrets and Variables All of these workflows depend on a few Actions secrets and variables, with some at the organization level, and others at the repository level. -### Organization secrets and variables +#### Organization secrets and variables -All of the organizational secrets and variables are located [here](https://github.com/organizations/berkeley-dsep-infra/settings/secrets/actions). +[GitHub Actions settings](https://github.com/organizations/berkeley-dsep-infra/settings/secrets/actions) contain all of the organizational secrets and variables. -#### Organization Secrets +##### Organization Secrets -**DATAHUB_CREATE_PR** +DATAHUB_CREATE_PR +: This secret is a fine-grained personal [access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens), and has the following permissions defined: -This secret is a fine-grained personal [access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens), and has the following permissions defined: + * Select repositories (only berkeley-dsep-infra/datahub) + * Repository permissions: Contents (read/write), Metadata (read only), Pull requests (read/write) -* Select repositories (only berkeley-dsep-infra/datahub) -* Repository permissions: Contents (read/write), Metadata (read only), Pull requests (read/write) - -When adding a new image repository in the berkeley-dsep-infra org, you must + When adding a new image repository in the berkeley-dsep-infra org, you must edit this secret and manually add this repository to the access list. -*IMPORTANT:* This PAT has an lifetime of 366 days, and should be rotated at the -beginning of every maintenance window! - -**GAR_SECRET_KEY** and **GAR_SECRET_KEY_EDX** +::: {.callout-important} +This PAT has an lifetime of 366 days, and should be rotated at the beginning of +every maintenance window. +::: -These secrets are for the GCP IAM roles for each GCP project given -`roles/storage.admin` permissions. This allows us to push the built images to -the Artifact Registry. +GAR_SECRET_KEY and GAR_SECRET_KEY_EDX +: These secrets are for the GCP IAM roles for each GCP project given `roles/storage.admin` permissions. This allows us to push the built images to the Artifact Registry. -When adding a new image repository in the berkeley-dsep-infra org, you must + When adding a new image repository in the berkeley-dsep-infra org, you must edit this secret and manually add this repository to the access list. -#### Organization Variables - -**IMAGE_BUILDER_BOT_EMAIL** and **IMAGE_BUILDER_BOT_NAME** - -These are used to set the git identity in the image build workflow step that -pushes a commit and creates a PR in the datahub repo. - -#### Repository secrets and variables - -##### Datahub repository secrets +##### Organization Variables -**GCP_PROJECT_ID** +IMAGE_BUILDER_BOT_EMAIL and IMAGE_BUILDER_BOT_NAME +: These are used to set the git identity in the image build workflow step that pushes a commit and creates a PR in the datahub repo. -This is the name of our GCP project. +###### Datahub repository secrets -**GKE_KEY** +GCP_PROJECT_ID +: This is the name of our GCP project. -This key is used in the workflows that deploy the `support` and -`node-placeholder` namespaces. It's attached to the `hubploy` service account, -and has the assigned roles of `roles/container.clusterViewer` and -`roles/container.developer`. +GKE_KEY +: This key is used in the workflows that deploy the `support` and `node-placeholder` namespaces. It's attached to the `hubploy` service account, and has the assigned roles of `roles/container.clusterViewer` and `roles/container.developer`. -**SOPS_KEY** +SOPS_KEY +: This key is used to decrypt our secrets using `sops`, and is attached to the `sopsaccount` service account and provides KMS access. -This key is used to decrypt our secrets using `sops`, and is attached to the -`sopsaccount` service account and provides KMS access. - -#### Image repository variables +##### User Image Repository Variables Each image repository contains two variables, which are used to identify the name of the hub, and the path within the Artifact Registry that it's published to. -**HUB** - -The name of the hub, natch! datahub, data100, etc. - -**IMAGE** +HUB +: The name of the hub, natch! `datahub`, `data100`, etc. -The path within the artifact registry: `ucb-datahub-2018/user-images/-user-image` +IMAGE +: The path within the artifact registry: `ucb-datahub-2018/user-images/-user-image` -## Single user server image modification workflow +### Single user server image modification workflow Each hub's user image is located in the berkeley-dsep-infra's organization. When a pull request is submitted, there are two workflows that run: @@ -114,9 +99,9 @@ This builds the image again, and when successful pushes it to our Google Artifact Registry and creates a pull request in the datahub repository with the updated image tag for that hub's deployment. -## Updating the datahub repository +### Updating the datahub repository -### Single user server image tag updates +#### Single user server image tag updates When a pull request is opened to update one or more image tags for deployments, the [labeler](https://github.com/berkeley-dsep-infra/datahub/blob/staging/.github/labeler.yml) @@ -132,7 +117,7 @@ a list of what's to be deployed. That list is iterated over, and [hubploy](https://github.com/berkeley-dsep-infra/hubploy) is used to deploy only the flagged hubs. -### Support and node-placeholder charts +#### Support and node-placeholder charts Each of these deployments has their own workflow, which only runs on pushes to `staging`: @@ -143,19 +128,16 @@ Each of these deployments has their own workflow, which only runs on pushes to If the correct labels are found, it will use the **GKE_KEY** secret to run `helm upgrade` for the necessary deployments. -### Misc workflows +#### Miscellaneous workflows There are also a couple of other workflows in the datahub repository: -* [ prevent-prod-merges.yml](https://github.com/berkeley-dsep-infra/datahub/blob/staging/.github/workflows/prevent-prod-merges.yml) - -This workflow will only allow us to merge to `prod` from `staging`. - -* [quarto-docs.yml](https://github.com/berkeley-dsep-infra/datahub/blob/staging/.github/workflows/quarto-docs.yml) +[ prevent-prod-merges.yml](https://github.com/berkeley-dsep-infra/datahub/blob/staging/.github/workflows/prevent-prod-merges.yml) +: This workflow will only allow us to merge to `prod` from `staging`. -This builds, renders and pushes our docs to Github Pages. +[quarto-docs.yml](https://github.com/berkeley-dsep-infra/datahub/blob/staging/.github/workflows/quarto-docs.yml) +: This builds, renders and pushes our docs to Github Pages. -# Original design document +## Original Design Document -Here are the [slides](https://docs.google.com/presentation/d/1o_P4H8XfbdgI5NMPnjojHZOcSNHRoP5twl0E8Ern1z4/edit?usp=sharing) -that describe the process in some more detail. +[Slides](https://docs.google.com/presentation/d/1o_P4H8XfbdgI5NMPnjojHZOcSNHRoP5twl0E8Ern1z4/edit?usp=sharing) describe the process in some more detail. From 9347e73fed98ecd05f4bfc1d5fc968c3e33fc21a Mon Sep 17 00:00:00 2001 From: Ryan Lovett Date: Fri, 25 Oct 2024 13:11:59 -0700 Subject: [PATCH 3/4] Fix capitalization. --- docs/admins/cicd-github-actions.qmd | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/admins/cicd-github-actions.qmd b/docs/admins/cicd-github-actions.qmd index dfced3c0c..3ff3ebece 100644 --- a/docs/admins/cicd-github-actions.qmd +++ b/docs/admins/cicd-github-actions.qmd @@ -1,14 +1,14 @@ --- -title: Datahub CI/CD +title: DataHub CI/CD --- -## Datahub CI/CD overview +## Overview -Datahub's continuous integration and deployment system uses both +DataHub's continuous integration and deployment system uses both [Github Actions](https://github.com/features/actions) and [workflows](https://docs.github.com/en/actions/writing-workflows). -These workflows are stored in the Datahub repo in the +These workflows are stored in the DataHub repo in the [.github/workflows/](https://github.com/berkeley-dsep-infra/datahub/tree/staging/.github/workflows) directory. The basic order of operations is as follows: @@ -59,7 +59,7 @@ edit this secret and manually add this repository to the access list. IMAGE_BUILDER_BOT_EMAIL and IMAGE_BUILDER_BOT_NAME : These are used to set the git identity in the image build workflow step that pushes a commit and creates a PR in the datahub repo. -###### Datahub repository secrets +###### DataHub repository secrets GCP_PROJECT_ID : This is the name of our GCP project. From 32265d6b0ed9f4c6c2930c39ccb78db5f6f91502 Mon Sep 17 00:00:00 2001 From: Ryan Lovett Date: Fri, 25 Oct 2024 14:06:50 -0700 Subject: [PATCH 4/4] Reference documentation's CI/CD process. --- docs/admins/cicd-github-actions.qmd | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/admins/cicd-github-actions.qmd b/docs/admins/cicd-github-actions.qmd index 3ff3ebece..7f29b1868 100644 --- a/docs/admins/cicd-github-actions.qmd +++ b/docs/admins/cicd-github-actions.qmd @@ -1,5 +1,5 @@ --- -title: DataHub CI/CD +title: Continuous Integration and Deployment --- ## Overview @@ -21,7 +21,7 @@ The basic order of operations is as follows: The hubs are deployed via [hubploy](https://github.com/berkeley-dsep-infra/hubploy), which is our custom wrapper for `gcloud`, `sops` and `helm`. -## Github Actions architecture +## Github Actions Architecture ### Secrets and Variables @@ -138,6 +138,10 @@ There are also a couple of other workflows in the datahub repository: [quarto-docs.yml](https://github.com/berkeley-dsep-infra/datahub/blob/staging/.github/workflows/quarto-docs.yml) : This builds, renders and pushes our docs to Github Pages. +### Documentation's Workflow + +This documentation is also [deployed by GitHub Actions](../tasks/documentation.html#action). + ## Original Design Document [Slides](https://docs.google.com/presentation/d/1o_P4H8XfbdgI5NMPnjojHZOcSNHRoP5twl0E8Ern1z4/edit?usp=sharing) describe the process in some more detail.