diff --git a/content/post/1brc.md b/content/post/1brc.md index 0710e45..50adbf2 100644 --- a/content/post/1brc.md +++ b/content/post/1brc.md @@ -3,7 +3,7 @@ draft: false title: 1️⃣🐝🏎️🦆 (1BRC in SQL with DuckDB) date: "2024-01-03T12:12:32Z" image: "/images/2024/01/h_IMG_6977.webp" -thumbnail: "/images/2024/01/1brc.png" +thumbnail: "/images/2024/01/1brc.webp" credit: "https://twitter.com/rmoff/" categories: - DuckDB diff --git a/content/post/antora-2.adoc b/content/post/antora-2.adoc new file mode 100644 index 0000000..e6e74b3 --- /dev/null +++ b/content/post/antora-2.adoc @@ -0,0 +1,190 @@ +--- +draft: false +title: 'Antora Deployment to Cloudflare Across Private Repositories with GitHub Actions' +date: "2024-01-17T12:09:23Z" +image: "/images/2024/01/h_IMG_7272.webp" +thumbnail: "/images/2024/01/t_IMG_7306.webp" +credit: "https://twitter.com/rmoff/" +categories: +- Antora +- GitHub +- Cloudflare +--- + +:source-highlighter: rouge +:icons: font +:rouge-css: style +:rouge-style: github + +At https://decodable.co[Decodable] we're migrating our docs platform onto https://antora.org/[Antora]. I wrote link:/2023/12/19/deploying-antora-with-github-actions-and-a-private-github-repo/[previously] about my escapades in getting cross-repository authentication working using https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#types-of-personal-access-tokens[Private Access Tokens] (PAT). These are fine for just a single user, but they're tied to that user, which isn't a good practice for deployment in this case. + +In this article I'll show how to use GitHub Apps and Installation Access Tokens (IAT) instead, and go into some detail on how we've deployed Antora. Our GitHub repositories are private which makes it extra-gnarly. + + + +== Overview + +The docs platform is built on https://docs.antora.org/antora/latest/[Antora], which generates a static site. This is hosted on https://pages.cloudflare.com/[Cloudflare Pages]. + +* The *docs content* is on our private `code_repo` GitHub repo. +* The Antora *platform configuration and build scripts* is held on the private `docs-platform` GitHub repo. +* The user interface of the documentation site is taken mostly from the vanilla https://gitlab.com/antora/antora-ui-default[antora-ui-default], with some small tweaks in this repo. + +GitHub Actions is used to trigger, build, and deploy the docs site. + +image::/images/2024/01/contents.webp[Diagram of what content is where] + +=== Why Two Repositories? + +One of the brilliant things that Antora supports is the ability to pull in the docs content from a git repository. This can be the same as the one that is hosting the Antora configuration—or as in our case, the repository that holds the code for which the documentation is written. + +This means that when a new feature is released the code and the docs can be kept in lock-step, both deploy and rollback if necessary. + +You can also pull in code from multiple repositories—Antora is super-flexible like this, and supports various styles of code organisation and control. + +== Build and Deploy + +The docs content lives in `code_repo` under `/docs`. The actual build and deploy happens on the `docs-platform` repository, and as a result the process is more complicated than it would be in a single repository. + +image::/images/2024/01/buildanddeploy.webp[Diagram of the build and deploy process] + +* For *docs changes*, when a change is pushed to `main` on `code_repo` that includes a file under `/docs/`, the https://gist.github.com/rmoff/6d06b0b258a65502828205733b6a8c8e#file-docs-trigger-deploy-yaml[`docs-trigger-deploy.yaml`] workflow runs. This in turn triggers the https://gist.github.com/rmoff/6d06b0b258a65502828205733b6a8c8e#file-docs-platform-cloudflare-yaml[`cloudflare.yaml`] workflow on the `docs-platform` repository. +* For a *platform change* the `cloudflare.yaml` workflow on the `docs-platform` repository will run directly. + +=== PR Preview builds + +Not everyone writes perfect docs PRs. I has ben knowwn to right mistaks with speling and grammer, and that's before you get onto the correctness of documentation which in software is particularly important to be precise and accurate. + +By being able to create a preview of PRs as they are raised it's easy to seek review from colleagues. However readable Asciidoc might be as a markup language, it's not going to be more readable than the parsed and rendered web page with its links and images in all their glory. The same goes for PRs against the Antora platform configuration itself—whether customising the skin, tweaking a config—all these things benefit from being able to see them in action *before* hitting the deploy button. + +Thus, when a PR is raised against `main` on the `docs-platform` or `code_repo` repos (and includes a file under `/docs/` in the case of the `code_repo` repo) a preview docs site is built and deployed to a unique URL. + +image::/images/2024/01/buildanddeploy_preview.webp[Diagram of the Preview build and deploy process] + +The build process does some additional things that the production one doesn't: + +1. The trigger action https://gist.github.com/rmoff/6d06b0b258a65502828205733b6a8c8e#file-docs-trigger-pr-preview-yml-L31-L35[passes details] about the source PR as https://gist.github.com/rmoff/6d06b0b258a65502828205733b6a8c8e#file-docs-platform-preview-cloudflare-yaml-L6-L27[inputs] to the `docs-platform` workflow + +2. The https://gist.github.com/rmoff/6d06b0b258a65502828205733b6a8c8e#file-docs-platform-preview-cloudflare-yaml[preview-cloudflare.yaml] workflow on the `docs-platform` modifies the Antora playbook to use the source branch of the PR on the `code-repo` repository (instead of `main`) ++ +[source,yaml] +---- +- name: If not a platform PR, set the branch of the source repo for antora content +if: github.event_name != 'pull_request' +id: override_antora_playbook_yml +run: | + sed -i '7s/main/${{ inputs.pr_branch }}/' antora-playbook.yml +---- + +3. It overlays a message on the preview site indicating the PR with which that it is associated. ++ +image::/images/2024/01/pr2.webp[Screenshot of a preview docs deployment with a PR message overlaid] + +4. Once deployed, it updates the originating PR with the URL of the preview site. ++ +image::/images/2024/01/pr1.webp[Screenshot of a PR showing the preview URL] + +== Security and Configuration in GitHub Workflows + +There are two areas of security that need to be correctly configured: + +* Cloudflare +* GitHub Intra-Repository interactions + +image::/images/2024/01/securityoverview.webp[Overview of interactions requiring additional authorisation steps] + +=== Cloudflare + +This one is fairly simple. Store these in the `docs-platform` repository settings: + +* Repository secret: `CLOUDFLARE_API_TOKEN` +* Repository variable: `CLOUDFLARE_ACCOUNT_ID` + +The GitHub Action then uses these when deploying the site to Cloudflare. + +=== GitHub Intra-Repository Interactions + +This is more complicated since the two repositories are private. The following interactions need authorisation beyond what happens by default within a GitHub Workflow: + +1. Actions running in `code_repo` repo + * Triggering a workflow in `docs-platform` from `code_repo` workflow +2. Actions running in `docs-platform` repo + * When Antora builds the docs site it needs to clone the `code_repo` repository + * Updating the PR issue on `code_repo` that triggered the preview build + +==== How It Works + +image::/images/2024/01/securitydetail.webp[Detail of how security works for interactions requiring additional authorisation steps] + +The auth for these is handled using a custom https://docs.github.com/en/apps[GitHub App] (`Antora Docs Build Bot`) https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/about-authentication-with-a-github-app#authentication-as-an-app-installation[installation] on each repository. + +NOTE: The auth https://rmoff.net/2023/12/19/deploying-antora-with-github-actions-and-a-private-github-repo/[can also be done] using Personal Access Tokens (PAT) but this would then be tied to a particular user's account and is therefore not suitable for an organisation. + +When each workflow runs its first step is to use the https://github.com/actions/create-github-app-token/tree/v1/?tab=readme-ov-file[create-github-app-token] action to https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/authenticating-as-a-github-app-installation#using-an-installation-access-token-to-authenticate-as-an-app-installation[generate a GitHub App installation access token (IAT)]. This is valid for the session only, and then provides the authorisation for the intra-repository actions. + +The IAT is used in two ways: + +1. From the https://github.com/actions/github-script?tab=readme-ov-file[github-script] action via https://github.com/actions/github-script?tab=readme-ov-file#using-a-separate-github-token[the optional `github-token` parameter]. This is used for two different interactions: + a. To trigger the `docs-platform` build and deploy workflows from the `code_repo` repository. + b. When the Preview workflow adds a comment to the PR that triggered it. If this PR came from the `docs-platform` repository (i.e. local to the action) then no additional auth is needed, but to comment on the `code_repo` repository it is. + +2. When Antora builds the site it clones the `code_repo` repository. Since this is run from a different repository the default authentication that would apply to an action running in the same repository doesn't exist. Antora https://docs.antora.org/antora/latest/playbook/private-repository-auth/#git-credentials-environment-variable[performs the authentication] using the pre-specified `GIT_CREDENTIALS` environment variable. This *must* follow the following syntax: ++ +[source] +---- +https://x-access-token:$IAT_TOKEN@github.com +---- + +==== Setting up the GitHub App + +This needs to be done by a user with `Owner` rights on the GitHub organisation. The App has to be created in the GitHub organisation, and from there is installed to the two repositories. The GitHub docs https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/registering-a-github-app[detail the process] - below is a short set of notes covering the essential settings: + +1. From https://github.com/settings/profile[your GitHub profile page] set the `settings context` to that of your organisation, and then click on `Developer settings` (at the very bottom of the page) and then `GitHub Apps` +2. Click on `New GitHub App`. + a. Give the new app a name (e.g. `Antora Docs Build Bot`) + b. Set the `Homepage URL` to that of `docs-platform` repo + c. Disable `Webhook` +3. Under `Repository permissions` set the following ++ +|=========================== +| Actions | Read/Write +| Contents | Read +| Issues | Read/Write +| Metadata | Read +| Pull Requests | Read/Write +|=========================== +4. Click `Create GitHub App` +5. Make a note of the App ID. You'll store this later on as a repository secret. +6. Scroll down to `Private keys` and click on `Generate a private key`. Save the resulting `.pem` file locally. +7. Click `Install App` + a. Install it to the account under which the the `docs-platform` and `code_repo` repos are (i.e. `decodeableco`). + b. When prompted which repositories it should be installed to, select `Only select repositories` and choose `docs-platform` and `code_repo` + +==== Configuring Repository Secrets and Variables + +As a repo admin, on the `code_repo` repository add the following repository secrets: + +[options="header"] +|=============================================================================================================================== +| Key | Value +| `DOCS_APP_ID` | GitHub App ID +| `DOCS_APP_PRIVATE_KEY` | The full text of the .pem, including the `BEGIN RSA PRIVATE KEY` and `END RSA PRIVATE KEY` header and footer +|=============================================================================================================================== + +As a repo admin, on the `docs-platform` add the following repository secrets + +[options="header"] +|=============================================================================================================================== +| Key | Value +| `DOCS_APP_ID` | GitHub App ID +| `DOCS_APP_PRIVATE_KEY` | The full text of the .pem, including the `BEGIN RSA PRIVATE KEY` and `END RSA PRIVATE KEY` header and footer +| `CLOUDFLARE_API_TOKEN` | API token from Cloudflare +|=============================================================================================================================== + +and the following repository variable + +[options="header"] +|=============================================================================================================================== +| Key | Value +| `CLOUDFLARE_ACCOUNT_ID` | Cloudflare Account ID +|=============================================================================================================================== diff --git a/static/images/2024/01/1brc.png b/static/images/2024/01/1brc.png deleted file mode 100644 index 9b0840c..0000000 Binary files a/static/images/2024/01/1brc.png and /dev/null differ diff --git a/static/images/2024/01/1brc.webp b/static/images/2024/01/1brc.webp new file mode 100644 index 0000000..1a52e0f Binary files /dev/null and b/static/images/2024/01/1brc.webp differ diff --git a/static/images/2024/01/buildanddeploy.webp b/static/images/2024/01/buildanddeploy.webp new file mode 100644 index 0000000..e6f67d3 Binary files /dev/null and b/static/images/2024/01/buildanddeploy.webp differ diff --git a/static/images/2024/01/buildanddeploy_preview.webp b/static/images/2024/01/buildanddeploy_preview.webp new file mode 100644 index 0000000..43d7327 Binary files /dev/null and b/static/images/2024/01/buildanddeploy_preview.webp differ diff --git a/static/images/2024/01/contents.webp b/static/images/2024/01/contents.webp new file mode 100644 index 0000000..c69cebb Binary files /dev/null and b/static/images/2024/01/contents.webp differ diff --git a/static/images/2024/01/h_IMG_7272.webp b/static/images/2024/01/h_IMG_7272.webp new file mode 100644 index 0000000..0df0f94 Binary files /dev/null and b/static/images/2024/01/h_IMG_7272.webp differ diff --git a/static/images/2024/01/pr1.webp b/static/images/2024/01/pr1.webp new file mode 100644 index 0000000..05ab9a4 Binary files /dev/null and b/static/images/2024/01/pr1.webp differ diff --git a/static/images/2024/01/pr2.webp b/static/images/2024/01/pr2.webp new file mode 100644 index 0000000..41ccbb6 Binary files /dev/null and b/static/images/2024/01/pr2.webp differ diff --git a/static/images/2024/01/securitydetail.webp b/static/images/2024/01/securitydetail.webp new file mode 100644 index 0000000..dca352d Binary files /dev/null and b/static/images/2024/01/securitydetail.webp differ diff --git a/static/images/2024/01/securityoverview.webp b/static/images/2024/01/securityoverview.webp new file mode 100644 index 0000000..f82f4b9 Binary files /dev/null and b/static/images/2024/01/securityoverview.webp differ diff --git a/static/images/2024/01/t_IMG_7306.webp b/static/images/2024/01/t_IMG_7306.webp new file mode 100644 index 0000000..7edd4dc Binary files /dev/null and b/static/images/2024/01/t_IMG_7306.webp differ