Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DH-301] Python script to replace bash scaffolding in CI/CD and simplify github workflows #6035

Merged
merged 14 commits into from
Aug 28, 2024
Merged

[DH-301] Python script to replace bash scaffolding in CI/CD and simplify github workflows #6035

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
6fd9cb1
script and workflow to get deployments from deployments/ subdir
shaneknapp Aug 27, 2024
d931fa1
forgot about staging
shaneknapp Aug 27, 2024
ac0f289
newline at EOF
shaneknapp Aug 27, 2024
84f6b64
docstring ftw
shaneknapp Aug 27, 2024
c888a45
must remember: double quotes, not single. double quotes, not single…
shaneknapp Aug 27, 2024
d462e9a
formatting
shaneknapp Aug 27, 2024
88fbf92
python really helps clean things up
shaneknapp Aug 28, 2024
f9ad125
add arg to allow us to limit hub deployments to only specific ones
shaneknapp Aug 28, 2024
ddacff1
seriously hating bash
shaneknapp Aug 28, 2024
739c19a
reformat comments to be a bit more clear
shaneknapp Aug 28, 2024
8d31b03
more explainy comments
shaneknapp Aug 28, 2024
132acfa
grammarizing the crap out of comments
shaneknapp Aug 28, 2024
9f64914
might as well explicitly call python to execute these scripts... can…
shaneknapp Aug 28, 2024
a7b591a
bashenannigans
shaneknapp Aug 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions .github/scripts/determine-hub-deployments.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
#! /usr/bin/env python
"""
Check the Github environment variables for hub deployments and determine if we
will deploy all hubs or just a subset.

All hubs will be deployed if the environment variable
GITHUB_PR_LABEL_JUPYTERHUB_DEPLOYMENT or GITHUB_PR_LABEL_HUB_IMAGES is set.

Otherwise, the environment variables GITHUB_PR_LABEL_HUB_<HUB_NAME> will be
checked to determine which hubs to deploy.

If no hubs need deploying, nothing will be emitted.
"""
import argparse
import os

def main(args):
hubs = []

# Deploy all hubs
if (
"GITHUB_PR_LABEL_JUPYTERHUB_DEPLOYMENT" or
"GITHUB_PR_LABEL_HUB_IMAGES"
) in os.environ.keys():
for deployment in next(os.walk(args.deployments))[1]:
if deployment not in args.ignore:
hubs.append(deployment)

# Deploy specific hubs
else:
hub_labels = [
k.lower() for k in os.environ.keys()
if k.startswith("GITHUB_PR_LABEL_HUB_")
]
hubs = [x.split("_")[-1] for x in hub_labels]
hubs = [x for x in hubs if x not in args.ignore]

hubs.sort()
for h in hubs:
if args.only_deploy and h not in args.only_deploy:
continue
print(h)

if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Get hubs that need to be deployed from environment variables."
)
parser.add_argument(
"--deployments",
"-d",
default="deployments",
help="The directory to search for deployments."
)
parser.add_argument(
"--ignore",
"-i",
nargs="*",
default=["template"],
help="Ignore one or more deployment targets."
)
parser.add_argument(
"--only-deploy",
"-o",
nargs="*",
help="Only deploy the specified hubs."
)
args = parser.parse_args()

main(args)
189 changes: 0 additions & 189 deletions .github/workflows/deploy-all-hubs.yaml
View file
Open in desktop

This file was deleted.

64 changes: 28 additions & 36 deletions .github/workflows/deploy-hubs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
# this workflow re-deploys SPECIFIC hubs to staging or prod if the single-user
# server image or config has changed based on the PR labels "hub: <hubname>".
#
# however, this workflow will be not run if the PR labels of "hub-images" or
# "jupyterhub-deployment" are present, as these labels will trigger the
# "deploy-jupyterhub-base-images.yaml" workflow which re-deploys every hub.
# This workflow will determine if the base hub image and/or single-user server
# image for any or all hubs has has changed, and if so, deploy accordingly.
#
name: Deploy staging and prod hubs
on:
Expand All @@ -24,25 +20,23 @@ jobs:
with:
github-token: ${{ secrets.GITHUB_TOKEN }}

- name: Pull out any hubs that need deploying from the labels on the merge commit to staging
- name: Check for hubs that need deploying from the labels on the merge commit to staging
run: |
echo "PR labels: ${{ steps.pr-labels.outputs.labels }}"
HUBS=()
# If the PR labels "hub-images" or "jupyterhub-deployment" are present, this
# means the base hub image has changed, and all hubs (staging or prod) need to
# be redeployed. The rest of this job will not run in that case.
if [ -c "${GITHUB_PR_LABEL_HUB_IMAGES}" ] || [ -c "${GITHUB_PR_LABEL_JUPYTERHUB_DEPLOYMENT}" ]; then
echo "Base hub image has changed, not deploying individual hubs to staging"
# If the PR labels "hub-images" or "jupyterhub-deployment" are
# present, this means the base hub image has changed, and all hubs
# (staging or prod) need to be redeployed.
#
# Otherwise, just check to see if the PR labels contain any hubs,
# and if so, deploy just those hubs to staging.
if [[ -n ${GITHUB_PR_LABEL_HUB_IMAGES} || -n ${GITHUB_PR_LABEL_JUPYTERHUB_DEPLOYMENT} ]]; then
echo "DEPLOY=1" >> $GITHUB_ENV
Copy link
Contributor Author

@shaneknapp shaneknapp Aug 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@felder : sanity check here. the github action that grabs labels from a push exports them to env vars in the format of GITHUB_PR_LABEL_<whatever>=1. in this case, i want to check for the labels of jupyterhub-deployment and/or hub-images (stored as the vars in the code snippet above).

if i want to check if these are indeed set, the way i do it here (if [[ -n ${GITHUB_PR_LABEL_HUB_IMAGES} || -n ${GITHUB_PR_LABEL_JUPYTERHUB_DEPLOYMENT} ]]) is correct, yes?

(i did a bunch of testing on the CLI and this seems to give us what we want but a 2nd pair of eyes is always nice etc)

Copy link
Contributor

@felder felder Aug 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shaneknapp -n evaluates to true if the variable is not set to the empty string. You may want to use -v instead which evaluates to true if the variable is set. -v also expects the name of the variable as opposed to an expansion of the variable.

So you may want to tryif [[ -v GITHUB_PR_LABEL_HUB_IMAGES || -v GITHUB_PR_LABEL_JUPYTERHUB_DEPLOYMENT ]]

Example:

#!/usr/bin/env bash

TEST=""

if [[ ! -n ${TEST} ]]; then
  echo "TEST is empty"
fi

if [[ -v TEST ]]; then
  echo "TEST is set"
fi

unset TEST

if [[ ! -v TEST ]]; then
  echo "TEST is no longer set"
fi
ras-29-57:~ felder$ ./test.sh
TEST is empty
TEST is set
TEST is no longer set

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alright, lemme try that... i'll set some labels to test this when i merge to staging

else
# deploy any hubs that have been labeled for deployment
for label in $(echo -e "${{ steps.pr-labels.outputs.labels }}"); do
if [[ "$label" == hub-* ]]; then
hub_name=$(echo $label | awk -F'-' '{print $2}')
HUBS+="$hub_name "
echo "DEPLOY=1" >> $GITHUB_ENV
fi
done
echo "DEPLOY_HUBS=${HUBS[@]}" >> $GITHUB_ENV
fi

- name: Check out the image repo
Expand Down Expand Up @@ -104,10 +98,10 @@ jobs:
- name: Deploy hubs to staging
if: ${{ env.DEPLOY }}
run: |
for hub in $(echo -e "${{ env.DEPLOY_HUBS }}"); do
echo "Deploying $hub to staging"
echo "hubploy --verbose deploy $hub hub staging"
done
while read deployment; do
echo "Pretending to deploy base hub image to ${deployment} :P"
echo "hubploy deploy --verbose ${deployment} hub staging"
done < <(.github/scripts/determine-hub-deployments.py)

deploy-hubs-to-prod:
if: github.event_name == 'push' && github.ref == 'refs/heads/prod'
Expand All @@ -119,25 +113,23 @@ jobs:
with:
github-token: ${{ secrets.GITHUB_TOKEN }}

- name: Pull out any hubs that need deploying from the labels on the merge commit to prod
- name: Check for hubs that need deploying from the labels on the merge commit to prod
run: |
echo "PR labels: ${{ steps.pr-labels.outputs.labels }}"
HUBS=()
# If the PR labels "hub-images" or "jupyterhub-deployment" are present, this
# means the base hub image has changed, and all hubs (staging or prod) need to
# be redeployed. The rest of this job will not run in that case.
if [ -c "${GITHUB_PR_LABEL_HUB_IMAGES}" ] || [ -c "${GITHUB_PR_LABEL_JUPYTERHUB_DEPLOYMENT}" ]; then
echo "Base hub image has changed, not deploying individual hubs to prod"
# If the PR labels "hub-images" or "jupyterhub-deployment" are
# present, this means the base hub image has changed, and all hubs
# (staging or prod) need to be redeployed.
#
# Otherwise, just check to see if the PR labels contain any hubs,
# and if so, deploy just those hubs to prod.
if [[ -n ${GITHUB_PR_LABEL_HUB_IMAGES} || -n ${GITHUB_PR_LABEL_JUPYTERHUB_DEPLOYMENT} ]]; then
echo "DEPLOY=1" >> $GITHUB_ENV
else
# deploy any hubs that have been labeled for deployment
for label in $(echo -e "${{ steps.pr-labels.outputs.labels }}"); do
if [[ "$label" == hub-* ]]; then
hub_name=$(echo $label | awk -F'-' '{print $2}')
HUBS+="$hub_name "
echo "DEPLOY=1" >> $GITHUB_ENV
fi
done
echo "DEPLOY_HUBS=${HUBS[@]}" >> $GITHUB_ENV
fi

- name: Check out the image repo
Expand Down Expand Up @@ -199,7 +191,7 @@ jobs:
- name: Deploy hubs to prod
if: ${{ env.DEPLOY }}
run: |
for hub in $(echo -e "${{ env.DEPLOY_HUBS }}"); do
echo "Deploying $hub to prod"
echo "hubploy --verbose deploy $hub hub prod"
done
while read deployment; do
echo "Pretending to deploy base hub image to ${deployment} :P"
echo "hubploy deploy --verbose ${deployment} hub prod"
done < <(.github/scripts/determine-hub-deployments.py)
Loading