Skip to content

Commit

Permalink
Now the manager also handles issues after release (on-sync)
Browse files Browse the repository at this point in the history 
a) keeping the current queue fed (like before release).
b) holding new features and improvements (with special on-syn message)

I've ended refactoring / reorganizing everything, splitting into
bash functions for easier reuse and future expansion.

Also, add support for $dryrun env variable to allow the script to be
executed without performing any change in the tracker.
  • Loading branch information
@stronk7
stronk7 committed Nov 10, 2020
1 parent a0993ac commit 44c50d6
Show file tree
Hide file tree
Showing 2 changed files with 344 additions and 161 deletions.
231 changes: 70 additions & 161 deletions tracker_automations/continuous_manage_queues/continuous_manage_queues.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,19 @@
# - current queue: issues under current integration.
#
# The automatisms are as follow:
# 0) Add the "integration_held" (+ std. comment) to new features & improvements issue missing it @ candidates.
# 1) Move "important" issues from candidates to current.
# 2) Move issues away from the candidates queue:
# 2a) Before a date (last week), keep the current queue fed with issues when it's under a threshold.
# 2b) After a date (last week), add the "integration_held" (+ std.comment) to bug issues.
# A) Before release only! (normally 6 weeks of continuous before release)
# 1) Add the "integration_held" (+ standard comment) to new features & improvements issue missing it @ candidates.
# 2) Move "important" issues from candidates to current.
# 3) Move issues away from the candidates queue.
# a) Before a date (last week), keep the current queue fed with bug issues when it's under a threshold.
# b) After a date (last week), add the "integration_held" (+ standard comment) to bug issues.
# B) After release only! (normally 2 weeks of on-sync continuous after release)
# 1) Move issues away from the candidates queue.
# a) Keep the current queue fed with bug issues when it's under a threshold.
# b) Add the "integration_held" (+ on-sync standard comment) to new features and improvements missing it @ candidates.
#
# The criteria to consider an issue "important" are:
# 1) It must be in the candidates queue, awating for integration. |
# 1) It must be in the candidates queue, awaiting for integration. |
# 2) It must not have the integration_held or security_held labels. | => filter=14000
# 3) It must not have the "agreed_to_be_after_release" text in a comment.| => NOT filter = 21366
# 4) At least one of this is true:
Expand All @@ -31,15 +36,17 @@
# jiraserver: jira server url we are going to connect to
# jirauser: user that will perform the execution
# jirapass: password of the user
# datetoholdbugs: first date to change behavior from 2a - feed current to 2b - held bug issues. (YYY-MM-DD)
# releasedate: Release date, used to decide between A (before release) and B (after release) behaviors. YYYY-MM-DD.
# lastweekdate: Last week date to decide between 2a - feed current and 2b - held bug issues. (YYY-MM-DD, defaults to release -1w)
# currentmin: optional, number of issue under which the current queue will be fed from the candidates one.
# movemax: optional, max number of issue that will be moved from candidates to current when under currentmin.
# dryrun: don't perfomr any write operation, only reads. Defaults to empty (false).

# Let's go strict (exit on error)
set -e

# Verify everything is set
required="WORKSPACE jiraclicmd jiraserver jirauser jirapass datetoholdbugs"
required="WORKSPACE jiraclicmd jiraserver jirauser jirapass releasedate"
for var in $required; do
if [ -z "${!var}" ]; then
echo "Error: ${var} environment variable is not defined. See the script comments."
Expand All @@ -59,175 +66,77 @@ mydir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
basereq="${jiraclicmd} --server ${jiraserver} --user ${jirauser} --password ${jirapass}"
BUILD_TIMESTAMP="$(date +'%Y-%m-%d_%H-%M-%S')"

source ${mydir}/lib.sh # Add all the functions.

# Set defaults
currentmin=${currentmin:-6}
movemax=${movemax:-3}
lastweekdate=${lastweekdate:-$(date -d "${releasedate} -7day" +%Y-%m-%d)}
dryrun=${dryrun:-}

# Verify that $releasedata has a correct YYYY-MM-DD format
if [[ ! ${releasedate} =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]]; then
echo "ERROR: \$releasedate. Incorrect YYYY-MM-DD format detected: ${releasedate}"
exit 1
fi

# Verify that $datetoholdbugs has a correct YYYY-MM-DD format
if [[ ! ${datetoholdbugs} =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]]; then
echo "ERROR: Incorrect YYYY-MM-DD format detected: ${datetoholdbugs}"
# Verify that $lastweekdate has a correct YYYY-MM-DD format
if [[ ! ${lastweekdate} =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]]; then
echo "ERROR: \$lastweekdate. Incorrect YYYY-MM-DD format detected: ${lastweekdate}"
exit 1
fi

# Decide if we are going to perform the behavior 2a) move to current
# or 2b) hold bugs, based on current date.
behavior2=
holddate=$(date -d "${datetoholdbugs}" +%Y%m%d)
# Today
nowdate=$(date +%Y%m%d)
if [ $nowdate -lt $holddate ]; then
behavior2="move"

# Decide if we are going to proceed with behaviour A (before release) or behaviour B (after release)
behaviorAB=
if [ $nowdate -lt $(date -d "${releasedate}" +%Y%m%d) ]; then
behaviorAB="before"
else
behavior2="hold"
behaviorAB="after"
fi

# Note this could be done by one unique "runFromIssueList" action, but we are splitting
# the search and the update in order to log all the closed issues within jenkins ($logfile)

# 0) Add the "integration_held" (+ std. comment) to new features & improvements issue missing it @ candidates.

# Basically get all the issues in the candidates queue (filter=14000), that are not bug
# and that haven't received any comment with the standard unholding text (NOT filter = 22054)

# Get the list of issues.
${basereq} --action getIssueList \
--search "filter=14000 \
AND type IN ('New Feature', Improvement) \
AND NOT filter = 22054" \
--file "${resultfile}"

# Iterate over found issues and perform the actions with them.
for issue in $( sed -n 's/^"\(MDL-[0-9]*\)".*/\1/p' "${resultfile}" ); do
echo "Processing ${issue}"
# Add the integration_held label.
${basereq} --action addLabels \
--issue ${issue} \
--labels "integration_held"
# Add the standard comment for held issues.
comment='This issue has been sent to integration after the freeze.
If you want Moodle HQ to consider including it into the incoming major release please add the "{{unhold_requested}}" label, and post a comment here outlining good reasons why you think it should be considered for late integration into the next major release.'
${basereq} --action addComment \
--issue ${issue} \
--comment "${comment}"
echo "$BUILD_NUMBER $BUILD_TIMESTAMP ${issue} integration_held added" >> "${logfile}"
done
# Decide if we are going to proceed with behaviour A3a (before last week, keep current queue fed)
# or behaviour A3b (last-week, add the integration_held + standard last week message to any issue).
behaviorA3=
if [ $behaviorAB == "before" ]; then # Only calculate this before release.
if [ $nowdate -lt $(date -d "${lastweekdate}" +%Y%m%d) ]; then
behaviorA3="move"
else
behaviorA3="hold"
fi
fi

# 1) Move "important" issues from candidates to current (cleaning integrator and tester).

# Get the list of issues.
${basereq} --action getIssueList \
--search "filter=14000
AND NOT filter = 21366
AND (
filter = 21363 OR
labels IN (mdlqa) OR
priority IN (Critical, Blocker) OR
level IS NOT EMPTY OR
component IN ('Privacy', 'Automated functional tests (behat)', 'Unit tests')
)" \
--file "${resultfile}"

# Iterate over found issues and perform the actions with them.
for issue in $( sed -n 's/^"\(MDL-[0-9]*\)".*/\1/p' "${resultfile}" ); do
echo "Processing ${issue}"
# For fields available in the default screen, it's ok to use updateIssue or SetField, but in this case
# we are setting some custom fields not available (on purpose) on that screen. So we have created a
# global transition, only available to the bots, not transitioning but bringing access to all the fields
# via special screen. So we'll ne using that global transition via progressIssue instead.
# Also, there is one bug in the 4.4.x series, setting the destination as 0, leading to error in the
# execution, so the form was hacked in the browser to store correct -1: https://jira.atlassian.com/browse/JRA-25002
# Commented below, it's the "ideal" code. If some day JIRA changes that restriction we could stop using
# that non-transitional transition and use normal update.
#${basereq} --action updateIssue \
# --issue ${issue} \
# --custom "customfield_10110:,customfield_10210:,customfield_10211:Yes"
${basereq} --action progressIssue \
--issue ${issue} \
--step "CI Global Self-Transition" \
--custom "customfield_10211:Yes,customfield_10110:,customfield_10011:" \
--comment "Continuous queues manage: Moving to current because it's important" \
--role "Integrators"
echo "$BUILD_NUMBER $BUILD_TIMESTAMP ${issue} moved to current: important" >> "${logfile}"
done
if [ -n "${dryrun}" ]; then
echo "Dry-run enabled, no changes will be performed to the tracker"
fi

# 2) Move issues away from the current queue:

# 2a) Before a date (last week), keep the current queue fed with issues when it's under a threshold.

if [[ "${behavior2}" == "move" ]]; then
# Count the list of issues in the current queue. (We cannot use getIssueCount till bumping to Jira CLI 8.1, hence, old way)
${basereq} --action getIssueList \
--search "project = MDL \
AND 'Currently in integration' IS NOT EMPTY \
AND status IN ('Waiting for integration review')" \
--file "${resultfile}"

# Iterate over found issues just to count them.
counter=0
for issue in $( sed -n 's/^"\(MDL-[0-9]*\)".*/\1/p' "${resultfile}" ); do
counter=$((counter+1))
done
echo "$counter issues awaiting integration in current queue"

# If there are < $currentmin issues, let's add up to $movemax issues from the candidates queue.
if [[ "$counter" -lt "$currentmin" ]]; then
# Get an ordered list of up to $movemax issues in the candidate queue.
${basereq} --action getIssueList \
--limit $movemax \
--search "filter=14000 \
ORDER BY 'Integration priority' DESC, \
priority DESC, \
votes DESC, \
'Last comment date' ASC" \
--file "${resultfile}"

# Iterate over found issues, moving them to the current queue (cleaning integrator and tester).
for issue in $( sed -n 's/^"\(MDL-[0-9]*\)".*/\1/p' "${resultfile}" ); do
echo "Processing ${issue}"
# For fields available in the default screen, it's ok to use updateIssue or SetField, but in this case
# we are setting some custom fields not available (on purpose) on that screen. So we have created a
# global transition, only available to the bots, not transitioning but bringing access to all the fields
# via special screen. So we'll ne using that global transition via progressIssue instead.
# Also, there is one bug in the 4.4.x series, setting the destination as 0, leading to error in the
# execution, so the form was hacked in the browser to store correct -1: https://jira.atlassian.com/browse/JRA-25002
# Commented below, it's the "ideal" code. If some day JIRA changes that restriction we could stop using
# that non-transitional transition and use normal update.
#${basereq} --action updateIssue \
# --issue ${issue} \
# --custom "customfield_10110:,customfield_10210:,customfield_10211:Yes"
${basereq} --action progressIssue \
--issue ${issue} \
--step "CI Global Self-Transition" \
--custom "customfield_10211:Yes,customfield_10110:,customfield_10011:" \
--comment "Continuous queues manage: Moving to current given we are below the threshold ($currentmin)" \
--role "Integrators"
echo "$BUILD_NUMBER $BUILD_TIMESTAMP ${issue} moved to current: threshold (before ${datetoholdbugs})" >> "${logfile}"
done
# Behaviour A, before the release (normally the 6 weeks of continuous).

if [ $behaviorAB == "before" ]; then
# A1, add the "integration_held" + standard comment to any new feature or improvement arriving to candidates.
run_A1
# A2, move "important" issues from candidates to current
run_A2
# A3, move all issues aways from candidates queue:
if [ $behaviorA3 == "move" ]; then
# A3a, keep the current queue fed with bug issues when it's under a threshold.
run_A3a
fi
if [ $behaviorA3 == "hold" ]; then
# A3b, add the "integration_held" + standard comment to any issue arriving to candidates.
run_A3b
fi
fi

# 2b) After a date (last week), add the "integration_held" (+ std.comment) to bug issues.

if [[ "${behavior2}" == "hold" ]]; then
# Get the list of issues in the candidates queue. All them will be held with last week comment.
${basereq} --action getIssueList \
--search "filter=14000" \
--file "${resultfile}"

# Iterate over found issues, moving them to the current queue.
for issue in $( sed -n 's/^"\(MDL-[0-9]*\)".*/\1/p' "${resultfile}" ); do
echo "Processing ${issue}"
# Add the integration_held label.
${basereq} --action addLabels \
--issue ${issue} \
--labels "integration_held"
# Add the standard comment for held issues the last week.
comment='We are currently in the final week before release ( https://docs.moodle.org/dev/Integration_Review#During_continuous_integration.2FFreeze.2FQA_period ) so this issue is being held until after release. Thanks for your patience!'
${basereq} --action addComment \
--issue ${issue} \
--comment "${comment}"
echo "$BUILD_NUMBER $BUILD_TIMESTAMP ${issue} integration_held added (since ${datetoholdbugs})" >> "${logfile}"
done
# Behaviour B, after the release (normally the 2 weeks of on-sync).

if [ $behaviorAB == "after" ]; then
# B1a, keep the current queue fed with bug issues when it's under a threshold.
run_B1a
# B1b, add the "integration_held" + standard on-sync comment to any new feature or improvement arriving to candidates.
run_B1b
fi

# Remove the resultfile. We don't want to disclose those details.
Expand Down
Loading

0 comments on commit 44c50d6

Please sign in to comment.