Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ci] Fix DRA artifacts permissions #5175

Merged
merged 14 commits into from
Jul 19, 2024
Merged

[Ci] Fix DRA artifacts permissions #5175

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
f803d48
Debug DRA
pazone Jul 19, 2024
3ce6955
Debug DRA
pazone Jul 19, 2024
3b7bac1
Debug DRA
pazone Jul 19, 2024
5f6356a
[CI] Fix packaging permissions
pazone Jul 19, 2024
24990dd
[CI] Fix packaging permissions
pazone Jul 19, 2024
2e38e39
CHanged added read permissions
pazone Jul 19, 2024
ee3e09b
CHanged added read permissions
pazone Jul 19, 2024
f591e83
Fix veriables interpolation
pazone Jul 19, 2024
fa0e113
Debug
pazone Jul 19, 2024
416485f
Debug
pazone Jul 19, 2024
e07a3b4
Debug
pazone Jul 19, 2024
f3b4e05
Debug
pazone Jul 19, 2024
73a6078
Cleanup
pazone Jul 19, 2024
33eb785
Cleanup
pazone Jul 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 17 additions & 14 deletions .buildkite/pipeline.elastic-agent-package.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
env:
DOCKER_REGISTRY: "docker.elastic.co"
SETUP_GVM_VERSION: 'v0.5.0' # https://github.com/andrewkroh/gvm/issues/44#issuecomment-1013231151
SETUP_MAGE_VERSION: '1.14.0'
# this is required in order to allow the build process to override the default PWD of the BEAT_NAME.
BEAT_NAME: "elastic-agent"
# after moving elastic-agent out of beats, we should update the URL of the packaging.
Expand Down Expand Up @@ -63,14 +61,14 @@ steps:
machineType: "c2-standard-16"
diskSizeGb: 400
command: |
if [[ -z "${MANIFEST_URL}" ]]; then
if [[ -z "$${MANIFEST_URL}" ]]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused. The switch to $$ would output the PID, right? So everywhere this is used, the string being checked will be non-null.

Copy link
Contributor Author

@pazone pazone Jul 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

before any pipeline execution builkite agent uploads the pipeline yaml with buildkite-agent pipeline upload. It interpolates the yaml; if it finds $somethig it considers it a variable and tries to resolve it during the upload. ${MANIFEST_URL} will be resolved to "" (empty string). The $$ is used to avoid variable substitution.
I still don't understand why it worked before.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, thanks for explaining. I guess I haven't noticed this because most pipelines have the command logic in separate .sh files.

export MANIFEST_URL=$(buildkite-agent meta-data get MANIFEST_URL --default "")
if [[ -z "${MANIFEST_URL}" ]]; then
if [[ -z "$${MANIFEST_URL}" ]]; then
echo ":broken_heart: Missing MANIFEST_URL variable or empty string provided"
exit 1
fi
fi
if [[ -z "${MAGEFILE_VERBOSE}" ]]; then
if [[ -z "$${MAGEFILE_VERBOSE}" ]]; then
export MAGEFILE_VERBOSE=$(buildkite-agent meta-data get MAGEFILE_VERBOSE --default "0")
fi
.buildkite/scripts/steps/package.sh
Expand All @@ -86,17 +84,18 @@ steps:
PLATFORMS: "linux/arm64"
PACKAGES: "docker"
command: |
if [[ -z "${MANIFEST_URL}" ]]; then
if [[ -z "$${MANIFEST_URL}" ]]; then
export MANIFEST_URL=$(buildkite-agent meta-data get MANIFEST_URL --default "")
if [[ -z "${MANIFEST_URL}" ]]; then
if [[ -z "$${MANIFEST_URL}" ]]; then
echo ":broken_heart: Missing MANIFEST_URL variable or empty string provided"
exit 1
fi
fi
if [[ -z "${MAGEFILE_VERBOSE}" ]]; then
if [[ -z "$${MAGEFILE_VERBOSE}" ]]; then
export MAGEFILE_VERBOSE=$(buildkite-agent meta-data get MAGEFILE_VERBOSE --default "0")
fi
.buildkite/scripts/steps/package.sh
ls -lahR build/
artifact_paths:
- "build/distributions/**/*"

Expand All @@ -111,18 +110,22 @@ steps:
DRA_PROJECT_ARTIFACT_ID: "agent-package"
command: |
echo "+++ Restoring Artifacts"
buildkite-agent artifact download "build/**/*" .
echo "+++ Changing permissions for the release manager"
buildkite-agent artifact download "build/**/*" .
echo "+++ Changing permissions for the release manager"
sudo chmod -R a+r build/distributions/
sudo chown -R :1000 build/distributions/
ls -lahR build/
echo "+++ Running DRA publish step"
if [[ -z "${MAGEFILE_VERBOSE}" ]]; then
if [[ -z "$${MAGEFILE_VERBOSE}" ]]; then
export MAGEFILE_VERBOSE=$(buildkite-agent meta-data get MAGEFILE_VERBOSE --default "0")
fi
if [[ -z "${DRA_DRY_RUN}" ]]; then
if [[ -z "$${DRA_DRY_RUN}" ]]; then
DRA_DRY_RUN=$(buildkite-agent meta-data get DRA_DRY_RUN --default "")
export DRA_DRY_RUN
fi
if [[ -z "${DRA_VERSION}" ]]; then
if [[ -z "$${DRA_VERSION}" ]]; then
DRA_VERSION=$(buildkite-agent meta-data get DRA_VERSION --default "")
export DRA_VERSION
fi
Expand Down Expand Up @@ -158,7 +161,7 @@ steps:
.buildkite/scripts/steps/dra-publish.sh
# Artifacts will be uploaded via the artifact_paths entry above
echo "+++ Set job metadata if TRIGGER_JOB_ID is properly set"
if [[ -z "${TRIGGER_JOB_ID}" ]]; then
if [[ -z "$${TRIGGER_JOB_ID}" ]]; then
echo "TRIGGER_JOB_ID is not set, so not setting metadata"
else
# If a pipeline that triggered this build passes in a "TRIGGER_JOB_ID" env var, that
Expand Down
5 changes: 2 additions & 3 deletions .buildkite/scripts/common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,11 @@ if [[ -z "${WORKSPACE-""}" ]]; then
WORKSPACE=$(git rev-parse --show-toplevel)
export WORKSPACE
fi
PIPELINE="${WORKSPACE}/.buildkite/pipeline.elastic-agent-package.yml"
if [[ -z "${SETUP_MAGE_VERSION-""}" ]]; then
SETUP_MAGE_VERSION=$(grep -oe "SETUP_MAGE_VERSION\: [\"'].*[\"']" "$PIPELINE" | awk '{print $2}' | sed "s/'//g" )
SETUP_MAGE_VERSION="1.14.0"
fi
if [[ -z "${SETUP_GVM_VERSION-""}" ]]; then
SETUP_GVM_VERSION=$(grep -oe "SETUP_GVM_VERSION\: [\"'].*[\"']" "$PIPELINE" | awk '{print $2}' | sed "s/'//g" )
SETUP_GVM_VERSION="v0.5.0" # https://github.com/andrewkroh/gvm/issues/44#issuecomment-1013231151
Comment on lines +10 to +13
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How are these version changes related to the permissions problem?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's indirectly related to the issue. Go installation started to fail. It tried to extract the SETUP_GVM_VERSION from pipeline.elastic-agent-package.yml and it stopped working after my changes. I just decided to simplify it because it's actually a very simple thing that was done in an overly complex way

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks 👍

fi
BEAT_VERSION=$(grep -oE '[0-9]+\.[0-9]+\.[0-9]+(\-[a-zA-Z]+[0-9]+)?' "${WORKSPACE}/version/version.go")
export BEAT_VERSION
Expand Down