Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Add support for running tests with stack 9.0.0 #11138

Merged
merged 23 commits into from
Sep 19, 2024
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .buildkite/pipeline.schedule-daily.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,18 @@ steps:
- step: "check"
allow_failure: false

- label: "Check integrations local stacks - Stack Version v9.0"
trigger: "integrations"
build:
env:
SERVERLESS: "false"
FORCE_CHECK_ALL: "true"
STACK_VERSION: 9.0.0-SNAPSHOT
PUBLISH_COVERAGE_REPORTS: "false"
depends_on:
- step: "check"
allow_failure: false
Comment on lines +62 to +64
Copy link
Contributor Author

@mrodm mrodm Sep 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be interesting to run this step after finishing the tests with 8.x ?
It would be updating the step defined in the depends_on field.

To avoid creating another build to test 290 packages at the same time (not sure if that could be an issue). WDYT?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, this is going to be temporary, I guess that some time after 9.0 is released, 7.x will become unmaintained and we will remove its stage.

If it depends on 8.x, will it be executed if 8.x fails? We want both to be executed even if the other fails, what is pretty common 😅

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it should be able to do that by setting depends_on as:

    depends_on:
      - step: "local-8-stack"
        allow_failure: true

Once it is removed 7.x, it can be reverted the depends_on configuration too.

Copy link
Contributor Author

@mrodm mrodm Sep 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add this for now at least for daily

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But is it a problem to run so many stages at the same time? It looks like we are fixing an issue we don't have (yet?).

I don't have a strong preference with any of the options, but it looks simpler to just run everything at the same time if this is not a problem by now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's true, I'm just moving ahead and probably there would not be any issue.
I'll change it back, and if there are issues we can do this change then.


- label: "Check integrations in serverless - project: Observability"
key: "trigger-integrations-serverless-obs"
trigger: "integrations-serverless"
Expand Down
13 changes: 13 additions & 0 deletions .buildkite/pipeline.schedule-weekly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,16 @@ steps:
depends_on:
- step: "check"
allow_failure: false

- label: "Check integrations local stacks and non-wolfi images for Elastic Agent - Stack Version v9.0"
trigger: "integrations"
build:
env:
SERVERLESS: "false"
FORCE_CHECK_ALL: "true"
STACK_VERSION: 9.0.0-SNAPSHOT
PUBLISH_COVERAGE_REPORTS: "false"
ELASTIC_PACKAGE_DISABLE_ELASTIC_AGENT_WOLFI: "true"
depends_on:
- step: "check"
allow_failure: false
Comment on lines +40 to +42
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as in the daily job? Would it be interesting to trigger this after the other step testing 8.x has finished? (update depends_on field).

3 changes: 2 additions & 1 deletion .buildkite/pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -102,5 +102,6 @@ steps:
# not fail build if this step fails
soft_fail: true
# run this step when if it is triggered by the daily job
# skipped for now notification for tests related to STACK_VERSION: 9
if: >
build.source == "trigger_job" && build.env('BUILDKITE_TRIGGERED_FROM_BUILD_PIPELINE_SLUG') == "integrations-schedule-daily"
build.source == "trigger_job" && build.env('BUILDKITE_TRIGGERED_FROM_BUILD_PIPELINE_SLUG') == "integrations-schedule-daily" && build.env('STACK_VERSION') !~ /^9/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

11 changes: 9 additions & 2 deletions .buildkite/scripts/common.sh
Original file line number Diff line number Diff line change
Expand Up @@ -464,12 +464,19 @@ is_supported_stack() {
if [ "${kibana_version}" == "null" ]; then
return 0
fi
if [[ ! "${kibana_version}" =~ \^7\. && "${STACK_VERSION}" =~ ^7\. ]]; then
if [[ ( ! "${kibana_version}" =~ \^7\. ) && "${STACK_VERSION}" =~ ^7\. ]]; then
return 1
fi
if [[ ! "${kibana_version}" =~ \^8\. && "${STACK_VERSION}" =~ ^8\. ]]; then
if [[ ( ! "${kibana_version}" =~ \^8\. ) && "${STACK_VERSION}" =~ ^8\. ]]; then
return 1
fi

# TODO: Allowed temporarily to test packages with stack version 9.0 if they have as constraint ^8.0 defined too.
# This workaround should be removed once packages have been updated their constraints for 9.0 stack.
if [[ ( ! ( "${kibana_version}" =~ \^9\. || "${kibana_version}" =~ \^8\. ) ) && "${STACK_VERSION}" =~ ^9\. ]]; then
return 1
fi

return 0
}

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: Bump latest elastic-stack test version
pipelineid: 'bump-latest-elastic-stack-version'
name: Bump latest 8.x-SNAPSHOT elastic-stack test version
pipelineid: 'bump-latest-8x-elastic-stack-version'

actions:
default:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
---
name: Bump latest 9.x-SNAPSHOT elastic-stack test version
pipelineid: 'bump-latest-9x-elastic-stack-version'

actions:
default:
title: '[updatecli] Update latest snapshot to {{ source "latestSnapshot" }}'
kind: github/pullrequest
scmid: default
spec:
labels:
- automation
- dependency

scms:
default:
kind: github
spec:
owner: '{{ .scm.owner }}'
repository: '{{ .scm.repository }}'
user: '{{ requiredEnv "GITHUB_ACTOR" }}'
username: '{{ requiredEnv "GITHUB_ACTOR" }}'
token: '{{ requiredEnv "GITHUB_TOKEN" }}'
commitusingapi: true
branch: main

sources:
latestSnapshot:
name: Get latest snapshot
kind: json
spec:
file: https://storage.googleapis.com/artifacts-api/snapshots/main.json
key: .version
latestSnapshotMajorMinor:
name: Get latest snapshort major and minor
kind: json
spec:
file: https://storage.googleapis.com/artifacts-api/snapshots/main.json
key: .version
transformers:
- findsubmatch:
pattern: '(^\d+\.\d+)\.[^\s]+$'
captureindex: 1

targets:
update-snapshot-daily:
name: '[updatecli] [daily] Update latest snapshot to {{ source "latestSnapshot" }}'
kind: file
sourceid: latestSnapshot
scmid: default
spec:
file: '.buildkite/pipeline.schedule-daily.yml'
matchpattern: '(STACK_VERSION:) 9\.[^\s]*\.[^\s]*'
replacepattern: '$1 {{ source "latestSnapshot" }}'

update-snapshot-label-daily:
name: '[updatecli] [daily] Update latest snapshot label step to {{ source "latestSnapshotMajorMinor" }}'
kind: file
sourceid: latestSnapshotMajorMinor
scmid: default
spec:
file: '.buildkite/pipeline.schedule-daily.yml'
matchpattern: '(Stack Version) v9\.\d+"'
replacepattern: 'Stack Version v{{ source "latestSnapshotMajorMinor" }}"'

update-snapshot-weekly:
name: '[updatecli] [weekly] Update latest snapshot to {{ source "latestSnapshot" }}'
kind: file
sourceid: latestSnapshot
scmid: default
spec:
file: '.buildkite/pipeline.schedule-weekly.yml'
matchpattern: '(STACK_VERSION:) 9\.[^\s]*\.[^\s]*'
replacepattern: '$1 {{ source "latestSnapshot" }}'

update-snapshot-label-weekly:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These steps are now failing in the current check, but they should work once this PR is merged and all the changes are in main branch.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the updatecli config too 👍

name: '[updatecli] [weekly] Update latest snapshot label step to {{ source "latestSnapshotMajorMinor" }}'
kind: file
sourceid: latestSnapshotMajorMinor
scmid: default
spec:
file: '.buildkite/pipeline.schedule-weekly.yml'
matchpattern: '(Stack Version) v9\.\d+"'
replacepattern: 'Stack Version v{{ source "latestSnapshotMajorMinor" }}"'
33 changes: 28 additions & 5 deletions docs/ci_pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ Currently, there are five different pipelines:
- https://buildkite.com/elastic/integrations-serverless: pipeline in charge of testing all packages using a Elastic Serverless project. More info at [section](#serverless-pipeline).
- https://buildkite.com/elastic/integrations-publish: pipeline to publish the new versions of packages. More info at [section](#publish-packages).
- https://buildkite.com/elastic/integrations-schedule-daily/: pipeline running every night to test packages in different scenarios. More info at [section](#daily-job).
- https://buildkite.com/elastic/integrations-schedule-weekly/: pipeline running once per week to test packages in different scenarios. More info at [section](#weekly-job).
- https://buildkite.com/elastic/integrations-backport/: pipeline to create backport branches (just from UI). More info at [section](#backport-branches-pipeline).

## Pull Requests and pushes to specific branches
Expand All @@ -22,14 +23,15 @@ Special comments that can be added in the Pull Request (by Elastic employees):
There are some environment variables that can be added into this pipeline:
- **FORCE_CHECK_ALL**: If `true`, this forces the CI to check all packages even if those packages have no file updated/added/deleted. Default: `false`.
- **STACK_VERSION**: Force the CI steps to spin up a specific Elastic stack version to run the tests. Default: unset.
- **STACK_LOGSDB_ENABLED**: Enable LogsDB setting in Elasticsearch service. Default: false.
- **PUBLISH_COVERAGE_REPORTS**: If `true`, it enables reporting coverage reports.
Currently, it is just set for the build triggered with the current major Elastic stack from the daily job. Default: `false`.

These environment variables can be defined in different locations:
- In the [global `env` section](https://github.com/elastic/integrations/blob/5276ef63712f8f2311818770881688870e8422fe/.buildkite/pipeline.yml#L2).
- In the case of the scheduled daily job here:
- [Running tests with previous major Elastic stack](https://github.com/elastic/integrations/blob/5276ef63712f8f2311818770881688870e8422fe/.buildkite/pipeline.schedule-daily.yml#L21).
- [Running tests with current major Elastic stack](https://github.com/elastic/integrations/blob/d6d99792b90838d18844f6df9343bc5f16130666/.buildkite/pipeline.schedule-daily.yml#L32).
- In the case of the scheduled daily job in each step. Some examples:
- [Running tests with 7.x major Elastic stack](https://github.com/elastic/integrations/blob/5276ef63712f8f2311818770881688870e8422fe/.buildkite/pipeline.schedule-daily.yml#L21).
- [Running tests with 8.x major Elastic stack](https://github.com/elastic/integrations/blob/d6d99792b90838d18844f6df9343bc5f16130666/.buildkite/pipeline.schedule-daily.yml#L32).

More details about this CI pipeline:

Expand Down Expand Up @@ -124,9 +126,13 @@ The schedule of this job can be checked [here](https://github.com/elastic/integr

The scenarios that are tested in this daily job are:

- Test packages with a local Elastic stack running the previous major version of the stack (7.17.X SNAPSHOT).
- Test packages with a local Elastic stack running the latest 7.x version of the stack (7.17.X SNAPSHOT).
- Triggered pipeline: https://buildkite.com/elastic/integrations
- Test packages with a local Elastic stack running the latest major version of the stack (currently 8.X.Y-SNAPSHOT).
- Test packages with a local Elastic stack running the latest 8.x version of the stack (8.X.Y-SNAPSHOT).
- Triggered pipeline: https://buildkite.com/elastic/integrations
- Test packages with a local Elastic stack running the latest 8.x version of the stack with LogsDB setting enabled (8.X.Y-SNAPSHOT).
- Triggered pipeline: https://buildkite.com/elastic/integrations
- Test packages with a local Elastic stack running the latest major version of the stack (currently 9.X.Y-SNAPSHOT).
- Triggered pipeline: https://buildkite.com/elastic/integrations
- Test packages with an Elastic Serverless Observability project.
- Triggered pipeline: https://buildkite.com/elastic/integration-serverless
Expand All @@ -141,6 +147,23 @@ have been published by triggering the pipeline https://buildkite.com/elastic/int
Each step triggering a new pipeline can be customized through environment variables. Environment variables that can
be used in each pipeline are detailed in the corresponding sections of each pipeline.

## Weekly job

**Note**: Just available for Employees at Elastic.

Every week it is configured to run a job that will be in charge of testing all packages with non-Wolfi Elastic Agent docker images: https://buildkite.com/elastic/integrations-schedule-weekly/

The schedule of this job can be checked [here](https://github.com/elastic/integrations/blob/2e72e8524728daca2d47c814d8042031b8f5804f/catalog-info.yaml#L145).

The scenarios that are tested in this weekly job are:

- Test packages with a local Elastic stack running the latest 8.x version of the stack with Elastic Agent images based on Ubuntu images (8.X.Y-SNAPSHOT).
- Triggered pipeline: https://buildkite.com/elastic/integrations
- Test packages with a local Elastic stack running the latest major version of the stack with Elastic Agent images based on non-Wolfi images (currently 9.X.Y-SNAPSHOT).
- Triggered pipeline: https://buildkite.com/elastic/integrations

Each step triggering a new pipeline can be customized through environment variables. Environment variables that can
be used in each pipeline are detailed in the corresponding sections of each pipeline.

## Backport branches pipeline

Expand Down
Loading