Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add go benchmark as part of the buildkite pipeline #3141

Merged
merged 33 commits into from
Dec 6, 2023
Merged

Add go benchmark as part of the buildkite pipeline #3141

merged 33 commits into from
Dec 6, 2023

Conversation

jlind23
Copy link
Contributor

@jlind23 jlind23 commented Dec 5, 2023 •
edited by alexsapran
Loading

What is the problem this PR solves?

We migrate the problematic GH workflow to a Buildkite pipeline.

How does this PR solve the problem?

Now when we run the buildkite pipeline we execute the go benchmark tests along side for the HEAD of the branch as well as the base branch and compare the 2.

How to test this PR locally

Design Checklist

  • I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
  • I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
  • I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool

Related issues

alexsapran
alexsapran reviewed Dec 5, 2023
View reviewed changes
.buildkite/pipeline.yml Outdated Show resolved Hide resolved
jlind23 and others added 23 commits December 5, 2023 14:54
@jlind23 @alexsapran
Update .buildkite/pipeline.yml
82bbd23 
Co-authored-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Add steps for comparing and baseline
155a4a5 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Make each annotate the gobench output
5cccf63 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Fix artifact paths
0bc1a95 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Compare dont need to checkout base
885a989 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Attempt to increase the cpu and mem for go bench
5ff8b5d 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Escape bash
a029392 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Remove the GH action for benchmark
3501064 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Increase memory
fd19ba6 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Formatting the annotations
c2384b9 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Change the type of the VMs for benchmarks
9caeb6f 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Ignore direnv files
05165ad 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Parse go benchmark results with gobenchdata
b684303 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Fix the artifact path
4aeec7a 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
fix the filepaths
3a1d18e 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Fix the path again
aaa85ed 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Make annotation reflect status
41ec3c8 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Fix failed json path
8f53958 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Suppress gobenchdata error exiting
12c0550 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Enable mem reporting for gobench by default
50874ef 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Add links to artifact download in bk annotation
b0e203a 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
change the gobenchdata check name
9846de7 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Improve the benchmark script for the step
45b5dd5 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Allow gobenchdata failure to not block
2aa34e9 
Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Reset Benchmark timer for Bulk
69a5cf7 
Attempt to fix the reporting of time of go benchmark

Signed-off-by: Alexandros Sapranidis <[email protected]>
@alexsapran
Revert "Reset Benchmark timer for Bulk"
2e8253d 
This reverts commit 69a5cf7.
@jlind23 jlind23 marked this pull request as ready for review December 6, 2023 14:53
@jlind23 jlind23 requested a review from a team as a code owner December 6, 2023 14:53
@jlind23 jlind23 enabled auto-merge (squash) December 6, 2023 14:57
@jlind23 jlind23 self-assigned this Dec 6, 2023
@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No Coverage information No data about Coverage
No Duplication information No data about Duplication

See analysis details on SonarQube

@nchaulet
Copy link
Member

nchaulet commented Dec 6, 2023

Should the benchmark fail the build in that case?

Screenshot 2023-12-06 at 11 41 12 AM

@jlind23
Copy link
Contributor Author

jlind23 commented Dec 6, 2023

@nchaulet For now we decided not to fail the build as it seems that benchmarks have some flakiness.
Example, this PR didn't change anything apart from the buildkite pipeline and even though the benchmarks are failing.

nchaulet
nchaulet approved these changes Dec 6, 2023
View reviewed changes
Copy link
Member

@nchaulet nchaulet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀 it will be great if it can report something in the PR when it failed, like a failed check that do not block the merge

@jlind23 jlind23 merged commit d8470ab into main Dec 6, 2023
8 checks passed
@jlind23 jlind23 deleted the jlind23-go-benchmark branch December 6, 2023 16:56
@jlind23 jlind23 mentioned this pull request Dec 7, 2023
Closed
1 task
jlind23 added a commit that referenced this pull request Dec 11, 2023
@apmmachine @jlind23 @elasticmachine
[updatecli] update elastic stack version for testing 8.12.0-9bbde39d-…
008af81 
…SNAPSHOT (#3153)

* Add go benchmark as part of the buildkite pipeline (#3141)

* add initial benchmark for PR
* Add steps for comparing and baseline
* Parse go benchmark results with gobenchdata
* Allow gobenchdata failure to not block
---------
Co-authored-by: Alexandros Sapranidis <[email protected]>

* [Release] update version (#3145)

Co-authored-by: Julien Lind <[email protected]>

* chore: Update snapshot.yml

Made with ❤️️ by updatecli

* chore: Update snapshot.yml

Made with ❤️️ by updatecli

* chore: Update snapshot.yml

Made with ❤️️ by updatecli

---------

Co-authored-by: Julien Lind <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: apmmachine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexsapran alexsapran alexsapran approved these changes

@nchaulet nchaulet nchaulet approved these changes

Assignees

@jlind23 jlind23

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jlind23 @nchaulet @alexsapran