[8.14](backport #4906) [CI] Monorepo diff plugin for integration tests

[mergify]

What does this PR do?

Use monorepo diff plugin to run integration tests for pull requests and branches.
The plugin specifies a list of watched paths for pull requests. If a PR modifies files under a path from the list - integration tests will be executed.

The main pipeline triggers elastic-agent-extended-testing if all conditions are met:

• At least one of the listed paths is modified by a PR
• skip-it label is not set

When a PR is merged to main or a release branch the integration tests will be triggered in any case

That's how it looks now:

Why is it important?

The previous approach wasn't flexible enough and the separate integration testing pipeline didn't allow to run integration tests conditionally due to required checks limitation.

Important: if we add skip-it label - the PR will be mergeable without running integration tests.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Additional checks

• Draft PRs shouldn't trigger integration tests by default (Need to enhance buildkite-pr-bot and pass GITHUB_PULL_REQUEST_DRAFT env variable to the build)
• /test comment must trigger unit and integration tests also for draft PRs
• A PR modifying the files from watch list must run integration tests
• If a PR doesn't modify the listed files integration tests shouldn't be launched
• A PR labeled with skip-it must not trigger integration tests

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

---

This is an automatic backport of pull request #4906 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of fd8d1ee has failed:

On branch mergify/bp/8.14/pr-4906
Your branch is up to date with 'origin/8.14'.

You are currently cherry-picking commit fd8d1ee1a0.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .buildkite/pipeline.yml

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   .buildkite/pull-requests.json

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube