Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup E2E test using NATS Backend #30

Merged
merged 59 commits into from
Jan 3, 2024
Merged
Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
883f97f
Setup E2E github action flow
grischperl Dec 20, 2023
ff276ce
Move hack Makefile
grischperl Dec 20, 2023
9164364
Move ci folder outside of e2e
grischperl Dec 20, 2023
412bb90
Correct makefile path
grischperl Dec 20, 2023
a2af197
Add missing step to e2e flow to checkout repo
grischperl Dec 20, 2023
c61ff0e
Correct path in Makefile
grischperl Dec 20, 2023
0982fa5
Add kustomize to Makefile
grischperl Dec 20, 2023
087b5cd
Add kustomization
grischperl Dec 20, 2023
c276ba7
Add LOCALBIN variable
grischperl Dec 21, 2023
f2dbfef
Add k3d cluster create step
grischperl Dec 21, 2023
4b264c1
Remove go step
grischperl Dec 21, 2023
288ecd8
Add missing variables to Makefile
grischperl Dec 21, 2023
1341054
Add kyma script
grischperl Dec 21, 2023
1fb940d
Make script executable
grischperl Dec 21, 2023
1b2dd7f
Checkout EM repositry
grischperl Dec 21, 2023
5e513b3
Correct repository checkout
grischperl Dec 21, 2023
b4d2709
Use EM repo for e2e tests
grischperl Dec 21, 2023
08a3ed9
Remove hack/Makefile
grischperl Dec 21, 2023
03edb93
Replace make command with local one
grischperl Dec 21, 2023
6c78a9a
Try adding sudo for `permission denied` problem
grischperl Dec 21, 2023
ba87682
Try using `kyma` instead of variable
grischperl Dec 21, 2023
4353671
Try using explicit path
grischperl Dec 21, 2023
07ae627
Use different make targets to deploy EM
grischperl Dec 21, 2023
dce976f
Update image
grischperl Dec 27, 2023
7c2ee4a
Fix reference of variable
grischperl Dec 27, 2023
260c0fe
Add step to create namespace
grischperl Dec 27, 2023
e6f803b
Fix Github token
grischperl Dec 27, 2023
b309ef4
Fix make calls in workflow
grischperl Dec 27, 2023
9cba00a
Add step to install NATS
grischperl Dec 27, 2023
f7dcbe2
Try to use Github variable
grischperl Dec 27, 2023
a989fc5
Deploy nats manager via makefile
grischperl Dec 27, 2023
cdd4190
Add step to install NATS CR
grischperl Dec 27, 2023
17e2b19
Fix reference of variable
grischperl Dec 28, 2023
47de078
Try using `create-k3d` from `eventing-manager` repo
grischperl Dec 28, 2023
616b945
Change KYMA_CLI path
grischperl Dec 28, 2023
12e3bc0
Undo variable changes
grischperl Dec 28, 2023
13d0d9a
Remove unused files
grischperl Dec 28, 2023
d6138d6
Add clean-up step
grischperl Dec 28, 2023
840c401
Improve patch command by using `jq`
grischperl Dec 28, 2023
64e162e
Add missing path to cleanup command
grischperl Dec 28, 2023
d254cbb
Try using `MODULE_VERSION` from EM Repo
grischperl Dec 28, 2023
2be5728
Echo Module Version
grischperl Dec 28, 2023
75fcad0
Print deployment image
grischperl Dec 28, 2023
06ca070
Fix kubectl command
grischperl Dec 28, 2023
bc2bda5
Try to use string
grischperl Dec 28, 2023
f8cec62
Try using `yg` to extract version
grischperl Dec 28, 2023
b14c40a
Fix command
grischperl Dec 28, 2023
acc5b7d
Use `grep` instead of `yq`
grischperl Dec 28, 2023
1e6bcb4
Use curl to get latest release of NM
grischperl Dec 29, 2023
c2e4a71
Use `curl` to get EM latest release
grischperl Dec 29, 2023
cdcc5f3
Use NM image used in release
grischperl Dec 29, 2023
ea5b8fb
Use `kubectl apply` to deploy latest NATS and Eventing Manager
grischperl Jan 2, 2024
10e2564
Remove step to test EM setup
grischperl Jan 2, 2024
8ffdb8c
Remove unused variable
grischperl Jan 2, 2024
d2c4e2d
Check for pod rollout
grischperl Jan 3, 2024
0192c4d
Add missing `$` to variable
grischperl Jan 3, 2024
43f7807
Wait for rollout to complete
grischperl Jan 3, 2024
22d997d
Add step to check for success of EPP image update
grischperl Jan 3, 2024
d69cdb3
Replace `k` with `kubectl`
grischperl Jan 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 118 additions & 0 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
name: E2E Test

on:
pull_request:
branches:
- main
- "release-*"
paths-ignore:
- "docs/**"
- "**.md"
- "sec-scanners-config.yaml"

env:
NATS_MANAGER_IMAGE: europe-docker.pkg.dev/kyma-project/prod/nats-manager:latest
EVENTING_MANAGER_IMAGE: europe-docker.pkg.dev/kyma-project/prod/eventing-manager:latest
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use latest image tag? or image from latest release of eventing+nats manager?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering the same. is there a way to always get latest-rel?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

latest will ensure the current EPP changes work with the newest changes of EM and NM. If we use a release tag, we need to update the image every time there is a new release. If we prefer a stable environment by using a release tag, I'd try and see if I can refer to these env variables from somewhere else, so we don't have to remember to bump them in the test itself.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used curl to get the latest release for the Eventing Manager. Same can be done for NATS Manager as soon as there is a new release.

EPP_IMAGE: europe-docker.pkg.dev/kyma-project/dev/eventing-publisher-proxy:PR-${{ github.event.number }}

jobs:
e2e-nats:
runs-on: ubuntu-latest
steps:
- name: Checkout EPP repository
uses: actions/checkout@v4
with:
path: main

- name: Checkout EM repository
uses: actions/checkout@v4
with:
repository: kyma-project/eventing-manager
path: eventing-manager

- name: Checkout NM repository
uses: actions/checkout@v4
with:
repository: kyma-project/nats-manager
path: nats-manager

- name: Cache binaries
id: cache-binaries
uses: actions/cache@v3
with:
path: bin
key: ${{ runner.os }}-bin

- name: Install k3d tools
run: |
make -C eventing-manager/hack/ci/ install-k3d-tools

- name: Install Kyma CLI & setup k3d cluster using kyma CLI
run: |
make -C eventing-manager kyma
make -C main/hack/ci/ create-k3d
kubectl version
kubectl cluster-info

- name: Create Namespace
run: |
kubectl create ns kyma-system

- name: Setup NATS Manager
run: |
make -C nats-manager install

- name: Deploy NATS Manager
run: |
make -C nats-manager deploy IMG=$NATS_MANAGER_IMAGE

- name: Create NATS CR and resources and check configuration
run: |
make -C nats-manager e2e-setup

- name: Setup Eventing Manager
run: |
make -C eventing-manager install

- name: Deploy Eventing Manager
run: |
make -C eventing-manager deploy IMG=$EVENTING_MANAGER_IMAGE

- name: Wait for build job to succeed
uses: kyma-project/wait-for-commit-status-action@2b3ffe09af8b6f40e1213d5fb7f91a7bd41ffb20
with:
context: "pull-eventing-publisher-proxy-build"
commit_ref: "${{ github.event.pull_request.head.sha }}" # Note: 'github.event.pull_request.head.sha' is not same as 'github.sha' on pull requests.
timeout: 600000 # 10 minutes in milliseconds
# The check interval is kept long otherwise it will exhaust the GitHub rate limit (More info: https://docs.github.com/en/rest/overview/resources-in-the-rest-api?apiVersion=2022-11-28#rate-limiting)
check_interval: 60000 # 1 minute in milliseconds
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
GITHUB_OWNER: "${{ github.repository_owner }}"
GITHUB_REPO: "eventing-publisher-proxy"

- name: Setup Eventing Manager with new EPP image
run: |
kubectl patch deployment eventing-manager -n kyma-system --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/env/8/value", "value": "'$EPP_IMAGE'"}]'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if we later do any changes to ENVs of eventing-manager deployment, will this command fail? as I see the indexes are hard coded 0 and 8?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can get the json and use jq to find the right env with key and replace its value and apply it back?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I'll check to improve the patch command to not rely on the indexes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


- name: Create eventing CR and resources and check configuration
run: |
make -C eventing-manager e2e-setup

- name: Setup subscriptions for testing
run: |
make -C eventing-manager e2e-eventing-setup

- name: Test eventing
run: |
make -C eventing-manager e2e-eventing
mfaizanse marked this conversation as resolved.
Show resolved Hide resolved

- name: On error get NATS CR
if: failure()
run: |
kubectl get nats -n kyma-system -o yaml

- name: On error get Eventing CR
if: failure()
run: |
kubectl get eventing -n kyma-system -o yaml
7 changes: 7 additions & 0 deletions hack/ci/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
KYMA_CLI ?= "/home/runner/work/eventing-publisher-proxy/eventing-publisher-proxy/eventing-manager/bin/kyma-unstable"
CLUSTER_NAME ?= kyma
REGISTRY_PORT ?= 5001

.PHONY: create-k3d
create-k3d: ## Create k3d with kyma CRDs.
"${KYMA_CLI}" provision k3d -p 8081:80@loadbalancer -p 8443:443@loadbalancer --registry-port ${REGISTRY_PORT} --name ${CLUSTER_NAME} --ci
15 changes: 15 additions & 0 deletions hack/get_kyma_file_name.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
#!/bin/bash

function get_kyma_file_name () {

local _OS_TYPE=$1
local _OS_ARCH=$2

[ "$_OS_TYPE" == "Linux" ] && [ "$_OS_ARCH" == "x86_64" ] && echo "kyma-linux" ||
[ "$_OS_TYPE" == "Linux" ] && [ "$_OS_ARCH" == "arm64" ] && echo "kyma-linux-arm" ||
[ "$_OS_TYPE" == "Windows" ] && [ "$_OS_ARCH" == "x86_64" ] && echo "kyma.exe" ||
[ "$_OS_TYPE" == "Windows" ] && [ "$_OS_ARCH" == "arm64" ] && echo "kyma-arm.exe" ||
[ "$_OS_TYPE" == "Darwin" ] && [ "$_OS_ARCH" == "x86_64" ] && echo "kyma-darwin"
}

get_kyma_file_name "$@"