Adding kustomize templates for k8s onboarding

[gizas]

• Enhancement

What does this PR do?

WHAT: Introduces the onboarding templates of kustomize for k8s observability
WHY: #4657

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [] I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

How to test this PR locally

1. Clone this repo
2. Change to directory elastic/elastic-agent/deploy/kubernetes
3. Use makefile to generate templates

GENERATEKUSTOMIZE=true make ci-create-kustomize

4. test the templates with

cd elastic-agent-kustomize/default
kubectl apply -k elastic-agent-managed
or 
kubectl apply -k elastic-agent-standalone

• Secrets are created here https://github.com/elastic/elastic-agent/pull/4754/files#diff-67cc02e5c01acb0ecf253b716b2c223ecd4e8366e865a3ff41b9817bfd3243f7R9
• Are referenced here https://github.com/elastic/elastic-agent/pull/4754/files#diff-9aeecc78b3187a6c2b495ae90fdf0555402dc580422a3192cd013696f742a83fR39

Examples of online references:

kubectl kustomize https://github.com/elastic/elastic-agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-managed\?ref\=kustomizepoc

kubectl kustomize https://github.com/elastic/elastic-agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-managed\?ref\=main

kubectl apply -k https://github.com/elastic/elastic-agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-managed\?ref\=v8.13.1

Testing for onboarding procedure

Managed:

❯ kubectl kustomize elastic-agent-kustomize/default/elastic-agent-managed/ | sed -e "s/JUVOUk9MTE1FTlRfVE9LRU4l/base64_ENCODED_ENROLLMENT_TOKEN/g" -e "s/%FLEET_URL%/https:\/\/localhost:9200/g" | kubectl apply -f-

Standalone

❯ kubectl kustomize elastic-agent-kustomize/default/elastic-agent-standalone/ | sed -e "s/JUFQSV9LRVkl/<base64_encoded_APIKEY>/g" -e "s/%ES_HOST%/https:\/\/localhost:9200/g" -e "s/%CA_TRUSTED%/ca_trusted_fingerprint/g" -e "s/%ELASTIC_AGENT_ID%/12345/g" | kubectl apply -f-

Eg.

❯ kubectl kustomize https://github.com/elastic/elastic-agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-managed\?ref\=kustomizepoc |  sed -e 's/JUFQSV9LRVkl/JUFQSV9LRVkl/g' -e "s/%ES_HOST%/https:\/\/localhost:9200/g" -e "s/%CA_TRUSTED%/ca_trusted_fingerprint/g" -e "s/%ELASTIC_AGENT_ID%/12345/g" |  kubectl apply -f-

Note: API_KEY should be given as bas64 encoded values from Kibana

Base 64 encoding decding:

❯ echo '${enrollment_token}' | base64
JHtlbnJvbGxtZW50X3Rva2VufQo=
❯ echo '${api_key}' | base64
JHthcGlfa2V5fQo=
> echo JHtlbnJvbGxtZW50X3Rva2VufQ== | base64 -D
${enrollment_token}

Images

Successful enrollement in managed

Successful Installation of kube-state-metrics and

kgp -n kube-system
NAME                                         READY   STATUS    RESTARTS       AGE
coredns-5d78c9869d-b6np9                     1/1     Running   6 (7d8h ago)   22d
coredns-5d78c9869d-p62sz                     1/1     Running   6 (7d8h ago)   22d
elastic-agent-tvskp                          1/1     Running   0              11s
etcd-kind-control-plane                      1/1     Running   0              7d8h
kindnet-qpxrb                                1/1     Running   6 (7d8h ago)   22d
kube-apiserver-kind-control-plane            1/1     Running   0              7d8h
kube-controller-manager-kind-control-plane   1/1     Running   6 (7d8h ago)   22d
kube-proxy-96hfl                             1/1     Running   6 (7d8h ago)   22d
kube-scheduler-kind-control-plane            1/1     Running   6 (7d8h ago)   22d
kube-state-metrics-5f89fb6d84-8f2sm          1/1     Running   0              11s

Related issues

• Relates #[POC]Automating the deployment of kube-state-metrics during K8s Observability onboarding  #4657

[mergify]

This pull request does not have a backport label. Could you fix it @gizas? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[constanca-m]

I am a bit confused. What is the different between default and onboarding directories?

[mykolaharmash]

@gizas one other thing came up about the elastic agent version. Right now in the template, version is set to non-existing 8.15.0. Can we somehow ensure that we always have a valid version there, or alternatively expose it as well as a variable, so for onboarding use-case we can set it on our side?

[gizas]

@mykolaharmash the way it works is that the Makefile computes the agent image version based on this file:

elastic-agent/version/version.go

Line 7 in a6dba3d

const defaultBeatVersion = "8.15.0"

So all the daemonset file has the elastic-agent:8.15.0 inside. This is the only place where I see that this image exists

So i guess another sed like: sed -e "s/elastic-agent:8.15.0/elastic-agent:VERSION/g" can do the magic.

Also, I think this is only now a problem until we merge the PR. After that we wont have the problem and the CI will make sure that this is always correct. And the user can choose the version with the following use of ref:
kubectl kustomize elastic-agent-kustomize/default/elastic-agent-managed?ref=v8.15.1

[gizas]

Updating my answer above, I think needs to be mandatory for kibana UI onboarding to update the link like
kubectl kustomize <url>?ref=v8.15.1 always with the existing version of the system of the user. Otherwise we will use always the main branch and this might break in the periods where the version.go file is ahead from the latest released versiosn

[flash1293]

@gizas Just to make sure - is there an automation that there is a version of the template for every stack version? Or is this something that needs to be done manually? We should make sure that this won't break in the next version.

[gizas]

is there an automation that there is a version of the template for every stack version?

Yes as part of this PR, I have added this in the ci:
https://github.com/elastic/elastic-agent/pull/4754/files#diff-744561ff1e6cb54740e099862a03543c589ecd5416130dab47ce3c4f1b6130a1R32

[mykolaharmash]

Got it, thank you! I'll update the onboarding logic so that URL always has the version ref

[gizas]

@cmacknz or @blakerouse if no other objection can you please approve this? We will need one more to merge

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[mykolaharmash]

@gizas one more things about the Elastic Agent version, am I correct that even after your change is merged, we still need to wait for Agent v8.15.0 to be released in order to use the new template in onboarding? Because if we reference ref=8.14.1 it will have the old one, right?

[flash1293]

Good point @mykolaharmash , ideally we could backport this so it will work for 8.14.x as well on serverless before the 8.15 release which is still some time in the future?

[gizas]

@mykolaharmash I added backport-v8.14.0 label. Thanks!
I will check the backport

@elastic/elastic-agent-control-plane team can I have an approval please?

[pierrehilbert]

@blakerouse / @andrzej-stencel could you please review here?