Problem in Kibana configuration

[antru6]

While installing the metrics subsystem, the status of pod "Kobe-kibana-configuration" is always "NotReady".

The machine used for setting up KOBE is a server that runs a 64-bit Debian GNU/Linux 10 server operating system. On this machine, I have installed Kubernetes (v1.22.4), nfs-commons (v1:1.3.4-6), minikube (v1.24.0), istio (v1.12.0), and helm (v2.17.0)

Issue identified by @chengsijin0817

1_Pods_status&kobe-kibina-configuration_log.pdf
2_kobe-kibina-configuration_NotReady.pdf

[gmouchakis]

Can you please provide the output of command kubectl describe pod <kobe-kibana-configuration-pod-name>?

[chengsijin0817]

@antru6, thank you for keeping track of the issue here.
Hi @gmouchakis, here is the output when I run the command as you indicated. Please check it.
out.txt

[chengsijin0817]

In addition, I just tried to manually install the metrics subsystem using the commands under "or alternatively the following". The problem still exists.

[gmouchakis]

@chengsijin0817 thanks for the input.

It seems like the issue is related to Istio. kobe-kibana-configuration is used to import the predefined dashboards and configurations in kibana through a HTTP request (see https://github.com/semagrow/kobe/blob/devel/operator/deploy/efk-config/kobe-kibana-configuration.yaml). For some reason this fails with error "No healthy upstream".

As a workaround, you could open http://<your_kibana_address>/app/kibana#/management/kibana/objects?_g=() and manually import the attached export.txt json file, but since Istio is not working properly it is possible that you'll have to deal with another related issue later on.

Could you use Kobe without Istio until we figure this out?

[antru6]

@gmouchakis So it seems that these Istio versions causes us problems (again...) :P

@chengsijin0817 If it's not too much trouble to you, you can give another try this time with Istio 1.11.3. I have performed some experiments by using this version (along with Kubernetes 1.20.7), so my bet is that it will work this time.

However, if the problem with Istio persists, I have another proposal. You can skip the installation of Istio, and by doing so, the only feature that you will miss is the simulated endpoint delay. In fact, if you don't want to use delays in your experiments, my suggestion is to not use Istio at all because it allocates extra containers and services to operate.

[chengsijin0817]

Thank you both for taking the time to check out this issue.

@gmouchakis So it seems that these Istio versions causes us problems (again...) :P

@chengsijin0817 If it's not too much trouble to you, you can give another try this time with Istio 1.11.3. I have performed some experiments by using this version (along with Kubernetes 1.20.7), so my bet is that it will work this time.

I tried to reset up KOBE using Istio 1.11.3 along with Kubernetes 1.20.7. Unfortunately, the status of kobe-kibana-configuration is still "notReady" (appears to be the same issue).

However, if the problem with Istio persists, I have another proposal. You can skip the installation of Istio, and by doing so, the only feature that you will miss is the simulated endpoint delay. In fact, if you don't want to use delays in your experiments, my suggestion is to not use Istio at all because it allocates extra containers and services to operate.

Thanks a lot for pointing it out. It makes sense to have some experiments without any simulated delay.
However, when I skip the installation of Istio, the development of kobe-operator and metrics subsystem also complains with an error message:
Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": service "istiod" not found

Is there any corresponding configuration or modification I need to make?

[antru6]

@chengsijin0817 Can you please provide us with the output of the commands
kubectl get namespace -L istio-injection and kubectl get pods?

[chengsijin0817]

Sure, here is the output of running the specified commands (with/ without using Istio):
output_withIstio.txt
output_withoutIstio.txt

[antru6]

It seems that in your installation the istio-related containers are injected in the default namespace (where the kibana pod is deployed), and it seems that this causes the problem. Can you run this command and try one more time?
kubectl label namespace default istio-injection-

[chengsijin0817]

That's it! The problem is solved!!!
Now KOBE runs well, with or without Istio. ;-)

Thank you so much for your help and valuable time!
Cheers!!

[antru6]

@chengsijin0817 Thank you again for reporting this problem and for the tests you have done for us. Please do not hesitate to post new issues if you encounter any further problems!

[stasinos]

Issue #47 should help users avoid this pithole in the future. Closing this issue now.