Initial version

[weissreto]

Monitoring a solution with a Reverse Proxy, Axon Ivy Engine and a Backend with a REST and SOAP endpoint. OpenTelemetry is used to export metrics to Prometheus. Grafana is used to display the data in a dashboard

[alexsuter]

I've worked with Denis on this topic a while ago https://github.com/axonivy/monitoring. Seems like that you use the JMX exporter. I guess Prometheus is requesting the Axon ivy Engine from time to time? The issue is (or at least was) that not all JMX values are cached. Some of them were calculated at runtime. Some of them were highly resource intensive. So if we add this kind of monitoring we gave a lot more pressure to the Axon Ivy Engine.

Furthermore I think we should design very carefully the prometheus API. Because customer's will create custom Graphana dashboards and we can not change keys anymore without migration pain.

The backend is highly outdated. Not sure how long we are able to build the backend. Maybe there are some simple docker containers out there, which we can use? Or even better, building a simple backend from source directly here.

[weissreto]

Seems like that you use the JMX exporter. I guess Prometheus is requesting the Axon ivy Engine from time to time? The issue is (or at least was) that not all JMX values are cached. Some of them were calculated at runtime. Some of them were highly resource intensive. So if we add this kind of monitoring we gave a lot more pressure to the Axon Ivy Engine.

-> This is not the case the JMX exporter exports the value to Prometheus. How often this is done can be configured. An there is a list of rules which MBeans are exported. There should not be any performance issues here.

Furthermore I think we should design very carefully the prometheus API. Because customer's will create custom Graphana dashboards and we can not change keys anymore without migration pain.

-> This is only a sample of how customers can do monitoring by themselves. The keys are either standard to OT or can be configured in this file https://github.com/axonivy/docker-samples/pull/196/files#diff-bcde297a529515ad4cb44a5c078a71a4257523904d1189496f12ab978103ab52 by the customer itself.

The backend is highly outdated. Not sure how long we are able to build the backend. Maybe there are some simple docker containers out there, which we can use? Or even better, building a simple backend from source directly here.

-> The same backend is also used in ivy-tracing-jaeger. I will update the backend in another PR that will do it in both examples.

[alexsuter]

I'm fine if we have no performance issues and we don't need to keep the jmx interface stable for ever.

For the backend I would use an nginx with static file service. There you can put a json file for a rest endpoint and a xml file for a web service endpoint. This should be enough for demo and should work for a long long time.

[alexsuter]

Axon Ivy Engine instead of Axon.Ivy