Pull Observability Operator objects from separate location

[cmwylie19]

Pull manifests from a separate location

Feature Request

Background

The Observability operator will be installed by default on OpenShift Dedicated clusters. We want to leverage the operator to monitor ISV metrics, however, there is still a need to configure the operator for a given ISV in the form of ServiceMonitors and PrometheusRules. Given that each ISV will have specific criteria from which to trigger alerts, record metrics, etc, we need a more automated form of configuration for the operator.

Description of proposed solution

Considering that users of this operator could exist in disconnected environments, we see two potential/necessary solutions for this problem. Adding the ability to pull manifests/objects from a remote url (manifestURL), and the ability to pull config from a secret or configmap.

The Kafka Observability Operator for reference. https://github.com/redhat-developer/observability-operator

Motivation

Low maintenance, repeatable configuration on OpenShift Dedicated environments.

thanks for the consideration

[simonpasquier]

My first instinct is to say that it's a configuration management issue which is a bit away from the Observability Operator goals. I guess that each ISV will have to configure resources other than monitoring anyway?

[cmwylie19]

@simonpasquier I hear you when you say config management issue, and I respectfully agree that maybe the Observability Operator shouldn't or does not care about the way in which it is configured, however, i think this transcends the scope of being just a config management issue and could add a lot of value on top of the already very powerful Observability Operator by providing more flexibility, efficiency and deployment options when working on disconnected or even OSD environments.

Let's think of a common use-case - monitor a specific app, configure alerts for the app, and configure the alerts to trigger a page (say from PagerDuty) when something goes wrong. This would involve ServiceMonitors to scrape the app itself, another ServiceMonitor to federate cluster style metrics (say from openshift-monitoring), the PrometheusRules from which to trigger alerts, an AlertManagerConfig with credentials to send alert to PagerDuty, the remoteWrite info if you are sending your metrics to Observatorium or another prometheus, and necessary RBAC required to achieve these pieces.

We know this flow well and can attest that it does work very well indeed. However, this is a lot of steps. Imagine being able to replicate this process by supplying a manifestURL, or secret/configmap from which to obtain manifests to ultimately configure the Observability Operator. You still have a clean infrastructure-as-code style operator but now configuration can be consolidated essentially to just one step.

This could also lower the barrier of entry and democratize usage the Observability Operator. Even a more unexperienced engineer to monitoring would be able to point the Observability Operator spec to a manifestURL or secret that has been pre-populated with the manifests created by a platform operator or SRE style engineer.

To answer your last questions with regards to each ISV will have to configure resources other than monitoring anyway- that is the motivation for feature-request. Each ISV will have a specific configuration to support monitoring of their given apps.

Thanks for the consideration and sorry for the long read. Just a thought though, do you think this would add value @simonpasquier? Or would this be more entirely out of scope?

[simonpasquier]

To answer your last questions with regards to each ISV will have to configure resources other than monitoring anyway- that is the motivation for feature-request. Each ISV will have a specific configuration to support monitoring of their given apps.

My comment was more that ISV may have to configure resources other than monitoring (deployments? services?). If the answer is "no we only ever need to configure monitoring resources for ISV" then yes it might make sense for OBO to grow this capability but if the answer is "yes ISVs will have to deploy other kubernetes resources not related to monitoring" then you'd need something else anyway.

Assuming that we go this path, OBO would need to get additional permissions to create/update/delete monitoring resources. And we need to think about the full lifecycle:

• how do you remove a resource?
• how do you provision secrets?

To be clear, I'm not saying that OBO isn't the right tool here but I want to make sure that we evaluate carefully the consequences/options :)

[cmwylie19]

Yes, these are all extremely valid points, @simonpasquier. Let's put this on hold for now. I may close this ticket. It seems like it may not be needed in the end for our team. Thank you!

[simonpasquier]

As you like but we can definitely discuss it during one of the projet's office hours :)

[cmwylie19]

@redmikhail