Relay(bmr) health monitoring

[andrii-kl]

From the moment when third-party systems began to use our bridge, there was a need to provide the ability to unify whether the relay is working or not. Since many business processes depend on it, it is easier to suspend the system if the relay does not work than to solve problems with clients whose money is frozen.
The importance of such functionality is obvious, there is only a question on the way of implementation.

I can see only one point for discussion:

Can we provide public access to the health check http relay endpoint?

Yes - the implementation will be simple

No - In this case I can see next solution: We can build a service that will have an http access to the all relays.
This service will collect, process and provide all information about all relays.
This service also can provide different information depending on you role. It should simplify the relay managing process.
But this decision will take much longer.

[CyrusVorwald]

Check out the icon-bridge-monitoring. It's hard coding some channels which is why I've refrained from open sourcing it, but I would like that to be changed.

The relay endpoint should not be public. Public info can be created via upptime. It would be great if you can add this.

[CyrusVorwald]

The way I understand it is that the service that queries the health of the relay should be ip whitelisted and allowed to access read-only methods, and that service can be accessed publicly. Why can't we do a relay health check call?

[andrii-kl]

Ip whitelist is a good idea but we can't use it for the GitHyb Action, because it doesn't have a static IP.

https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/about-githubs-ip-addresses

[CyrusVorwald]

Should be doable via self hosted runners.

Also, there's this https://github.blog/changelog/2022-09-01-github-actions-larger-runners-are-now-in-public-beta/#fixed-ip-ranges

[andrii-kl]

This should work. But I'm thinking about different solution that can cover two problems.
So we have two needs:

1. Relay health check for a third party services.
2. Relay monitoring tools for internal use with extensive information. (Relay account balances, current sync/processing heights, system metrics, memory usage/threads, etc.)

upptime Can solve only the first one.

I have read a bit more about Grafana, and it looks a good choose for both cases.
I found a good solution for the first case, Grafana has a functionality to make a public statistic dashboard. I think it can be really cool feature.

https://grafana.com/docs/grafana/latest/dashboards/dashboard-public/

[CyrusVorwald]

The 2nd bullet point is covered between a mix of Golang code on the relay and the icon-bridge-monitoring repo. ICONLOOP monitors the relay with this information internally.

[robcxyz]

My 2 cents, have you considered instrumenting your application with prometheus? It's relatively easy to setup (I can PR) and get going to build dashboards in grafana of any custom metric within the relay. Thinking things like last_processed_block or num_transactions_sent which would be useful to build alarms around. Anyways, the metrics are easy to integrate into any data that exists in the runtime.

Let me know what you think. Also here are a couple examples how I instrument the tracker backend.

• Init metrics
• Create metrics
• Count number of errors in application

I then have some internal grafana dashboards that I use to watch backfill runs to fill in tracker data. Most of my alarms are built from other sources but the relay I am sure will have some metrics that when they go out of wack, someone should get a call.

Anyways, my 2 cents and can PR some of the setup scripts and a path towards instrumentation.

[andrii-kl]

Yes, we have some developments in the direction of Prometheus stack. I don't know what condition they are in yet. But in any case, it will be very valuable to get your recommendations.

But now the discussion is more about how to provide a public health check without revealing the relay.

[andrii-kl]

If we decide to try the Prometheus stack, the component structure can be next.

In this way only Grafana app expose public access.

Some points:

1. Admin / developer password to the system should be resistant to brute force.

2. Hacking this machine should not cause damage to the system.

[robcxyz]

@andrii-kl - Hey so a few quick thoughts.

• I think these metrics endpoints can be exposed publicly initially but if they are in a private network, that is fine. Lots of layers of security you can add like firewall rules / tls certs but these can all be layered in on an as needed basis. But if you are running prometheus outside of your network then you will need to figure out routes. Inside then it is no big deal but ultimately if I am not mistaken there are going to be many of these relays on different private networks in which case you will want to figure something out that can scrape metrics from all these locations. If this is the case then I would not stress the networking side of this. Prometheus metrics ports should not be too much of a security issue at least initially. Prometheus just needs to access them.
• Grafana can be deployed with both public read only and password protected dashboards. Not sure if you want anything public but it is nice to also show stats of the network this way.
• Not sure what the github actions are there for. Can you explain that?

RE your points:

1. Passwords can be complex etc. There can be multiple users provisioned manually or you can hook it up to oauth but that gets more complicated and probably not needed.
2. Prometheus should be in its own namespace / VM. There should be no issues here.

Ultimately if I were to monitor these relays, I would setup a new prometheus distribution using the prometheus operator to get prometheus and alertmanager on k8s. Alertmanager would send alarms to pagerduty and there should be an on call rotation for that. This is all because it is trivial to set it up this way since I have done a lot of the hard parts already. If you want to setup your own VM with prometheus, then I would simply fork dockprom and build everything out that way but in my experience this is harder to manage than k8s. If you are going to get your own k8s cluster eventually, I would just focus on the operator instead of a VM.