Nodies Monitoring

Nodies Monitoring is a customizable and extensible monitoring solution for monitoring host machine metrics, container metrics, node status and node logs.

Our project is split into a server (monitoring) stack, and a client (exporter) stack.

Please see architecture documentation for a deeper analysis of the server/client stack.

Want to Contribute?

TBD instructions on how to contribute to this project.

Table of content

• Installation
  • Dependencies
• Getting started
  • Settings
  • Server Setup
  • Client Setup
  • Run Individual Clients
• Customization
  • Datasources
  • Dashboards
  • Alerts
• Links/Contact

Installation

NOTE: These dependencies are required on both the server and client stack.

Note: Tested and recommended installation on Ubuntu 22.04.1 LTS host OS

Dependencies

Python

Python 3.10.6

Docker

Uninstall existing docker

sudo apt-get remove docker docker-engine docker.io containerd runc

Install required packages

sudo apt-get update
sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

Add docker official GPG key

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

Setup docker repo

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Update apt package index

sudo apt-get update

Install latest docker

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin

Install python3-pip

apt install python3-pip

Install requirements

pip3 install -r requirements.txt

Getting Started

Configuration Setup (REQUIRED)

NOTE: This step is required on both the server and the client stack. When setting up this stack, you should follow the principles of least privilege when allowing users to access your exposed ports by setting up network or host based firewalls.

1. Run sudo python3 setup.py to generate a template settings.yml.new file.
2. Update clients.promtail.loki_endpoint with the ip address of the host that will run the monitoring stack (loki, grafana, minio, prometheus, alertmanager)
3. Update server.prometheus.exporter_endpoint with the endpoints any services of the exporter_stack (blockchain_exporter, cadvisor, node_exporter, promtail)
4. Update server.alerts.contactpoints with the contact points of your alert recievers to send grafana-managed alerts to.
5. Update clients.promtail.log_root_path with the root path of your nginx logs.
6. Rename settings.yml.new to settings.yml

---

Server setup

On your monitoring server:

1. Run setup.py again to save your configurations and set folder permissions.

sudo python3 setup.py

NOTE: Sudo is required here to set some folder permissions

2. Change directory into the server subfolder, and boot up all server services

cd nodies_monitoring/server && docker compose up -d

---

Client setup

On your exporter server:

Blockchain exporter setup

1. Include your blockchain accessible endpoints inside a chains.json (./templates/chains.json) file. An example file is provided for you in chains.example.json.

Log aggregation setup (Promtail)

2. Ensure that your access logs are formatted with the following format:

    log_format json_combined escape=json
    '{'
        '"region":"us-east-1",'
        '"time_local":"$time_local",'
        '"request":"$scheme://$host$request_uri",'
        '"method":"$request_method",'
        '"protocol":"$server_protocol",'
        '"status":"$status",'
        '"request_body":"$request_body",'
        '"body_bytes_sent":"$body_bytes_sent",'
        '"http_referrer":"$http_referer",'
        '"http_user_agent":"$http_user_agent",'
        '"upstream_addr":"$upstream_addr",'
        '"upstream_status":"$upstream_status",'
        '"remote_addr":"$remote_addr",'
        '"remote_user":"$remote_user",'
        '"upstream_response_time":"$upstream_response_time",'
        '"upstream_connect_time":"$upstream_connect_time",'
        '"upstream_header_time":"$upstream_header_time",'
        '"request_time":"$request_time"'
    '}';

Note: Replace the region value with a unique region identifer, i.e us-east-1, eu-central-1, etc based off where your nginx server is located.

3. Run setup.py again to save your configurations and set folder permissions.

sudo python3 setup.py

NOTE: Sudo is required here to set some folder permissions

4. Change directory into the clients subfolder, and boot up all client services

cd nodies_monitoring/server && docker compose up -d

---

Run individual clients

setup.py has an optional CLI flag that allows control over which clients are ran on the exporter stack. For example, if you don't want to run log aggregation, then you can disable the log shipper promtail.

sudo python3 setup.py --clients blockchain_exporter cadvisor node_exporter

---

Customization

Datasources

Default datasources have been provisioned in ./templates/datasources

To add additional datasources, please refer to grafana datasource documentation

Dashboards

Default dashboards have been provisioned in ./server/grafana/dashboards

To add additional dashboards, please refer to grafana dashboard documentation

Alerts

Default alerting has been provisioned in ./server/grafana_provisioning/alerting

To add additional alerting, please refer to grafana alerting documentation

Links/Contact

For any inquiries, please reach out to PoktBlade(PoktBlade#5970) or poktdachi(dachi#0005) on the pokt discord

Pokt Log

Chain Metrics

Machine Metrics

Container Metrics