Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: setup devenv observability #1105

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

djordon
Copy link
Collaborator

@djordon djordon commented Dec 12, 2024

Description

Closes #1104

This just sets up metrics. Logs will come in a separate PR.

Changes

  • Add prometheus, grafana, and vector, as devenv services.
  • Export metrics in devenv.
  • Log in JSON in devenv.

Testing Information

I tested this locally and everything worked as expected.

Checklist:

  • I have performed a self-review of my code

- observability

grafana:
image: grafana/grafana:10.2.3-ubuntu
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: 11.4.0

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 28a0436.

@@ -610,3 +614,54 @@ services:
command: "./wait-for-it.sh mariadb:3306 --timeout=720 --strict -- ./start.sh"
profiles:
- bitcoin-mempool

prometheus:
image: prom/prometheus:v2.45.2
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: v3.0.1

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 28a0436.

Copy link
Collaborator

@aldur aldur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is nice! 👏

Comment on lines 15 to 21
type = "prometheus_scrape"
endpoints = [
"http://host.docker.internal:9181",
"http://host.docker.internal:9182",
"http://host.docker.internal:9183",
]

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh! I didn't knew we could do this with vector! So we can use the same stack to push signer metrics (if they allow us to!)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, it's a great tool.

Makefile Show resolved Hide resolved
@djordon djordon requested a review from matteojug January 28, 2025 18:46
Copy link
Collaborator

@matteojug matteojug left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this locally but I couldn't find any metric/dashboard on the grafana instance.

@@ -67,7 +67,7 @@ exec_run() {
. "$ENV_PATH/signer-$i.env" \
&& export RUST_LOG="$LOG_SETTINGS" \
&& export SIGNER_SIGNER__BOOTSTRAP_SIGNING_SET="$BOOTSTRAP_SIGNER_SET" \
&& cargo run --bin signer -- --config "$SIGNER_CONFIG" --migrate-db > "$PWD/target/signer-$i.log" 2>&1 \
&& cargo run --bin signer -- --config "$SIGNER_CONFIG" --migrate-db -o json > "$PWD/target/signer-$i.log" 2>&1 \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a flag to control the -o json? Otherwise checking the logs when running this natively will be quite annoying.

@@ -30,6 +30,10 @@ x-common-vars:
- &REWARD_RECIPIENT ${REWARD_RECIPIENT:-STQM73RQC4EX0A07KWG1J5ECZJYBZS4SJ4ERC6WN} # priv: 6ad9cadb42d4edbfbe0c5bfb3b8a4125ddced021c4174f829b714ccbf527f02001
- &EXIT_FROM_MONITOR 1 # set to "1" to automatically shut down via monitor.ts

volumes:
# prometheus-data: {}
grafana-data1: {}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: do we want the trailing 1?
qq: why do we want a volume for one but not the other?

- grafana-data1:/var/lib/grafana
- ./grafana/provisioning/:/etc/grafana/provisioning/
ports:
- 3030:3000
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This port is already used for the blocklist client.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: In Review
Development

Successfully merging this pull request may close these issues.

[Feature]: Setup devenv observability
3 participants