Analytics backend long term design notes

[joe-getcouragenow]

This is just a running design doc

Good overview of same analysis from Grafana lead: https://gouthamve.dev/tempo-a-game-of-trade-offs/

grafana tempoDB.

Can run as 100% local, or with s3 store.

Have they tested it with minio in distributed setup ? Do they support it ? Do they run it with minio gateway ? https://docs.min.io/docs/minio-gateway-for-s3.html

Retention times we need are way higher than normal, and they are controlled here:
https://grafana.com/docs/tempo/latest/configuration/#compactor

Thanos

Another Option from tempo DB is Thanos
https://github.com/thanos-io/thanos

• uses the minio client and store which makes it more agnostic for us: https://thanos.io/tip/thanos/storage.md/
• Its single binary :)
• huge adaption and project is run by Improbable and so likely to not be hampered.
• has configs fully setup for Grafana dashboard: https://github.com/thanos-io/thanos/blob/master/examples/dashboards/dashboards.md
• uses jssonet and mixin too: https://github.com/thanos-io/thanos/tree/master/mixin
• Stateless components
• Stable StoreAPI between services
• Historic metrics and the whole state are persisted to object storage
• Last but not least it is a CNCF project

observatorium

https://github.com/observatorium/observatorium

• uses thansos
• uses OTel
• suports logs, traces, metrics.
• uses jsonnet and all the bits we do for deployments: https://github.com/observatorium/deployments/blob/master/Makefile
  • like how they do it. We should maybe also adopt this ?

Where a system demands minio, https://github.com/RTradeLtd/s3x might help.