Monitoring of snowflake servers

[francisco-core]

Track our usage of servers:

https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40121

[francisco-core]

I looked a bit into this and here are my findings:

We need three components

1. snowflake logging + expose to prometheus (to extract data from snowflake and present to prometheous)
2. promethous + grafana instance (to visualize data)
3. authentication in all places to avoid exposing granular metrics

1. Snowflake logging

Upstream initiatives (1, 2) to have snowflake expose directly data to prometheus appear to be stalled.

However, I found this work, which gets around that limitation by logging snowflake data to a file and then reading from that file and exposing metrics to prometheous.

[francisco-core]

Investigation Results

snowflake.py

# This Script uses the following dependencies
# pip install nums-from-string
# pip install datetime
#
# To Run this script type:
# python main.py <Log File Name>
#
# The default <Log File Name> is ./docker_snowflake.log
#
# Example:
# python main.py snow.log
#
# Written By Allstreamer_
# Licenced Under MIT
#
# Enhanced by MariusHerget
# Further enhanced and modified by mrdrache333

import sys
import re
from datetime import datetime, timedelta
from http.server import HTTPServer, BaseHTTPRequestHandler

def nums_from_string(string):
    return [int(num) for num in re.findall(r"\d+", string)]

def readFile():
    # Read in log file as lines
    lines_all = []
    with open(logfile_path, "r") as file:
        lines_all = file.readlines()
    return lines_all


# Catchphrase for lines who do not start with a timestamp
def catchTimestampException(rowSubString, timestampFormat):
    try:
        return datetime.strptime(rowSubString, timestampFormat)
    except Exception:
        return datetime.strptime("1970/01/01 00:00:00", "%Y/%m/%d %H:%M:%S")


# Filter the log lines based on a time delta in hours
def filterLinesBasedOnTimeDelta(log_lines, hours):
    now = datetime.now()
    length_timestamp_format = len(datetime.strftime(now, timestamp_format))
    return filter(lambda row: now - timedelta(hours=hours) <= catchTimestampException(row[0:length_timestamp_format],
                                                                                      timestamp_format) <= now,
                  log_lines)


# Convert traffic information (in B, KB, MB, or GB) to B (Bytes) and add up to a sum
def get_byte_count(log_lines):
    byte_count = 0
    for row in log_lines:
        symbols = row.split(" ")

        # Use a dictionary to map units to their byte conversion values
        units = {
            "B": 1,
            "KB": 1024,
            "MB": 1024 * 1024,
            "GB": 1024 * 1024 * 1024
        }

        # Use the dictionary to get the byte conversion value for the current unit
        byte_count += int(symbols[1]) * units[symbols[2]]
    return byte_count


# Filter important lines from the log
# Extract number of connections, uploaded traffic in GB and download traffic in GB
def getDataFromLines(lines):
    # Filter out important lines (Traffic information)
    lines = [row.strip() for row in lines if "In the" in row]
    lines = [row.split(",", 1)[1] for row in lines]

    # Filter out all traffic log lines who did not had any connection
    lines = [row for row in lines if nums_from_string(row)[0] != 0]

    # Extract number of connections as a sum
    connections = sum([nums_from_string(row)[0] for row in lines])
    
    # Extract upload and download data
    lines = [row.split("Relayed")[1] for row in lines]
    upload = [row.split(",")[0].strip() for row in lines]
    download = [row.split(",")[1].strip()[:-1] for row in lines]

    # Convert upload/download data to GB
    upload_gb = get_byte_count(upload) / 1024 / 1024 / 1024
    download_gb = get_byte_count(download) / 1024 / 1024 / 1024
    
    # Return information as a dictionary for better structure
    return {'connections': connections, 'upload_gb': upload_gb, 'download_gb': download_gb}


def main():
    # Read file
    lines_all = readFile()
    
    # Get the statistics for various time windows
    # e.g. all time  => getDataFromLines(lines_all, 24)
    # e.g. last 24h  => getDataFromLines(filterLinesBasedOnTimeDelta(lines_all, 24))
    # e.g. last Week => getDataFromLines(filterLinesBasedOnTimeDelta(lines_all, 24 * 7))
    stats = {
        'All time': getDataFromLines(lines_all),
        'Last 24h': getDataFromLines(filterLinesBasedOnTimeDelta(lines_all, 24)),
        'Last Week': getDataFromLines(filterLinesBasedOnTimeDelta(lines_all, 24 * 7)),
    }

    # Print all the results in the Prometheus metric format
    for time in stats:
        stat = stats[time]
        # Write the text message to the response body
        print(
            f"snowflake_served_people{{time=\"{time}\"}} {stat['connections']}\n" +
            f"snowflake_upload_gb{{time=\"{time}\"}} {round(stat['upload_gb'], 4)}\n" +
            f"snowflake_download_gb{{time=\"{time}\"}} {round(stat['download_gb'], 4)}"
        )

# Format of your timestamps in the beginning of the log
# e.g. "2022/01/01 16:50:30 <LOG ENTRY>" => "%Y/%m/%d %H:%M:%S"
timestamp_format = "%Y/%m/%d %H:%M:%S"

# Log file path from arguments (default: ./docker_snowflake.log)
logfile_path = sys.argv[1] if len(sys.argv) > 1 else "./docker_snowflake.log"

main()

[anadahz]

Notes