Monitoring Liquidsoap server metrics

[vitoyucepi]

Introduction

Effectively monitoring server health and resource usage is crucial for maintaining optimal performance and reliability of any system. Liquidsoap is no exception.

As a powerful audio streaming server, Liquidsoap has many moving parts that need monitoring, including CPU usage, memory consumption, and garbage collection activity. Keeping tabs on these key metrics allows you to:

• Identify any emerging issues or bottlenecks before they cause disruptions.
• Inform optimization and tuning of configuration and infrastructure.
• Understand typical resource usage patterns for capacity planning.
• Pinpoint abnormalities that need investigation.

In this post, I will cover how to monitor core Liquidsoap server health metrics by:

1. Writing a Liquidsoap script to collect key resource usage data.
2. Visualizing trends and patterns using Python and matplotlib.

Monitoring these indicators will provide invaluable visibility into the inner workings of your Liquidsoap server. Let's get started!

Keeping Tabs on Resource Usage

To start collecting data on Liquidsoap's resource usage, I can write a monitoring script that will:

• Gather CPU, memory, and GC statistics
• Output the metrics to a JSON file
• Run periodically to log over time

Here is an example script:

# Script to collect and log resource usage statistics

def statistics(path, ~id=null(), ~every=30.)
  # The name of the statistics module
  id = string.id.default(id, default="statistics")
  
  # Output file path 
  filename = "#{path}/#{id}_#{time.string()}.json" 
  
  # Open file for writing
  statistics_file = file.open(filename, write=true, append=true)
  
  # Path to the log file where the statistics will be saved
  log.important("Writing server statistics to #{string.quote(filename)}.", label=id)

  def save_statistics()
    # Get CPU usage
    let {user, system} = process.time()
    cpu = {user = user, system = system, uptime = time.up()}
    
    # Get GC stats
    gc = runtime.gc.stat()  
    
    # Get memory usage
    let {pretty, ...memory} = runtime.memory()

    # Stringify metrics as JSON 
    data = json.stringify({cpu = cpu, gc = gc, memory = memory}, compact=true) ^ "\n"

    # Write to file
    statistics_file.write(data)
  end

  # Run every N seconds
  thread.run(save_statistics, every=every, fast=false)
end

This script:

• Collects CPU, memory, and GC metrics using Liquidsoap APIs.
• Writes the data to a JSON file.
• Runs periodically in a thread to log over time.

Next I can visualize this data to monitor usage trends...

Visualizing Metrics with Matplotlib

Now that I have metrics being logged to a JSON file, I can load and visualize the data in Python with matplotlib.

1. Loading the JSON Data

To start, I'll open the JSON log file and parse the data into dictionaries for each metric category:

import json


def read_log(file_name):
    result = {"cpu": {}, "gc": {}, "memory": {}}
    init = True    
    with open(file_name) as file:
        for line in file:
            try:
                record = json.loads(line)
            except json.decoder.JSONDecodeError:
                continue

            # Initialize dicts
            if init:
                init = False
                for parameter_set in result:
                    for parameter in record[parameter_set]:
                        result[parameter_set][parameter] = []
            
            # Extract metrics
            for parameter_set in result:
                for parameter in result[parameter_set]:
                    result[parameter_set][parameter].append(record[parameter_set][parameter])

    return result

This:

• Opens the JSON log file.
• Initializes empty dicts for CPU, memory, and GC.
• Loads each JSON record.
• Extracts metrics into the corresponding dicts.
• Returns structured data ready for plotting.

Now the metrics are loaded into memory and ready for preparation and visualization.

2. Preparing and Plotting the Data

With the metrics loaded, I can now process the data and visualize it.

First I define some helper functions:

from statistics import mean
import matplotlib.pyplot as plt
from scipy.ndimage import uniform_filter1d


# Calculate CPU usage beetwen steps
def cpu_filter(user_time, elapsed_time):
    result = []
    result.append(user_time[0] / elapsed_time[0] * 100)
    for i in range(1, len(user_time)):
        result.append((user_time[i] - user_time[i-1]) / (elapsed_time[i] - elapsed_time[i - 1]) * 100)
    return result


# Calcuate difference in percents
def percent_filter(items, size = 120):
    average = mean(items[:size])
    return [item / average * 100 for item in items]


# Format x-axis as time
def format_date(x, pos = None):
    x = round(x)
    m, s = divmod(x, 60)
    h, m = divmod(m, 60)
    return f'{h}:{m:02d}:{s:02d}'


# Format y-axis memory as MB
def format_memory(x, pos = None):
    return x / 1000 ** 2

Next I can plot the metrics:

def plot_main(data):
    elapsed_time = data['cpu']['uptime']
    user_time = data['cpu']['user']
    system_time = data['cpu']['system']

    user_cpu = cpu_filter(user_time, elapsed_time)
    system_cpu = cpu_filter(system_time, elapsed_time)
    user_cpu_uniform = uniform_filter1d(user_cpu, size = 100)
    system_cpu_uniform = uniform_filter1d(system_cpu, size = 100)
    user_cpu_cumulative = [i[0] / i[1] * 100 for i in zip(user_time, elapsed_time)]
    system_cpu_cumulative = [i[0] / i[1] * 100 for i in zip(system_time, elapsed_time)]

    figure = plt.figure('Main')
    ax = figure.subplots()
    ax.xaxis.set_major_formatter(format_date)
    ax1 = ax.twinx()
    ax.set_ylabel('CPU (%)')
    ax1.set_ylabel('Memory (MB)')
    ax1.ticklabel_format(axis = 'y', useOffset = False)
    ax1.yaxis.set_major_formatter(format_memory)
    ax1._get_lines = ax._get_lines

    ax.plot(elapsed_time, user_cpu_uniform, label = 'Average User CPU')
    ax.plot(elapsed_time, user_cpu_cumulative, label = 'Cumulative User CPU')
    ax.plot(elapsed_time, system_cpu_uniform, label = 'Average System CPU')
    ax.plot(elapsed_time, system_cpu_cumulative, label = 'Cumulative System CPU')
    ax1.plot(elapsed_time, data['memory']['process_physical_memory'], label = 'Process physical memory')
    figure.legend()

This:

• Calculates derived metrics like CPU usage %.
• Sets human-readable axes formatting.
• Plots CPU and memory as line graphs.
• Uses twin axes to show both metrics.

In addition to the overall memory usage plot, I can also visualize specific memory metrics.

def plot_memory(data):
    elapsed_time = data['cpu']['uptime']
    figure = plt.figure('Memory')
    ax = figure.subplots()
    ax.xaxis.set_major_formatter(format_date)
    ax.set_ylabel('Relative memory (%)')
    for key, value in data['memory'].items():
        if key in ['total_used_virtual_memory', 'total_used_physical_memory']:
            ax.plot(
                elapsed_time,
                percent_filter(uniform_filter1d(value, size = 100)),
                label = f'Relative average {key}'
            )
        else:
            ax.plot(elapsed_time, percent_filter(value), label = f'Relative {key}')
    figure.legend()

I can also visualize Liquidsoap's GC activity over time.

def plot_gc(data):
    elapsed_time = data['cpu']['uptime']
    figure = plt.figure('Garbage collector')
    ax = figure.subplots()
    ax.xaxis.set_major_formatter(format_date)
    ax.set_ylabel('Relative GC params (%)')
    for key, value in data['gc'].items():
        if key in [
            'minor_words',
            'promoted_words', 
            'major_words', 
            'minor_collections', 
            'major_collections', 
            'compactions'
        ]:
            continue
        ax.plot(elapsed_time, percent_filter(uniform_filter1d(value, size = 100)), label = f'Relative average {key}')
    figure.legend()

3. Bringing It All Together

Finally, I can call the parsing, preparation, and plotting functions to generate the visualizations:

import sys


if __name__ == '__main__':
    if len(sys.argv) == 2:
        # Load data
        data = read_log(sys.argv[1])
        
        # Plot overall metrics
        plot_main(data)
        
        # Plot memory detail
        plot_memory(data)
        
        # Plot GC detail
        plot_gc(data)

		plt.show()

Now when executed, the script will read the Liquidsoap JSON metrics log, and output a set of insightful graphs for monitoring server resource usage and health over time.

The different visualizations provide granular visibility into how CPU, memory, and GC are behaving in Liquidsoap.

Full Script

import sys
import json
import matplotlib.pyplot as plt
from statistics import mean
from scipy.ndimage import uniform_filter1d


def read_log(file_name):
    result = {"cpu": {}, "gc": {}, "memory": {}}
    init = True    
    with open(file_name) as file:
        for line in file:
            try:
                record = json.loads(line)
            except json.decoder.JSONDecodeError:
                continue

            # Initialize dicts
            if init:
                init = False
                for parameter_set in result:
                    for parameter in record[parameter_set]:
                        result[parameter_set][parameter] = []
            
            # Extract metrics
            for parameter_set in result:
                for parameter in result[parameter_set]:
                    result[parameter_set][parameter].append(record[parameter_set][parameter])

    return result


# Calculate CPU usage beetwen steps
def cpu_filter(user_time, elapsed_time):
    result = []
    result.append(user_time[0] / elapsed_time[0] * 100)
    for i in range(1, len(user_time)):
        result.append((user_time[i] - user_time[i-1]) / (elapsed_time[i] - elapsed_time[i - 1]) * 100)
    return result


# Calcuate difference in percents
def percent_filter(items, size = 120):
    average = mean(items[:size])
    return [item / average * 100 for item in items]


# Format x-axis as time
def format_date(x, pos = None):
    x = round(x)
    m, s = divmod(x, 60)
    h, m = divmod(m, 60)
    return f'{h}:{m:02d}:{s:02d}'


# Format y-axis memory as MB
def format_memory(x, pos = None):
    return x / 1000 ** 2


def plot_main(data):
    elapsed_time = data['cpu']['uptime']
    user_time = data['cpu']['user']
    system_time = data['cpu']['system']

    user_cpu = cpu_filter(user_time, elapsed_time)
    system_cpu = cpu_filter(system_time, elapsed_time)
    user_cpu_uniform = uniform_filter1d(user_cpu, size = 100)
    system_cpu_uniform = uniform_filter1d(system_cpu, size = 100)
    user_cpu_cumulative = [i[0] / i[1] * 100 for i in zip(user_time, elapsed_time)]
    system_cpu_cumulative = [i[0] / i[1] * 100 for i in zip(system_time, elapsed_time)]

    figure = plt.figure('Main')
    ax = figure.subplots()
    ax.xaxis.set_major_formatter(format_date)
    ax1 = ax.twinx()
    ax.set_ylabel('CPU (%)')
    ax1.set_ylabel('Memory (MB)')
    ax1.ticklabel_format(axis = 'y', useOffset = False)
    ax1.yaxis.set_major_formatter(format_memory)
    ax1._get_lines = ax._get_lines

    ax.plot(elapsed_time, user_cpu_uniform, label = 'Average User CPU')
    ax.plot(elapsed_time, user_cpu_cumulative, label = 'Cumulative User CPU')
    ax.plot(elapsed_time, system_cpu_uniform, label = 'Average System CPU')
    ax.plot(elapsed_time, system_cpu_cumulative, label = 'Cumulative System CPU')
    ax1.plot(elapsed_time, data['memory']['process_physical_memory'], label = 'Process physical memory')
    figure.legend()


def plot_memory(data):
    elapsed_time = data['cpu']['uptime']
    figure = plt.figure('Memory')
    ax = figure.subplots()
    ax.xaxis.set_major_formatter(format_date)
    ax.set_ylabel('Relative memory (%)')
    for key, value in data['memory'].items():
        if key in ['total_used_virtual_memory', 'total_used_physical_memory']:
            ax.plot(
                elapsed_time,
                percent_filter(uniform_filter1d(value, size = 100)),
                label = f'Relative average {key}'
            )
        else:
            ax.plot(elapsed_time, percent_filter(value), label = f'Relative {key}')
    figure.legend()


def plot_gc(data):
    elapsed_time = data['cpu']['uptime']
    figure = plt.figure('Garbage collector')
    ax = figure.subplots()
    ax.xaxis.set_major_formatter(format_date)
    ax.set_ylabel('Relative GC params (%)')
    for key, value in data['gc'].items():
        if key in [
            'minor_words',
            'promoted_words',
            'major_words',
            'minor_collections',
            'major_collections',
            'compactions'
        ]:
            continue
        ax.plot(elapsed_time, percent_filter(uniform_filter1d(value, size = 100)), label = f'Relative average {key}')
    figure.legend()


if __name__ == '__main__':
    if len(sys.argv) == 2:
        # Load data
        data = read_log(sys.argv[1])

        # Plot overall metrics
        plot_main(data)

        # Plot memory detail
        plot_memory(data)

        # Plot GC detail
        plot_gc(data)
        
        plt.show()

Setting Up the Python Environment

Before running the metrics visualization script, I'll need to set up a Python virtual environment.

This keeps the script's dependencies isolated from the rest of the system.

Creating the Virtualenv

First create the virtualenv directory:

python -m venv .env

Next activate the environment:

. .env/bin/activate

The virtual environment is now ready to install packages into.

Installing Dependencies

With the venv active, install the required packages:

pip install matplotlib scipy

This will download matplotlib and scipy into the virtual environment.

Running the Script

Finally, with the venv activated you can run the metrics visualization script:

python visualize_metrics.py statistics_2024-01-12T10:10:00+0000.json

This ensures a clean, reproducible environment for the script.

Now the metrics graphs will render correctly!

[toots]

Awesome one, again. Thanks so much!

[nigeyuk]

Just a FYI i get this when running the stats script

At stats.liq, line 25, char 30-46:
    let {pretty, ...memory} = runtime.memory()

Error 5: this value has no method `pretty`
  Its type is {process_managed_memory : int}.

[toots]

You might need to install mem_info

[nigeyuk]

I've got mem_usage but it says there is not mem_info package ?

[gAlleb]

If you are using 2.2.5 version - it should work just fine. Unless you are using opam. Then you have to install mem_usage explicitly.

P.S. Oh, sorry, wasn't fast enough))

[nigeyuk]

` Additional libraries

• FFmpeg filters : yes
• FFmpeg devices : yes
• inotify : no (requires inotify)
• irc : no (requires irc-client-unix)
• jemalloc : no (requires jemalloc)
• lastfm : no (requires lastfm)
• lo : no (requires lo)
• memtrace : yes
• mem_usage : yes
• osc : no (requires osc-unix)
• ssl : yes
• tls : yes
• posix-time2 : no (requires posix)
• windows service : no (requires winsvc)
• YAML support : no (requires yaml)
• XML playlists : no (requires xmlplaylist)`

Got mem_trace and mem_usage both installed via opam so .. yeah i'm not sure.