vapormark is a benchmark framework developed for measuring various performance metrics (e.g., throughput, latency, and tail latency) and the process states (e.g., backend stall, energy consumption) while running a program on Linux. It especially targets SteamOS -- a Linux-based gaming device --but most features are genetically useful in regular Linux environments.

Three Phases

vapormark consists of three phases:

1. running a benchmark (i.e., collecting performance data)
2. analyzing the collected data
3. generating a report.

Only the first step should run on a target device, such as SteamDeck. The others can be run on almost any Linux box.

External Dependencies

vapormark uses the following in each phase:

• running a benchmark (i.e., collecting performance data)
  • schbench, stress-ng, and gbench for micro-benchmarking of scheduler performance
  • MangoHud for measuring FPS (frame per second) during a running game
  • strace, trace-cmd, cpupower, turbostat, chcpu, taskset, and perf for collecting processor states
• analyzing the collected data
  • matplotlib and graphviz python library for generating graphs
• generating a report
  • pandoc for generating a report in HTML format
• for all phases
  • python3

Installation

Just clone this repository and hit make on the top of the directory. The build procedure will clone and build schbench. All the binaries will be put under the vapormark/bin directory.

If you want to measure FPS, install MangoHud. For SteamDeck, please refer the following steps:

1. Press STEAM button then choose Power -> Switch to Desktop
2. On Plasma Desktop, launch Discover Center. Then find and install MangoHud
3. Copy the ManguHud configuration file, vapormark/config/MangoHud.conf to /home/deck/.config/MangoHud/MangoHud.conf. This is the minimal MangoHud configuration that vapormark expects. MangoHud logs will be stored to the /home/deck/mangologs-vapormark/ directory.

Running a benchmark and collecting performance data

scmon: collecting system usage of a process tree

scmon collects a system call usage of a process tree. It generates per-task system call trace file under OUTDIR with prefix LOG and suffix -scmon.*. It imposes noticeable performance overhead so it should not be used when collecting performance numbers. scmon is useful to understand the high-level behavioral traits of an application.

usage: scmon [-h] -o OUTDIR -l LOG [-p PID] [-r ROOT] [-n NAME] [-c CMD [CMD ...]]

Collect system call usage statistics of a program

options:
  -h, --help            show this help message and exit
  -o OUTDIR, --outdir OUTDIR
                        output directory
  -l LOG, --log LOG     log file prefix
  -p PID, --pid PID     process id to monitor
  -r ROOT, --root ROOT  root process id to monitor (all decendents will be monitored)
  -n NAME, --name NAME  name of a process to monitor
  -c CMD [CMD ...], --cmd CMD [CMD ...]
                        command to execute

For example, 'scmon -o log -l steam -n steam' to log the system call usage of 'steam' 
and all its decendents under log/steam*-scmon*.

procmon: collecting processor and scheduling statistics

procmon collects four types of information: 1) scheduler's wakeup events, 2) CPU's c-state, 3) CPU's energy consumption, and 4) processor's performance monitoring data (e.g., instruction per cycle). Similar to scmon, it generates logs under OUTDIR with prefix LOG and suffix -procmon.*. It collects information while it runs. The runtime overhead is not marginal so it can be run with an application level benchmark (like game). However, it is not recommended with a micro-benchmark (schdbench), which is much more sensitive to any noises.

usage: procmon [-h] -o OUTDIR -l LOG [-s] [-c] [-e] [-p] [-a]

Collect CPU statistics and system-wide scheduling statistics

options:
  -h, --help            show this help message and exit
  -o OUTDIR, --outdir OUTDIR
                        output directory
  -l LOG, --log LOG     log file prefix
  -s, --sched           trace wake-up events of process scheduler
  -c, --cstate          trace c-state of all CPUs
  -e, --energy          trace energy consumption of all CPUs
  -p, --perf            trace performance statistics of all CPUs
  -a, --all             trace all statistics

procmon internally uses 'trace-cmd', 'cpupower', and 'perf'.

mbench: running a micro-benchmark

mbench is a wrapper which runs schbench with a pre-configured settings. For convenience, it launches procmon if necessary. However, to get accurate performance results, it is recommended mbench with and without procmon. Also, make sure there is no heavy background tasks: for example, in SteamOS, steam, mangoapp, gamemoded, gamescope, and steamwebhelper.

usage: mbench [-h] -o OUTDIR -l LOG [-b BG] [-f FG] [-c CONFIG] [-r RUNTIME] [-p]

Run a micro-benchmark with a pre-configured setting

options:
  -h, --help            show this help message and exit
  -o OUTDIR, --outdir OUTDIR
                        output directory
  -l LOG, --log LOG     log file prefix
  -b BG, --bg BG        command line of a background task
  -f FG, --fg FG        command line of a foreground task for benchmarking
  -c CONFIG, --config CONFIG
                        run a benchmark with preconfigured setting: `schbench50`, 
                        `schbench100`, and `schbench200`, each of which runs `schbench` with 
                        50%, 100%, and 200% CPU utilization, respectively
  -r RUNTIME, --runtime RUNTIME
                        benchmark running time in seconds (default = 90sec)
  -p, --procmon         run with profiling on

Performance monitoring (-p) WILL interfere the results of micro-benchmark. Do NOT use -p when 
you collect performance results. Instead, run the same benchmark twice: one without profiling 
for performance comparison and another with profiling for analysis. Also, make sure there is 
no heavy background task running. 

MangoHud: measuring FPS, CPU/GPU utilization, etc.

Launching, starting, and stopping MangoHud is not integrated with vapormark. Hence vapormark just follows the standard MangoHud usage. Especially in SteamDeck, please refer to the following procedure:

• For a game to FPS logging, go to Properties -> General -> Launch Options and add mangohud %command%. The game must be launched in Desktop Mode (not in Gaming Mode) to log FPS and other system stats.

• Now, you will see the overlay window showing FPS when launching the game. You can start and stop FPS logging by hitting Shift_L+F2. The log will be stored at /home/deck/mangologs-vapormark. Some games hang when MangoHud is enabled. Other useful MangoHud shortcuts are as follows:

  Shift_L+F2 : Toggle Logging
  Shift_L+F4 : Reload Config
  Shift_R+F12 : Toggle Hud        
  

• Once you finish FPS logging by hitting Shift_L+F2, MangoHud will generate a csv log file under /home/deck/mangologs-vapormark. Please copy and rename it ending with -mangohud.csv for analysis and report generation.

• Following games provide in-game benchmarks:

  Game	How to start an in-game benchmark
  Far Cry: New Dawn	Options -> Benchmark
  A Total War Saga: Troy	Options -> Graphics -> Advanced -> Benchmark
  Cyber Punk 2077	Settings -> Graphics -> Quick Preset, Run Benchmark
  Factorio	On terminal: factorio --benchmark map.zip --benchmark-ticks 1000 --disable-audio

schedmon: collecting the detailed scheduling activities

schedmon collects the detailed system-wide scheduling activities. It internally relies on perf sched record command.

usage: schedmon [-h] -o OUTDIR -l LOG

Collect the detailed scheduler activity internally using `perf sched`

options:
  -h, --help            show this help message and exit
  -o OUTDIR, --outdir OUTDIR
                        output directory
  -l LOG, --log LOG     log file prefix

energyprof: collecting energy and performance statistics according to CPU load

energyprof measures energy consumption and performance while running stress-ng with various CPU loads and various numbers of online CPUs. It internally relies on turbostat, stress-ng, chcpu, taskset, and perf.

usage: energyprof [-h] -c NUM_CPUS [-t TIME_SEC] -o OUTDIR -l LOG

Collect energy and performance statistics according to CPU load

options:
  -h, --help            show this help message and exit
  -c NUM_CPUS, --num_cpus NUM_CPUS
                        number of CPUs of this machine
  -t TIME_SEC, --time_sec TIME_SEC
                        time in sec to run stress-_ng
  -o OUTDIR, --outdir OUTDIR
                        output directory
  -l LOG, --log LOG     log file prefix

Analyzing the collected data

Once the performance data is collected, it is time to analyze the results. In this phase, vapormark transforms various log files into the standard CSV format and produces the latency distribution graphs. Specifically, it provides the following commands. The generated files have a suffix of its program, *-scinsight*, *-procinsight*, *-ginsight*, and *-energyprof*.

scinsight: analyzing scmon logs

usage: scinsight [-h] -o OUTDIR -l LOG [-q]

Report system call usage statistics of a program

options:
  -h, --help            show this help message and exit
  -o OUTDIR, --outdir OUTDIR
                        output directory
  -l LOG, --log LOG     log file prefix, or path to directory containing log files
  -q, --quiet           do not print result to stdout

procinsight: analyzing procmon logs

usage: procinsight [-h] -o OUTDIR -l LOG [-q]

Report CPU statistics and system-wide scheduling statistics

options:
  -h, --help            show this help message and exit
  -o OUTDIR, --outdir OUTDIR
                        output directory
  -l LOG, --log LOG     log file prefix
  -q, --quiet           do not print result to stdout

ginsight: analyzing a MangoHud log

usage: ginsight [-h] -l LOG -o OUTDIR -p PREFIX [-q]

Generarte a report from MangoHud log

options:
  -h, --help            show this help message and exit
  -l LOG, --log LOG     MangoHud log file in a CSV format
  -o OUTDIR, --outdir OUTDIR
                        output directory
  -p PREFIX, --prefix PREFIX
                        output file prefix
  -q, --quiet           do not print result to stdout

schedinsight: analyzing the results of schedmon

usage: schedinsight [-h] -l LOGDIR -p PREFIX -o OUTPUT [-i IMGTYPE] [-k] [-s MINSCHED] [-t TIMELIMIT]

Report the detailed analysis of scheduliing activities collected by `perf sched record`

options:
  -h, --help            show this help message and exit
  -l LOGDIR, --logdir LOGDIR
                        a log directory
  -p PREFIX, --prefix PREFIX
                        log file prefix
  -o OUTPUT, --output OUTPUT
                        a target report file name in markdown format
  -i IMGTYPE, --imgtype IMGTYPE
                        type of image format (png, svg)
  -k, --pickle          use pickle whenever possible
  -s MINSCHED, --minsched MINSCHED
                        set the minimum number of schedules for task analysis
  -t TIMELIMIT, --timelimit TIMELIMIT
                        time limit to draw a graph in seconds

energyinsight: analyzing the results of energyprof

usage: energyinsight [-h] -o OUTDIR -l LOG [-q]

Report energy usage per CPU load and number of online CPUs

options:
  -h, --help            show this help message and exit
  -o OUTDIR, --outdir OUTDIR
                        output directory
  -l LOG, --log LOG     output log file prefix
  -q, --quiet           do not print result to stdout

Generating a (comparison) report

vapormark provides a reporting feature that compares the results of multiple configurations. This is especially useful when checking the impact of a certain optimization. When more than one log directories are given (with multiple -l options), report uses the logs in the first directory as a baseline and shows the relative delta in percent. Note that report does not support the comparison of schedmon logs yet.

usage: report [-h] -l LOGDIR -p PREFIX -o OUTPUT [-f] [-g]

Generate a report of given log directories

options:
  -h, --help            show this help message and exit
  -l LOGDIR, --logdir LOGDIR
                        a log directory. When mulltiple `-l` options are given, comparison 
                        will be reported using the first one as a baseline.
  -p PREFIX, --prefix PREFIX
                        log file prefix for report generation
  -o OUTPUT, --output OUTPUT
                        target report file name in markdown format
  -f, --force           force to regenerate all CSV files
  -g, --debug           print out debug messages

For example, `report -l base_dir -l cmp_dir -p game1 -o report.md` compares `game1` logs 
in two directories -- `base_dir` and `cmp_dir` -- and generates `report.md`. `base_dir`
is used in calculating the relative difference. When only one log directory is given, 
only the summary of results without comparison is provided. It expects certain file 
extensions: `*.factorio_out` for factorio benchmark and `*.schbench_out` for schbench 
benchmark.

Misc tools

sched-config: save and restore key scheduler parameters from debugfs

usage: sched-config [-h] [-g GET] [-s SET]

Set or get the scheduler config parameters

options:
  -h, --help         show this help message and exit
  -g GET, --get GET  Get the scheculer parameters
  -s SET, --set SET  Set the scheculer parameters
sched-config: error: either '-g' or '-s' should be specified