Overhaul of the results presented in the readme with new charts 🤩

JohannesDeml · May 5, 2021 · 579cf23 · 579cf23
1 parent 15c9876
commit 579cf23
Show file tree

Hide file tree

Showing 7 changed files with 80 additions and 99 deletions.
diff --git a/Docs/Results-Ubuntu20.04.1-.NETCore5.0.png b/Docs/Results-Ubuntu20.04.1-.NETCore5.0.png
diff --git a/Docs/Results-Windows10-.NETCore5.0.png b/Docs/Results-Windows10-.NETCore5.0.png
diff --git a/Docs/nbn-overview-1.0.0.png b/Docs/nbn-overview-1.0.0.png
diff --git a/Docs/nbn-pingpongbatchedunreliable-1.0.0.png b/Docs/nbn-pingpongbatchedunreliable-1.0.0.png
diff --git a/Docs/nbn-pingpongreliable-1.0.0.png b/Docs/nbn-pingpongreliable-1.0.0.png
diff --git a/Docs/nbn-pingpongunreliable-1.0.0.png b/Docs/nbn-pingpongunreliable-1.0.0.png
diff --git a/README.md b/README.md
@@ -9,7 +9,7 @@
 ## Table of Contents
 
 1. [Description](#description)
-2. [Benchmarks](#benchmarks)
+2. [Benchmark Setup](#benchmark-setup)
 3. [Benchmark Results](#benchmark-results)
 4. [Installation](#installation)
 5. [Usage](#usage)
@@ -42,100 +42,62 @@ NBN is a benchmark for low level networking libraries using UDP and can be used
   * [Unity Client Example](https://github.com/JohannesDeml/Unity-Net-Core-Networking-Sockets)
 
 
+## Benchmark Setup
 
-## Benchmarks
+### Used Hardware
 
-### Benchmark [PingPongUnreliable](./NetworkBenchmarkDotNet/PredefinedBenchmarks/UnreliablePerformanceBenchmark.cs)
+* Ubuntu VPS
+  * Virtual private server with dedicated CPU's running - [Hardware](https://www.netcup.eu/bestellen/produkt.php?produkt=2624)
+  * Ubuntu 20.04.2 LTS x86-64 Kernel 5.4.0-72-generic
 
-Runs the benchmark with **500** clients, which pingpong **1 message** each with the server with **unreliable** transmission. The benchmark runs until a total of **500,000** messages are sent to the server and back to the clients. Message size is **32 bytes**.  
-This test is for getting an idea of an average roundtrip time.
+* Ubuntu Desktop / Windows Desktop
+  * Desktop PC from 2020 - [Hardware](https://pcpartpicker.com/user/JohannesDeml/saved/zz7yK8)
+  * Windows 10 Pro 20H2 x86-64 Build 19042.844
+  * Ubuntu 20.04.2 LTS x86-64 Kernel 5.8.0-50-generic
 
-### Benchmark [PingPongBatchedUnreliable](./NetworkBenchmarkDotNet/PredefinedBenchmarks/UnreliablePerformanceBenchmark.cs)
+### Procedure
+For the two desktop setups, the benchmarks are run on a restarted system with 5 minutes idle time before starting the benchmark. They are run with admin privileges and all unnecessary other processes are killed before running the benchmarks. For Ubuntu VPS, the benchmarks are run through continuous integration on a typical indie server setup with other processes running as well. After the benchmarks are run, a list of all running processes to make them more reproducible. To reproduce the benchmarks, run `sudo sh linux-benchmark.sh` or `win-benchmark.bat` . If you want to execute directly from the compiled program, run `./NetworkBenchmarkDotNet -b Essential`.
 
-Runs the benchmark with **500** clients, which pingpong **10 messages** each with the server with **unreliable** transmission. The benchmark runs until a total of **500,000** messages are sent to the server and back to the clients. Message size is **32 bytes**.  
-This test is for multiplexing / message merging performance.
+## Benchmark Results
+The raw data and additional files can be downloaded from the [release section](../../releases).
+
+### Benchmark [PingPongUnreliable](./NetworkBenchmarkDotNet/PredefinedBenchmarks/UnreliablePerformanceBenchmark.cs)
+
+Runs the benchmark with **500** clients, which pingpong **1 message** each with the server with **unreliable** transmission. The benchmark runs until a total of **500,000** messages are sent to the server and back to the clients. Message size is **32 bytes**.  
+This test is for getting an idea of an average roundtrip time (lower is better).
+![PingPong Unreliable .NET Benchmark chart](./Docs/nbn-pingpongunreliable-1.0.0.png)
 
 ### Benchmark [PingPongReliable](./NetworkBenchmarkDotNet/PredefinedBenchmarks/ReliablePerformanceBenchmark.cs)
 
 Runs the benchmark with **500** clients, which pingpong **1 message** each with the server with **reliable** transmission. The benchmark runs until a total of **500,000** messages are sent to the server and back to the clients. Message size is **32 bytes**.  
-This test is for getting an idea of an average roundtrip time.
+This test is for getting an idea of an average roundtrip time (lower is better).
+![PingPong Reliable .NET Benchmark chart](./Docs/nbn-pingpongreliable-1.0.0.png)
+
+### Benchmark [PingPongBatchedUnreliable](./NetworkBenchmarkDotNet/PredefinedBenchmarks/UnreliablePerformanceBenchmark.cs)
+
+Runs the benchmark with **500** clients, which pingpong **10 messages** each with the server with **unreliable** transmission. The benchmark runs until a total of **500,000** messages are sent to the server and back to the clients. Message size is **32 bytes**.  
+This test is for multiplexing / message merging performance (higher is better).
+![PingPong Batched Unreliable .NET Benchmark chart](./Docs/nbn-pingpongbatchedunreliable-1.0.0.png)
 
 ### Benchmark [SampleEchoSimple](./NetworkBenchmarkDotNet/PredefinedBenchmarks/SamplingBenchmark.cs)
 
 Runs the benchmark with **1** client, which pingpong **10 messages** each with the server. The benchmark runs until a total of **100,000** messages are sent to the server and back to the clients. Message size is **128 bytes**.  
 This test collects information about generated garbage and CPU times while running the benchmark. Those results can be analyzed with [PerfView](https://github.com/microsoft/perfview) on Windows.
 
-## Benchmark Results
-
-### Ubuntu 20.04
-
-To reproduce the benchmarks, run `./NetworkBenchmarkDotNet -b Essential`.   
-[Hardware Details](https://pcpartpicker.com/b/Wtykcf)
+### Overview
+This is a comparison between all tests with their message throughput (higher is better).
 
-``` ini
-BenchmarkDotNet=v0.12.1, OS=ubuntu 20.04
-Intel Core i5-3570K CPU 3.40GHz (Ivy Bridge), 1 CPU, 4 logical and 4 physical cores
-.NET Core SDK=5.0.103
-  [Host]     : .NET Core 5.0.3 (CoreCLR 5.0.321.7203, CoreFX 5.0.321.7203), X64 RyuJIT
-  Job-YNISTP : .NET Core 5.0.3 (CoreCLR 5.0.321.7203, CoreFX 5.0.321.7203), X64 RyuJIT
-
-Platform=X64  Runtime=.NET Core 5.0  Concurrent=True  
-Force=True  Server=True  InvocationCount=1  
-IterationCount=10  LaunchCount=1  UnrollFactor=1  
-WarmupCount=1  Version=0.9.0  OS=Linux 5.8.0-43-generic #49~20.04.1-Ubuntu SMP Fri Feb 5 09:57:56 UTC 2021  
-DateTime=02/18/2021 16:18:27  
-```
-|                    Method |       Library | Transmission | Clients |      Throughput |       Mean |    Error |   StdDev |
-|-------------------------- |-------------- |------------- |--------:|----------------:|-----------:|---------:|---------:|
-| **PingPongReliable** |       **ENet** |     **Reliable** |     **500** | **90,333 msg/s** | **5.535 s** | **0.0311 s** | **0.0205 s** |
-|        PingPongUnreliable |          ENet |   Unreliable |     500 |   185,112 msg/s | 2,701.1 ms | 21.57 ms | 12.83 ms |
-| PingPongBatchedUnreliable |          ENet |   Unreliable |     500 | 1,129,598 msg/s |   442.6 ms |  4.83 ms |  2.87 ms |
-|        **PingPongUnreliable** | **NetCoreServer** |   **Unreliable** |     **500** |    **96,514 msg/s** | **5,180.6 ms** | **64.21 ms** | **42.47 ms** |
-| PingPongBatchedUnreliable | NetCoreServer |   Unreliable |     500 |    97,245 msg/s | 5,141.6 ms | 55.11 ms | 36.45 ms |
-| **PingPongReliable** | **LiteNetLib** |     **Reliable** |     **500** | **82,804 msg/s** | **6.038 s** | **0.0671 s** | **0.0444 s** |
-|        PingPongUnreliable |    LiteNetLib |   Unreliable |     500 |    91,222 msg/s | 5,481.2 ms | 51.56 ms | 34.11 ms |
-| PingPongBatchedUnreliable |    LiteNetLib |   Unreliable |     500 |   251,421 msg/s | 1,988.7 ms | 56.09 ms | 33.38 ms |
-
-![Benchmark Results](./Docs/Results-Ubuntu20.04.1-.NETCore5.0.png)
-
-### Windows 10
-To reproduce the benchmarks, run `./NetworkBenchmarkDotNet -b Essential`.  
-[Hardware Details](https://pcpartpicker.com/b/8MMcCJ) (Note that this machine has a lot more performance than the linux machine)
-
-``` ini
-BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19042
-AMD Ryzen 7 3700X, 1 CPU, 16 logical and 8 physical cores
-.NET Core SDK=5.0.103
-  [Host]     : .NET Core 5.0.3 (CoreCLR 5.0.321.7212, CoreFX 5.0.321.7212), X64 RyuJIT
-  Job-ODIPRB : .NET Core 5.0.3 (CoreCLR 5.0.321.7212, CoreFX 5.0.321.7212), X64 RyuJIT
-
-Platform=X64  Runtime=.NET Core 5.0  Concurrent=True  
-Force=True  Server=True  InvocationCount=1  
-IterationCount=10  LaunchCount=1  UnrollFactor=1  
-WarmupCount=1  Version=0.9.0  OS=Microsoft Windows 10.0.19042  
-DateTime=02/18/2021 16:18:02  
-```
-|                    Method |       Library | Transmission | Clients |    Throughput |       Mean |       Error |    StdDev |
-|-------------------------- |-------------- |------------- |--------:|--------------:|-----------:|------------:|----------:|
-| **PingPongReliable** |       **ENet** |     **Reliable** |     **500** | **51,718 msg/s** | **9.668 s** | **0.1468 s** | **0.0768 s** |
-|        PingPongUnreliable |          ENet |   Unreliable |     500 |  93,960 msg/s | 5,321.4 ms | 1,040.97 ms | 688.54 ms |
-| PingPongBatchedUnreliable |          ENet |   Unreliable |     500 | 687,775 msg/s |   727.0 ms |    84.13 ms |  55.65 ms |
-|        **PingPongUnreliable** | **NetCoreServer** |   **Unreliable** |     **500** |  **72,975 msg/s** | **6,851.6 ms** |    **41.01 ms** |  **27.12 ms** |
-| PingPongBatchedUnreliable | NetCoreServer |   Unreliable |     500 |  78,644 msg/s | 6,357.8 ms |    42.78 ms |  25.46 ms |
-| **PingPongReliable** | **LiteNetLib** |     **Reliable** |     **500** | **88,463 msg/s** | **5.652 s** | **0.0231 s** | **0.0138 s** |
-|        PingPongUnreliable |    LiteNetLib |   Unreliable |     500 |  90,985 msg/s | 5,495.4 ms |    30.93 ms |  20.46 ms |
-| PingPongBatchedUnreliable |    LiteNetLib |   Unreliable |     500 | 771,852 msg/s |   647.8 ms |    13.48 ms |   8.92 ms |
-
-![Benchmark Results](./Docs/Results-Windows10-.NETCore5.0.png)
+![Network Benchmark .NET results overview](./Docs/nbn-overview-1.0.0.png)
 
 ### Notes
 
-* The tests perform very different on Linux compared to Windows 10, since there are a lot of client threads involved and Linux seems to handle them a lot better.
+* Ubuntu outperforms windows in almost all tests and libraries. 
+* The benchmarks are run locally and therefor a lot of processing power is used for simulating the clients. Therefore, this benchmark does not include a CCU comparison, but is sufficient to compare different libraries for the same setup.
+* Also the network latency results do not show any costs of the network itself, but only the cost of the application layer in the round trip time. Since multiple clients are run at the same time, this time with be smaller than presented here.
 * Creation, Connection and Disconnection and Disposal of the Server and Clients is not included in the performance benchmarks, but is included in the .nettrace files from the Sampling benchmark.
-* Since the clients and the server run on the same machine, there is a lot less network latency as in a real world application. On the other hand, the CPU pressure is a lot higher than for a normal server, since all the clients get there own threads and run on the same machine. Take the results with a grain of salt.
 * To access the Sampling results, you can use [PerfView](https://github.com/microsoft/perfview) to open the `.nettrace` files.
 * Kcp2k has been recently added and might have some room for improvements. Especially using `Thread.Sleep` on Windows creates [noticeable delays](https://social.msdn.microsoft.com/Forums/vstudio/en-US/facc2b57-9a27-4049-bb32-ef093fbf4c29/threadsleep1-sleeps-for-156-ms?forum=clr). For now it is excluded of the predefined benchmarks, until its execution and cleanup are improved.
-
+* This is not a general purpose benchmark for every and any game type. Different games have different requirements. I hope these benchmarks help you as a smoke test and give you the possibility to quickly test how your hardware handles the different libraries.
 
 
 ## Installation
@@ -155,26 +117,39 @@ You can run custom benchmarks through the command-line. Use can test multiple se
 ```
 Usage:
   NetworkBenchmarkDotNet [options]
-
 Options:
-  -b, --benchmark <All|Custom|Essential|Performance|Quick|Sampling>    Run predefined benchmarks [default: Custom]
-  -m, --execution-mode <Client|Complete|Server>                        Control what parts to run [default: Complete]
-  -t, --test <PingPong>                                                Test type [default: PingPong]
-  --transmission <Reliable|Unreliable>                                 Transmission type [default: Unreliable]
-  -l, --library <ENet|Kcp2k|LiteNetLib|NetCoreServer>                  Library target [default: ENet]
-  -d, --duration <duration>                                            Test duration in seconds (-1 for manual stopping) [default: 10]
-  --address <address>                                                  IP Address, can be ipv4 (e.g. 127.0.0.1) or ipv6 (e.g. ::1) [default: ::1]
-  --port <port>                                                        Socket Port [default: 3330]
-  --clients <clients>                                                  # Simultaneous clients [default: 500]
-  --parallel-messages <parallel-messages>                              # Parallel messages per client [default: 1]
-  --message-byte-size <message-byte-size>                              Message byte size sent by clients [default: 32]
-  --message-payload <Ones|Random|Zeros>                                Message load sent by clients [default: Random]
-  --verbose                                                            Verbose output of test steps and errors [default: True]
-  --client-tick-rate <client-tick-rate>                                Client ticks per second if supported [default: 60]
-  --server-tick-rate <server-tick-rate>                                Server ticks per second if supported [default: 60]
-  --version                                                            Show version information
-  -?, -h, --help                                                       Show help and usage information
-
+  -b, --benchmark                         Run predefined benchmarks [default:
+  <All|Custom|Essential|Performance|Qu    Custom]
+  ick|Sampling>
+  -m, --execution-mode                    Control what parts to run [default:
+  <Client|Complete|Server>                Complete]
+  -t, --test <Manual|PingPong>            Test type [default: PingPong]
+  --transmission <Reliable|Unreliable>    Transmission type [default:
+                                          Unreliable]
+  -l, --library                           Library target [default: ENet]
+  <ENet|Kcp2k|LiteNetLib|NetCoreServer
+  >
+  -d, --duration <duration>               Test duration in seconds (-1 for
+                                          manual stopping) [default: 10]
+  --address <address>                     IP Address, can be ipv4 (e.g.
+                                          127.0.0.1) or ipv6 (e.g. ::1)
+                                          [default: ::1]
+  --port <port>                           Socket Port [default: 3330]
+  --clients <clients>                     #Simultaneous clients [default: 500]
+  --parallel-messages                     #Parallel messages per client
+  <parallel-messages>                     [default: 1]
+  --message-byte-size                     Message byte size sent by clients
+  <message-byte-size>                     [default: 32]
+  --message-payload                       Message load sent by clients
+  <Ones|Random|Zeros>                     [default: Random]
+  --verbose                               Verbose output of test steps and
+                                          errors [default: True]
+  --client-tick-rate                      Client ticks per second if supported
+  <client-tick-rate>                      [default: 60]
+  --server-tick-rate                      Server ticks per second if supported
+  <server-tick-rate>                      [default: 60]
+  --version                               Show version information
+  -?, -h, --help                          Show help and usage information
 ```
 
 ### Predefined Benchmarks
@@ -228,4 +203,4 @@ Your new proposed library ...
 
 ## License
 
-[MIT](./LICENSE)
+[MIT](./LICENSE)