From 5b7992496812899d9e9c2f277104f53110860a07 Mon Sep 17 00:00:00 2001 From: Sayan Date: Fri, 19 Apr 2024 12:21:44 +0530 Subject: [PATCH] benchmarking: Improve docs on workload (#119) * benchmarking: Improve docs on workload * benchmarking: Fix typo in docs --- docs/14.benchmarking.md | 72 ++++++++++++++++++++++++----------------- docs/2.installation.md | 6 ++++ 2 files changed, 48 insertions(+), 30 deletions(-) diff --git a/docs/14.benchmarking.md b/docs/14.benchmarking.md index 26bb2e141..c6d0fe1a3 100644 --- a/docs/14.benchmarking.md +++ b/docs/14.benchmarking.md @@ -3,33 +3,45 @@ id: benchmarking title: Benchmarking --- -Due to Skytable having in-house implementations of almost everything, starting from the protocol, the storage engine and query -language — we have our own custom load testing tool called `sky-bench` that is distributed with the bundle. - -## Setting up for benchmarking - -**Quick notes**: -- The benchmark tool will create: - - a space called `bench` - - a model called `bench` -- **Be sure that these objects don't already exist!** But don't worry, if they do — the benchmark tool will error. You won't lose any data -- **Once the benchmark is complete, the `bench` space is removed!** -- **Do not use the `bench` space during the benchmark**: You might lose data - -**The benchmark tool will:** -- Run a total of 4,000,000 queries (don't worry, they run pretty fast!): - - Run 1,000,000 `INSERT`s - - Run 1,000,000 `SELECT`s - - Run 1,000,000 `UPDATE`s - - Run 1,000,000 `DELETE`s -- The model used has the declaration `(un: string, pw: uint8)` -- The `SELECT` will select and return all fields -- The `UPDATE` will increment the value of `pw` like this `pw += 1` -- The `DELETE` removes the entire collection -- The default primary key size is 7 bytes. All generated keys will be padded with zeros from to 0 to 999,999 like this: `0000000` or `0999999` - -## Off to the races - -1. Start up the database server -2. Run `./sky-bench --password `. We need your `root` password because only the root account can create, alter and drop models and the benchmark tool needs to run these queries -3. Wait for it to happen. You may not believe your eyes, so we recommend that you keep your eyes hydrated 🔥🚀✨💣 +All of Skytable's components are developed from scratch in-house, which makes it hard (if not impossible) to utilize other benchmarking suites. This is why all of Skytable's release bundles come with the `sky-bench` benchmarking tool. It's important to understand how the tool works to understand what you're benchmarking and what the results mean. + +The overall goal with the benchmark tool is to simulate how the database would perform in the real-world and reduce synthetic factors. + +:::info +The ability to simulate different workloads is currently being worked on. This means that across releases, the default workload that the benchmark tool runs may vary. See the release notes to see if the benchmark workload has changed. +::: + +## Benchmark workload + +The workload that the engine currently uses (as of v0.8.1) is the following: +- A model `bench.bench` is created with a primary key (of type `string` and with a column of type `uint8`) +- Multiple clients are created (simulating "application servers") +- Queries are run against **unique rows**. This means that unlike `redis-benchmark` **every query group touches a different row, as it would generally do in the real-world** +- The following queries (collectively a "query group") are run for each unique row: + - The row is first created with an `INSERT` + - All columns of the row are returned with a `SELECT` + - The integer column is incremented with an `UPDATE` + - The row is finally removed with a `DELETE` +- **By default, 1,000,000 rows are created and manipulated** +- **The time taken for each row to be sent, read back and decoded into a readable form is recorded** (time taken to parse into actual language structures such as maps and lists) towards the total time taken, once again unlike many other benchmark tools +- In total 4,000,000 queries are run (by default) + +:::caution +The benchmark tool will create a space `bench` and a model `bench` and will completely remove the space and associated data once the benchmark is complete. **Do not use this space!** +::: + +## Running the benchmark + +Now that you know how the benchmark engine works, go ahead and benchmark for yourself. + +1. Run: + ```sh + sky-bench --password + ``` + **Note**: You can ignore the `--password` argument if you have already set it using the `SKYDB_PASSWORD` environment variable +2. The benchmark engine will run the full workload (as described earlier) +3. Witness 4,000,000 queries being executed in real-time. Good luck and enjoy the results! + +:::tip +You can tune the number of threads, connections, rows created and such to your liking to simulate the environment that you think matches your production setting. +::: diff --git a/docs/2.installation.md b/docs/2.installation.md index 07d2e8307..787a3f6a0 100644 --- a/docs/2.installation.md +++ b/docs/2.installation.md @@ -10,6 +10,12 @@ Getting started with Skytable involves choosing a mode of installation, download - [**A Docker image**](#docker-image): We generally recommend using a Docker image for experimenting with Skytable on your local system during development and you want to keep your local system *clean*. If you want to use a Docker image for deployment, you're always free to do so! > **Note:** You might experience slightly degraded performance from the storage engine due to Docker engine's abstractions. +:::tip +All client tools (such as `skysh` and `sky-bench`) *can* use the `SKYDB_PASSWORD` variable for authentication. If you're using Skytable in a testing environment and frequently need to use `skysh`, you may consider setting this variable to your password to avoid having to pass the `--password` argument every time. + +However, we strongly recommend **not** using it outside testing environments. +::: + ## Native binaries To use native binaries you need to download a bundle which is simply a ZIP file with all the necessary binaries that you'll ever need to develop on and deploy Skytable.