Generic performance testing system

[eddyashton]

Creating this discussion to flesh out our planning around #848. Our existing performance testing framework is brittle and was designed to support a custom protocol which is no longer needed. We plan to replace it with a simpler system which clearly separates the steps of generating a request stream, submitting it to CCF, and analysing the results.

For parity with the current performance tests this needs to:

• submit hundreds of thousands of requests, faster than CCF can process them
• submit Websockets as well as HTTP requests
• detect when the final transaction has been committed
• confirm that the responses had successful status codes, and that the final state matches a known target state
• get a final transactions-per-second throughput rate for the test

We're hoping the new system will do this more cheaply and flexibly, while also supporting additional functionality:

• setting request parameters in URL query and headers, rather than only in the request body
• calculating additional metrics from the results, such as:
  • response and commit latencies
  • intermediate throughput rates
  • per-endpoint metrics

I see the new system as 3 separate tools with 3 supporting file formats, which can be chained together for a simple test or run separately/offline when needed. I'll discuss these in more detail in separate comments below, but at the top level these are:

1. Generate the requests, pre-serialised as HTTP or Websockets, dumped to a file
2. Submit the requests, to a CCF node, recording the request send and response receive times as well as the response contents, to a file
3. Analyse the request and response times to calculate performance metrics

[eddyashton]

Generating requests

CCF nodes receive commands as HTTP (or Websockets) requests over a TLS connection. These commands can be created and fully serialised independently of their submission (eg, using Python requests' Prepared Requests or CCF's http_builder.h utils).

Our existing tests using the scenario_perf_client and Vegeta do a similar thing already - they generate a specification for the requests (known as targets in Vegeta) ahead of time, and submit this later. We think we can go one step further and create fully serialised HTTP request bodies in advance, removing the need to do this work during the perf measurement.

We have several different applications that we need to run performance tests on, and they each have different HTTP APIs. To benchmark the logging sample app we call /app/log/private, and for smallbank we're calling endpoints like /app/SmallBank_transact_savings. So the frontend for this stage will need to be able to generate requests for different targets, ideally by being extended/called differently per-app.

The output of this step can be very simple, a table where each entry contains message ID and serialised HTTP request. It would look something like this:

Message ID	Serialised Request
0	POST /app/log/private HTTP/1.1 Content-type: application/json Content-Length: 52 {"id": 0, "msg": "Some arbitrary message payload 0"}
1	POST /app/log/private HTTP/1.1 Content-type: application/json Content-Length: 52 {"id": 1, "msg": "Some arbitrary message payload 1"}
2	POST /app/log/private HTTP/1.1 Content-type: application/json Content-Length: 52 {"id": 2, "msg": "Some arbitrary message payload 2"}
...	...

The message ID should be a unique value generated by this tool, used to correlate the requests and responses which are later submitted (ie - the response is keyed by this message ID, rather than needing to store request and response together). The HTTP requests will be application specific - this example shows what the expected stream for a perf test our logging sample app might look like (all calling /app/log/private, each given a unique "id" so that they're all retained in memory, each entry given a unique message so we can clearly distinguish them+validate the results).

This can easily be ingested by another tool which creates a TLS connection and submits these to CCF. Where there's a lot of repeated data (eg - common headers), this file should compress very well, so it should be fine to check in pre-generated request loads to version control for consistent, reproducible testing. Parquet looks like a good file format for this.

[eddyashton]

Submitting requests

Given a stream of requests produced by the tool above, a simple client should be able to:

1. Open a single TLS connection with a CCF node, and keep this connection open. For websockets, this also does the initial websockets handshake/upgrade
2. Submit all of the given requests in a pipeline, continuing to send outgoing requests without blocking and waiting for each read
3. Record the message ID and timestamp of every sent request
4. Read responses over the same connection, parsing just enough to find the end of each response, doing minimal or ideally no validation, and explicitly not trying to parse the body
5. Record the message ID, timestamp, and raw response for every received response

In the simplest case, it should submit all of these, wait for their responses, and then terminate, but it should also have some configurable options:

1. Repeat all the requests N times
2. Repeat all the requests infinitely, until you're terminated (and leave the files in a consistent state when that happens)
3. Repeat the requests until you've seen the same response N times (with a binary comparison of the response, not a parsed interpretation)
4. Sleep x seconds between each repetition
5. Pipeline at most N requests ahead, to avoid overloading CCF

Additionally we want to poll until these transactions have been committed (calling the /commit endpoint on the CCF node), and record these responses as well, because the metrics for this performance test should be "how long did it take all of these transactions to commit", not just "how long did it take to receive a response for each of these transactions". We could add this as a hard-coded step in the runner, but a simpler solution would be to have a separate instance of this submitting client which just repeatedly calls /commit (recording the timestamps like every other client), and terminating it when the commit has reached the desired point.

The output of this step is 2 files, one containing the sent request and another the received responses, all with timestamps:

Sends

Message ID	Send Time
0	1615895076.6057985
1	1615895076.6058214
2	1615895076.6058254
...	...

Receives

Message ID	Receive Time	Raw Response
0	1615895077.6057985	HTTP/1.1 200 OK content-length: 4 content-type: application/json x-ms-ccf-transaction-id: 2.17 true
1	1615895077.6058214	HTTP/1.1 200 OK content-length: 4 content-type: application/json x-ms-ccf-transaction-id: 2.18 true
2	1615895077.6058254	HTTP/1.1 200 OK content-length: 4 content-type: application/json x-ms-ccf-transaction-id: 2.19 true
...	...	...

[achamayou]

Reading and writing from and to parquet files in C++: https://arrow.apache.org/docs/cpp/parquet.html#filereader

[eddyashton]

Analysing results

Given those sent and received files, it should be easy to load them, parse any details that need to be parsed, and produce performance metrics. For instance, calculating the total time between the first send and the last receive, and a throughput rate derived from that. This kind of analysis (especially if the files are a common format like Parquet) is a natural fit for pandas.

As a minimum, we need a tool which takes those files and prints the transaction rate.

Ideally, this prints several more metrics, and is configurable/extensible to add more metrics in future.

[fotiskoun]

Description of Performance Test Tools

Artillery

• It has two types of performance tests: loads on a system (such as stress tests) and system-is-working verification.
• Supports variety of protocols: HTTP/1.1, WebSocket, and Socket.io out of the box, and many additional protocols added via plugin interface such as HLS, Kinesis, and Kafka
• Uses scenarios which define complex user behaviour with multiple steps transactions, verbs, loops, condition and custom js code.
• Load- & Smoke- Testing: re-use scenarios, use assertions on the response
• Extensible: extension APIs to extend in js with custom engines, plugins
• Yaml based human readable scripts to run and configure the requests allowing input from csv files
• Artillery-Pro (which is closed-source!) option for parallel tests from AWS account
• Definition of different running phases specifying arrival rate and duration sending requests
• Tracks and records performance metrics, such as API response times, throughput, and errors. Reports some metrics on the go but also provide a summary report at the end
• Have different configuration profiles (environments), where you can keep some scenario definitions but specify differently limited configuration variables, such as the target path or one phase. This prevents large scenarios re-writing for different tests with mostly the same definitions.
• Variables that you can pass from the response of one request to the description of a following one, or from a csv to different scenarios that may need authorization
• Conditional checks for the time of requests, or if the response met the requirements
• Loop a virtual user to request N times with the same confs and ability to take the number of iterations in a variable
• You can tell Artillery to write a JSON report of a test run into a file. The JSON report can be used to generate an HTML report with the report command, or to run queries on with a tool like jq
• Separate config and scenarios yaml files for reusability
• Add on cli different variables that would be taken inside the yaml file or override scenarios/configurations
• Create html report
• Ability to pause between phases

Cons:

• Plugins configuration can be troublesome and is needed for everything that is more complex than a basic scenario-running testing
• Not ideal for reporting your data results in plots or formats other than json/html
• TLS configuration plugin comes with Artillery Pro, which is charged for organizations
• It doesn’t support root custom CA
• HTTP/2 protocols are not supported yet

K6

• A high-loading testing (stress- sike- soak- tests) tool for performance and synthetic monitoring and chaos and reliability testing
• It has a CLI tool wit API and js scripting to struct the script.
• It allows checks and thresholds for test termination
• In CLI you can define the js script to read the scenario from, the number of virtual users, the duration etc but you can also specify these values inside the script
• Availability for different stages with different time duration and variety of virtual users for each stage
• Supports three execution modes a) local, b) distributed with Kubernetes and c) cloud on k6 cloud
• Provides stdout summary and streams live test data to an external output
• During outputs provides a bar of completion and percentage of time spent (fancy but why not, it engages the user)
• Each scenario can have a unique name which is printed in the result screen and can map with the metrics
• In the stdout prints reports for time taken and throughput for data received and sent, as well as statistics (min, max, percentiles) for the phases of requests (blocked, connect, send, receive, handshake etc)
• Cli customization of the time units and the starts that you want to be reported
• Ability to send the report on JSON, CSV, XML and other formats
• Automatic support to HTTP/2: if the server reports back that it supports HTTP/2 then k6 will upgrade the connection

Cons:

• It does not support out of the box root custom CA, you need to go through insecure option, which should not be the case in production environments
• Report analysis with basic metrics, no plotting

Vegeta

• A command-line tool, which is very easy to learn and straightforward
• You set the configurations on the command line, and you also provide json files for the request tasks
• Straightforward for TLS connection with root custom CA through command-line options
• Great for reporting to a file where you can further use Vegeta command line to export it in terminal or even plot it
• Has no requirements for other languages to be installed from the user
• Supports HTTP/2
• Allows setting number of requests per time unit with default 50/1s
• Can be merged with command line options to generate targets or variables for the targets
• Can be integrated with jplot to plot real-time analysis data
• Comes also as an independent library

Cons:

• Can be very messy on describing complex scenarios in command line
• Does not allow as complex scenarios as the other tools do

Locust

• Performance testing tool where you can describe the behaviour of the test in Python language.
• You can create loops, condition checks, and perform calculations
• You write plain Python code, and the system takes over to run the code as a test
• It is based on gevent so, a single process can handle multiple concurrent users.
• Web-based UI to get the analysis results with alternative to run in command line and get text report as well or export in CSV
• It can support multiple Python files to run scenarios
• You can run multiple Python processes distributed by setting master/worker processes in command line
• It uses annotation semantics and a class-must-inherit system to recognise the test functions
• It can optionally sleep between tests
• Uses a class which inherits Python’s requests.Session to send HTTP requests and handle the response codes where the user can further validate/invalidate a failed/successful response.
• A handful of command line options to further setup TLS configuration, running time, host name etc
• It supports TLS configuration with root custom CA
• It supports HTTP/1.1 and HTTP/2 requests
• Debugger option for running tests
• Can run on Docker or distributed on Kubernetes and AWS using Terraform module
• It has a class option (FastHttpUser) which uses geventhttpclient to support a higher throughput than the usual for more demanding request throughputs “best case scenario a test using FastHttpUsers will be able to do close to 5000 requests per second per core, instead of around 850 for HttpUser (tested on a 2018 MacBook Pro i7 2.6GHz)”
• It uses Python logging framework to handle logging for warn/errors/debug with command line option to specify the log-file

Cons:

• It can be used in existing Python scripts as a library but lacks features that the command line provides
• FastHttpUser does not support TLS authentication with custom root CA as a regular HttpUser

h2load

• It is a command line benchmarking tool suitable for HTTP/2 requests, with support for HTTP/1.1 and HTTP/3
• It straightforwardly supports TLS
• Number of requests, clients or seconds for the test to run are all specified on the command line followed by a list of URIs where the tests will run in round robin format
• Ability to provide multiple headers for the requests
• Command line output of basic metrics for request and connection and traffic.
• It can apply request on streams allowing also the window specification

Cons:

• While it is very straightforward to use, with a detailed documentation for the command line options, it lacks multiple scenarios build cases, other than single requests
• Provides only basic metrics
• Does not support root custom CA
• Issues with the report, even if the requests are received from an HTTP/1.1 server it reports the requests as failed and errored.

Performance Testing Client

• A command line tool written in C# and Python for HTTP/1.1 requests
• It supports easy TLS configuration and custom root CA
• It is a tool based on 3 components (Request generator, Request sender and result analysis) that uses parquet files to send data between components
• It allows very simple scenario creation
• Exports the analysis report on files

Cons:

• Not suitable for complex scenarios
• Doesn’t support HTTP/2
• Not a great variety of report options

Other performance testing tools:

• Httperf – Supports HTTP/1.1 pipelining
• Ab from httpd
• Hey
• Jmeter
• Wrk – One of the best at generating high volumes of traffic (source)