Performance Metrics

The performance metrics are defined based on the following three notions:

• Query execution time (QET) is the amount of time (typically given in milliseconds, but may also be specified otherwise) that passes from the begin of sending a given query to a GraphQL server under test until the complete query result has been received in return.
• Query response time (QRT) is the amount of time (typically given in milliseconds, but may also be specified otherwise) that passes from the begin of sending a given query to a GraphQL server under test until the begin of receiving the query result in return.
• Throughput (TP) is the number of a given set of queries that are processed completely by a GraphQL-based client-server system within a specified time interval, where a query is considered to be processed completely after its complete result has been received by the client that requested the execution of the query. By default, the time intervals considered have a duration of 30 seconds; however, the interval duration may also be specified differently for a specific experiment. Throughput may be considered for a single client or for a multiple clients. The number of clients has to be clearly specified when reporting measurements for throughput-based metrics.

Metrics for Single Queries

• Average query execution time (aQETq) is the average of the QETs measured when executing an individual query multiple times with the GraphQL server under test. When reporting measurements for this metric, the corresponding standard deviation has to be reported as well.
• Average query response time (aQRTq) is the average of the QRTs measured when executing an individual query multiple times with the GraphQL server under test. When reporting measurements for this metric, the corresponding standard deviation has to be reported as well.

Metrics for Query Templates

• Query execution times per query template (QETt) is the distribution of the individual QETs measured for multiple queries of the same template.
• Average query execution time per query template (aQETt) is the average of the QETs measured when executing multiple instances of a specific query template by the GraphQL server under test. When reporting measurements for this metric, the corresponding standard deviation has to be reported as well.
• Overall query execution time per query template (oQETt) is the overall amount of time that passes when executing multiple instances of a specific query template by the GraphQL server under test; i.e., if all the query instances are executed sequentially, then oQETt is the sum of the individual QETs (assuming no delay between the queries).
• Average query response time per query template (aQRTt) is the average of the QRTs measured when executing multiple instances of a specific query template by the GraphQL server under test. When reporting measurements for this metric, the corresponding standard deviation has to be reported as well.
• Average throughput per query template (aTPt) is the average of the throughput measured when running the same query workload multiple times using the same client(s)-server system, where the queries in the workload are all from the same template. When reporting measurements for this metric, the corresponding standard deviation has to be reported as well.