HP - Lesson 3 - 「9547」

[9547]

通过 go-ycsb 测试 TiDB 性能

集群配置

发包机为本地 macOS 中跑的 Ubuntu VM，4Core， TiDB 集群部署在一台物理机上，本地与 TiDB 之间通过 1G 网卡相连：

[ go-ycsb] <----> [ TiDB/TiKV/PD]
   macOS               baremetal

TiDB 使用的 TiUP 配置如下：

global:
  user: "tidb"
  ssh_port: 22
  deploy_dir: "/home/tidb/deploy"
  data_dir: "/home/tidb/data"
  arch: "amd64"

pd_servers:
  - host: 192.168.1.62

tikv_servers:
  - host: 192.168.1.62

tidb_servers:
  - host: 192.168.1.62

monitoring_servers:
  - host: 192.168.1.62

grafana_servers:
  - host: 192.168.1.62

alertmanager_servers:
  - host: 192.168.1.62

TiDB, TiKV, PD 部署于一台物理机器上，物理机的配置如下：

CPU 6core

Intel(R) Core(TM) i5-9400 CPU @ 2.90GHz

Memory 32GB

Disk nvme

性能待测试

测试过程

1. go-ycsb 只起 4 个线程（与本机的 CPU core 数相等）

(base) root@v:go-ycsb <master() ✖️  ># ./bin/go-ycsb load mysql -P workloads/workloada -p recordcount=1000000 -p mysql.host=192.168.1.62 -p mysql.port=4000 -p mysql.db=test --threads 4
***************** properties *****************
"readproportion"="0.5"
"operationcount"="1000"
"updateproportion"="0.5"
"dotransactions"="false"
"threadcount"="4"
"insertproportion"="0"
"readallfields"="true"
"mysql.port"="4000"
"mysql.db"="test"
"recordcount"="1000000"
"mysql.host"="192.168.1.62"
"scanproportion"="0"
"requestdistribution"="uniform"
"workload"="core"
**********************************************
INSERT - Takes(s): 9.7, Count: 2160, OPS: 222.2, Avg(us): 18406, Min(us): 8082, Max(us): 297018, 99th(us): 41000, 99.9th(us): 291000, 99.99th(us): 298000
INSERT - Takes(s): 19.7, Count: 4458, OPS: 226.1, Avg(us): 17860, Min(us): 7558, Max(us): 297018, 99th(us): 36000, 99.9th(us): 175000, 99.99th(us): 298000
INSERT - Takes(s): 29.7, Count: 6647, OPS: 223.7, Avg(us): 17968, Min(us): 7558, Max(us): 297018, 99th(us): 37000, 99.9th(us): 171000, 99.99th(us): 298000
INSERT - Takes(s): 39.7, Count: 8804, OPS: 221.7, Avg(us): 18096, Min(us): 7558, Max(us): 297018, 99th(us): 37000, 99.9th(us): 165000, 99.99th(us): 298000
INSERT - Takes(s): 49.7, Count: 10971, OPS: 220.7, Avg(us): 18153, Min(us): 7558, Max(us): 297018, 99th(us): 39000, 99.9th(us): 60000, 99.99th(us): 291000
INSERT - Takes(s): 59.7, Count: 13276, OPS: 222.3, Avg(us): 18001, Min(us): 7558, Max(us): 297018, 99th(us): 38000, 99.9th(us): 54000, 99.99th(us): 295000

QPS 在 220 上下，观察 Grafana 上 CPU Disk 等数值均较低，猜测是发包机的并发度不够。于是增加发包机的并发度再测试（发现一个小 trick，此时需要先将 TiDB 中这个待测试的表（usertable)数据清除，不然会影响到测试结果）:

truncate table usertable;

2. 清除完已有的数据之后，将发包机的并发增加到 16 线程再测试：

(base) root@v:go-ycsb <master() ✖️  ># ./bin/go-ycsb load mysql -P workloads/workloada -p recordcount=1000000 -p mysql.host=192.168.1.62 -p mysql.port=4000 -p mysql.db=test --threads 16
***************** properties *****************
"scanproportion"="0"
"mysql.port"="4000"
"recordcount"="1000000"
"workload"="core"
"threadcount"="16"
"updateproportion"="0.5"
"requestdistribution"="uniform"
"insertproportion"="0"
"mysql.db"="test"
"readallfields"="true"
"dotransactions"="false"
"mysql.host"="192.168.1.62"
"readproportion"="0.5"
"operationcount"="1000"
**********************************************
INSERT - Takes(s): 10.0, Count: 7014, OPS: 703.6, Avg(us): 22640, Min(us): 8202, Max(us): 237799, 99th(us): 46000, 99.9th(us): 227000, 99.99th(us): 238000
INSERT - Takes(s): 20.0, Count: 12442, OPS: 622.1, Avg(us): 25474, Min(us): 8202, Max(us): 287557, 99th(us): 92000, 99.9th(us): 206000, 99.99th(us): 233000
INSERT - Takes(s): 30.1, Count: 17273, OPS: 574.2, Avg(us): 27547, Min(us): 8202, Max(us): 287557, 99th(us): 109000, 99.9th(us): 232000, 99.99th(us): 247000
INSERT - Takes(s): 40.0, Count: 23264, OPS: 582.1, Avg(us): 27258, Min(us): 8202, Max(us): 342918, 99th(us): 112000, 99.9th(us): 238000, 99.99th(us): 288000
INSERT - Takes(s): 50.0, Count: 29144, OPS: 583.1, Avg(us): 27186, Min(us): 8202, Max(us): 342918, 99th(us): 106000, 99.9th(us): 238000, 99.99th(us): 327000
INSERT - Takes(s): 60.0, Count: 35581, OPS: 593.0, Avg(us): 26732, Min(us): 7250, Max(us): 342918, 99th(us): 103000, 99.9th(us): 246000, 99.99th(us): 333000
INSERT - Takes(s): 70.0, Count: 40789, OPS: 582.4, Avg(us): 27180, Min(us): 7250, Max(us): 342918, 99th(us): 109000, 99.9th(us): 261000, 99.99th(us): 332000
INSERT - Takes(s): 80.0, Count: 46224, OPS: 578.0, Avg(us): 27416, Min(us): 7250, Max(us): 447200, 99th(us): 106000, 99.9th(us): 258000, 99.99th(us): 333000
INSERT - Takes(s): 90.0, Count: 52329, OPS: 581.6, Avg(us): 27256, Min(us): 7250, Max(us): 456519, 99th(us): 105000, 99.9th(us): 285000, 99.99th(us): 335000
INSERT - Takes(s): 100.0, Count: 58093, OPS: 581.1, Avg(us): 27282, Min(us): 7250, Max(us): 456519, 99th(us): 103000, 99.9th(us): 275000, 99.99th(us): 335000
INSERT - Takes(s): 110.0, Count: 63348, OPS: 576.1, Avg(us): 27534, Min(us): 7250, Max(us): 456519, 99th(us): 101000, 99.9th(us): 261000, 99.99th(us): 334000
INSERT - Takes(s): 120.0, Count: 68475, OPS: 570.8, Avg(us): 27799, Min(us): 7250, Max(us): 456519, 99th(us): 99000, 99.9th(us): 255000, 99.99th(us): 334000

相比前者，QPS 提升了 2 倍，而 99th latency 也提升了 2 倍左右

需要说明的是：

1.从图中看， QPS 的统计结果与 go-ycsb 是一致的，在前后两个过程中变化较明显；
2. 但是 99th duration 前后的数值『却』不符合 go-ycsb 的统计数值

仔细观察 server 与 go-ycsb 统计的结果还是能发现一些问题：server 端统计的数值比 go-ycsb 统计的要小。这也符合常理，说明了 go-ycsb 与 server 之间的网络占了近一半（40/20）以上的开销，在这个 case 中 go-ycsb 发包机的网络直接决定了 TiDB 的上限。

3. 在 TiDB 本机上直接跑 go-ycsb 测试

直接在 TiDB 机器上测试，跑 32 线程

[root@rabbit ~]# ./go-ycsb load mysql -P workloads/workloada -p recordcount=1000000 -p mysql.host=192.168.1.62 -p mysql.port=4000 -p mysql.db=test --threads 32
***************** properties *****************
"requestdistribution"="uniform"
"workload"="core"
"readproportion"="0.5"
"readallfields"="true"
"mysql.db"="test"
"insertproportion"="0"
"recordcount"="1000000"
"operationcount"="1000"
"mysql.port"="4000"
"dotransactions"="false"
"updateproportion"="0.5"
"threadcount"="32"
"scanproportion"="0"
"mysql.host"="192.168.1.62"
**********************************************
INSERT - Takes(s): 10.0, Count: 184599, OPS: 18468.2, Avg(us): 1709, Min(us): 329, Max(us): 39673, 99th(us): 6000, 99.9th(us): 12000, 99.99th(us): 33000
INSERT - Takes(s): 20.0, Count: 199956, OPS: 10000.0, Avg(us): 3176, Min(us): 329, Max(us): 39900, 99th(us): 23000, 99.9th(us): 29000, 99.99th(us): 35000
INSERT - Takes(s): 30.0, Count: 219018, OPS: 7301.6, Avg(us): 4358, Min(us): 329, Max(us): 46841, 99th(us): 24000, 99.9th(us): 29000, 99.99th(us): 38000
INSERT - Takes(s): 40.0, Count: 234711, OPS: 5868.4, Avg(us): 5430, Min(us): 329, Max(us): 46841, 99th(us): 26000, 99.9th(us): 32000, 99.99th(us): 39000
INSERT - Takes(s): 50.0, Count: 250478, OPS: 5010.0, Avg(us): 6362, Min(us): 329, Max(us): 53608, 99th(us): 26000, 99.9th(us): 34000, 99.99th(us): 52000
INSERT - Takes(s): 60.0, Count: 267683, OPS: 4461.7, Avg(us): 7149, Min(us): 329, Max(us): 53608, 99th(us): 26000, 99.9th(us): 33000, 99.99th(us): 52000
INSERT - Takes(s): 70.0, Count: 285395, OPS: 4077.3, Avg(us): 7824, Min(us): 329, Max(us): 53608, 99th(us): 26000, 99.9th(us): 35000, 99.99th(us): 52000
INSERT - Takes(s): 80.0, Count: 300807, OPS: 3760.3, Avg(us): 8485, Min(us): 329, Max(us): 53608, 99th(us): 26000, 99.9th(us): 35000, 99.99th(us): 46000
INSERT - Takes(s): 90.0, Count: 315768, OPS: 3508.7, Avg(us): 9095, Min(us): 329, Max(us): 53608, 99th(us): 28000, 99.9th(us): 38000, 99.99th(us): 52000
INSERT - Takes(s): 100.0, Count: 332533, OPS: 3325.5, Avg(us): 9598, Min(us): 329, Max(us): 53608, 99th(us): 28000, 99.9th(us): 38000, 99.99th(us): 50000
INSERT - Takes(s): 110.0, Count: 349307, OPS: 3175.6, Avg(us): 10053, Min(us): 329, Max(us): 53608, 99th(us): 28000, 99.9th(us): 38000, 99.99th(us): 51000
INSERT - Takes(s): 120.0, Count: 365404, OPS: 3045.1, Avg(us): 10484, Min(us): 329, Max(us): 53608, 99th(us): 28000, 99.9th(us): 38000, 99.99th(us): 50000
INSERT - Takes(s): 130.0, Count: 382238, OPS: 2940.4, Avg(us): 10858, Min(us): 329, Max(us): 53608, 99th(us): 28000, 99.9th(us): 37000, 99.99th(us): 46000
INSERT - Takes(s): 140.0, Count: 398683, OPS: 2847.8, Avg(us): 11212, Min(us): 329, Max(us): 53608, 99th(us): 28000, 99.9th(us): 37000, 99.99th(us): 49000
INSERT - Takes(s): 150.0, Count: 414022, OPS: 2760.2, Avg(us): 11569, Min(us): 329, Max(us): 53608, 99th(us): 29000, 99.9th(us): 37000, 99.99th(us): 49000
INSERT - Takes(s): 160.0, Count: 428523, OPS: 2678.3, Avg(us): 11923, Min(us): 329, Max(us): 61974, 99th(us): 29000, 99.9th(us): 38000, 99.99th(us): 54000
INSERT - Takes(s): 170.0, Count: 442965, OPS: 2605.7, Avg(us): 12256, Min(us): 329, Max(us): 61974, 99th(us): 29000, 99.9th(us): 38000, 99.99th(us): 52000
INSERT - Takes(s): 180.0, Count: 460891, OPS: 2560.6, Avg(us): 12473, Min(us): 329, Max(us): 61974, 99th(us): 29000, 99.9th(us): 38000, 99.99th(us): 52000
INSERT - Takes(s): 190.0, Count: 475667, OPS: 2503.6, Avg(us): 12757, Min(us): 329, Max(us): 61974, 99th(us): 29000, 99.9th(us): 38000, 99.99th(us): 52000
INSERT - Takes(s): 200.0, Count: 491343, OPS: 2456.8, Avg(us): 13000, Min(us): 329, Max(us): 61974, 99th(us): 29000, 99.9th(us): 38000, 99.99th(us): 52000

总结

性能测试不仅仅于待测试的服务端的性能有关，也取决于客户端的性能，特别是网络（不过在数据中心内部，这一问题并没有那么严竣）。另外，在使用 TiUP 部署集群的过程中发现几个问题:

1. TiUP 部署之后的集群使用 tiup cluster start xxx 无法启动，到 TiDB 机器上查看日志，发现是 systemctl 无法将服务启动，journalctl 日志中报下列错误：

   Failed at step EXEC spawning /home/tidb/deploy/tidb-4000/scripts/run_tidb.sh: Permission denied

   经过一番 Google, 发现是 TiDB 机器上 selinux 没有关的原因；如果能在 `tiup check 中有相关功能的检测倒是不错的主义

2. tidb dashboard 中看不到 qps 等 metric 的数据，通过 diagnostic report 输出如下：

通过 report 中说明的方法去更新 metric-storage 的值之后，发现 tidb dashboard 还是报一样的错误：

而此是 tidb 数据库中已经有对应的值了：

[AndreMouche]

LGTM