3. Join Migration
Using the following as a guide, we will walk you through the setup of BullFrog. To make it smoother, we record terminal sessions and share them on the web.
This tutorial assumes you have a current version of Docker installed on your machine. If you do not have Docker installed, choose your preferred operating system below to download Docker:
After you installed Docker, you can issue a command to pull our docker image and start a container:
# Pulls Docker image and run it on your local machine.
$ docker run --rm -it -d --name bullfrog gangliao/bullfrog:join_migration bash
# Output:
#
# Unable to find image 'gangliao/bullfrog:join_migration' locally
# latest: Pulling from gangliao/bullfrog
# ......: Pull complete
# ......: Pull complete
# ......: Pull complete
# Digest: sha256:......
# Status: Downloaded newer image for gangliao/bullfrog:join_migration
# Enters the container.
$ docker exec -u postgres -it bullfrog bash
# Output:
#
# postgres@968c58c4b04d:/$-
Rebooting a DB instance:
# Deploys the postgres backend $ cd /home/postgres/BullFrog && ./deploy.sh # Output: # # +++ case $- in # +++ return # ++ rm -rf /home/postgres/BullFrog/build/data # ++ initdb -D /home/postgres/BullFrog/build/data # The files belonging to this database system will be owned by user "postgres". # This user must also own the server process. # # The database cluster will be initialized with locale "C". # The default database encoding has accordingly been set to "SQL_ASCII". # The default text search configuration will be set to "english". # # Data page checksums are disabled. # # creating directory /home/postgres/BullFrog/build/data ... ok # creating subdirectories ... ok # selecting default max_connections ... 100 # selecting default shared_buffers ... 128MB # selecting dynamic shared memory implementation ... posix # creating configuration files ... ok # running bootstrap script ... Shared Global Bitmap created! # ok # performing post-bootstrap initialization ... ok # syncing data to disk ... ok # # WARNING: enabling "trust" authentication for local connections # You can change this by editing pg_hba.conf or using the option -A, or # --auth-local and --auth-host, the next time you run initdb. # # Success. You can now start the database server using: # # pg_ctl -D /home/postgres/BullFrog/build/data -l logfile start # # ++ cp /home/postgres/postgresql.conf /home/postgres/BullFrog/build/data/ # ++ pg_ctl -D /home/postgres/BullFrog/build/data -o '-F -p 5433' start
-
Run a TPC-C Benchmark where data is already loaded into the database.
- Please re-execute all commands in the step 2 if the benchmark fails to run.
$ cd /home/postgres/BullFrog-Oltpbench # Clean tuples in new tables with new schemas $ psql -h localhost -p 5433 tpcc -f /home/postgres/BullFrog-Oltpbench/clean_new_tables.sql # Output: # # DROP TABLE # CREATE TABLE # DROP TABLE # CREATE TABLE # CREATE INDEX # CREATE INDEX # DROP TABLE # CREATE TABLE # CREATE INDEX # CREATE INDEX # CREATE INDEX # CREATE INDEX # CREATE INDEX # CREATE INDEX # CREATE VIEW # Clean shared memory via restarting database $ pg_ctl -D $PGDATA restart # Output: # # waiting for server to shut down.... # 2021-02-15 05:22:27.763 UTC [1208] LOG: background worker "logical replication launcher" (PID 1214) exited with exit code 1 # 2021-02-15 05:22:27.964 UTC [1210] LOG: shutting down # 2021-02-15 05:22:28.557 UTC [1208] LOG: database system is shut down # done # server stopped # waiting for server to start.....2021-02-15 05:22:29.579 UTC [1275] LOG: listening on IPv4 address "127.0.0.1", port 5433 # 2021-02-15 05:22:29.579 UTC [1275] LOG: could not bind IPv6 address "::1": Cannot assign requested address # 2021-02-15 05:22:29.579 UTC [1275] HINT: Is another postmaster already running on port 5433? If not, wait a few seconds and retry. # 2021-02-15 05:22:29.579 UTC [1275] LOG: listening on Unix socket "/tmp/.s.PGSQL.5433" # Shared Global Bitmap created! # 2021-02-15 05:22:29.863 UTC [1276] LOG: database system was shut down at 2021-02-15 05:22:28 UTC # 2021-02-15 05:22:29.865 UTC [1275] LOG: database system is ready to accept connections # done # server started # run benchmark $ ./oltpbenchmark -b tpcc -c config/pgtpcc_lazy_join.xml --execute=true -s 1 -o lazy_join --port=5433 --bgthread=join # Output: # ... # 05:37:19,860 (ThreadBench.java:473) INFO - TERMINATE :: Waiting for all terminals to finish .. # 05:37:20,010 (ThreadBench.java:534) INFO - Attempting to stop worker threads and collect measurements # 05:37:20,012 (ThreadBench.java:255) INFO - Starting WatchDogThread # 05:37:20,059 (DBWorkload.java:886) INFO - ====================================================================== # 05:37:20,059 (DBWorkload.java:887) INFO - Rate limited reqs/s: Results(nanoSeconds=60000173489, measuredRequests=25774) = 429.56542458540093 requests/sec # 05:37:20,071 (DBWorkload.java:708) INFO - Upload Results URL: com.oltpbenchmark.util.ResultUploader@4690b489 # 05:37:20,073 (DBWorkload.java:741) INFO - Output Raw data into file: results/lazy_join.csv # 05:37:20,597 (DBWorkload.java:760) INFO - Output summary data into file: results/lazy_join.summary # 05:37:20,625 (DBWorkload.java:767) INFO - Output DBMS parameters into file: results/lazy_join.params # 05:37:20,647 (DBWorkload.java:774) INFO - Output DBMS metrics into file: results/lazy_join.metrics # 05:37:20,720 (DBWorkload.java:781) INFO - Output experiment config into file: results/lazy_join.expconfig # 05:37:20,806 (DBWorkload.java:798) INFO - Output throughput samples into file: results/lazy_join.res # 05:37:20,806 (DBWorkload.java:801) INFO - Grouped into Buckets of 1 seconds
Experiment Results
-
Note: We have simplified the experimental environment and dataset size, so the results may be different from the paper. Since Docker for Mac runs in a LinuxKit VM, to get the best performance, please run the experiment on bare metal machines.
time(sec), throughput(req/sec), avg_lat(ms), min_lat(ms), 25th_lat(ms), median_lat(ms), 75th_lat(ms), 90th_lat(ms), 95th_lat(ms), 99th_lat(ms), max_lat(ms), tp (req/s) scaled 0,702.000,853.670,19.257,710.622,888.379,1096.760,1286.477,1331.835,1355.632,1393.684,0.001 1,700.000,1474.251,1339.103,1418.365,1475.823,1521.727,1575.126,1593.868,1616.345,1683.607,0.001 2,700.000,1811.737,1589.018,1685.246,1819.383,1935.552,1966.866,1974.636,1990.719,2084.343,0.001 3,700.000,2147.102,1967.488,2029.107,2123.769,2230.773,2324.802,2397.429,2419.072,2476.172,0.000 4,700.000,2547.627,2411.769,2476.075,2563.215,2596.331,2637.840,2669.772,2691.902,2723.438,0.000 5,700.000,2821.600,2673.001,2751.853,2789.201,2849.442,3012.314,3039.110,3072.193,3105.470,0.000 6,695.000,3336.237,3059.301,3210.660,3340.564,3467.351,3536.360,3552.601,3588.349,3651.358,0.000 7,703.000,3946.451,3567.329,3794.532,3980.403,4073.833,4224.702,4271.707,4293.933,4348.321,0.000 8,702.000,4483.016,4280.283,4415.277,4477.357,4560.028,4621.622,4631.646,4647.446,4702.741,0.000 9,700.000,4866.388,4630.259,4742.254,4880.324,4986.937,5048.317,5060.744,5084.106,5148.873,0.000 10,697.000,5365.324,5067.286,5270.699,5368.705,5477.988,5530.465,5555.109,5598.699,5659.181,0.000 11,703.000,5821.157,5586.420,5694.597,5831.846,5947.864,5990.920,6004.994,6026.042,6064.524,0.000 12,700.000,6257.668,6009.671,6106.622,6238.068,6409.481,6498.794,6528.212,6569.409,6628.143,0.000 13,700.000,6775.881,6564.242,6683.447,6786.034,6871.846,6900.874,6920.403,6942.998,6966.934,0.000 14,699.000,7123.783,6923.313,7056.542,7129.343,7194.743,7238.234,7263.726,7287.269,7332.929,0.000 15,701.000,7584.517,7277.061,7374.348,7615.673,7749.125,7823.164,7855.090,7892.987,7953.127,0.000 16,699.000,8206.570,7896.816,8073.878,8234.464,8336.581,8409.798,8418.629,8444.327,8480.162,0.000 17,701.000,8695.844,8438.230,8576.343,8700.308,8828.262,8910.521,8928.053,8945.056,9410.964,0.000 18,700.000,9637.615,8928.858,9460.934,9681.641,9830.084,9974.545,9997.797,10032.429,10068.481,0.000 19,700.000,10288.628,10011.346,10187.044,10283.065,10414.243,10485.879,10513.247,10543.094,10563.939,0.000 20,700.000,10745.597,10527.655,10644.695,10756.805,10845.463,10900.345,10915.405,10938.152,10989.170,0.000 21,698.000,11237.838,10925.069,11130.175,11228.911,11357.546,11439.395,11448.909,11475.829,11551.977,0.000 22,702.000,11730.004,11454.927,11571.074,11717.014,11878.842,11989.089,12006.337,12031.817,12083.986,0.000 23,700.000,12249.150,12014.997,12121.831,12241.752,12366.818,12457.249,12482.295,12510.256,12568.382,0.000 24,700.000,12888.781,12498.576,12679.831,12882.080,13107.145,13232.189,13261.063,13300.104,13347.126,0.000 25,700.000,13600.447,13285.167,13458.250,13590.827,13753.513,13853.771,13871.935,13892.372,13955.255,0.000 26,657.000,14134.623,13874.586,14050.808,14133.130,14228.707,14290.502,14298.084,14328.910,14419.296,0.000 27,409.000,14303.541,14285.985,14291.362,14297.235,14309.933,14323.369,14335.672,14388.889,14439.024,0.000 28,435.000,14302.374,14282.035,14290.184,14296.558,14307.394,14322.543,14333.780,14393.179,14429.283,0.000 29,413.000,14303.257,14283.881,14291.546,14297.130,14309.143,14322.480,14332.954,14380.781,14457.706,0.000 30,447.000,14302.348,14284.919,14290.292,14297.073,14307.532,14319.114,14337.315,14393.089,14445.123,0.000 31,449.000,14301.235,14279.956,14290.175,14296.260,14306.101,14318.714,14332.055,14374.225,14406.820,0.000 32,443.000,14302.104,14281.044,14290.441,14297.407,14308.521,14320.587,14326.676,14377.560,14440.626,0.000 33,430.000,14303.341,14285.192,14290.450,14297.143,14308.338,14323.319,14342.874,14382.513,14466.994,0.000 34,378.000,14305.313,14283.675,14291.392,14297.368,14310.867,14327.151,14342.419,14410.405,14513.131,0.000 35,409.000,14303.149,14282.115,14291.253,14296.705,14309.442,14325.701,14337.079,14383.169,14425.652,0.000 36,397.000,14304.338,14284.037,14291.470,14298.270,14310.346,14323.510,14337.098,14393.134,14422.581,0.000 37,388.000,14305.055,14286.511,14291.955,14297.773,14311.677,14327.433,14336.922,14399.755,14426.531,0.000 38,446.000,14301.799,14285.762,14290.946,14296.419,14308.115,14321.825,14332.289,14359.928,14400.220,0.000 39,454.000,14301.594,14282.388,14290.615,14295.934,14308.742,14319.067,14330.027,14359.224,14436.623,0.000 40,402.000,14303.905,14283.700,14290.962,14295.982,14309.784,14321.580,14336.912,14392.822,14465.575,0.000 41,503.000,14299.451,14282.559,14289.962,14296.079,14305.972,14312.666,14320.037,14358.280,14408.962,0.000 42,557.000,14297.528,14282.540,14289.341,14292.911,14303.654,14311.363,14318.851,14351.095,14390.290,0.000 43,504.000,14299.802,14281.787,14289.731,14294.220,14305.750,14313.343,14319.712,14371.157,14402.551,0.000 44,537.000,14298.536,14280.226,14289.377,14295.253,14304.563,14309.821,14316.933,14359.051,14374.114,0.000 45,340.000,14299.251,14278.900,14290.087,14294.001,14305.560,14314.207,14319.761,14357.679,14403.131,0.000
-
You may want to shut down the database for any reason:
$ cd /home/postgres/BullFrog && ./shutdown.sh # Output: # # ++ source /home/postgres/.bashrc # +++ case $- in # +++ return # ++ pg_ctl -D /home/postgres/BullFrog/build/data stop # waiting for server to shut down.... done # server stopped # ++ pg_ctl -D /home/postgres/BullFrog/build/data status # pg_ctl: no server running
-
You may stop a running container when you finish the tutorial:
$ docker stop bullfrog