Latency test inter process take too long [14815]

[xixioba]

Is there an already existing issue for this?

• I have searched the existing issues

Expected behavior

As mentioned here, intra process (zero copy) can achieve very small latency

./LatencyTest both --data_sharing=on --data_loans --shared_memory=on 
Performing intraprocess test with 1 subscribers and 10000 samples
Overhead 0.0265564 ns
Printing round-trip times in us, statistics for 10000 samples
   Bytes, Samples,   stdev,    mean,     min,     50%,     90%,     99%,  99.99%,     max
--------,--------,--------,--------,--------,--------,--------,--------,--------,--------,
      16,   10000,   0.790,   1.152,   0.702,   0.890,   2.265,   2.765,  15.598,  21.577 
    1024,   10000,   0.879,   1.132,   0.702,   0.869,   2.327,   2.807,  22.765,  28.390 
   64512,   10000,   1.516,   3.163,   2.285,   2.807,   3.786,   8.160,  33.202,  39.807 
 1048576,   10000,   2.558,  34.246,  32.389,  33.703,  35.306,  46.161,  73.619,  75.285

There should be similar benchmarks for inter process

Current behavior

But in inter process(zero copy) test , I can't achieve a similar benchmark with poor results (tens of milliseconds)

Steps to reproduce

Terminal 1:

./LatencyTest subscriber --data_sharing=on --data_loans --shared_memory=on
EVERYTHING STOPPED FINE

Terminal 2:

./LatencyTest publisher -s 1000 --data_sharing=on --data_loans --shared_memory=on
Performing test with 1 subscribers and 1000 samples
Overhead 0.0267233 ns
Printing round-trip times in us, statistics for 1000 samples
   Bytes, Samples,   stdev,    mean,     min,     50%,     90%,     99%,  99.99%,     max
--------,--------,--------,--------,--------,--------,--------,--------,--------,--------,
      16,    1000,1200.594,3506.894,  97.494,3126.119,5954.682,6613.974,7540.744,7714.078 
    1024,    1000,1283.282,3671.125, 586.848,3117.973,5968.098,6481.265,6972.494,7507.953 
   64512,     998,1225.901,3550.602,1446.536,3107.264,5892.035,6517.932,8191.681,8254.410 
 1048576,    1000,1263.561,3399.523, 650.494,2955.161,5780.411,6431.577,7300.890,7386.181 
EVERYTHING STOPPED FINE

Fast DDS version/commit

v2.6.0

Platform/Architecture

MacOS Mojave 10.14

Transport layer

Default configuration, UDPv4 & SHM, Zero copy

Additional context

No response

XML configuration file

No response

Relevant log output

No response

Network traffic capture

No response

[JLBuenoLopez]

Hi @xixioba,

I have moved your issue to the proper forum according with the project CONTRIBUTING guidelines.

Regarding your issue, when using zero copy no transport layer is involved because data sharing delivery is directly used. Consequently, please run again those commands without the --shared-memory option enabled.

With respect to the interprocess benchmark, the second figure that is titled Fast DDS v2.0.0 SHM Reliable is just doing that. Probably a better title (not prone to misunderstandings) should have been chosen.

Finally, find below a capture from Fast DDS CI so you can see how the performance tests are run:

          Start 2066: performance.latency.interprocess_best_effort_shm.data_loans_and_sharing

2066: Test command: /usr/bin/python3 "/home/jenkins/workspace/fastdds_linux/src/fastrtps/test/performance/latency/latency_tests.py" "--xml_file" "/home/jenkins/workspace/fastdds_linux/src/fastrtps/test/performance/latency/xml/interprocess_best_effort_shm.xml" "--demands_file" "/home/jenkins/workspace/fastdds_linux/src/fastrtps/test/performance/latency/payloads_demands.csv" "--interprocess" "--data_loans" "--data_sharing=on"
2066: Environment variables: 
2066:  LATENCY_TEST_BIN=/home/jenkins/workspace/fastdds_linux/build/fastrtps/test/performance/latency/LatencyTest
2066: Test timeout computed to be: 300
2066: Publisher command: /home/jenkins/workspace/fastdds_linux/build/fastrtps/test/performance/latency/LatencyTest publisher --samples 10000 --export_raw_data ./measurements_interprocess_best_effort_shm_data_loans_and_sharing.csv --domain 229 --xml /home/jenkins/workspace/fastdds_linux/src/fastrtps/test/performance/latency/xml/interprocess_best_effort_shm.xml --file /home/jenkins/workspace/fastdds_linux/src/fastrtps/test/performance/latency/payloads_demands.csv --data_sharing=on --data_loans --reliability=besteffort
2066: Subscriber command: /home/jenkins/workspace/fastdds_linux/build/fastrtps/test/performance/latency/LatencyTest subscriber --domain 229 --xml /home/jenkins/workspace/fastdds_linux/src/fastrtps/test/performance/latency/xml/interprocess_best_effort_shm.xml --file /home/jenkins/workspace/fastdds_linux/src/fastrtps/test/performance/latency/payloads_demands.csv --data_sharing=on --data_loans --reliability=besteffort
2066: Reading demands file: /home/jenkins/workspace/fastdds_linux/src/fastrtps/test/performance/latency/payloads_demands.csv
2066: Performing test with 1 subscribers and 10000 samples
2066: Reading demands file: /home/jenkins/workspace/fastdds_linux/src/fastrtps/test/performance/latency/payloads_demands.csv
2066: Overhead 0.0275005 ns
2066: �[32mEVERYTHING STOPPED FINE�[m
2066: Printing round-trip times in us, statistics for 10000 samples
2066:    Bytes, Samples,   stdev,    mean,     min,     50%,     90%,     99%,  99.99%,     max
2066: --------,--------,--------,--------,--------,--------,--------,--------,--------,--------,
2066:       16,   10000,  67.392,  17.851,   9.889,  14.874,  16.000,  19.145,2510.860,2533.227 
2066:    16384,   10000,   1.946,  15.368,  11.614,  14.911,  18.073,  21.281,  37.762,  40.836 
2066: �[32mEVERYTHING STOPPED FINE�[m

[xixioba]

Thanks for such a quick reply, unfortunately doesn't seem to solve the problem :(
I did the following tests(),

1. remove the --shared-memory option

./LatencyTest subscriber --data_sharing=on --data_loans

./LatencyTest publisher -s 1000 --data_sharing=on --data_loans
Performing test with 1 subscribers and 1000 samples
Overhead 0.0265574 ns
Printing round-trip times in us, statistics for 1000 samples
   Bytes, Samples,   stdev,    mean,     min,     50%,     90%,     99%,  99.99%,     max
--------,--------,--------,--------,--------,--------,--------,--------,--------,--------,
      16,    1000,1153.506,3507.524, 876.598,3127.536,5859.036,6304.244,7624.077,7659.932 
    1024,     999,1329.039,3657.350, 795.994,3096.432,5946.619,6352.057,7449.244,7484.556 
   64512,    1000,1231.320,3532.051,1158.140,3095.182,5917.702,6394.265,7740.265,7901.578 
 1048576,    1000,1224.764,3357.492, 633.535,2961.723,5734.077,6808.640,8339.452,8639.515

2. declare not to use --shared-memory option

./LatencyTest subscriber --data_sharing=on --data_loans --shared_memory=off

./LatencyTest publisher -s 1000 --data_sharing=on --data_loans 
--shared_memory=off
Performing test with 1 subscribers and 1000 samples
Overhead 0.0274725 ns
Printing round-trip times in us, statistics for 1000 samples
   Bytes, Samples,   stdev,    mean,     min,     50%,     90%,     99%,  99.99%,     max
--------,--------,--------,--------,--------,--------,--------,--------,--------,--------,
      16,    1000,1163.682,3531.233,1506.660,3128.119,5966.036,6287.494,6391.264,6861.473 
    1024,    1000,1175.270,3501.046, 576.202,3117.348,5857.390,6340.911,7454.848,8998.494 
   64512,     999,1280.792,3618.527, 785.265,3109.826,5901.681,6660.660,8336.014,8453.077 
 1048576,    1000,1229.975,3374.198, 761.431,2954.431,5757.431,6231.598,6538.014,7591.244

3. with --reliability=besteffort option

./LatencyTest subscriber --data_sharing=on --data_loans --shared_memory=off --reliability=besteffort

./LatencyTest publisher -s 1000 --data_sharing=on --data_loans --reliability=besteffort
Performing test with 1 subscribers and 1000 samples
Overhead 0.0196474 ns
Printing round-trip times in us, statistics for 1000 samples
   Bytes, Samples,   stdev,    mean,     min,     50%,     90%,     99%,  99.99%,     max
--------,--------,--------,--------,--------,--------,--------,--------,--------,--------,
      16,    1000,1093.284,3459.697,1142.751,3127.626,5856.335,6277.439,9384.292,9680.105 
    1024,    1000,1135.477,3440.155, 336.917,3121.314,5810.855,6650.459,7730.293,8241.377 
   64512,    1000,1326.141,3727.371,1121.772,3111.855,6061.522,6351.417,6534.918,6595.418 
 1048576,    1000,1212.574,3363.573,1152.980,2941.876,5741.168,6180.396,6479.439,6932.938

4. with --xml option

./LatencyTest subscriber --xml ./xml/interprocess_best_effort_shm.xml --data_sharing=on --data_loans --reliability=besteffort

./LatencyTest publisher -s 1000 --xml ./xml/interprocess_best_effort_shm.xml --data_sharing=on --data_loans  --reliability=besteffort 
Performing test with 1 subscribers and 1000 samples
Overhead 0.0216034 ns
Printing round-trip times in us, statistics for 1000 samples
   Bytes, Samples,   stdev,    mean,     min,     50%,     90%,     99%,  99.99%,     max
--------,--------,--------,--------,--------,--------,--------,--------,--------,--------,
      16,    1000, 705.589,3228.578,1739.416,3127.645,3139.937,6259.478,6279.770,6330.979 
    1024,    1000,1257.386,3611.339, 254.228,3115.770,5924.874,6416.832,7104.062,7254.312 
   64512,    1000,1123.486,3418.330, 289.895,3094.937,5783.520,6384.353,7738.582,8128.708 
 1048576,    1000,1232.479,3374.710, 648.582,2933.729,5719.708,6172.707,6921.582,7743.936

[JLBuenoLopez]

Hi @xixioba,

Understand that the performance results are dependent on your platform, operating system, resources, dedicated threads, etc. These tests are helpful to benchmark between different transport mechanisms so you may find helpful running them with some more option that allow you to compare.

Finally, you may be interested in contacting our support team for further help with your use case.