Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Patch 8 - check but is redundant? #8

Open
wants to merge 54 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
5cc5dad
rmw_fastrtps_cpp report 2021 (#44)
IkerLuengo Oct 5, 2021
ed10fb7
Change ROS2 -> ROS 2 in the eProsima Humble report. (#60)
clalancette Oct 5, 2021
526a461
Remove trailing whitespace from eProsima report. (#61)
clalancette Oct 5, 2021
78a7049
In the eProsima report, change cycloneDDS -> Cyclone DDS (#62)
clalancette Oct 5, 2021
f8a7eef
Fix up image rendering in the eProsima response. (#63)
clalancette Oct 5, 2021
a5ef75d
Fix the links to the eProsima images. (#64)
clalancette Oct 5, 2021
3ecc957
Fix a number of very minor typos in the eProsima part of the report. …
clalancette Oct 5, 2021
4b919f5
The Humble report for Eclipse Cyclone DDS (#66)
clalancette Oct 7, 2021
e2edf23
Remove trailing whitespace from Cyclone DDS report.
clalancette Oct 6, 2021
2f61a6e
Remove width=50% from images in Cyclone DDS report.
clalancette Oct 6, 2021
18e2814
Get the report skeleton in place. (#68)
clalancette Oct 7, 2021
1504969
Fix cyclone typos and fix quality link (#69)
paudrow Oct 7, 2021
4a67e9c
Minor fixes to links in eProsima response. (#70)
clalancette Oct 8, 2021
7cea742
Try to fix the table formatting for REP-2004. (#71)
clalancette Oct 8, 2021
14c8052
Add in a discussion for the REP-2004 table. (#72)
clalancette Oct 8, 2021
57450b4
Rerun the objective measurements for Humble (#73)
clalancette Oct 8, 2021
d09ffe2
change link caption for Fast DDS response (#74)
IkerLuengo Oct 11, 2021
cb5ec42
Write large pieces of the Humble report.
clalancette Oct 8, 2021
b333231
Typos and formatting changes
paudrow Oct 12, 2021
1e7ecee
Exec summary + number sections
paudrow Oct 12, 2021
cd45b25
Add an Appendix and do some editing.
clalancette Oct 12, 2021
a6ec342
Updated dates on data to correct date, made nb easier to change date,…
kscottz Oct 12, 2021
2267687
Merge pull request #76 from osrf/humble-additions
kscottz Oct 12, 2021
39ff434
Early redacted data, will update later.
kscottz Oct 12, 2021
c369466
plots for new data.
kscottz Oct 12, 2021
6290e66
Merge pull request #77 from osrf/humble-additions
kscottz Oct 12, 2021
44fc7f7
A few more minor edits. (#78)
clalancette Oct 13, 2021
301bc85
Final Data
kscottz Oct 13, 2021
7fdb873
Last plot tweaks.
kscottz Oct 13, 2021
2f4c9cf
Rough plots.
kscottz Oct 13, 2021
3ce0db9
Merge pull request #80 from osrf/humble-additions
kscottz Oct 13, 2021
81dcf96
fixed typos and minor omissions (#79)
joespeed Oct 13, 2021
b7cc5d8
Adding the text to go with plots and quotes.
kscottz Oct 13, 2021
5194a66
Merge pull request #81 from osrf/humble-additions
kscottz Oct 13, 2021
bb548d9
Formatting fix.
kscottz Oct 13, 2021
eb8bde2
Merge pull request #82 from osrf/humble-additions
kscottz Oct 13, 2021
94907d9
Formatting test.
kscottz Oct 13, 2021
adcea34
Merge pull request #83 from osrf/humble-additions
kscottz Oct 13, 2021
795ec30
Proof reading, formatting, consistency.
kscottz Oct 13, 2021
9ba7f44
Merge pull request #84 from osrf/humble-additions
kscottz Oct 13, 2021
aba75d2
Uhg, open quote fix.
kscottz Oct 13, 2021
5375d50
Merge pull request #85 from osrf/humble-additions
kscottz Oct 13, 2021
bbc6314
Adding more quotes.
kscottz Oct 13, 2021
1a5e777
Merge pull request #86 from osrf/humble-additions
kscottz Oct 13, 2021
ef6e9d9
Fill in executive summary, Fast DDS to Fast RTPS
kscottz Oct 13, 2021
b73088c
Merge branch 'main' into humble-additions
kscottz Oct 13, 2021
3ca2132
Merge pull request #87 from osrf/humble-additions
kscottz Oct 13, 2021
8cea2dc
"manual and nightly performance testing"
joespeed Oct 14, 2021
f3230bc
Update README.md
joespeed Oct 14, 2021
cad6299
Update README.md
joespeed Oct 14, 2021
97eeaaa
Update README.md
joespeed Oct 14, 2021
2800974
Update README.md
joespeed Oct 14, 2021
d8a9a54
Update README.md
joespeed Oct 14, 2021
d5dd864
OR build farm nightly performance testing
joespeed Oct 14, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
A few more minor edits. (osrf#78)
Signed-off-by: Chris Lalancette <clalancette@openrobotics.org>
  • Loading branch information
clalancette authored Oct 13, 2021
commit 44fc7f7a5f6434e65ecf9154215a66a1119ac892
16 changes: 9 additions & 7 deletions humble/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,18 +30,19 @@ In order to be considered for this report, RMW implementations needed to meet a

Two RMW implementations currently meet this minimum bar: `rmw_cyclonedds_cpp` based on Cyclone DDS and `rmw_fastrtps_cpp` based on Fast RTPS.
From here on out, Cyclone DDS will be used synonymously with `rmw_cyclonedds_cpp` and Fast RTPS will be used synonymously with `rmw_fastrtps_cpp` unless otherwise specified.
This report evaluates these two DDS implementations along with their RMW implementations for ROS 2, namely Cyclone DDS and Fast RTPS (this is now called Fast DDS, but this report will continue to refer to it as Fast RTPS).
Note that Fast RTPS is now known as Fast DDS, but for historical reasons this report will continue to refer to it as Fast RTPS.
This report evaluates these two DDS implementations along with their RMW implementations for ROS 2.

The application performance and community engagement is measured objectively by Open Robotics along 4 axes:

* [Build Farm Performance Metrics](#BuildFarm) - this dataset covers basic RMW performance in terms of memory, CPU utilitization, and lost messages using a simplified network under optimal conditions
* [REP-2004 Code Quality Data](#CodeQuality) - this simple table represents the [REP-2004](https://www.ros.org/reps/rep-2004.html) code quality standards as implemented for both the RMW and the underlying DDS implementation
* [REP-2004 Code Quality Data](#CodeQuality) - this simple table represents the [REP-2004](https://www.ros.org/reps/rep-2004.html) code quality standards as implemented for both the RMWs and DDS implementations
* [GitHub User Statistics](#GitHubStats) - this section looks at GitHub community engagement data over the preceding six months for both the RMWs and DDS implementations
* [User Survey Results](#Survey) - this section presents the results of a survey of the ROS 2 community asking them about the overall end-user experience
* [User Survey Results](#Survey) - this section presents the results of a survey of the ROS 2 community asking about the overall end-user experience

The RMW providers were each asked a series of questions that are current concerns of the ROS 2 TSC.
In the [DDS Provider Reponses](#DDSProviderResponse) section, each of the RMW providers were asked a series of questions that are current concerns of the ROS 2 TSC.
The questionnaire that was provided is available [here](dds_provider_question_template.md).
The results in each of the responses are necessarily biased, and because the hardware and software used for each providers response is different, no direct comparison is possible.
The responses from the providers are necessarily biased, and because the hardware and software used for each providers response is different, no direct comparison is possible.
However, the manner in which the providers responded to the questionnaire should give some insights into how they are thinking about the problems that ROS 2 users are facing.

# <a id="ExecutiveSummary"></a> Executive Summary
Expand Down Expand Up @@ -142,7 +143,7 @@ However, comparing the results to the [2020 report](../galactic/README.md), it i
In terms of messages received both RMW implementations appear to perform well up until the 1MB message size.
After that point, we see a divergence in the implementations.
Cyclone DDS starts dropping messages at the 2MB size, and continues to drop more messages as the size increases.
Fast RTPS async receives all messages at the 2MB size, and starts dropping messages after that.
Fast RTPS async drops a small number of messages at the 1MB and 2MB size, but starts dropping a large number of messages after that.
Fast RTPS sync mode receives all messages up until 4MB size, and starts dropping messages after that.
It should be noted that compared to the [2020 report](../galactic/README.md), both implementations now deal with 2MB sizes better than before, with Fast RTPS showing a larger improvement.

Expand All @@ -157,7 +158,8 @@ After that, the Cyclone DDS CPU usage goes up quickly with the amount of data be
The Fast RTPS async CPU usage spikes at 2MB, and then starts going down a lot more dramatically.
In both of those cases, that is likely happening because more messages are being dropped after 2MB.
The Fast RTPS sync CPU usage goes up at the PointCloud512k size, drops dramatically for the 1MB, 2MB, and 4MB sizes, and then spikes again for the 8MB size.
It isn't completely clear on why that happens.
This is an unexpected result, since CPU usage should increase as the data size goes up.
This may be a measurement error or other problem in that particular test.
Compared to the [2020 report](../galactic/README.md), Cyclone DDS sync and Fast RTPS async are approximately the same, with Fast RTPS sync showing a very different performance curve.

# <a id="CodeQuality"></a> 2. REP-2004 Code Quality Metrics
Expand Down