Skip to content

Commit

Permalink
feat: improve additional-settings-for-developers section (autowarefou…
Browse files Browse the repository at this point in the history 
…ndation#587)

Signed-off-by: M. Fatih Cırıt <[email protected]>
  • Loading branch information
@xmfcx
xmfcx authored Jul 17, 2024
1 parent 064f63d commit 7cd0261
Show file tree
Hide file tree
Showing 10 changed files with 347 additions and 210 deletions.
2 changes: 2 additions & 0 deletions docs/installation/additional-settings-for-developers/.pages
View file
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
nav:
- index.md
- console-settings.md
- network-configuration
29 changes: 29 additions & 0 deletions docs/installation/additional-settings-for-developers/console-settings.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Console settings for ROS 2

## Colorizing logger output

By default, ROS 2 logger doesn't colorize the output.
To colorize it, add the following to your `~/.bashrc`:

```bash
export RCUTILS_COLORIZED_OUTPUT=1
```

## Customizing the format of logger output

By default, ROS 2 logger doesn't output detailed information such as file name, function name, or line number.
To customize it, add the following to your `~/.bashrc`:

```bash
export RCUTILS_CONSOLE_OUTPUT_FORMAT="[{severity} {time}] [{name}]: {message} ({function_name}() at {file_name}:{line_number})"
```

For more options, see [here](https://docs.ros.org/en/rolling/Tutorials/Demos/Logging-and-logger-configuration.html#console-output-formatting).

## Colorized GoogleTest output

Add `export GTEST_COLOR=1` to your `~/.bashrc`.

For more details, refer to [Advanced GoogleTest Topics: Colored Terminal Output](https://google.github.io/googletest/advanced.html#colored-terminal-output).

This is useful when running tests with `colcon test`.
136 changes: 2 additions & 134 deletions docs/installation/additional-settings-for-developers/index.md
View file
Original file line number Diff line number Diff line change
@@ -1,136 +1,4 @@
# Additional settings for developers

## Console settings for ROS 2

### Colorizing logger output

By default, ROS 2 logger doesn't colorize the output.
To colorize it, write the following in your `.bashrc`:

```bash
export RCUTILS_COLORIZED_OUTPUT=1
```

### Customizing the format of logger output

By default, ROS 2 logger doesn't output detailed information such as file name, function name, or line number.
To customize it, write the following in your `.bashrc`:

```bash
export RCUTILS_CONSOLE_OUTPUT_FORMAT="[{severity} {time}] [{name}]: {message} ({function_name}() at {file_name}:{line_number})"
```

For more options, see [here](https://docs.ros.org/en/rolling/Tutorials/Logging-and-logger-configuration.html#console-output-formatting).

## Network settings for ROS 2

ROS 2 employs DDS, and the configuration of ROS 2 and DDS is described separately.
For ROS 2 networking concepts, refer to the [official documentation](http://design.ros2.org/articles/ros_on_dds.html).

### ROS 2 network setting

ROS 2 multicasts data on the local network by default.
Therefore, when you develop in an office, the data flows over the local network of your office.
It may cause collisions of packets or increases in network traffic.

To avoid these, there are two options.

- Localhost-only communication
- Same domain only communication on the local network

Unless you plan to use multiple host computers on the local network, localhost-only communication is recommended.
For details, refer to the sections below.

#### Enabling localhost-only communication

Write the following in your `.bashrc`:
For more information, see the [ROS 2 documentation](https://docs.ros.org/en/humble/Tutorials/Beginner-CLI-Tools/Configuring-ROS2-Environment.html#the-ros-localhost-only-variable).

```bash
export ROS_LOCALHOST_ONLY=1
```

If you export `ROS_LOCALHOST_ONLY=1`, `MULTICAST` must be enabled at the loopback address.
To verify that `MULTICAST` is enabled, use the following command.

```console
$ ip link show lo
1: lo: <LOOPBACK,MULTICAST,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
```

If the word `MULTICAST` is not printed, use the following command to enable it.

```bash
sudo ip link set lo multicast on
```

#### Same domain only communication on the local network

ROS 2 uses `ROS_DOMAIN_ID` to create groups and communicate between machines in the groups.
Since all ROS 2 nodes use domain ID `0` by default, it may cause unintended interference.

To avoid it, set a different domain ID for each group in your `.bashrc`:

```bash
# Replace X with the Domain ID you want to use
# Domain ID should be a number in range [0, 101] (inclusive)
export ROS_DOMAIN_ID=X
```

Also confirm that `ROS_LOCALHOST_ONLY` is `0` by using the following command.

```bash
echo $ROS_LOCALHOST_ONLY # If the output is 1, localhost has priority.
```

For more information, see the [ROS 2 Documentation](https://docs.ros.org/en/humble/Tutorials/Beginner-CLI-Tools/Configuring-ROS2-Environment.html#the-ros-domain-id-variable).

### DDS settings

Autoware uses DDS for inter-node communication. [ROS 2 documentation](https://docs.ros.org/en/humble/How-To-Guides/DDS-tuning.html) recommends users to tune DDS to utilize its capability. Especially, receive buffer size is the critical parameter for Autoware. If the parameter is not large enough, Autoware will failed in receiving large data like point cloud or image.

#### Tuning DDS

Unless customized, CycloneDDS is adopted by default. For example, to execute Autoware with CycloneDDS, prepare a config file. A sample config file is given below. Save it as `cyclonedds_config.xml`.

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
<Domain Id="any">
<General>
<Interfaces>
<NetworkInterface autodetermine="true" priority="default" multicast="default" />
</Interfaces>
<AllowMulticast>default</AllowMulticast>
<MaxMessageSize>65500B</MaxMessageSize>
</General>
<Internal>
<SocketReceiveBufferSize min="10MB"/>
<Watermarks>
<WhcHigh>500kB</WhcHigh>
</Watermarks>
</Internal>
</Domain>
</CycloneDDS>
```

This configuration is mostly taken from [Eclipse Cyclone DDS:Run-time configuration documentation](https://github.com/eclipse-cyclonedds/cyclonedds/tree/a10ced3c81cc009e7176912190f710331a4d6caf#run-time-configuration).
You can see why each value is set as such under the documentation link.

Set the config file path and enlarge the Linux kernel maximum buffer size before launching Autoware.

```bash
export CYCLONEDDS_URI=file:///absolute/path/to/cyclonedds_config.xml
sudo sysctl -w net.core.rmem_max=2147483647
```

For more information, Refer to [ROS 2 documentation](https://docs.ros.org/en/humble/How-To-Guides/DDS-tuning.html). Reading user guide for chosen DDS is helpful for more understanding.

#### Tuning DDS for multiple host computers (for advanced users)

When Autoware runs on multiple host computers, IP Fragmentation should be taken into account. As [ROS 2 documentation](https://docs.ros.org/en/humble/How-To-Guides/DDS-tuning.html#cross-vendor-tuning) recommends, parameters for IP Fragmentation should be set as shown in the following example.

```bash
sudo sysctl -w net.ipv4.ipfrag_time=3
sudo sysctl -w net.ipv4.ipfrag_high_thresh=134217728 # (128 MB)
```
- [Console settings for ROS 2](./console-settings.md)
- [Network configuration for ROS 2](./network-configuration/index.md)
5 changes: 5 additions & 0 deletions docs/installation/additional-settings-for-developers/network-configuration/.pages
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
nav:
- index.md
- dds-settings.md
- enable-multicast-for-lo.md
- multiple-computers.md
128 changes: 128 additions & 0 deletions ...lation/additional-settings-for-developers/network-configuration/dds-settings.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# DDS settings for ROS 2 and Autoware

## Enable localhost-only communication

1. [Enable `multicast` for `lo`](./enable-multicast-for-lo.md)
2. Make sure `export ROS_LOCALHOST_ONLY=1` **is NOT** present in `.bashrc`.
- See [About `ROS_LOCALHOST_ONLY` environment variable](#about-ros_localhost_only-environment-variable) for more information.

## Tune DDS settings

Autoware uses DDS for internode communication. [ROS 2 documentation](https://docs.ros.org/en/humble/How-To-Guides/DDS-tuning.html) recommends users to tune DDS to utilize its capability.

!!! note

CycloneDDS is the recommended and most tested DDS implementation for Autoware.

!!! warning

If you don't tune these settings, Autoware will fail to receive large data like point clouds or images.

### Tune system-wide network settings

Set the config file path and enlarge the Linux kernel maximum buffer size before launching Autoware.

```bash
# Increase the maximum receive buffer size for network packets
sudo sysctl -w net.core.rmem_max=2147483647 # 2 GiB, default is 208 KiB

# IP fragmentation settings
sudo sysctl -w net.ipv4.ipfrag_time=3 # in seconds, default is 30 s
sudo sysctl -w net.ipv4.ipfrag_high_thresh=134217728 # 128 MiB, default is 256 KiB
```

To make it permanent,

```bash
sudo nano /etc/sysctl.d/10-cyclone-max.conf
```

Paste the following into the file:

```bash
# Increase the maximum receive buffer size for network packets
net.core.rmem_max=2147483647 # 2 GiB, default is 208 KiB

# IP fragmentation settings
net.ipv4.ipfrag_time=3 # in seconds, default is 30 s
net.ipv4.ipfrag_high_thresh=134217728 # 128 MiB, default is 256 KiB
```

Details of each parameter here is explained in the [ROS 2 documentation](https://docs.ros.org/en/humble/How-To-Guides/DDS-tuning.html#cross-vendor-tuning).

#### Validate the sysctl settings

```console
user@pc$ sysctl net.core.rmem_max net.ipv4.ipfrag_time net.ipv4.ipfrag_high_thresh
net.core.rmem_max = 2147483647
net.ipv4.ipfrag_time = 3
net.ipv4.ipfrag_high_thresh = 134217728
```

### CycloneDDS Configuration

Save the following file as `~/cyclonedds.xml`.

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
<Domain Id="any">
<General>
<Interfaces>
<NetworkInterface autodetermine="false" name="lo" priority="default" multicast="default" />
</Interfaces>
<AllowMulticast>default</AllowMulticast>
<MaxMessageSize>65500B</MaxMessageSize>
</General>
<Internal>
<SocketReceiveBufferSize min="10MB"/>
<Watermarks>
<WhcHigh>500kB</WhcHigh>
</Watermarks>
</Internal>
</Domain>
</CycloneDDS>
```

Then add the following lines to your `~/.bashrc` file.

```bash
export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp

export CYCLONEDDS_URI=file:///absolute/path/to/cyclonedds.xml
# Replace `/absolute/path/to/cyclonedds.xml` with the actual path to the file.
# Example: export CYCLONEDDS_URI=file:///home/user/cyclonedds.xml
```

You can refer to [Eclipse Cyclone DDS: Run-time configuration documentation](https://github.com/eclipse-cyclonedds/cyclonedds/tree/a10ced3c81cc009e7176912190f710331a4d6caf#run-time-configuration) for more details.

!!! warning

`RMW_IMPLEMENTATION` variable might be already set with [Ansible/RMW Implementation](https://github.com/autowarefoundation/autoware/tree/main/ansible/roles/rmw_implementation#manual-installation).

Check and remove the duplicate line if necessary.

## Additional information

### About `ROS_LOCALHOST_ONLY` environment variable

Previously, we used to set `export ROS_LOCALHOST_ONLY=1` to enable localhost-only communication.
But because of [an ongoing issue](https://github.com/ros2/rmw_cyclonedds/issues/370), this method doesn't work.

!!! warning

Do not set `export ROS_LOCALHOST_ONLY=1` in `~/.bashrc`.

If you do so, it will cause an error with RMW.

Remove it from `~/.bashrc` if you have set it.

### About `ROS_DOMAIN_ID` environment variable

We can also set `export ROS_DOMAIN_ID=3(or any number 1 to 255)` (`0` by default) to avoid interference with other ROS 2 nodes on the same network.

But since `255` is a very small number, it might interfere with other computers on the same network unless you make sure everyone has a unique domain ID.

Another problem is that if someone runs a test that uses ROS 2 [launch_testing](https://github.com/ros2/launch/blob/a317c54bbbf2dfeec35fbb6d2b5913939d02750d/launch_testing/README.md) framework,
by default it will use a random domain ID to isolate between tests even on the same machine.
See [this PR](https://github.com/ros2/launch/pull/251) for more details.
Loading

0 comments on commit 7cd0261

Please sign in to comment.