Skip to content

Commit

Permalink
Merge pull request #132 from open-dynamic-robot-initiative/fkloss/doc_rt
Browse files Browse the repository at this point in the history 
doc: Update RT kernel pages
  • Loading branch information
@luator
luator authored Jul 22, 2024
2 parents 934eff7 + 6bf7e15 commit ae3b768
Show file tree
Hide file tree
Showing 3 changed files with 171 additions and 88 deletions.
2 changes: 1 addition & 1 deletion doc/about_architecture.rst
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.. _architecture::
.. _architecture:

About the Software Architecture
===============================
Expand Down
107 changes: 40 additions & 67 deletions doc/preempt_rt_kernel.rst
View file
Original file line number Diff line number Diff line change
@@ -1,19 +1,16 @@
:orphan:

***********************
PREEMPT_RT Linux Kernel
***********************
*********************************************
Build and Install the PREEMPT_RT Linux Kernel
*********************************************

To get good real-time performance on Linux, we recommend using a Linux kernel
with the `PREEMPT_RT patch`_. Unfortunately, this requires one to build the kernel
from source (at least on Ubuntu). We provide some help for this in the
following.

Note that this page only covers installation of the kernel, for further
real-time-related configuration of the system see :doc:`real_time`.

Unfortunately, Nvidia does not support the PREEMPT_RT patch, so you may not be
able to use Nvidia drivers (and thus also no CUDA) when using this kernel.
Note that Nvidia drivers do officially not support the PREEMPT_RT patch. It may still
work (see :ref:`nvidia_preempt_rt` below) but you are at your own risk.

An alternative may be the "lowlatency" kernel. It is easier to install and
works with Nvidia drivers but has weaker real-time capabilities (see
Expand All @@ -24,86 +21,62 @@ Install PREEMPT_RT Kernel
=========================

To install the patched kernel on Ubuntu, you may use our script
install_rt_preempt_kernel.sh_. It is tested for Ubuntu 18.04 but may also work
with other recent versions.

Before running the script, you may want to modify it a bit, though:
``install_rt_preempt_kernel.sh``. We provide adapted versions for different Ubuntu
versions, which can be found `here <install_rt_preempt_kernel.sh_>`__.

- The ``VERSION_`` variables at the top refer to the kernel version that will be
installed. Usually you can leave the default values. See
preempt_rt_versions_ for available versions in case you want to change it.
The ``VERSION_`` variables at the top of the script refer to the kernel version that
will be installed. Usually you can leave the default values, but you can change it here
if you want a different version. See preempt_rt_versions_ for available versions.

Then simply execute the script in a terminal. Internally, sudo is used in some
steps, so the user executing it needs to have sudo-permission.

In the beginning (after downloading some things) you will be asked to manually
adjust some configuration settings. Before entering the menu, the script prints
the following instructions::
.. note::

Please apply the following configurations in the next step:
In the beginning (after downloading some things) you will be asked to manually
adjust some configuration settings. Before entering the menu, the script prints
instructions like the following::

General setup [Enter]
Local version - append to kernel release: [Enter] Add '-preempt-rt'
Please apply the following configurations in the next step:

General setup [Enter]
Preemption Model (Voluntary Kernel Preemption (Desktop)) [Enter]
Fully Preemptible Kernel (RT) [Enter] #Select
[...]

However, depending on the kernel version the "Preemption Model" setting may be
found in the "Processor type and features" menu instead.
General setup [Enter]
Preemption Model (Voluntary Kernel Preemption (Desktop)) [Enter]
Fully Preemptible Kernel (RT) [Enter] #Select

However, depending on the kernel version the "Preemption Model" setting may be
found in the "Processor type and features" menu instead.

The script will automatically download, build and install the kernel. This will
take a while.

When finished, go back to :doc:`real_time` and follow the further steps to configure
your system for real-time usage.

.. _boot_rt_kernel:

Boot with the PREEMPT_RT Kernel
===============================

Once the PREEMPT_RT kernel is installed, you need to reboot and select the
"preempt-rt" kernel in the GRUB menu (go to "Advanced options for Ubuntu", it
should be listed there).

When the system is running, you can check which kernel is running with
``uname -a``. It should print something containing "PREEMPT_RT".


Select PREEMPT_RT Kernel by Default
-----------------------------------

You can configure GRUB to automatically select this kernel by setting
``GRUB_DEFAULT`` in ``/etc/default/grub``. For this, first the identifier of
the kernel needs to be determined. Open a terminal and run

.. code-block:: bash
cat /boot/grub/grub.cfg | grep -w -e menuentry -e submenu
It should print something like this::

menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-1a26991b-b045-48dd-bb12-064a2725b80b' {
submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-1a26991b-b045-48dd-bb12-064a2725b80b' {
menuentry 'Ubuntu, with Linux 5.4.93-rt51-preempt-rt' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.93-rt51-preempt-rt-advanced-1a26991b-b045-48dd-bb12-064a2725b80b' {
menuentry 'Ubuntu, with Linux 5.4.93-rt51-preempt-rt (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.93-rt51-preempt-rt-recovery-1a26991b-b045-48dd-bb12-064a2725b80b' {
menuentry 'Ubuntu, with Linux 5.4.0-65-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.0-65-generic-advanced-1a26991b-b045-48dd-bb12-064a2725b80b' {
menuentry 'Ubuntu, with Linux 5.4.0-65-generic (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.0-65-generic-recovery-1a26991b-b045-48dd-bb12-064a2725b80b' {

.. _nvidia_preempt_rt:

For ``GRUB_DEFAULT`` the path full submenu/menuentry is needed, using the id of
each step (the last part in the line, "gnulinux-..."), separated by ">". In
this specific case, the setting for starting the rt-kernel would be::
Using Nvidia Drivers with the PREEMPT_RT Kernel
===============================================

GRUB_DEFAULT = "gnulinux-advanced-1a26991b-b045-48dd-bb12-064a2725b80b>gnulinux-5.4.93-rt51-preempt-rt-advanced-471e9718-013f-4cbb-91a7-d22635173b70"
Officially, Nvidia drivers do not support the PREEMPT_RT kernel. However, at least with
more recent versions, it seems to work in practice. However, you need set the
environment variable ``IGNORE_PREEMPT_RT_PRESENCE=1`` when installing it with apt.
Complete steps:

After saving the changes in ``/etc/default/grub`` you need to run the following
command for the changes to become active::
1. First uninstall any Nvidia drivers
2. Install the PREEMPT_RT kernel (see above)
3. Install the drivers with the following command (adjust driver version to the desired
one)::

sudo update-grub
sudo IGNORE_PREEMPT_RT_PRESENCE=1 apt install nvidia-driver-530

Then reboot and verify that the correct kernel is used.
Please note that this variable also needs to be set when upgrading packages, so you may
want to set it in a global place like `/etc/environment` and disable unattended upgrades
for the driver.


.. _PREEMPT_RT patch: https://wiki.linuxfoundation.org/realtime/start
.. _install_rt_preempt_kernel.sh: https://github.com/machines-in-motion/ubuntu_installation_scripts/blob/master/rt-preempt/ubuntu18.04/install_rt_preempt_kernel.sh
.. _install_rt_preempt_kernel.sh: https://github.com/machines-in-motion/ubuntu_installation_scripts/tree/master/rt-preempt
.. _preempt_rt_versions: https://wiki.linuxfoundation.org/realtime/preempt_rt_versions
150 changes: 130 additions & 20 deletions doc/real_time.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,36 +6,144 @@ Since a constant update rate of the :cpp:class:`~robot_interfaces::RobotBackend`
is important for stable control of the robot, a real-time capable operating
system should be used.

We are using Ubuntu with a PREEMPT_RT kernel patch. Below are instructions on
how to install the patched kernel and how to configure the system to use it.
For this we are using a standard Linux distribution (Ubuntu in our case) but install a
kernel with better real-time capabilities. There are two options: the "low-latency"
kernel which is easy to install but doesn't give full real-time guarantees, and the
preempt_rt kernel, which has better real-time performance but is more tedious to
install.

Below are instructions for both of them and on how to configure the system to use them.

Install PREEMPT_RT Kernel
=========================

See :doc:`preempt_rt_kernel`.
.. _lowlatency_kernel:

Installation: Low-latency vs PREEMPT_RT Kernel
==============================================

.. _lowlatency_kernel:
We recommend one of the following Linux kernel variants:

**low-latency:**

- Fewer latency guarantees as with PREEMPT_RT but often good enough. Latency can
increase if the system load is high, though. It can be useful to assign separate CPU
cores for the real-time critical processes.
- Can be used with Nvidia drivers.
- Easy to install. On Ubuntu it's just::

sudo apt install linux-lowlatency


**PREEMPT_RT patch:**

- Better real-time guarantees.
- Officially not supported by Nvidia drivers (but may still kind of work, see
:ref:`nvidia_preempt_rt`).
- Difficult to install as it has to be built from source. See :doc:`preempt_rt_kernel`.


Historically we used the PREEMPT_RT kernel (and still do on many of our robots) but
lately also used the low-latency kernel in some projects without problems (you have to
be a bit more careful to not overload the CPU, though).
Since the low-latency Kernel is much easier to install, you may want to try with that
one first and only move to the PREEMPT_RT version if you notice timing-related issues.

The further system configuration described in the following is the same in both cases.


.. _boot_rt_kernel:

Boot with the real-time Kernel
==============================

Once the real-time kernel is installed, you need to reboot and select the corresponding
kernel ("preempt-rt" or "low-latency") kernel in the GRUB menu (go to "Advanced options
for Ubuntu", it should be listed there).

When the system is running, you can check which kernel is running with
``uname -a``. It should print something containing "PREEMPT_RT"/"lowlatency".


Select real-time Kernel by Default
----------------------------------

You can configure GRUB to automatically select a specific kernel when booting the
computer. We provide a script to semi-automatically update the configuration (option
1), but you may also edit it manually (option 2).

Option 1: Semi-automatic
~~~~~~~~~~~~~~~~~~~~~~~~

We provide a script grub_select_default_kernel.py_ for this. Download it and execute
with

Alternative: The Low Latency Kernel
===================================
.. code-block:: bash
A potential alternative to the PREEMPT_RT patch is using the "lowlatency"
kernel. It is easier to install and does not conflict with Nvidia drivers. On
the other hand, the real-time capabilities it provides are weaker than those of
the PREEMPT_RT kernel. Whether it is sufficient for you depends on your
application (e.g. how much computational load you put on the system while the
robot is running). We still recommend using the PREEMPT_RT kernel but you may
test with the lowlatency kernel first.
sudo python3 grub_select_default_kernel.py
To install on Ubuntu 20.04 (on other versions of Ubuntu adjust the version at
the end)::
It will automatically parse the GRUB configuration and list the available boot options.
Select the desired kernel and press enter. It will then show a diff of the change it
intends to apply to ``/etc/default/grub``. **Review the diff carefully and only confirm
if it looks good!** It should only modify the line starting with ``GRUB_DEFAULT``, i.e.
like this:

sudo apt install linux-lowlatency-hwe-20.04
.. code-block:: diff
--- current
+++ new
@@ -3,7 +3,7 @@
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'
-GRUB_DEFAULT=0
+GRUB_DEFAULT="gnulinux-advanced-cc71b1fb-b530-4694-a839-aecf600f1f49>gnulinux-5.15.0-112-generic-advanced-cc71b1fb-b530-4694-a839-aecf600f1f49"
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
If all is good and you confirmed, you still need to run

.. code-block:: bash
sudo update-grub
to apply the change. Then reboot and verify that the correct kernel is used.


Option 2: Manually
~~~~~~~~~~~~~~~~~~

If you don't want to use the script mentioned above, you may also edit the GRUB
configuration manually.

For this, first the identifier of the kernel needs to be determined. Open a terminal and run

.. code-block:: bash
cat /boot/grub/grub.cfg | grep -w -e menuentry -e submenu
It should print something like this::

menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-1a26991b-b045-48dd-bb12-064a2725b80b' {
submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-1a26991b-b045-48dd-bb12-064a2725b80b' {
menuentry 'Ubuntu, with Linux 5.4.93-rt51-preempt-rt' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.93-rt51-preempt-rt-advanced-1a26991b-b045-48dd-bb12-064a2725b80b' {
menuentry 'Ubuntu, with Linux 5.4.93-rt51-preempt-rt (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.93-rt51-preempt-rt-recovery-1a26991b-b045-48dd-bb12-064a2725b80b' {
menuentry 'Ubuntu, with Linux 5.4.0-65-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.0-65-generic-advanced-1a26991b-b045-48dd-bb12-064a2725b80b' {
menuentry 'Ubuntu, with Linux 5.4.0-65-generic (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.0-65-generic-recovery-1a26991b-b045-48dd-bb12-064a2725b80b' {


For ``GRUB_DEFAULT`` the path full submenu/menuentry is needed, using the id of
each step (the last part in the line, "gnulinux-..."), separated by ">". In
this specific case, the setting for starting the rt-kernel would be::

GRUB_DEFAULT = "gnulinux-advanced-1a26991b-b045-48dd-bb12-064a2725b80b>gnulinux-5.4.93-rt51-preempt-rt-advanced-471e9718-013f-4cbb-91a7-d22635173b70"

After saving the changes in ``/etc/default/grub`` you need to run the following
command for the changes to become active::

sudo update-grub

Then reboot and verify that the correct kernel is used.

Configuration to use this kernel by default works the same as for the PREEMPT_RT
kernel, see :ref:`boot_rt_kernel`.


System Configuration
Expand Down Expand Up @@ -120,3 +228,5 @@ following command::
Warning this thread is not going to be real time.
.. _grub_select_default_kernel.py: https://github.com/machines-in-motion/ubuntu_installation_scripts/blob/master/rt-preempt/grub_select_default_kernel.py

0 comments on commit ae3b768

Please sign in to comment.