diff --git a/doc/about_architecture.rst b/doc/about_architecture.rst index 13590ee..94ad9cd 100644 --- a/doc/about_architecture.rst +++ b/doc/about_architecture.rst @@ -1,4 +1,4 @@ -.. _architecture:: +.. _architecture: About the Software Architecture =============================== diff --git a/doc/preempt_rt_kernel.rst b/doc/preempt_rt_kernel.rst index 9639b85..b7fd31c 100644 --- a/doc/preempt_rt_kernel.rst +++ b/doc/preempt_rt_kernel.rst @@ -1,19 +1,16 @@ :orphan: -*********************** -PREEMPT_RT Linux Kernel -*********************** +********************************************* +Build and Install the PREEMPT_RT Linux Kernel +********************************************* To get good real-time performance on Linux, we recommend using a Linux kernel with the `PREEMPT_RT patch`_. Unfortunately, this requires one to build the kernel from source (at least on Ubuntu). We provide some help for this in the following. -Note that this page only covers installation of the kernel, for further -real-time-related configuration of the system see :doc:`real_time`. - -Unfortunately, Nvidia does not support the PREEMPT_RT patch, so you may not be -able to use Nvidia drivers (and thus also no CUDA) when using this kernel. +Note that Nvidia drivers do officially not support the PREEMPT_RT patch. It may still +work (see :ref:`nvidia_preempt_rt` below) but you are at your own risk. An alternative may be the "lowlatency" kernel. It is easier to install and works with Nvidia drivers but has weaker real-time capabilities (see @@ -24,86 +21,62 @@ Install PREEMPT_RT Kernel ========================= To install the patched kernel on Ubuntu, you may use our script -install_rt_preempt_kernel.sh_. It is tested for Ubuntu 18.04 but may also work -with other recent versions. - -Before running the script, you may want to modify it a bit, though: +``install_rt_preempt_kernel.sh``. We provide adapted versions for different Ubuntu +versions, which can be found `here `__. -- The ``VERSION_`` variables at the top refer to the kernel version that will be - installed. Usually you can leave the default values. See - preempt_rt_versions_ for available versions in case you want to change it. +The ``VERSION_`` variables at the top of the script refer to the kernel version that +will be installed. Usually you can leave the default values, but you can change it here +if you want a different version. See preempt_rt_versions_ for available versions. Then simply execute the script in a terminal. Internally, sudo is used in some steps, so the user executing it needs to have sudo-permission. -In the beginning (after downloading some things) you will be asked to manually -adjust some configuration settings. Before entering the menu, the script prints -the following instructions:: +.. note:: - Please apply the following configurations in the next step: + In the beginning (after downloading some things) you will be asked to manually + adjust some configuration settings. Before entering the menu, the script prints + instructions like the following:: - General setup [Enter] - Local version - append to kernel release: [Enter] Add '-preempt-rt' + Please apply the following configurations in the next step: - General setup [Enter] - Preemption Model (Voluntary Kernel Preemption (Desktop)) [Enter] - Fully Preemptible Kernel (RT) [Enter] #Select + [...] -However, depending on the kernel version the "Preemption Model" setting may be -found in the "Processor type and features" menu instead. + General setup [Enter] + Preemption Model (Voluntary Kernel Preemption (Desktop)) [Enter] + Fully Preemptible Kernel (RT) [Enter] #Select + + However, depending on the kernel version the "Preemption Model" setting may be + found in the "Processor type and features" menu instead. The script will automatically download, build and install the kernel. This will take a while. +When finished, go back to :doc:`real_time` and follow the further steps to configure +your system for real-time usage. -.. _boot_rt_kernel: - -Boot with the PREEMPT_RT Kernel -=============================== - -Once the PREEMPT_RT kernel is installed, you need to reboot and select the -"preempt-rt" kernel in the GRUB menu (go to "Advanced options for Ubuntu", it -should be listed there). - -When the system is running, you can check which kernel is running with -``uname -a``. It should print something containing "PREEMPT_RT". - - -Select PREEMPT_RT Kernel by Default ------------------------------------ - -You can configure GRUB to automatically select this kernel by setting -``GRUB_DEFAULT`` in ``/etc/default/grub``. For this, first the identifier of -the kernel needs to be determined. Open a terminal and run - -.. code-block:: bash - - cat /boot/grub/grub.cfg | grep -w -e menuentry -e submenu - -It should print something like this:: - - menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-1a26991b-b045-48dd-bb12-064a2725b80b' { - submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-1a26991b-b045-48dd-bb12-064a2725b80b' { - menuentry 'Ubuntu, with Linux 5.4.93-rt51-preempt-rt' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.93-rt51-preempt-rt-advanced-1a26991b-b045-48dd-bb12-064a2725b80b' { - menuentry 'Ubuntu, with Linux 5.4.93-rt51-preempt-rt (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.93-rt51-preempt-rt-recovery-1a26991b-b045-48dd-bb12-064a2725b80b' { - menuentry 'Ubuntu, with Linux 5.4.0-65-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.0-65-generic-advanced-1a26991b-b045-48dd-bb12-064a2725b80b' { - menuentry 'Ubuntu, with Linux 5.4.0-65-generic (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.0-65-generic-recovery-1a26991b-b045-48dd-bb12-064a2725b80b' { +.. _nvidia_preempt_rt: -For ``GRUB_DEFAULT`` the path full submenu/menuentry is needed, using the id of -each step (the last part in the line, "gnulinux-..."), separated by ">". In -this specific case, the setting for starting the rt-kernel would be:: +Using Nvidia Drivers with the PREEMPT_RT Kernel +=============================================== - GRUB_DEFAULT = "gnulinux-advanced-1a26991b-b045-48dd-bb12-064a2725b80b>gnulinux-5.4.93-rt51-preempt-rt-advanced-471e9718-013f-4cbb-91a7-d22635173b70" +Officially, Nvidia drivers do not support the PREEMPT_RT kernel. However, at least with +more recent versions, it seems to work in practice. However, you need set the +environment variable ``IGNORE_PREEMPT_RT_PRESENCE=1`` when installing it with apt. +Complete steps: -After saving the changes in ``/etc/default/grub`` you need to run the following -command for the changes to become active:: +1. First uninstall any Nvidia drivers +2. Install the PREEMPT_RT kernel (see above) +3. Install the drivers with the following command (adjust driver version to the desired + one):: - sudo update-grub + sudo IGNORE_PREEMPT_RT_PRESENCE=1 apt install nvidia-driver-530 -Then reboot and verify that the correct kernel is used. +Please note that this variable also needs to be set when upgrading packages, so you may +want to set it in a global place like `/etc/environment` and disable unattended upgrades +for the driver. .. _PREEMPT_RT patch: https://wiki.linuxfoundation.org/realtime/start -.. _install_rt_preempt_kernel.sh: https://github.com/machines-in-motion/ubuntu_installation_scripts/blob/master/rt-preempt/ubuntu18.04/install_rt_preempt_kernel.sh +.. _install_rt_preempt_kernel.sh: https://github.com/machines-in-motion/ubuntu_installation_scripts/tree/master/rt-preempt .. _preempt_rt_versions: https://wiki.linuxfoundation.org/realtime/preempt_rt_versions diff --git a/doc/real_time.rst b/doc/real_time.rst index 45646ad..8ef0c17 100644 --- a/doc/real_time.rst +++ b/doc/real_time.rst @@ -6,36 +6,144 @@ Since a constant update rate of the :cpp:class:`~robot_interfaces::RobotBackend` is important for stable control of the robot, a real-time capable operating system should be used. -We are using Ubuntu with a PREEMPT_RT kernel patch. Below are instructions on -how to install the patched kernel and how to configure the system to use it. +For this we are using a standard Linux distribution (Ubuntu in our case) but install a +kernel with better real-time capabilities. There are two options: the "low-latency" +kernel which is easy to install but doesn't give full real-time guarantees, and the +preempt_rt kernel, which has better real-time performance but is more tedious to +install. +Below are instructions for both of them and on how to configure the system to use them. -Install PREEMPT_RT Kernel -========================= -See :doc:`preempt_rt_kernel`. +.. _lowlatency_kernel: +Installation: Low-latency vs PREEMPT_RT Kernel +============================================== -.. _lowlatency_kernel: +We recommend one of the following Linux kernel variants: + +**low-latency:** + +- Fewer latency guarantees as with PREEMPT_RT but often good enough. Latency can + increase if the system load is high, though. It can be useful to assign separate CPU + cores for the real-time critical processes. +- Can be used with Nvidia drivers. +- Easy to install. On Ubuntu it's just:: + + sudo apt install linux-lowlatency + + +**PREEMPT_RT patch:** + +- Better real-time guarantees. +- Officially not supported by Nvidia drivers (but may still kind of work, see + :ref:`nvidia_preempt_rt`). +- Difficult to install as it has to be built from source. See :doc:`preempt_rt_kernel`. + + +Historically we used the PREEMPT_RT kernel (and still do on many of our robots) but +lately also used the low-latency kernel in some projects without problems (you have to +be a bit more careful to not overload the CPU, though). +Since the low-latency Kernel is much easier to install, you may want to try with that +one first and only move to the PREEMPT_RT version if you notice timing-related issues. + +The further system configuration described in the following is the same in both cases. + + +.. _boot_rt_kernel: + +Boot with the real-time Kernel +============================== + +Once the real-time kernel is installed, you need to reboot and select the corresponding +kernel ("preempt-rt" or "low-latency") kernel in the GRUB menu (go to "Advanced options +for Ubuntu", it should be listed there). + +When the system is running, you can check which kernel is running with +``uname -a``. It should print something containing "PREEMPT_RT"/"lowlatency". + + +Select real-time Kernel by Default +---------------------------------- + +You can configure GRUB to automatically select a specific kernel when booting the +computer. We provide a script to semi-automatically update the configuration (option +1), but you may also edit it manually (option 2). + +Option 1: Semi-automatic +~~~~~~~~~~~~~~~~~~~~~~~~ + +We provide a script grub_select_default_kernel.py_ for this. Download it and execute +with -Alternative: The Low Latency Kernel -=================================== +.. code-block:: bash -A potential alternative to the PREEMPT_RT patch is using the "lowlatency" -kernel. It is easier to install and does not conflict with Nvidia drivers. On -the other hand, the real-time capabilities it provides are weaker than those of -the PREEMPT_RT kernel. Whether it is sufficient for you depends on your -application (e.g. how much computational load you put on the system while the -robot is running). We still recommend using the PREEMPT_RT kernel but you may -test with the lowlatency kernel first. + sudo python3 grub_select_default_kernel.py -To install on Ubuntu 20.04 (on other versions of Ubuntu adjust the version at -the end):: +It will automatically parse the GRUB configuration and list the available boot options. +Select the desired kernel and press enter. It will then show a diff of the change it +intends to apply to ``/etc/default/grub``. **Review the diff carefully and only confirm +if it looks good!** It should only modify the line starting with ``GRUB_DEFAULT``, i.e. +like this: - sudo apt install linux-lowlatency-hwe-20.04 +.. code-block:: diff + + --- current + +++ new + @@ -3,7 +3,7 @@ + # For full documentation of the options in this file, see: + # info -f grub -n 'Simple configuration' + + -GRUB_DEFAULT=0 + +GRUB_DEFAULT="gnulinux-advanced-cc71b1fb-b530-4694-a839-aecf600f1f49>gnulinux-5.15.0-112-generic-advanced-cc71b1fb-b530-4694-a839-aecf600f1f49" + GRUB_TIMEOUT_STYLE=hidden + GRUB_TIMEOUT=0 + GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` + +If all is good and you confirmed, you still need to run + +.. code-block:: bash + + sudo update-grub + +to apply the change. Then reboot and verify that the correct kernel is used. + + +Option 2: Manually +~~~~~~~~~~~~~~~~~~ + +If you don't want to use the script mentioned above, you may also edit the GRUB +configuration manually. + +For this, first the identifier of the kernel needs to be determined. Open a terminal and run + +.. code-block:: bash + + cat /boot/grub/grub.cfg | grep -w -e menuentry -e submenu + +It should print something like this:: + + menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-1a26991b-b045-48dd-bb12-064a2725b80b' { + submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-1a26991b-b045-48dd-bb12-064a2725b80b' { + menuentry 'Ubuntu, with Linux 5.4.93-rt51-preempt-rt' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.93-rt51-preempt-rt-advanced-1a26991b-b045-48dd-bb12-064a2725b80b' { + menuentry 'Ubuntu, with Linux 5.4.93-rt51-preempt-rt (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.93-rt51-preempt-rt-recovery-1a26991b-b045-48dd-bb12-064a2725b80b' { + menuentry 'Ubuntu, with Linux 5.4.0-65-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.0-65-generic-advanced-1a26991b-b045-48dd-bb12-064a2725b80b' { + menuentry 'Ubuntu, with Linux 5.4.0-65-generic (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.0-65-generic-recovery-1a26991b-b045-48dd-bb12-064a2725b80b' { + + +For ``GRUB_DEFAULT`` the path full submenu/menuentry is needed, using the id of +each step (the last part in the line, "gnulinux-..."), separated by ">". In +this specific case, the setting for starting the rt-kernel would be:: + + GRUB_DEFAULT = "gnulinux-advanced-1a26991b-b045-48dd-bb12-064a2725b80b>gnulinux-5.4.93-rt51-preempt-rt-advanced-471e9718-013f-4cbb-91a7-d22635173b70" + +After saving the changes in ``/etc/default/grub`` you need to run the following +command for the changes to become active:: + + sudo update-grub + +Then reboot and verify that the correct kernel is used. -Configuration to use this kernel by default works the same as for the PREEMPT_RT -kernel, see :ref:`boot_rt_kernel`. System Configuration @@ -120,3 +228,5 @@ following command:: Warning this thread is not going to be real time. + +.. _grub_select_default_kernel.py: https://github.com/machines-in-motion/ubuntu_installation_scripts/blob/master/rt-preempt/grub_select_default_kernel.py