Update CI setup in general and add more detailed explanations into do…

…cs. (StoglRobotics#49)

Co-authored-by: Manuel M <[email protected]>

docs/use-cases/ros_packages/configure_repository.rst

@@ -10,10 +10,10 @@ Setup Repository CI configuration (GitHub)
 The script **has to be executed** from the *main* folder of your package.
 
 .. code-block:: bash
-   :caption: Usage of script for setting up new packages.
+   :caption: Usage of the script for setting up new packages.
    :name: setup-package-ci
 
-   setup-repository-ci "repo_name" "repo_namespace" ["first_package" "second_package" ...]
+   setup-repository-ci "repo_name" "repo_namespace" ["first_package second_package ..."]
 
 When executing the script, read all output carefully.
 If you make any wrong decision or enter incorrect data, use <CTRL>+C keys to terminate the script.
@@ -22,5 +22,38 @@ The script can add setup for multiple ROS2 versions.
 Simply follow the output.
 
 After a setup is created, you should go through the files and check if they are correct.
-Then try to push this to a GitHub-repository and open a PR.
+Please remember that each workflow you would like to run on schedule has to be in the default branch of your repository.
+Then try to push new files to a GitHub-repository and open a PR.
 You should then already see new workflows active.
+
+
+Details about CI setup and created workflows
+---------------------------------------------
+Code formatters and linters
+,,,,,,,,,,,,,,,,,,,,,,,,,,,,
+For executing formatters ``pre-commit`` program is used and automatically setup.
+To use it in a cloned repositors (if there is ``.pre-commit-conifg.yaml`` file) execute once ``pre-commit install``.
+Clang-Format is used as the main code formatting program.
+
+
+Building packages
+,,,,,,,,,,,,,,,,,,
+This script creates three stages of build configuration for each ROS2 version.
+Each of those stages test different compatibility levels where users can experience issues with actively developed repositories and ensures future compatibility of the repository.
+In the following each stage is explained.
+
+``binary``
+  Building against released packages in a ROS distribution. The stage ensures that isolated build on a local machine and build-farm is possible. Unreleased dependencies can be defined in ``.repos``-file with suffix ``-not-released.<ros-distro>.repos``. This functionality is useful for new projects where packages are located in multiple repositories and when packages are released in a new ROS distribution (and you are not releasing regularly to ``rolling``).
+  The ``binary`` build has two workflows to build against ``main`` and ``testing`` ROS debian repositories. This is useful when for example API changes are done that involve multiple packages. Then it is expected that build against binary packages in the ``main`` repository fails, but if all the changes are released, building against ``testing`` repository has to work. Otherwise it means that the code has some regressions and cannot be synced to the ``main`` repositories (This is what is happening after maintainers from OpenRobotics announce a "Sync" or a ros distribution).
+  Scheduled ``binary`` build warns you that your package is incompatible with released dependencies (``main``) or that a package is breaking changes that will be released (``testing``). The first case should actually never happen if you are using all three building stages.
+
+``semi-binary``
+  Building against released core packages in a ROS distribution, but the main dependencies are build from source. The dependencies are defined in ``.repos``-file with suffix ``.<ros-distro>.repos``. This functionality is useful during development process when you have to sync changes in multiple repositories which are not released yet. This is the stage which should never fail if you are keeping your code up to date. Scheduled ``semi-binary`` builds warn you that some important dependency has changed something in their code so you can prepare yourself better for the next release.
+  There are two workflows building against ``main`` and ``testing`` ROS repositories with released packages.
+
+``source``
+  Core ROS packages are build from source. Expect the ``.repos`` files for ROS core it uses is the ``.repos``-file with suffix ``.<ros-distro>.repos`` for the package-specific dependencies. This stage makes mostly sense when working with ``rolling`` since other distribution of ROS should be API stable once released. This stage helps to anticipate braking changes in the ROS API that will be relevant the mid-term future.
+
+
+For the better overview and to avoid confusion for users there are two different tables with the workflow overview.
+One in the README file showing only the ``binary`` and ``semi-binary`` builds against ``main`` ROS repository, and another in the ``.github/workflows/README.md`` that shows all the workflows.

scripts/_Team_Defines.bash

@@ -55,7 +55,7 @@ alias create-new-package=$RosTeamWS_FRAMEWORK_SCRIPTS_PATH/create-new-package.ba
 
 alias setup-repository=$RosTeamWS_FRAMEWORK_SCRIPTS_PATH/setup-repository.bash
 
-alias setup-repository-ci=$RosTeamWS_FRAMEWORK_SCRIPTS_PATH/setup-ci-config.bash
+alias setup-repository-ci=$RosTeamWS_FRAMEWORK_SCRIPTS_PATH/setup-repository-ci.bash
 
 alias setup-ros-workspace=$RosTeamWS_FRAMEWORK_SCRIPTS_PATH/setup-ros-workspace.bash
 

scripts/setup-repository-ci.bash

@@ -15,13 +15,11 @@
 # limitations under the License.
 
 
-usage='setup-ci-config.bash "repo_name" "repo_namespace"'
-#
+usage='setup-repository-ci.bash "repo_name" "repo_namespace" ["first_package second_package ..."]'
 
 # Load Framework defines
 script_own_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" > /dev/null && pwd )"
 source $script_own_dir/../setup.bash
-check_and_set_ros_distro_and_version ${ROS_DISTRO}
 
 repo_name=$1
 if [ -z "$1" ]; then
@@ -43,7 +41,7 @@ read -p "${RAW_TERMINAL_COLOR_BROWN}Are you setting CI for private repository?${
 private=${private:="no"}
 
 CI_FORMAT="ci-format"
-if  [[ "$choice" == "yes" ]]; then
+if  [[ "$private" == "yes" ]]; then
   CI_FORMAT="ci-format-private"
 fi
 
@@ -55,8 +53,10 @@ CI_REUSABLE_ROS_TOOLING="reusable-ros-tooling-source-build"
 PRE_RELEASE="prerelease-check"
 
 CI_ABI_COMPATIBILITY="abi-compatibility"
-CI_BINARY_BUILD="binary-build"
-CI_SEMI_BINARY_BUILD="semi-binary-build"
+CI_BINARY_BUILD_MAIN="binary-build-main"
+CI_BINARY_BUILD_TESTING="binary-build-testing"
+CI_SEMI_BINARY_BUILD_MAIN="semi-binary-build-main"
+CI_SEMI_BINARY_BUILD_TESTING="semi-binary-build-testing"
 CI_SOURCE_BUILD="source-build"
 CI_RHEL_BINARY_BUILD="rhel-binary-build"
 
@@ -73,14 +73,18 @@ CI_GENERAL_FILES=(
 
 CI_DISTRIBUTION_FILES=(
   $CI_ABI_COMPATIBILITY
-  $CI_BINARY_BUILD
-  $CI_SEMI_BINARY_BUILD
+  $CI_BINARY_BUILD_MAIN
+  $CI_BINARY_BUILD_TESTING
+  $CI_SEMI_BINARY_BUILD_MAIN
+  $CI_SEMI_BINARY_BUILD_TESTING
   $CI_SOURCE_BUILD
   $CI_RHEL_BINARY_BUILD
 )
 
-read -p "${RAW_TERMINAL_COLOR_BROWN}Name of the default ROS distro?${RAW_TERMINAL_COLOR_NC} [default: rolling] " default_ros_distro
+read -p "${RAW_TERMINAL_COLOR_BROWN}Name of the default ROS distro? Supported distributions(${rtw_supported_ros_distributions[*]})${RAW_TERMINAL_COLOR_NC} [default: rolling] " default_ros_distro
 default_ros_distro=${default_ros_distro:="rolling"}
+check_and_set_ros_distro_and_version "$default_ros_distro"
+default_ros_distro=${ros_distro}
 read -p "${RAW_TERMINAL_COLOR_BROWN}Name of the repository default branch?${RAW_TERMINAL_COLOR_NC} [default: master]: " default_branch
 default_branch=${default_branch:="master"}
 
@@ -90,7 +94,7 @@ mkdir -p .github/workflows
 # Coping and SEDing general CI files
 for CI_FILE in "${CI_GENERAL_FILES[@]}"; do
   cp -n ${PACKAGE_TEMPLATES}/CI-github_${CI_FILE}.yml .github/workflows/${CI_FILE}.yml
-  sed -i 's/\$ROS_DISTRO\$/'${default_ros_distro}'/g' .github/workflows/${CI_FILE}.yml
+  sed -i 's/\$ros_distro\$/'${default_ros_distro}'/g' .github/workflows/${CI_FILE}.yml
   sed -i 's/\$branch\$/'${default_branch}'/g' .github/workflows/${CI_FILE}.yml
   sed -i 's/\$NAME\$/'${repo_name}'/g' .github/workflows/${CI_FILE}.yml
   if [[ "$package_names" != "<none>" ]]; then
@@ -112,13 +116,15 @@ some_ros_distro=${default_ros_distro}
 some_branch=${default_branch}
 
 TMP_FILE=".f_tmp"
+TMP_FILE_2=".f_tmp2"
 touch $TMP_FILE
+touch $TMP_FILE_2
 
 while true; do
   for CI_FILE in "${CI_DISTRIBUTION_FILES[@]}"; do
     output_file_name="${some_ros_distro}-${CI_FILE}"
     cp -n ${PACKAGE_TEMPLATES}/CI-github_${CI_FILE}.yml .github/workflows/${output_file_name}.yml
-    sed -i 's/\$ROS_DISTRO\$/'${some_ros_distro}'/g' .github/workflows/${output_file_name}.yml
+    sed -i 's/\$ros_distro\$/'${some_ros_distro}'/g' .github/workflows/${output_file_name}.yml
     sed -i 's/\$Ros_distro\$/'${some_ros_distro^}'/g' .github/workflows/${output_file_name}.yml
     sed -i 's/\$branch\$/'${some_branch}'/g' .github/workflows/${output_file_name}.yml
     sed -i 's/\$NAME\$/'${repo_name}'/g' .github/workflows/${output_file_name}.yml
@@ -149,17 +155,30 @@ while true; do
   head -$CUT_LINE README.md >> $TMP_FILE
   cat ${PACKAGE_TEMPLATES}/_append_to_README_build_status_table.md >> $TMP_FILE
 
-  sed -i 's/\$branch\$/'${some_branch}'/g' $TMP_FILE
-  sed -i 's/\$ros_distro\$/'${some_ros_distro}'/g' $TMP_FILE
-  sed -i 's/\$Ros_distro\$/'${some_ros_distro^}'/g' $TMP_FILE
-  sed -i 's/\$NAME\$/'${repo_name}'/g' $TMP_FILE
-  sed -i 's/\$NAMESPACE\$/'${repo_namespace}'/g' $TMP_FILE
+  # Update .github/workflows/README.md file
+  cat ${PACKAGE_TEMPLATES}/_append_to_workflows_README_build_status_table.md > $TMP_FILE_2
+
+  # SED temp files
+  FILES_TO_UPDATE=("$TMP_FILE" "$TMP_FILE_2")
+  for SED_FILE in "${FILES_TO_UPDATE[@]}"; do
+    sed -i 's/\$branch\$/'${some_branch}'/g' $SED_FILE
+    sed -i 's/\$ros_distro\$/'${some_ros_distro}'/g' $SED_FILE
+    sed -i 's/\$Ros_distro\$/'${some_ros_distro^}'/g' $SED_FILE
+    sed -i 's/\$NAME\$/'${repo_name}'/g' $SED_FILE
+    sed -i 's/\$NAMESPACE\$/'${repo_namespace}'/g' $SED_FILE
+  done
 
+  # append to readme files
   tail -n +$CUT_LINE README.md >> $TMP_FILE
   mv $TMP_FILE README.md
 
+  cat ${PACKAGE_TEMPLATES}/_append_to_README_build_status.md >> $TMP_FILE_2
+
+  cat $TMP_FILE_2 >> .github/workflows/README.md
+  rm $TMP_FILE_2
+
   echo ""
-  echo -e "${TERMINAL_COLOR_BLUE}Added setup for ros distro '${some_ros_distro} on branch ${some_branch}.${TERMINAL_COLOR_NC}"
+  echo -e "${TERMINAL_COLOR_BLUE}Added setup for ros distro '${RAW_TERMINAL_COLOR_BROWN}${some_ros_distro}${TERMINAL_COLOR_BLUE}' on branch '${RAW_TERMINAL_COLOR_BROWN}${some_branch}${TERMINAL_COLOR_BLUE}'.${TERMINAL_COLOR_NC}"
   echo ""
 
   read -p "${RAW_TERMINAL_COLOR_BROWN}Do you want to configure ci for another ros distro?${RAW_TERMINAL_COLOR_NC} (yes/no) [no] " another
@@ -170,17 +189,22 @@ while true; do
   fi
 
   echo "" > $TMP_FILE
+  echo "" > $TMP_FILE_2
 
   read -p "${RAW_TERMINAL_COLOR_BROWN}Name of the ROS distro? ${RAW_TERMINAL_COLOR_NC}" some_ros_distro
   read -p "${RAW_TERMINAL_COLOR_BROWN}Name of the repository branch?: ${RAW_TERMINAL_COLOR_NC}" some_branch
 done
 
 # Setting up formatting
-cp -n ${PACKAGE_TEMPLATES}/.clang-format .
-cp -n ${PACKAGE_TEMPLATES}/.pre-commit-config.yaml .
-pre-commit install
-pre-commit autoupdate
-
+read -p "${RAW_TERMINAL_COLOR_BROWN}Do you want to setup formatting using pre-commit?${RAW_TERMINAL_COLOR_NC} (yes/no) [no]: " formatting
+formatting=${formatting:="no"}
+
+if  [[ "$formatting" == "yes" ]]; then
+  cp -n ${PACKAGE_TEMPLATES}/.clang-format .
+  cp -n ${PACKAGE_TEMPLATES}/.pre-commit-config.yaml .
+  pre-commit install
+  pre-commit autoupdate
+fi
 
 echo ""
 echo -e "${TERMINAL_COLOR_BLUE}FINISHED: Please check the generated file and setup package names.${TERMINAL_COLOR_NC}"

templates/package/.pre-commit-config.yaml

@@ -15,7 +15,7 @@
 repos:
   # Standard hooks
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.1.0
+    rev: v4.3.0
     hooks:
       - id: check-added-large-files
       - id: check-ast
@@ -33,13 +33,13 @@ repos:
 
   # Python hooks
   - repo: https://github.com/asottile/pyupgrade
-    rev: v2.31.0
+    rev: v2.37.3
     hooks:
     -   id: pyupgrade
         args: [--py36-plus]
 
   - repo: https://github.com/psf/black
-    rev: 21.12b0
+    rev: 22.6.0
     hooks:
       - id: black
         args: ["--line-length=99"]
@@ -52,7 +52,7 @@ repos:
       args: ["--ignore=D100,D101,D102,D103,D104,D105,D106,D107,D203,D212,D404"]
 
   - repo: https://github.com/pycqa/flake8
-    rev: 4.0.1
+    rev: 5.0.4
     hooks:
     - id: flake8
       args: ["--ignore=E501"]
@@ -63,7 +63,7 @@ repos:
       - id: clang-format
         name: clang-format
         description: Format files with ClangFormat.
-        entry: clang-format-10
+        entry: clang-format-12
         language: system
         files: \.(c|cc|cxx|cpp|frag|glsl|h|hpp|hxx|ih|ispc|ipp|java|js|m|proto|vert)$
         args: ['-fallback-style=none', '-i']
@@ -119,7 +119,7 @@ repos:
 
   # Docs - RestructuredText hooks
   - repo: https://github.com/PyCQA/doc8
-    rev: 0.10.1
+    rev: v1.0.0
     hooks:
       - id: doc8
         args: ['--max-line-length=100', '--ignore=D001']

templates/package/.pre-commit_wrapper.shb

@@ -0,0 +1,6 @@
+#!/bin/bash
+source /opt/ros/rolling/setup.bash
+
+cmd=$1
+shift
+$cmd "$@"

templates/package/CI-github_abi-compatibility.yml

@@ -1,5 +1,8 @@
-name: ABI Compatibility Check
+name: $Ros_distro$ - ABI Compatibility Check
 on:
+  workflow_dispatch:
+    branches:
+      - $branch$
   pull_request:
     branches:
       - $branch$
@@ -11,7 +14,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: ros-industrial/industrial_ci@master
         env:
-          ROS_DISTRO: $ROS_DISTRO$
+          ROS_DISTRO: $ros_distro$
           ROS_REPO: main
           ABICHECK_URL: github:${{ github.repository }}#${{ github.base_ref }}
           NOT_TEST_BUILD: true

templates/package/CI-github_binary-build.yml → templates/package/CI-github_binary-build-main.yml

@@ -1,8 +1,11 @@
-name: $Ros_distro$ Binary Build
+name: $Ros_distro$ Binary Build - main
 # author: Denis Štogl <[email protected]>
 # description: 'Build & test all dependencies from released (binary) packages.'
 
 on:
+  workflow_dispatch:
+    branches:
+      - $branch$
   pull_request:
     branches:
       - $branch$
@@ -17,6 +20,7 @@ jobs:
   binary:
     uses: ./.github/workflows/reusable-industrial-ci-with-cache.yml
     with:
-      ros_distro: $ROS_DISTRO$
-      upstream_workspace: $NAME$-not-released.$ROS_DISTRO$.repos
+      ros_distro: $ros_distro$
+      ros_repo: main
+      upstream_workspace: $NAME$-not-released.$ros_distro$.repos
       ref_for_scheduled_build: $branch$

templates/package/CI-github_binary-build-testing.yml

@@ -0,0 +1,26 @@
+name: $Ros_distro$ Binary Build - testing
+# author: Denis Štogl <[email protected]>
+# description: 'Build & test all dependencies from released (binary) packages.'
+
+on:
+  workflow_dispatch:
+    branches:
+      - $branch$
+  pull_request:
+    branches:
+      - $branch$
+  push:
+    branches:
+      - $branch$
+  schedule:
+    # Run every morning to detect flakiness and broken dependencies
+    - cron: '03 1 * * *'
+
+jobs:
+  binary:
+    uses: ./.github/workflows/reusable-industrial-ci-with-cache.yml
+    with:
+      ros_distro: $ros_distro$
+      ros_repo: testing
+      upstream_workspace: $NAME$-not-released.$ros_distro$.repos
+      ref_for_scheduled_build: $branch$

templates/package/CI-github_ci-coverage-build.yml

@@ -1,23 +1,26 @@
 name: Coverage Build
 on:
+  workflow_dispatch:
+    branches:
+      - $branch$
   pull_request:
     branches:
       - $branch$
 
 jobs:
   coverage:
     name: coverage build
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
     strategy:
       fail-fast: false
     env:
-      ROS_DISTRO: $ROS_DISTRO$
+      ROS_DISTRO: $ros_distro$
     steps:
-      - uses: ros-tooling/setup-ros@v0.2
+      - uses: ros-tooling/setup-ros@0.3.4
         with:
           required-ros-distributions: ${{ env.ROS_DISTRO }}
       - uses: actions/checkout@v3
-      - uses: ros-tooling/action-ros-ci@v0.2
+      - uses: ros-tooling/action-ros-ci@0.2.6
         with:
           target-ros2-distro: ${{ env.ROS_DISTRO }}
           import-token: ${{ secrets.GITHUB_TOKEN }}
@@ -33,13 +36,12 @@ jobs:
               }
             }
           colcon-mixin-repository: https://raw.githubusercontent.com/colcon/colcon-mixin-repository/master/index.yaml
-          skip-tests: true
-      - uses: codecov/[email protected]
+      - uses: codecov/[email protected]
         with:
           file: ros_ws/lcov/total_coverage.info
           flags: unittests
           name: codecov-umbrella
-      - uses: actions/upload-artifact@v1
+      - uses: actions/upload-artifact@v3.1.0
         with:
-          name: colcon-logs-${{ matrix.os }}
+          name: colcon-logs-coverage-rolling
           path: ros_ws/log

templates/package/CI-github_ci-format-private.yml

@@ -23,7 +23,7 @@ jobs:
       with:
         python-version: 3.9.7
     - name: Install system hooks
-      run: sudo apt install -qq clang-format-11 cppcheck
+      run: sudo apt install -qq clang-format-12 cppcheck
     - uses: pre-commit/[email protected]
       with:
         extra_args: --all-files --hook-stage manual