Added ros_buildfarm_config to be used in the Gazebo buildfarm (#1153)

ros_buildfarm_config to be used with ros_buildfarm code to generate jobs in the build.o.o Jenkins. --------- Signed-off-by: Jose Luis Rivero <[email protected]> Co-authored-by: Scott K Logan <[email protected]>
gazebo-tooling · Aug 2, 2024 · 92674e6 · 92674e6
1 parent 77de389
commit 92674e6
Show file tree

Hide file tree

Showing 4 changed files with 265 additions and 0 deletions.
diff --git a/jenkins-scripts/ros_buildfarm_config/README.md b/jenkins-scripts/ros_buildfarm_config/README.md
@@ -0,0 +1,67 @@
+# ros_buildfam configuration for the Gazebo buildfarm
+
+This directory hosts the ros_buildfarm configuration used to create some jobs
+in the Gazebo buildfarm. Mostly inspired by the ROS buildfarm configuration 
+repository: https://github.com/ros2/ros_buildfarm_config/
+
+## Rationale
+
+The [ros_buildfarm](https://github.com/ros-infrastructure/ros_buildfarm) code 
+is able to generate Jenkins jobs designed mainly for the ROS buildfarm that 
+are also applicable to other Jenkins buildfarms. In this case the Gazebo
+development team was looking for a colcon based jobs able to receive
+as an input a VCS file. Instead of creating these jobs from scratch, the use of
+ros_buildfarm was chosen to provide the required features and many others more.
+
+## Jobs generated
+
+The jobs generated by ros_buildfarm are easily recognizable since they are 
+placed under the Jenkins views `ci` (non ROS related jobs) or `Xci` 
+(where X is the initial of a ROS release). The job names also uses these
+same prefixes: `ci__` or `Xci__`.
+
+## Generating jobs
+
+### Initial setup
+
+Be sure of going through the 
+[Setup environment to deploy configuration](https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/environment.rst)
+document in the ros_buildfarm as the first step to proceed with the jobs
+generation.
+
+### Running job generation
+
+Note that the Gazebo buildfarm can not execute the script
+`ros_buildfarm/scripts/generate_all_jobs.py` since it will create some
+maintenance views and jobs that are not in the scope of this buildfarm.
+
+Instead of that, the job generation is done manually per configuration
+file using the following (empty quotes are mandatory):
+
+```
+# remove the --dry-run for real execution
+./generate_ci_jobs.py --dry-run \
+    https://raw.githubusercontent.com/gazebo-tooling/release-tools/jrivero/ros_buildfarm/jenkins-scripts/ros_buildfarm_config/index.yaml \
+    "" <ci_build>
+```
+
+The different `<ci_build>` are defined in the index file just before the key
+`ci_builds:`. So for example 
+
+```
+# remove the --dry-run for real execution
+./generate_ci_jobs.py --dry-run \
+    https://raw.githubusercontent.com/gazebo-tooling/release-tools/jrivero/ros_buildfarm/jenkins-scripts/ros_buildfarm_config/index.yaml \
+    "" colcon_any-manual
+```
+
+To automate the process using the `yq` tool:
+
+```
+wget https://raw.githubusercontent.com/gazebo-tooling/release-tools/master/jenkins-scripts/ros_buildfarm_config/index.yaml
+for ci_build in $(yq '.ci_builds | keys' < index.yaml | sed 's:^- ::'); do 
+./generate_ci_jobs.py --dry-run \
+    https://raw.githubusercontent.com/gazebo-tooling/release-tools/jrivero/ros_buildfarm/jenkins-scripts/ros_buildfarm_config/index.yaml \
+    "" $ci_build
+done
+```
diff --git a/jenkins-scripts/ros_buildfarm_config/global/colcon_any-manual.yaml b/jenkins-scripts/ros_buildfarm_config/global/colcon_any-manual.yaml
@@ -0,0 +1,92 @@
+%YAML 1.1
+# ROS buildfarm ci-build file
+---
+build_tool: colcon
+build_tool_args: '--cmake-args -DCMAKE_BUILD_TYPE=Release --no-warn-unused-cli'
+build_tool_test_args: '--retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail"'
+jenkins_job_label: docker
+jenkins_job_priority: 200  # same that pr-any
+jenkins_job_timeout: 300
+repos_files:
+- https://raw.githubusercontent.com/gazebo-tooling/gazebodistro/master/collection-harmonic.yaml
+repositories:
+  keys:
+  - |
+    -----BEGIN PGP PUBLIC KEY BLOCK-----
+    Version: GnuPG v1
+
+    mQENBFUcKaEBCAD1ajXaWLnow3pZEv44Lypt6s5jAh1rYPN6zpaFZWdkzpwTdYU1
+    Rpw/0hPzIoiyOPNwCti4E3+dSrv1ogEBu85P2XSy67RnabxF4/z7mPG/++u0EQav
+    CwfrsN8OpJTtTxk+nKIhVwpAtob+KOLATerTPETrdrKh7qJ/FE8cw/XXbknjwywf
+    R8uJqaKTu7mWNrTFaS3P5GZF5ss+ztf0EHcyYFMvzEVnSiOGBBL9pw91P1qpggBa
+    lKL1Ilmf6zZBPihORJ/iTH5qMCAPDdR5BaxxEUHgz+pg+RkLKd2ENEaO+SCDVRhP
+    yNdkYHpuIslyMHfXrh4y5nHclJ+bNXKXDcudABEBAAG0R09TUkYgUmVwb3NpdG9y
+    eSAoT1NSRiBSZXBvc2l0b3J5IEdQRyBrZXkpIDxvc3JmYnVpbGRAb3NyZm91bmRh
+    dGlvbi5vcmc+iQE4BBMBAgAiBQJVHCmhAhsDBgsJCAcDAgYVCAIJCgsEFgIDAQIe
+    AQIXgAAKCRBnFwWYrySXQ/D4CACVnSdHT/1dEkOrYkCnaFLNBrG2tJdBrbIZOxKy
+    +xV0yGniqsQFAxLESoy+ygaiKdTnAFlA24ozoNY8ur+oKMFt6CrUY01ItTq/WMA1
+    iper0TO935SpDzNIPjPnD2WUSXShISWP0bFg64g0aAD1S7Yg/v7/eOmMSoeMav0T
+    h8KOo6yhJuhgGp3lHKAKLppH94b77d8JYqGeP03Gv6gcaqNojyKccdXrKTugZui5
+    +7V/cOJTo9XqzXjkpfwp24jR8FlKI7EWqCVqtRAXHeqRgo3OaKmuoKLcJ4/8BjSU
+    +ppmJtEstSaL+qw49P/GQHwUkCHlx1mV5dSdVFLBPreli1ChuQENBFUcKaEBCAC7
+    ZgTdYubw1sU/4A6+NvW/poBfh2DDOeh3uHJc0y235JFjr+tC1AwouaxLOUm8FE9k
+    7qzwnyXbeklmXAHxw6wXZdE4PEYA/sgBYhTQy+s4PHlI6TGhwgcROkJKlW4Lld+W
+    IJ/fzW93DXyhEkV3AAhkrVcOLOgCPdpK5EXxJ3p6dCOKC5Vjyz1PxTNcRaLpp9w6
+    J0hLIXmmoCN4aoYSXWtL/C9J+B5Cr+HHgrmFsGNrHmmVv1gMXLcVzw5p3Z4d8SuT
+    g9a1CemSE5bFIoOHKEQRwv/CGpoviAr+T3za3dPFTcSMOoJuYvoheTJ6fhf2sj74
+    bp2Fwi4L7am/asfa7xWVABEBAAGJAR8EGAECAAkFAlUcKaECGwwACgkQZxcFmK8k
+    l0OX9Af+IrzUChXf6H0nZZY77gcjwFgVChRX1RLzHTTHum4WNKGP9Sw1aGdHpmdt
+    LhypQImxdT2yhCPEyB8EQxhgPHjqZ6UUMeYMw5rAvrcb3/ercy5pG7O8Z+Bea6hu
+    TAXquJ1tsFessZwMS3RUXp/gtZCHbESR7PeBlZJWBWxG/lOmX7Z4fa88dWRU0Pl/
+    nfns7v6eb57HXbf0teCitRRsJwCMhYbHj2m1slZHMjhEc6kv2bgPmAFb04bcyEAP
+    BAo3BKu2XUVqE1t7Q2EfsItL/0FpfDY6zGKM6NIi+C40CsRl4W0o6egUhiDqsMYX
+    9Su5aZdCoxMhzy5QxS3sXcpNAWH2gw==
+    =YM5F
+    -----END PGP PUBLIC KEY BLOCK-----
+  - |
+    -----BEGIN PGP PUBLIC KEY BLOCK-----
+
+    mQINBF/btn4BEADSD9KhOgzOwDFOw+ivJXt3lw6Ubyo6c2yXEEASnQpzQ2IPJBpQ
+    zAX+rncd76xNNgtAUmtPFdsWojwIOYvHLnCusmjB5zmpr08alKKUqKfjKI0ALfq8
+    xrv2F7AFeaV2FWaCrH8ZrNBxLUnm/1MZbrknvnh4J+1Bpzi4JQvYJ1hb2BUYCPBj
+    kXTpShj58yzlPvr5oDzmy++evPjVf8Lh5f0LxWtDlYK8NED0dBzHZlL1eeuEmNJP
+    Iw/UNVlhVgPv3j329sdmGja+uck5JEAlL0uRU4D83f/YndwUUX+I+Um5baGuyJ42
+    FXFtw9JHF3DJiERpYd/ZCPP8jQwAxKee2zukxUh/L1MCWBFoZCIuuy30ACq20gWZ
+    PAnDpqF9H/fzg3FQiwAaeF7ZCZPNvSCnV7myqdZ7C/adgayuRhT6vpMP48WACqV6
+    ndmYEJL2WC+eT7eDD6hEZpVnBNeuRA/RnU6aRgybZ2+rB+hhXTfX3xzLxqJeLIZ7
+    SWpXGNRZAmOy9Eqt/dKrzUUyrOXfgqHd75mWFlbHrQZ8tt0+1PE76XG4ZgT3pqOk
+    bHHqywJfJXpy5U62YBy0TruMM7WyKN8IrkvUdYkMGbyfVKudQha0lTIAvX8zeW1z
+    AHovXBFkdR9NgdggZV1XWdbBlscLOKLcg9t+HT2EKSvgx1sPJEhzBFdahwARAQAB
+    tFpST1MgQm9vdHN0cmFwIFJlcG9zaXRvcnkgKFNpZ25pbmcgS2V5IGZvciBST1Mg
+    Qm9vdHN0cmFwIFJlcG9zaXRvcnkpIDxyb3NAb3BlbnJvYm90aWNzLm9yZz6JAk4E
+    EwEIADgWIQRHMs5wbNexnbD/506O2y72YfyIDgUCX9u2fgIbAwULCQgHAgYVCgkI
+    CwIEFgIDAQIeAQIXgAAKCRCO2y72YfyIDksLD/9awISK9MlJJZ2W769Fj8D2JOEM
+    splwtVNBuADyH2rT2TjL5aY+V8/awq/DApl3ty/QpfM8qcybUyi94faAhdvYygMr
+    AvU3/VusaxexlJHKO9GkdFfFQTtcksWhV7+z21h1omq5RimInJA6eyrhsNjFnOiI
+    we4SG3Jwi+Scr0xPLIyuboacD+3AXxPaA3mwQjLnMyYMyBCZs7tTtET5FR6hAGIg
+    eptpm+Fiaosjub35/VB9r/JOjJCuiwKObMye+8jcDdR8umFxzqcgEe1TNQVBj5+V
+    I4ssbrzKvwAkte3mBEdndEBPbSlTAo0kd4ByMRZ5AWhqRdu4iabw8cz2IpGErUcb
+    K0ws76+D8PqnqjPqd5GkGxB54zLMMrrBya5+hnzyDifKzvv1M+L2xvFuICBLBTR7
+    FnRownLKmDXLXp/EloRpTvchsO9W+Y3cDTkzkKpaUgs8Gl4dI/zPkHEp8iZBZNsk
+    mEibjJ5g2qBAYr9GbyaUiifdmpUiDoJpCyDq22rKfFkZl7t1+1MVbjz92A8Vjesb
+    QHt0icnkrvwg7mYfmMcdVF4TVNaR2XzLTkAz62r1kOoXnPbzUTA1uiLakagsUG5G
+    YXyUVOJlJDTWr784FVSsEQIslvbO8dk3gmiOpurxBrLGn70SdhCnQRqSrjOnvzX7
+    3ih4hgyGHQOJ/7p5Rg==
+    =v6u9
+    -----END PGP PUBLIC KEY BLOCK-----
+  urls:
+  - https://packages.osrfoundation.org/gazebo/ubuntu-stable
+  - http://repos.ros.org/repos/ros_bootstrap  # Needed for getting the ROS building tools packages like python3-rosdep
+skip_rosdep_keys: 
+- python3-distutils
+targets:
+  ubuntu:
+    noble:
+      amd64:
+    jammy:
+      amd64:
+  _config:
+    custom_rosdep_urls:
+      - https://raw.githubusercontent.com/osrf/osrf-rosdep/master/gz/00-gazebo.list
+type: ci-build
+version: 1
diff --git a/jenkins-scripts/ros_buildfarm_config/global/colcon_gpu_any-manual.yaml b/jenkins-scripts/ros_buildfarm_config/global/colcon_gpu_any-manual.yaml
@@ -0,0 +1,93 @@
+%YAML 1.1
+# ROS buildfarm ci-build file
+---
+build_tool: colcon
+build_tool_args: '--cmake-args -DCMAKE_BUILD_TYPE=Release --no-warn-unused-cli'
+build_tool_test_args: '--retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail"'
+jenkins_job_label: gpu-reliable
+jenkins_job_priority: 200  # same than pr-any
+jenkins_job_timeout: 300
+repos_files:
+- https://raw.githubusercontent.com/gazebo-tooling/gazebodistro/master/collection-harmonic.yaml
+repositories:
+  keys:
+  - |
+    -----BEGIN PGP PUBLIC KEY BLOCK-----
+    Version: GnuPG v1
+
+    mQENBFUcKaEBCAD1ajXaWLnow3pZEv44Lypt6s5jAh1rYPN6zpaFZWdkzpwTdYU1
+    Rpw/0hPzIoiyOPNwCti4E3+dSrv1ogEBu85P2XSy67RnabxF4/z7mPG/++u0EQav
+    CwfrsN8OpJTtTxk+nKIhVwpAtob+KOLATerTPETrdrKh7qJ/FE8cw/XXbknjwywf
+    R8uJqaKTu7mWNrTFaS3P5GZF5ss+ztf0EHcyYFMvzEVnSiOGBBL9pw91P1qpggBa
+    lKL1Ilmf6zZBPihORJ/iTH5qMCAPDdR5BaxxEUHgz+pg+RkLKd2ENEaO+SCDVRhP
+    yNdkYHpuIslyMHfXrh4y5nHclJ+bNXKXDcudABEBAAG0R09TUkYgUmVwb3NpdG9y
+    eSAoT1NSRiBSZXBvc2l0b3J5IEdQRyBrZXkpIDxvc3JmYnVpbGRAb3NyZm91bmRh
+    dGlvbi5vcmc+iQE4BBMBAgAiBQJVHCmhAhsDBgsJCAcDAgYVCAIJCgsEFgIDAQIe
+    AQIXgAAKCRBnFwWYrySXQ/D4CACVnSdHT/1dEkOrYkCnaFLNBrG2tJdBrbIZOxKy
+    +xV0yGniqsQFAxLESoy+ygaiKdTnAFlA24ozoNY8ur+oKMFt6CrUY01ItTq/WMA1
+    iper0TO935SpDzNIPjPnD2WUSXShISWP0bFg64g0aAD1S7Yg/v7/eOmMSoeMav0T
+    h8KOo6yhJuhgGp3lHKAKLppH94b77d8JYqGeP03Gv6gcaqNojyKccdXrKTugZui5
+    +7V/cOJTo9XqzXjkpfwp24jR8FlKI7EWqCVqtRAXHeqRgo3OaKmuoKLcJ4/8BjSU
+    +ppmJtEstSaL+qw49P/GQHwUkCHlx1mV5dSdVFLBPreli1ChuQENBFUcKaEBCAC7
+    ZgTdYubw1sU/4A6+NvW/poBfh2DDOeh3uHJc0y235JFjr+tC1AwouaxLOUm8FE9k
+    7qzwnyXbeklmXAHxw6wXZdE4PEYA/sgBYhTQy+s4PHlI6TGhwgcROkJKlW4Lld+W
+    IJ/fzW93DXyhEkV3AAhkrVcOLOgCPdpK5EXxJ3p6dCOKC5Vjyz1PxTNcRaLpp9w6
+    J0hLIXmmoCN4aoYSXWtL/C9J+B5Cr+HHgrmFsGNrHmmVv1gMXLcVzw5p3Z4d8SuT
+    g9a1CemSE5bFIoOHKEQRwv/CGpoviAr+T3za3dPFTcSMOoJuYvoheTJ6fhf2sj74
+    bp2Fwi4L7am/asfa7xWVABEBAAGJAR8EGAECAAkFAlUcKaECGwwACgkQZxcFmK8k
+    l0OX9Af+IrzUChXf6H0nZZY77gcjwFgVChRX1RLzHTTHum4WNKGP9Sw1aGdHpmdt
+    LhypQImxdT2yhCPEyB8EQxhgPHjqZ6UUMeYMw5rAvrcb3/ercy5pG7O8Z+Bea6hu
+    TAXquJ1tsFessZwMS3RUXp/gtZCHbESR7PeBlZJWBWxG/lOmX7Z4fa88dWRU0Pl/
+    nfns7v6eb57HXbf0teCitRRsJwCMhYbHj2m1slZHMjhEc6kv2bgPmAFb04bcyEAP
+    BAo3BKu2XUVqE1t7Q2EfsItL/0FpfDY6zGKM6NIi+C40CsRl4W0o6egUhiDqsMYX
+    9Su5aZdCoxMhzy5QxS3sXcpNAWH2gw==
+    =YM5F
+    -----END PGP PUBLIC KEY BLOCK-----
+  - |
+    -----BEGIN PGP PUBLIC KEY BLOCK-----
+
+    mQINBF/btn4BEADSD9KhOgzOwDFOw+ivJXt3lw6Ubyo6c2yXEEASnQpzQ2IPJBpQ
+    zAX+rncd76xNNgtAUmtPFdsWojwIOYvHLnCusmjB5zmpr08alKKUqKfjKI0ALfq8
+    xrv2F7AFeaV2FWaCrH8ZrNBxLUnm/1MZbrknvnh4J+1Bpzi4JQvYJ1hb2BUYCPBj
+    kXTpShj58yzlPvr5oDzmy++evPjVf8Lh5f0LxWtDlYK8NED0dBzHZlL1eeuEmNJP
+    Iw/UNVlhVgPv3j329sdmGja+uck5JEAlL0uRU4D83f/YndwUUX+I+Um5baGuyJ42
+    FXFtw9JHF3DJiERpYd/ZCPP8jQwAxKee2zukxUh/L1MCWBFoZCIuuy30ACq20gWZ
+    PAnDpqF9H/fzg3FQiwAaeF7ZCZPNvSCnV7myqdZ7C/adgayuRhT6vpMP48WACqV6
+    ndmYEJL2WC+eT7eDD6hEZpVnBNeuRA/RnU6aRgybZ2+rB+hhXTfX3xzLxqJeLIZ7
+    SWpXGNRZAmOy9Eqt/dKrzUUyrOXfgqHd75mWFlbHrQZ8tt0+1PE76XG4ZgT3pqOk
+    bHHqywJfJXpy5U62YBy0TruMM7WyKN8IrkvUdYkMGbyfVKudQha0lTIAvX8zeW1z
+    AHovXBFkdR9NgdggZV1XWdbBlscLOKLcg9t+HT2EKSvgx1sPJEhzBFdahwARAQAB
+    tFpST1MgQm9vdHN0cmFwIFJlcG9zaXRvcnkgKFNpZ25pbmcgS2V5IGZvciBST1Mg
+    Qm9vdHN0cmFwIFJlcG9zaXRvcnkpIDxyb3NAb3BlbnJvYm90aWNzLm9yZz6JAk4E
+    EwEIADgWIQRHMs5wbNexnbD/506O2y72YfyIDgUCX9u2fgIbAwULCQgHAgYVCgkI
+    CwIEFgIDAQIeAQIXgAAKCRCO2y72YfyIDksLD/9awISK9MlJJZ2W769Fj8D2JOEM
+    splwtVNBuADyH2rT2TjL5aY+V8/awq/DApl3ty/QpfM8qcybUyi94faAhdvYygMr
+    AvU3/VusaxexlJHKO9GkdFfFQTtcksWhV7+z21h1omq5RimInJA6eyrhsNjFnOiI
+    we4SG3Jwi+Scr0xPLIyuboacD+3AXxPaA3mwQjLnMyYMyBCZs7tTtET5FR6hAGIg
+    eptpm+Fiaosjub35/VB9r/JOjJCuiwKObMye+8jcDdR8umFxzqcgEe1TNQVBj5+V
+    I4ssbrzKvwAkte3mBEdndEBPbSlTAo0kd4ByMRZ5AWhqRdu4iabw8cz2IpGErUcb
+    K0ws76+D8PqnqjPqd5GkGxB54zLMMrrBya5+hnzyDifKzvv1M+L2xvFuICBLBTR7
+    FnRownLKmDXLXp/EloRpTvchsO9W+Y3cDTkzkKpaUgs8Gl4dI/zPkHEp8iZBZNsk
+    mEibjJ5g2qBAYr9GbyaUiifdmpUiDoJpCyDq22rKfFkZl7t1+1MVbjz92A8Vjesb
+    QHt0icnkrvwg7mYfmMcdVF4TVNaR2XzLTkAz62r1kOoXnPbzUTA1uiLakagsUG5G
+    YXyUVOJlJDTWr784FVSsEQIslvbO8dk3gmiOpurxBrLGn70SdhCnQRqSrjOnvzX7
+    3ih4hgyGHQOJ/7p5Rg==
+    =v6u9
+    -----END PGP PUBLIC KEY BLOCK-----
+  urls:
+  - https://packages.osrfoundation.org/gazebo/ubuntu-stable
+  - http://repos.ros.org/repos/ros_bootstrap  # Needed for getting the ROS building tools packages like python3-rosdep
+skip_rosdep_keys: 
+- python3-distutils
+targets:
+  ubuntu:
+    noble:
+      amd64:
+    jammy:
+      amd64:
+  _config:
+    custom_rosdep_urls:
+      - https://raw.githubusercontent.com/osrf/osrf-rosdep/master/gz/00-gazebo.list
+tests_require_gpu: true
+type: ci-build
+version: 1
diff --git a/jenkins-scripts/ros_buildfarm_config/index.yaml b/jenkins-scripts/ros_buildfarm_config/index.yaml
@@ -0,0 +1,13 @@
+%YAML 1.1
+# ROS buildfarm index file
+---
+ci_builds:
+  colcon_any-manual: global/colcon_any-manual.yaml
+  colcon_gpu_any-manual: global/colcon_gpu_any-manual.yaml
+jenkins_url: https://build.osrfoundation.org
+notification_emails:
+- [email protected]
+prerequisites:
+rosdistro_index_url: https://raw.githubusercontent.com/ros/rosdistro/master/index-v4.yaml
+type: buildfarm
+version: 1