diff --git a/jenkins-scripts/ros_buildfarm_config/README.md b/jenkins-scripts/ros_buildfarm_config/README.md new file mode 100644 index 000000000..973be5d6f --- /dev/null +++ b/jenkins-scripts/ros_buildfarm_config/README.md @@ -0,0 +1,67 @@ +# ros_buildfam configuration for the Gazebo buildfarm + +This directory hosts the ros_buildfarm configuration used to create some jobs +in the Gazebo buildfarm. Mostly inspired by the ROS buildfarm configuration +repository: https://github.com/ros2/ros_buildfarm_config/ + +## Rationale + +The [ros_buildfarm](https://github.com/ros-infrastructure/ros_buildfarm) code +is able to generate Jenkins jobs designed mainly for the ROS buildfarm that +are also applicable to other Jenkins buildfarms. In this case the Gazebo +development team was looking for a colcon based jobs able to receive +as an input a VCS file. Instead of creating these jobs from scratch, the use of +ros_buildfarm was chosen to provide the required features and many others more. + +## Jobs generated + +The jobs generated by ros_buildfarm are easily recognizable since they are +placed under the Jenkins views `ci` (non ROS related jobs) or `Xci` +(where X is the initial of a ROS release). The job names also uses these +same prefixes: `ci__` or `Xci__`. + +## Generating jobs + +### Initial setup + +Be sure of going through the +[Setup environment to deploy configuration](https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/environment.rst) +document in the ros_buildfarm as the first step to proceed with the jobs +generation. + +### Running job generation + +Note that the Gazebo buildfarm can not execute the script +`ros_buildfarm/scripts/generate_all_jobs.py` since it will create some +maintenance views and jobs that are not in the scope of this buildfarm. + +Instead of that, the job generation is done manually per configuration +file using the following (empty quotes are mandatory): + +``` +# remove the --dry-run for real execution +./generate_ci_jobs.py --dry-run \ + https://raw.githubusercontent.com/gazebo-tooling/release-tools/jrivero/ros_buildfarm/jenkins-scripts/ros_buildfarm_config/index.yaml \ + "" +``` + +The different `` are defined in the index file just before the key +`ci_builds:`. So for example + +``` +# remove the --dry-run for real execution +./generate_ci_jobs.py --dry-run \ + https://raw.githubusercontent.com/gazebo-tooling/release-tools/jrivero/ros_buildfarm/jenkins-scripts/ros_buildfarm_config/index.yaml \ + "" colcon_any-manual +``` + +To automate the process using the `yq` tool: + +``` +wget https://raw.githubusercontent.com/gazebo-tooling/release-tools/master/jenkins-scripts/ros_buildfarm_config/index.yaml +for ci_build in $(yq '.ci_builds | keys' < index.yaml | sed 's:^- ::'); do +./generate_ci_jobs.py --dry-run \ + https://raw.githubusercontent.com/gazebo-tooling/release-tools/jrivero/ros_buildfarm/jenkins-scripts/ros_buildfarm_config/index.yaml \ + "" $ci_build +done +``` diff --git a/jenkins-scripts/ros_buildfarm_config/global/colcon_any-manual.yaml b/jenkins-scripts/ros_buildfarm_config/global/colcon_any-manual.yaml new file mode 100644 index 000000000..be91438e4 --- /dev/null +++ b/jenkins-scripts/ros_buildfarm_config/global/colcon_any-manual.yaml @@ -0,0 +1,92 @@ +%YAML 1.1 +# ROS buildfarm ci-build file +--- +build_tool: colcon +build_tool_args: '--cmake-args -DCMAKE_BUILD_TYPE=Release --no-warn-unused-cli' +build_tool_test_args: '--retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail"' +jenkins_job_label: docker +jenkins_job_priority: 200 # same that pr-any +jenkins_job_timeout: 300 +repos_files: +- https://raw.githubusercontent.com/gazebo-tooling/gazebodistro/master/collection-harmonic.yaml +repositories: + keys: + - | + -----BEGIN PGP PUBLIC KEY BLOCK----- + Version: GnuPG v1 + + mQENBFUcKaEBCAD1ajXaWLnow3pZEv44Lypt6s5jAh1rYPN6zpaFZWdkzpwTdYU1 + Rpw/0hPzIoiyOPNwCti4E3+dSrv1ogEBu85P2XSy67RnabxF4/z7mPG/++u0EQav + CwfrsN8OpJTtTxk+nKIhVwpAtob+KOLATerTPETrdrKh7qJ/FE8cw/XXbknjwywf + R8uJqaKTu7mWNrTFaS3P5GZF5ss+ztf0EHcyYFMvzEVnSiOGBBL9pw91P1qpggBa + lKL1Ilmf6zZBPihORJ/iTH5qMCAPDdR5BaxxEUHgz+pg+RkLKd2ENEaO+SCDVRhP + yNdkYHpuIslyMHfXrh4y5nHclJ+bNXKXDcudABEBAAG0R09TUkYgUmVwb3NpdG9y + eSAoT1NSRiBSZXBvc2l0b3J5IEdQRyBrZXkpIDxvc3JmYnVpbGRAb3NyZm91bmRh + dGlvbi5vcmc+iQE4BBMBAgAiBQJVHCmhAhsDBgsJCAcDAgYVCAIJCgsEFgIDAQIe + AQIXgAAKCRBnFwWYrySXQ/D4CACVnSdHT/1dEkOrYkCnaFLNBrG2tJdBrbIZOxKy + +xV0yGniqsQFAxLESoy+ygaiKdTnAFlA24ozoNY8ur+oKMFt6CrUY01ItTq/WMA1 + iper0TO935SpDzNIPjPnD2WUSXShISWP0bFg64g0aAD1S7Yg/v7/eOmMSoeMav0T + h8KOo6yhJuhgGp3lHKAKLppH94b77d8JYqGeP03Gv6gcaqNojyKccdXrKTugZui5 + +7V/cOJTo9XqzXjkpfwp24jR8FlKI7EWqCVqtRAXHeqRgo3OaKmuoKLcJ4/8BjSU + +ppmJtEstSaL+qw49P/GQHwUkCHlx1mV5dSdVFLBPreli1ChuQENBFUcKaEBCAC7 + ZgTdYubw1sU/4A6+NvW/poBfh2DDOeh3uHJc0y235JFjr+tC1AwouaxLOUm8FE9k + 7qzwnyXbeklmXAHxw6wXZdE4PEYA/sgBYhTQy+s4PHlI6TGhwgcROkJKlW4Lld+W + IJ/fzW93DXyhEkV3AAhkrVcOLOgCPdpK5EXxJ3p6dCOKC5Vjyz1PxTNcRaLpp9w6 + J0hLIXmmoCN4aoYSXWtL/C9J+B5Cr+HHgrmFsGNrHmmVv1gMXLcVzw5p3Z4d8SuT + g9a1CemSE5bFIoOHKEQRwv/CGpoviAr+T3za3dPFTcSMOoJuYvoheTJ6fhf2sj74 + bp2Fwi4L7am/asfa7xWVABEBAAGJAR8EGAECAAkFAlUcKaECGwwACgkQZxcFmK8k + l0OX9Af+IrzUChXf6H0nZZY77gcjwFgVChRX1RLzHTTHum4WNKGP9Sw1aGdHpmdt + LhypQImxdT2yhCPEyB8EQxhgPHjqZ6UUMeYMw5rAvrcb3/ercy5pG7O8Z+Bea6hu + TAXquJ1tsFessZwMS3RUXp/gtZCHbESR7PeBlZJWBWxG/lOmX7Z4fa88dWRU0Pl/ + nfns7v6eb57HXbf0teCitRRsJwCMhYbHj2m1slZHMjhEc6kv2bgPmAFb04bcyEAP + BAo3BKu2XUVqE1t7Q2EfsItL/0FpfDY6zGKM6NIi+C40CsRl4W0o6egUhiDqsMYX + 9Su5aZdCoxMhzy5QxS3sXcpNAWH2gw== + =YM5F + -----END PGP PUBLIC KEY BLOCK----- + - | + -----BEGIN PGP PUBLIC KEY BLOCK----- + + mQINBF/btn4BEADSD9KhOgzOwDFOw+ivJXt3lw6Ubyo6c2yXEEASnQpzQ2IPJBpQ + zAX+rncd76xNNgtAUmtPFdsWojwIOYvHLnCusmjB5zmpr08alKKUqKfjKI0ALfq8 + xrv2F7AFeaV2FWaCrH8ZrNBxLUnm/1MZbrknvnh4J+1Bpzi4JQvYJ1hb2BUYCPBj + kXTpShj58yzlPvr5oDzmy++evPjVf8Lh5f0LxWtDlYK8NED0dBzHZlL1eeuEmNJP + Iw/UNVlhVgPv3j329sdmGja+uck5JEAlL0uRU4D83f/YndwUUX+I+Um5baGuyJ42 + FXFtw9JHF3DJiERpYd/ZCPP8jQwAxKee2zukxUh/L1MCWBFoZCIuuy30ACq20gWZ + PAnDpqF9H/fzg3FQiwAaeF7ZCZPNvSCnV7myqdZ7C/adgayuRhT6vpMP48WACqV6 + ndmYEJL2WC+eT7eDD6hEZpVnBNeuRA/RnU6aRgybZ2+rB+hhXTfX3xzLxqJeLIZ7 + SWpXGNRZAmOy9Eqt/dKrzUUyrOXfgqHd75mWFlbHrQZ8tt0+1PE76XG4ZgT3pqOk + bHHqywJfJXpy5U62YBy0TruMM7WyKN8IrkvUdYkMGbyfVKudQha0lTIAvX8zeW1z + AHovXBFkdR9NgdggZV1XWdbBlscLOKLcg9t+HT2EKSvgx1sPJEhzBFdahwARAQAB + tFpST1MgQm9vdHN0cmFwIFJlcG9zaXRvcnkgKFNpZ25pbmcgS2V5IGZvciBST1Mg + Qm9vdHN0cmFwIFJlcG9zaXRvcnkpIDxyb3NAb3BlbnJvYm90aWNzLm9yZz6JAk4E + EwEIADgWIQRHMs5wbNexnbD/506O2y72YfyIDgUCX9u2fgIbAwULCQgHAgYVCgkI + CwIEFgIDAQIeAQIXgAAKCRCO2y72YfyIDksLD/9awISK9MlJJZ2W769Fj8D2JOEM + splwtVNBuADyH2rT2TjL5aY+V8/awq/DApl3ty/QpfM8qcybUyi94faAhdvYygMr + AvU3/VusaxexlJHKO9GkdFfFQTtcksWhV7+z21h1omq5RimInJA6eyrhsNjFnOiI + we4SG3Jwi+Scr0xPLIyuboacD+3AXxPaA3mwQjLnMyYMyBCZs7tTtET5FR6hAGIg + eptpm+Fiaosjub35/VB9r/JOjJCuiwKObMye+8jcDdR8umFxzqcgEe1TNQVBj5+V + I4ssbrzKvwAkte3mBEdndEBPbSlTAo0kd4ByMRZ5AWhqRdu4iabw8cz2IpGErUcb + K0ws76+D8PqnqjPqd5GkGxB54zLMMrrBya5+hnzyDifKzvv1M+L2xvFuICBLBTR7 + FnRownLKmDXLXp/EloRpTvchsO9W+Y3cDTkzkKpaUgs8Gl4dI/zPkHEp8iZBZNsk + mEibjJ5g2qBAYr9GbyaUiifdmpUiDoJpCyDq22rKfFkZl7t1+1MVbjz92A8Vjesb + QHt0icnkrvwg7mYfmMcdVF4TVNaR2XzLTkAz62r1kOoXnPbzUTA1uiLakagsUG5G + YXyUVOJlJDTWr784FVSsEQIslvbO8dk3gmiOpurxBrLGn70SdhCnQRqSrjOnvzX7 + 3ih4hgyGHQOJ/7p5Rg== + =v6u9 + -----END PGP PUBLIC KEY BLOCK----- + urls: + - https://packages.osrfoundation.org/gazebo/ubuntu-stable + - http://repos.ros.org/repos/ros_bootstrap # Needed for getting the ROS building tools packages like python3-rosdep +skip_rosdep_keys: +- python3-distutils +targets: + ubuntu: + noble: + amd64: + jammy: + amd64: + _config: + custom_rosdep_urls: + - https://raw.githubusercontent.com/osrf/osrf-rosdep/master/gz/00-gazebo.list +type: ci-build +version: 1 diff --git a/jenkins-scripts/ros_buildfarm_config/global/colcon_gpu_any-manual.yaml b/jenkins-scripts/ros_buildfarm_config/global/colcon_gpu_any-manual.yaml new file mode 100644 index 000000000..9eb0188d2 --- /dev/null +++ b/jenkins-scripts/ros_buildfarm_config/global/colcon_gpu_any-manual.yaml @@ -0,0 +1,93 @@ +%YAML 1.1 +# ROS buildfarm ci-build file +--- +build_tool: colcon +build_tool_args: '--cmake-args -DCMAKE_BUILD_TYPE=Release --no-warn-unused-cli' +build_tool_test_args: '--retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail"' +jenkins_job_label: gpu-reliable +jenkins_job_priority: 200 # same than pr-any +jenkins_job_timeout: 300 +repos_files: +- https://raw.githubusercontent.com/gazebo-tooling/gazebodistro/master/collection-harmonic.yaml +repositories: + keys: + - | + -----BEGIN PGP PUBLIC KEY BLOCK----- + Version: GnuPG v1 + + mQENBFUcKaEBCAD1ajXaWLnow3pZEv44Lypt6s5jAh1rYPN6zpaFZWdkzpwTdYU1 + Rpw/0hPzIoiyOPNwCti4E3+dSrv1ogEBu85P2XSy67RnabxF4/z7mPG/++u0EQav + CwfrsN8OpJTtTxk+nKIhVwpAtob+KOLATerTPETrdrKh7qJ/FE8cw/XXbknjwywf + R8uJqaKTu7mWNrTFaS3P5GZF5ss+ztf0EHcyYFMvzEVnSiOGBBL9pw91P1qpggBa + lKL1Ilmf6zZBPihORJ/iTH5qMCAPDdR5BaxxEUHgz+pg+RkLKd2ENEaO+SCDVRhP + yNdkYHpuIslyMHfXrh4y5nHclJ+bNXKXDcudABEBAAG0R09TUkYgUmVwb3NpdG9y + eSAoT1NSRiBSZXBvc2l0b3J5IEdQRyBrZXkpIDxvc3JmYnVpbGRAb3NyZm91bmRh + dGlvbi5vcmc+iQE4BBMBAgAiBQJVHCmhAhsDBgsJCAcDAgYVCAIJCgsEFgIDAQIe + AQIXgAAKCRBnFwWYrySXQ/D4CACVnSdHT/1dEkOrYkCnaFLNBrG2tJdBrbIZOxKy + +xV0yGniqsQFAxLESoy+ygaiKdTnAFlA24ozoNY8ur+oKMFt6CrUY01ItTq/WMA1 + iper0TO935SpDzNIPjPnD2WUSXShISWP0bFg64g0aAD1S7Yg/v7/eOmMSoeMav0T + h8KOo6yhJuhgGp3lHKAKLppH94b77d8JYqGeP03Gv6gcaqNojyKccdXrKTugZui5 + +7V/cOJTo9XqzXjkpfwp24jR8FlKI7EWqCVqtRAXHeqRgo3OaKmuoKLcJ4/8BjSU + +ppmJtEstSaL+qw49P/GQHwUkCHlx1mV5dSdVFLBPreli1ChuQENBFUcKaEBCAC7 + ZgTdYubw1sU/4A6+NvW/poBfh2DDOeh3uHJc0y235JFjr+tC1AwouaxLOUm8FE9k + 7qzwnyXbeklmXAHxw6wXZdE4PEYA/sgBYhTQy+s4PHlI6TGhwgcROkJKlW4Lld+W + IJ/fzW93DXyhEkV3AAhkrVcOLOgCPdpK5EXxJ3p6dCOKC5Vjyz1PxTNcRaLpp9w6 + J0hLIXmmoCN4aoYSXWtL/C9J+B5Cr+HHgrmFsGNrHmmVv1gMXLcVzw5p3Z4d8SuT + g9a1CemSE5bFIoOHKEQRwv/CGpoviAr+T3za3dPFTcSMOoJuYvoheTJ6fhf2sj74 + bp2Fwi4L7am/asfa7xWVABEBAAGJAR8EGAECAAkFAlUcKaECGwwACgkQZxcFmK8k + l0OX9Af+IrzUChXf6H0nZZY77gcjwFgVChRX1RLzHTTHum4WNKGP9Sw1aGdHpmdt + LhypQImxdT2yhCPEyB8EQxhgPHjqZ6UUMeYMw5rAvrcb3/ercy5pG7O8Z+Bea6hu + TAXquJ1tsFessZwMS3RUXp/gtZCHbESR7PeBlZJWBWxG/lOmX7Z4fa88dWRU0Pl/ + nfns7v6eb57HXbf0teCitRRsJwCMhYbHj2m1slZHMjhEc6kv2bgPmAFb04bcyEAP + BAo3BKu2XUVqE1t7Q2EfsItL/0FpfDY6zGKM6NIi+C40CsRl4W0o6egUhiDqsMYX + 9Su5aZdCoxMhzy5QxS3sXcpNAWH2gw== + =YM5F + -----END PGP PUBLIC KEY BLOCK----- + - | + -----BEGIN PGP PUBLIC KEY BLOCK----- + + mQINBF/btn4BEADSD9KhOgzOwDFOw+ivJXt3lw6Ubyo6c2yXEEASnQpzQ2IPJBpQ + zAX+rncd76xNNgtAUmtPFdsWojwIOYvHLnCusmjB5zmpr08alKKUqKfjKI0ALfq8 + xrv2F7AFeaV2FWaCrH8ZrNBxLUnm/1MZbrknvnh4J+1Bpzi4JQvYJ1hb2BUYCPBj + kXTpShj58yzlPvr5oDzmy++evPjVf8Lh5f0LxWtDlYK8NED0dBzHZlL1eeuEmNJP + Iw/UNVlhVgPv3j329sdmGja+uck5JEAlL0uRU4D83f/YndwUUX+I+Um5baGuyJ42 + FXFtw9JHF3DJiERpYd/ZCPP8jQwAxKee2zukxUh/L1MCWBFoZCIuuy30ACq20gWZ + PAnDpqF9H/fzg3FQiwAaeF7ZCZPNvSCnV7myqdZ7C/adgayuRhT6vpMP48WACqV6 + ndmYEJL2WC+eT7eDD6hEZpVnBNeuRA/RnU6aRgybZ2+rB+hhXTfX3xzLxqJeLIZ7 + SWpXGNRZAmOy9Eqt/dKrzUUyrOXfgqHd75mWFlbHrQZ8tt0+1PE76XG4ZgT3pqOk + bHHqywJfJXpy5U62YBy0TruMM7WyKN8IrkvUdYkMGbyfVKudQha0lTIAvX8zeW1z + AHovXBFkdR9NgdggZV1XWdbBlscLOKLcg9t+HT2EKSvgx1sPJEhzBFdahwARAQAB + tFpST1MgQm9vdHN0cmFwIFJlcG9zaXRvcnkgKFNpZ25pbmcgS2V5IGZvciBST1Mg + Qm9vdHN0cmFwIFJlcG9zaXRvcnkpIDxyb3NAb3BlbnJvYm90aWNzLm9yZz6JAk4E + EwEIADgWIQRHMs5wbNexnbD/506O2y72YfyIDgUCX9u2fgIbAwULCQgHAgYVCgkI + CwIEFgIDAQIeAQIXgAAKCRCO2y72YfyIDksLD/9awISK9MlJJZ2W769Fj8D2JOEM + splwtVNBuADyH2rT2TjL5aY+V8/awq/DApl3ty/QpfM8qcybUyi94faAhdvYygMr + AvU3/VusaxexlJHKO9GkdFfFQTtcksWhV7+z21h1omq5RimInJA6eyrhsNjFnOiI + we4SG3Jwi+Scr0xPLIyuboacD+3AXxPaA3mwQjLnMyYMyBCZs7tTtET5FR6hAGIg + eptpm+Fiaosjub35/VB9r/JOjJCuiwKObMye+8jcDdR8umFxzqcgEe1TNQVBj5+V + I4ssbrzKvwAkte3mBEdndEBPbSlTAo0kd4ByMRZ5AWhqRdu4iabw8cz2IpGErUcb + K0ws76+D8PqnqjPqd5GkGxB54zLMMrrBya5+hnzyDifKzvv1M+L2xvFuICBLBTR7 + FnRownLKmDXLXp/EloRpTvchsO9W+Y3cDTkzkKpaUgs8Gl4dI/zPkHEp8iZBZNsk + mEibjJ5g2qBAYr9GbyaUiifdmpUiDoJpCyDq22rKfFkZl7t1+1MVbjz92A8Vjesb + QHt0icnkrvwg7mYfmMcdVF4TVNaR2XzLTkAz62r1kOoXnPbzUTA1uiLakagsUG5G + YXyUVOJlJDTWr784FVSsEQIslvbO8dk3gmiOpurxBrLGn70SdhCnQRqSrjOnvzX7 + 3ih4hgyGHQOJ/7p5Rg== + =v6u9 + -----END PGP PUBLIC KEY BLOCK----- + urls: + - https://packages.osrfoundation.org/gazebo/ubuntu-stable + - http://repos.ros.org/repos/ros_bootstrap # Needed for getting the ROS building tools packages like python3-rosdep +skip_rosdep_keys: +- python3-distutils +targets: + ubuntu: + noble: + amd64: + jammy: + amd64: + _config: + custom_rosdep_urls: + - https://raw.githubusercontent.com/osrf/osrf-rosdep/master/gz/00-gazebo.list +tests_require_gpu: true +type: ci-build +version: 1 diff --git a/jenkins-scripts/ros_buildfarm_config/index.yaml b/jenkins-scripts/ros_buildfarm_config/index.yaml new file mode 100644 index 000000000..3f2cb8997 --- /dev/null +++ b/jenkins-scripts/ros_buildfarm_config/index.yaml @@ -0,0 +1,13 @@ +%YAML 1.1 +# ROS buildfarm index file +--- +ci_builds: + colcon_any-manual: global/colcon_any-manual.yaml + colcon_gpu_any-manual: global/colcon_gpu_any-manual.yaml +jenkins_url: https://build.osrfoundation.org +notification_emails: +- jrivero@osrfoundation.org +prerequisites: +rosdistro_index_url: https://raw.githubusercontent.com/ros/rosdistro/master/index-v4.yaml +type: buildfarm +version: 1