Merge changes from master branch into main branch #303
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* generators: update for new formatting E741 * CI was now failing with: /home/travis/build/ros-infrastructure/superflore/tests/../superflore/generators/bitbake/yocto_recipe.py:266:33: E741 ambiguous variable name l /home/travis/build/ros-infrastructure/superflore/tests/../superflore/generators/bitbake/yocto_recipe.py:407:37: E741 ambiguous variable name l /home/travis/build/ros-infrastructure/superflore/tests/../superflore/generators/ebuild/ebuild.py:164:42: E741 ambiguous variable name l /home/travis/build/ros-infrastructure/superflore/tests/../superflore/generators/ebuild/ebuild.py:170:61: E741 ambiguous variable name l /home/travis/build/ros-infrastructure/superflore/tests/../superflore/utils.py:148:17: E741 ambiguous variable name 'l' Signed-off-by: Martin Jansa <[email protected]> * PackageMetadata.py: use get_build_type() from catkin_pkg The implementation here didn't respect conditionals at all, but the function in catkin_pkg does since v0.4.10 with: ros-infrastructure/catkin_pkg@0d85ff9 Increase the minimal version of catkin_pkg from v0.4.0 to v0.4.10. Signed-off-by: Martin Jansa <[email protected]>
* noetic should use 3 Signed-off-by: Martin Jansa <[email protected]>
* when regenerating the recipes I often see change like this: suitesparse: -- suitesparse-cholmod@meta-ros - suitesparse-cxsparse@meta-ros +- suitesparse-cholmod@meta-ros * because the order of the list is not defined Signed-off-by: Martin Jansa <[email protected]>
* it will be used for creating context for conditional evaluations in rosdistro as well Signed-off-by: Martin Jansa <[email protected]>
* Use only ROS_OS_OVERRIDE, ROS_DISTRO, ROS_PYTHON_VERSION, ROS_VERSION variables with ROS_PYTHON_VERSION, ROS_VERSION derived from distro value instead of depending on ros-generate-recipes.sh script from meta-ros to export correct value in shell before calling superflore * this is needed by DependencyWalker evaluate_condition_context parameter to correctly resolve the conditionals when evaluating dependencies and build_type * there are 3 variables used in currently generated package.xml files not covered by this context: "ros_version" (lowercase) in lgsvl-msgs (all ROS distributions) https://raw.githubusercontent.com/ros2-gbp/lgsvl_msgs-release/release/rolling/lgsvl_msgs/0.0.3-2/package.xml "BUILD_WITH_LDMRS_SUPPORT" in sick_scan2 (only in foxy) "IGNITION_VERSION" in ros_ign_image (only in noetic) https://raw.githubusercontent.com/ros-gbp/ros_ign-release/release/noetic/ros_ign_image/0.111.0-1/package.xml Signed-off-by: Martin Jansa <[email protected]>
…ound * the package names aren't the same as generated recipes, so it's easy to make mistake and call e.g. sh scripts/ros-generate-recipes.sh noetic ros-core instead of sh scripts/ros-generate-recipes.sh noetic ros_core * the KeyError doesn't seem to be thrown from regenerate_pkg in this case as it raises RuntimeError, but extend the error message in both cases (as it's possible that KeyError might be raised in some other code-paths) Signed-off-by: Martin Jansa <[email protected]>
* set preserve_existing to False in regenerate_pkg call otherwise it will consider old recipe to be up to date (even without comparing its version) * call add_generated_files before commit, otherwise it generates a commit which removes the old version, but the new version is left untracked Signed-off-by: Martin Jansa <[email protected]>
…ot ROS 1 (#267) * _get_ros_version() expects just a string (as in distro.name) not whole RosDistro class This is a bit confusing, because in many places we use the same distro variable as this string, rename it to make it more clear in the next commit. Signed-off-by: Martin Jansa <[email protected]>
#267) * if get_build_type() returns 'catkin' and we're generating some ROS 2 distribution change it to 'ament_cmake' and show an error message * there are couple cases where <export><build_type> is missing completely and then catkin_pkg get_build_type() will return the default value as 'catkin' as specified in https://www.ros.org/reps/rep-0149.html#build-type-multiple "If no <build_type> is provided, catkin is assumed." but this assumption doesn't work for ROS 2 distributions and in meta-ros we had to override these cases (as of 2020-12-05): dashing: https://raw.github.com/ros2-gbp/async_web_server_cpp-release/release/dashing/async_web_server_cpp/1.0.0-1/package.xml https://raw.github.com/boschresearch/fmilibrary_vendor-release/release/dashing/fmilibrary_vendor/0.2.0-1/package.xml https://raw.github.com/ros-geographic-info/geographic_info-release/release/dashing/geographic_info/1.0.1-1/package.xml https://raw.github.com/swri-robotics-gbp/gps_umd-release/release/dashing/gps_umd/1.0.4-1/package.xml https://raw.github.com/ros2-gbp/web_video_server-release/release/dashing/web_video_server/1.0.0-1/package.xml https://raw.github.com/KIT-MRT/mrt_cmake_modules-release/release/dashing/mrt_cmake_modules/1.0.8-1/package.xml eloquent: https://raw.github.com/boschresearch/fmilibrary_vendor-release/release/eloquent/fmilibrary_vendor/0.2.0-1/package.xml https://raw.github.com/ros-geographic-info/geographic_info-release/release/eloquent/geographic_info/1.0.1-1/package.xml https://raw.github.com/swri-robotics-gbp/gps_umd-release/release/eloquent/gps_umd/1.0.2-1/package.xml foxy: https://raw.github.com/boschresearch/fmilibrary_vendor-release/release/foxy/fmilibrary_vendor/0.2.0-1/package.xml https://raw.github.com/swri-robotics-gbp/gps_umd-release/release/foxy/gps_umd/1.0.4-1/package.xml https://raw.github.com/KIT-MRT/mrt_cmake_modules-release/release/foxy/mrt_cmake_modules/1.0.8-1/package.xml rolling: https://raw.github.com/ros2-gbp/fmilibrary_vendor-release/release/rolling/fmilibrary_vendor/0.2.0-2/package.xml https://raw.github.com/ros2-gbp/gps_umd-release/release/rolling/gps_umd/1.0.3-2/package.xml Signed-off-by: Martin Jansa <[email protected]>
* e.g. _get_ros_version() expects just a string (as in distro.name) not whole RosDistro class this is a bit confusing, because in many places we use the same distro variable as this string, rename it to make it more clear Signed-off-by: Martin Jansa <[email protected]>
…pe names * with https://raw.githubusercontent.com/ros-gbp/moveit-release/release/noetic/moveit_kinematics/1.1.0-1/package.xml <depend condition="$ROS_DISTRO != noetic">orocos_kdl</depend> <depend condition="$ROS_DISTRO == noetic">liborocos-kdl-dev</depend> we have accidentally got orocos_kdl dependency (because ROS_DISTRO was missing in evaluate_condition_context) which results in dependency on ROS_UNRESOLVED_PLATFORM_PKG_orocos_kdl * but ROS_UNRESOLVED_PLATFORM_PKG_orocos_kdl = "UNRESOLVED-orocos_kdl" never got generated in generated/superflore-ros-distro.inc simply because https://github.com/ros-infrastructure/superflore/blob/649543e232fe1f44fcdf12c90ed5e0fc19bad4b8/superflore/generators/bitbake/yocto_recipe.py#L671 takes last underscore-separated field as PN, so it generated ROS_UNRESOLVED_PLATFORM_PKG_kdl = "UNRESOLVED-kdl" instead which in the end caused unexpanded dependency issue like: ERROR: Nothing PROVIDES '${ROS_UNRESOLVED_PLATFORM_PKG_orocos_kdl}' (but meta-ros/meta-ros1-noetic/generated-recipes/robot-calibration/robot-calibration_0.6.4-1.bb, meta-ros/meta-ros1-noetic/generated-recipes/moveit/moveit-kinematics_1.1.0-1.bb DEPENDS on or otherwise requires it) * similar issue is reproducible e.g. with jsk_pcl_ros: https://raw.githubusercontent.com/tork-a/jsk_recognition-release/release/noetic/jsk_pcl_ros/1.2.15-1/package.xml which has: <exec_depend>ml_classifiers</exec_depend> but there is no ml_classifiers package in noetic, it's only in melodic and dashing where it also generated: ROS_UNRESOLVED_PLATFORM_PKG_classifiers = "UNRESOLVED-classifiers" instead of: ROS_UNRESOLVED_PLATFORM_PKG_ml-classifiers = "UNRESOLVED-ml-classifiers" and similarly openni_launch which also doesn't exist in noetic, only in melodic ROS_UNRESOLVED_PLATFORM_PKG_launch = "UNRESOLVED-launch" ROS_UNRESOLVED_PLATFORM_PKG_openni-launch = "UNRESOLVED-openni-launch" * we could easily fix that by using whole "ROS_UNRESOLVED_PLATFORM_PKG_" prefix as a separator before PN, but there is still a risk of using underscore in bitbake variable names, because if some build happens to include one of these underscore separated strings (e.g. 'kdl') in OVERRIDES variable, then the ROS_UNRESOLVED_PLATFORM_PKG_orocos_kdl = "UNRESOLVED-orocos_kdl" assignment will be expanded as assignment to variable "ROS_UNRESOLVED_PLATFORM_PKG" with "kdl" and "orocos" as an overrides and ${ROS_UNRESOLVED_PLATFORM_PKG_orocos_kdl} will still end undefined - using fewer underscores here is better * but there is still an option that the whole ROS package name will match with something in OVERRIDES (e.g. MACHINE name can be pretty much anything) and then these variables will fail the same, to fix it properly we should use uppercase for dependency name as well (while possibly changing the prefix to use dash as separators for easier readability), e.g.: ROS-UNRESOLVED-PLATFORM-PKG_OROCOS-KDL = "UNRESOLVED-OROCOS-KDL" but that would require migration of all manually overriden mappings: $ git grep ^ROS_UNRESOLVED_PLATFORM_PKG_ | grep '/ros-distro\.inc' | wc -l 491 which is easy, but would cause unwelcome diff between newer and older meta-ros branches. To mitigate this we plan to upstream all these manual overrides from meta-ros to rosdistro rosdep files and then we can re-evaluate this change to upper case when we don't use so many overrides Signed-off-by: Martin Jansa <[email protected]>
* use DEPENDENCY instead of PLATFORM_PKG in UNRESOLVED_* variables in some cases as documented in previous commit the unresolved dependency might be ROS package which just happens to be missing in pkg_names, so it's considered as external/system/platform package * parse dependency name by skipping whole UNRESOLVED_DEPENDENCY_REF_PREFIX instead of relying on last underscore separated string Signed-off-by: Martin Jansa <[email protected]>
* without split() it was trying to remove all these files as one
path in quotes and failing as shown after removing --ignore-unmatch
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
cmdline: git rm -rf meta-ros2-eloquent/generated-recipes meta-ros2-eloquent/conf/ros-distro/include/eloquent/generated meta-ros2-eloquent/files/eloquent/generated/newer-platform-components.list meta-ros2-eloquent/files/eloquent/generated/rosdep-resolve.yaml meta-ros2-eloquent/files/eloquent/generated/superflore-change-summary.txt
stderr: 'fatal: pathspec 'meta-ros2-eloquent/generated-recipes meta-ros2-eloquent/conf/ros-distro/include/eloquent/generated meta-ros2-eloquent/files/eloquent/generated/newer-platform-components.list meta-ros2-eloquent/files/eloquent/generated/rosdep-resolve.yaml meta-ros2-eloquent/files/eloquent/generated/superflore-change-summary.txt ' did not match any files'
with split() it correctly complains only about superflore-change-summary.txt
which will be fixed by returning --ignore-unmatch
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
cmdline: git rm -rf meta-ros2-eloquent/generated-recipes meta-ros2-eloquent/conf/ros-distro/include/eloquent/generated meta-ros2-eloquent/files/eloquent/generated/newer-platform-components.list meta-ros2-eloquent/files/eloquent/generated/rosdep-resolve.yaml meta-ros2-eloquent/files/eloquent/generated/superflore-change-summary.txt
stderr: 'fatal: pathspec 'meta-ros2-eloquent/files/eloquent/generated/superflore-change-summary.txt' did not match any files'
Signed-off-by: Martin Jansa <[email protected]>
* this way the lowercase dependency name won't ever be parsed as an
override
* use the same prefix in the value (instead of 'UNRESOLVED-')
* there was still a risk of using underscore
in bitbake variable names, because if some build happens to include
one of these underscore separated strings (e.g. 'orocos-kdl') in OVERRIDES variable,
then the ROS_UNRESOLVED_PLATFORM_PKG_orocos_kdl = "UNRESOLVED-orocos_kdl" assignment will
be expanded as assignment to variable "ROS_UNRESOLVED_PLATFORM_PKG" with
"orocos-kdl" as an override and ${ROS_UNRESOLVED_PLATFORM_PKG_orocos_kdl}
will still end undefined - using fewer underscores here is better
* there was still an option that the whole ROS package name will match with
something in OVERRIDES (e.g. MACHINE name can be pretty much anything) and then
these variables will fail, to fix it properly we should either use uppercase
for dependency name or change the prefix separator from underscore to dash.
* this will require migration of all manually overriden mappings:
$ git grep ^ROS_UNRESOLVED_PLATFORM_PKG_ | grep '/ros-distro\.inc' | wc -l
491
so I've updated rosdistro/rosdep to include most of them first and
regenerated all ROS distributions with updated rosdistro.
Signed-off-by: Martin Jansa <[email protected]>
* instead of trying to guess what the owner meant
* exact string replacements are safer than parsing with regular expressions
because the license value is often quite creative
* these statistics from old regexp implementation are just from regenerated melodic recipes:
A) clearly wrong replacements
1 "BSD & CreativeCommons-by-nc-4.0" -> "BSD & CC-BY-SA-3.0"
1 "BSD & GPLv3 & LGPLv3" -> "BSD & GPL-3 & LGPL-2"
1 "BSD,-Apache-2.0" -> "BSD-2"
1 "CreativeCommons-Attribution-NonCommercial-ShareAlike-4.0-International" -> "CC-BY-SA-3.0"
1 "LGPL3" -> "LGPL-2"
2 "Creative-Commons-BY-NC-ND-3.0" -> "CC-BY-NC-ND-4.0"
4 "GPLv3 & LGPLv3 & BSD" -> "GPL-3 & LGPL-2 & BSD"
17 "GPLv3" -> "GPL-3"
21 "LGPLv3" -> "LGPL-2"
B) not clearly wrong, but guessing too much about version and often incorrectly
(after inspection of the actual source files)
1 "Apache-License-2.0 & LGPL" -> "Apache-2.0 & LGPL-2"
1 "BSD & LGPL & GPL-for-sigblock" -> "BSD & LGPL-2 & GPL-1"
1 "LGPL & proprietary" -> "LGPL-2 & proprietary"
2 "BSD & Creative-Commons" -> "BSD & CC-BY-SA-3.0"
4 "Apache" -> "Apache-1.0"
5 "GNU-Lesser-General-Public-License-LGPL-" -> "LGPL-2"
15 "GPL" -> "GPL-1"
23 "LGPL" -> "LGPL-2"
* and in some cases it was loosing some useful information:
1 "BSD,-Apache-2.0" -> "BSD-2"
1 "BSD,-LGPL" -> "BSD"
1 "BSD,-some-icons-are-licensed-under-the-GNU-Lesser-General-Public-License-LGPL-or-Creative-Commons-Attribution-Noncommercial-3.0-License" -> "BSD"
1 "BSD,GPL-because-of-list.h;-other-files-released-under-BSD,GPL" -> "BSD"
1 "BSD,LGPL,Apache-2.0" -> "BSD-2"
1 "BSD,LGPL,LGPL-amcl-" -> "BSD"
1 "GPL-because-of-list.h;-other-files-released-under-BSD" -> "GPL-1"
1 "GPLv2-with-linking-exception" -> "GPL-2"
2 "BSD,-GPL" -> "BSD"
8 "BSD-3-Clause" -> "BSD"
* fixes:
#271
* 'Apache' doesn't always mean version 1 in 'Apache-1.0' * 'GPL' doesn't always mean version 1 in 'GPL-1' * 'Mozilla Public License' doesn't always mean version 2.0 in 'MPL-2.0' * 'BSD License 2.0' doesn't always mean clause-2 in 'BSD-2' * 'Creative Commons' doesn't always mean Creative Commons Attribution Share Alike 3.0 in 'CC-BY-SA-3.0' * to be safe leave the orignal value and catkin_pkg package.xml validation will hopefully notify component developer to better specify the version based on the real source files - instead of superflore guessing what was meant by that * fixes: #271 Signed-off-by: Martin Jansa <[email protected]>
Signed-off-by: Martin Jansa <[email protected]>
…t original value in comment * print the original value from package.xml whenever the get_license function modified, this makes it easier to review what superflore did in the recipe and makes some more creative values more readable e.g. "HEBI C++ Software License (https://www.hebirobotics.com/softwarelicense)" instead of: "HEBI-C-Software-License-https-www.hebirobotics.com-softwarelicense-" or "N/A" instead of "N-A" Signed-off-by: Martin Jansa <[email protected]>
* replace spaces with dashes in these 4 cases which weren't mapped to valid SPDX (which would be without spaces) Signed-off-by: Martin Jansa <[email protected]>
Also removes defunct travis.yml config.
* Set up tag for 0.3.3 Signed-off-by: Hunter L. Allen <[email protected]> * Update changelog Signed-off-by: Hunter L. Allen <[email protected]>
* generate new syntax for OE recipes to match: https://lists.openembedded.org/g/openembedded-architecture/message/1260 https://lists.openembedded.org/g/openembedded-architecture/message/1279 https://lists.openembedded.org/g/openembedded-architecture/message/1291 * this was already done with conversion script in meta-ros with: ros/meta-ros#902 but for new ROS Distribution releases we want to generate new syntax directly with superflore. * it's not conditional based on OE release series, because all currently supported versions (dunfell, hardknott, honister) do support new syntax (when latest bitbake revision from corresponding branch is being used). * fixes #284 Signed-off-by: Martin Jansa <[email protected]>
…UILDTOOLS Signed-off-by: Martin Jansa <[email protected]>
Signed-off-by: Rob Woolley <[email protected]>
This prevents trailing zeroes from being removed by making the values strings not numbers. Signed-off-by: Rob Woolley <[email protected]>
Use pynose for Python 3.11+ support Signed-off-by: Rob Woolley <[email protected]>
Excluding tests for docker and for creating invalid directory paths. These tests do not seem relevant to testing superflore in a containerized CI/CD pipeline. Signed-off-by: Rob Woolley <[email protected]>
Update the checkout and setup-python actions to the latest versions. Remove Ubuntu 18.04 as a runner image as it has been deprecated by GitHub Actions and no longer has runners available. Signed-off-by: Rob Woolley <[email protected]>
These steps follow the installation instructions from the Rolling Ridley documentation for setting up a new system. These changes are necessary to use the ros2 package repository for newer versions of Ubuntu. https://docs.ros.org/en/rolling/Installation/Ubuntu-Install-Debs.html Signed-off-by: Rob Woolley <[email protected]>
Temporarily disable the ebuild unit tests. Support for Gentoo needs to be updated and the tests currently fail. Signed-off-by: Rob Woolley <[email protected]>
Adding --verbose to python nose will print out the test cases names. This is useful for seeing what tests are being run when viewing the logs. Signed-off-by: Rob Woolley <[email protected]>
Signed-off-by: Hunter L. Allen <[email protected]>
This `pkg` argument is shadowed by a local variable without first being used.
Ebuild generation was not utilizing package conditions. Packages with multiple build types were causing hard failures during ebuild generation but it's possible that invalid ebuilds were being generated for other packages as well due to the lack of dependency conditions.
|
Closing this PR for now, will reopen later. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is in anticipation of changing the default branch to main.
See #278 and https://discourse.ros.org/t/consider-renaming-default-branches/15385 for details.