CI incorrectly runs build tool tests on output of test tool

[rotu]

When a test writes output that mimics build output, like "[ERROR] []: Original error msg: custom exception raised in configure callback" (as seen in ros2/rclcpp#777), it falsely flags it as a MSBuild error. The text output of colcon test-result should never cause a CI failure.

This might be a bug in the deprecated Warnings plugin currently in use or might be a misconfiguration of CI rules.

[dirk-thomas]

When a test writes output that mimics build output

Since the Jenkins plugin parses the output to find known patterns I don't see how we could avoid this. Do you have a specific idea how that could be achieved? If not, I am inclined to close this as wontfix since it is outside our hands.

[rotu]

@dirk-thomas Yes, both Warnings and Warnings-NG plugins can (should) be configured to analyze a file other than consoleText. By default, colcon emits build and test output to log files already.

[j-rivero]

reopening as per #359 (comment)

[hidmic]

@j-rivero @rotu considering #404 is in, can this issue be closed?

[j-rivero]

@j-rivero @rotu considering #404 is in, can this issue be closed?

I don't think that this can be closed since #404 is not limiting the scope of the warnings in any way. To fix this issue we should inject the logs from colcon into the warnings-ng plugin instead of parsing all the output. We still have some false positives due to information coming from random test context (see http://build.ros.org/job/Mpr__gazebo_ros_pkgs__ubuntu_bionic_amd64/36/gcc/#issuesContent)

[dirk-thomas]

@j-rivero the referenced Melodic job is not using colcon but catkin_make_isolated which doesn't capture any log files.

[j-rivero]

@j-rivero the referenced Melodic job is not using colcon but catkin_make_isolated which doesn't capture any log files.

Right, sorry, the fix for the ROS1 buildfarm could be obvious. The example for the ROS2 buildfarm should be: http://build.ros2.org/job/Edev__gazebo_ros_pkgs__ubuntu_bionic_amd64/12/gcc/

[dirk-thomas]

Addressed by #477 and ros-infrastructure/ros_buildfarm#812.

[rotu]

Awesome job, @dirk-thomas! Thanks for making the CI so much more usable!