Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepare Rolling for platform transition. #32036

Merged
merged 5 commits into from
Feb 9, 2022

Conversation

nuclearsandwich
Copy link
Member

@nuclearsandwich nuclearsandwich commented Feb 3, 2022

As part of the transition process using the rosdistro migration tools we need an empty target distribution with the intended target platforms.

The first commit of this PR empties the repositories list which will get re-populated when the migration tool runs.
The second commit removes ubuntu focal, adds ubuntu jammy, and adds debian bullseye leaving RHEL8 in place.
It is easier to review each commit separately.

I'd like to get review of this pull request in two phases.

I'd like on review phase to approve the platform list and source ref as I prepare to run the migration tool. This PR won't be merged as-is once approved. Instead it will be used as the foundation for a migration using 7eafe04 as the source ref.

Once the migration is complete, I'll update this PR and request a second phase of reviews for merging the completed migration.


Migration command:

python3 migration-tools/migrate-rosdistro.py --source rolling --dest rolling \
  --source-ref https://github.com/ros/rosdistro/commit/7eafe041ed8058a17f0fcd0a0c70290bf310ea5a \
  --release-org ros2-gbp \

Of 299 repositories, 22 failed to bloom successfully.
This is most frequently caused by missing rosdep dependencies.

The following packages failed to bloom:


Follow-up migration command using tool version at 38741a4 and destination distribution at c066067

python3 migration-tools/migrate-rosdistro.py --source rolling --dest rolling \
  --source-ref 7eafe041ed8058a17f0fcd0a0c70290bf310ea5a \
  --release-org ros2-gbp

@nuclearsandwich nuclearsandwich self-assigned this Feb 3, 2022
@github-actions github-actions bot added the rolling Issue/PR is for the ROS 2 Rolling distribution label Feb 3, 2022
@nuclearsandwich
Copy link
Member Author

Regarding the failing test (complaining about repositories: {} rather than repositories: without the empty hash literal the rosdistro tools don't work properly. I haven't looked closely to see if there's a way to make the test match the required behavior.

@ros-discourse
Copy link

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/holding-releases-to-perform-rolling-transition-to-new-platforms/24191/1

@nuclearsandwich nuclearsandwich force-pushed the prepare-rolling-jammy-transition branch from ae7289e to 68a46ef Compare February 4, 2022 22:31
nuclearsandwich added a commit that referenced this pull request Feb 7, 2022
This release repository was forked into ros2-gbp for the Galactic
migration in April 2021 and has not been released into the upstream
repository since.

Since the [upcoming Rolling platform
migration](#32036) will again
branch the release repository into ros2-gbp I recommend that we update
it pre-emptively to reduce churn in the bloom configuration branches.
nuclearsandwich added a commit that referenced this pull request Feb 7, 2022
These release repositories were forked into ros2-gbp for the Galactic migration in April 2021.
Some packages have not been released into the upstream repository since but some have.

For the packages which have had releases these have been merged into the ros2-gbp branch on 2022-02-07.

Since the [upcoming Rolling platform migration](#32036) will again branch the release repositories into ros2-gbp I recommend that we update them pre-emptively to reduce churn in the bloom configuration branches.
nuclearsandwich added a commit that referenced this pull request Feb 7, 2022
This release repository was forked into ros2-gbp for the Galactic migration in April 2021 and has not been released into the upstream repository since.

Since the [upcoming Rolling platform migration](#32036) will again branch the release repository into ros2-gbp I recommend that we update it pre-emptively to reduce churn in the bloom configuration branches.
nuclearsandwich added a commit that referenced this pull request Feb 7, 2022
This release repository was forked into ros2-gbp for the Galactic migration in April 2021 and has not been released into the upstream repository since.

Since the [upcoming Rolling platform migration](#32036) will again branch the release repository into ros2-gbp I recommend that we update it pre-emptively to reduce churn in the bloom configuration branches.
nuclearsandwich added a commit that referenced this pull request Feb 7, 2022
This release repository was forked into ros2-gbp for the Galactic migration in April 2021 and has not been released into the upstream repository since.

Since the [upcoming Rolling platform migration](#32036) will again branch the release repository into ros2-gbp I recommend that we update it pre-emptively to reduce churn in the bloom configuration branches.
nuclearsandwich added a commit that referenced this pull request Feb 7, 2022
This release repository was forked into ros2-gbp for the Galactic migration in April 2021 and has not been released into the upstream repository since.

Since the [upcoming Rolling platform migration](#32036) will again branch the release repository into ros2-gbp I recommend that we update it pre-emptively to reduce churn in the bloom configuration branches.
nuclearsandwich added a commit that referenced this pull request Feb 7, 2022
This release repository was forked into ros2-gbp for the Galactic migration in April 2021 and has not been released into the upstream repository since.

Since the [upcoming Rolling platform migration](#32036) will again branch the release repository into ros2-gbp I recommend that we update it pre-emptively to reduce churn in the bloom configuration branches.
nuclearsandwich added a commit that referenced this pull request Feb 7, 2022
The release repository was forked into ros2-gbp for the Galactic migration in April 2021.  Since then there have been releases into the upstream release repository.

These releases were merged into the ros2-gbp branch on 2022-02-07.

Since the [upcoming Rolling platform migration](#32036) will again branch the release repositories into ros2-gbp and the ros2-gbp repository is now up-to-date with the external repository I recommend that we update the release repository pre-emptively to reduce churn in the bloom configuration branches.
nuclearsandwich added a commit that referenced this pull request Feb 7, 2022
The release repository was forked into ros2-gbp for the Galactic migration in April 2021.  Since then there have been releases into the upstream release repository.

These releases were merged into the ros2-gbp branch on 2022-02-07.

Since the [upcoming Rolling platform migration](#32036) will again branch the release repositories into ros2-gbp and the ros2-gbp repository is now up-to-date with the external repository I recommend that we update the release repository pre-emptively to reduce churn in the bloom configuration branches.
Using the migration-tool version from 38741a4

Migration command:

```
python3 migration-tools/migrate-rosdistro.py --source rolling --dest rolling \
  --source-ref 7eafe04 \
  --release-org ros2-gbp \
```

These packages did not build (as expected from previous runs)

* aws-robomaker-small-warehouse-world
* control_box_rst
* fmi_adapter
* fmilibrary_vendor
* gazebo_ros2_control
* gazebo_ros_pkgs
* ign_rviz
* ignition_cmake2_vendor
* ignition_math6_vendor
* locator_ros_bridge
* moveit
* ntpd_driver
* ompl
* rmw_connextdds
* rmw_gurumdds
* ros_ign
* rosidl_typesupport_gurumdds
* rtabmap
* sdformat_urdf
* usb_cam
* warehouse_ros_mongo
* webots_ros2
Copy link
Contributor

@clalancette clalancette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just did a spot check to make sure the platforms are what I expected, and that the release repos generally looked good. It all looked good to me, and CI is green now, so approved!

@ralph-lange
Copy link
Contributor

That's a pretty good yield. I looked at the CI failures and it seems that they're failing for packages that aren't in this diff?

@tfoote, @nuclearsandwich, where can I find the CI reports please, to work on the issues with fmilibrary_vendor and locator_ros_bridge? I've setup ROS 2 Rolling on a Jammy VM and could build both repos. Hence the problems seem to occur at the actual bloom release.

@nuclearsandwich
Copy link
Member Author

nuclearsandwich commented Feb 15, 2022

where can I find the CI reports please,

The CI failures that tfoote was referring to were the the checks for this distro file which run the rosdistro cache build process and identified missing release artifacts as a result of #32112

The failures for packages listed above were all failures during the bloom release process for the migration tool, which runs non-interactively.

fmilibrary_vendor failed because of the ':{ask}' in the release_tag field

Interactivity is disabled but release_tag is set to :{ask}

locator_ros_bridge failed because libpoco-dev is not available for RHEL 8. During an interactive bloom process there would be a prompt to proceed anyway but because the migration tool runs non-interactively it fails.

Running bloom-release manually for each package should be enough to resolve the bloom issues and get them back into the distribution once we lift the current migration hold](https://discourse.ros.org/t/holding-releases-to-perform-rolling-transition-to-new-platforms/24191). fmilibrary_vendor could be adjusted so that interactivity is no longer required for migration releases. I would have to look more closely at the git-bloom-release interface to determine whether I could make failing on 'non-critical' platforms non-blocking of the complete release.

That being said, if you'd prefer to leave things as they are and make manual bloom-releases after subsequent migrations (such as the creation of ROS 2 Humble in a couple of months) I think that's also fine.

@nuclearsandwich
Copy link
Member Author

Since I was canny enough to backup every ros2-gbp repository locally before running this migration. I had snapshots of the Rolling release branches prior to migration which I could use to find patches which had been clobbered by #32128

I used the following script to identify release branches with non-standard bloom commits and ran it over each repository in my backup to produce a TODO list of repositories that require attention. As of now each repository has been seen to and most have had their patches restored where necessary. There are a few packages where I am waiting for the maintainer's approval before I perform a non-maintainer release.

#!/usr/bin/env ruby

require "ostruct"
require "optparse"

BLOOM_DEFAULT_COMMIT_MESSAGES = [
  %r{Created orphaned branch \w+},
  %r{Imported Upstream version [\.\d]+},
  %r{Imported upstream version '[\.\d]+' of '\w+'},
  %r{Imported upstream version '[\.\d]+' from '[^']+'},
  %r{Rebase from '[\w+/]+'},
  %r{Placing (debian|rpm) template files},
  %r{Trimmed the branch to only the [\w/]+ sub directory},
  %r{Generated (debian|RPM) files for \w+},
  %r{Generated RPM spec file for \w+},
  %r{Renamed RPM spec file for \w+},
  %r{Overlaid patches from '\w+'},
  %r{Revert "Trimmed},
]

BRANCH_BASES = %w[debian release rpm]
options = OpenStruct.new
OptionParser.new do |opts|
  opts.on("-rROSDISTRO", "--rosdistro ROSDISTRO") do |rosdistro|
    options.rosdistro = rosdistro
  end

  opts.on("-bBASE", "--branch-base BASE") do |branch_base|
    if branch_base == "ALL"
      options.branch_base = BRANCH_BASES
    elsif not BRANCH_BASES.include? branch_base
      raise ArgumentError, "#{branch_base} must be one of: #{BRANCH_BASES.join(" ")} or ALL"
    else
      options.branch_base = [branch_base]
    end
  end
end.parse!


if options.rosdistro.nil?
  options.rosdistro = "rolling"
end

if options.branch_base.nil?
  options.branch_base = BRANCH_BASES
end

refs_to_check = Array.new
options.branch_base.each do |base|
  refs_to_check += `git for-each-ref --format='%(refname)' 'refs/heads/#{base}/#{options.rosdistro}'`.lines.map(&:strip)
end

reponame = File.basename Dir.pwd

refs_to_check.each do |ref|
  non_default = false
  `git log --pretty=%s #{ref}`.lines.map(&:strip).each do |commit_message|
    unless BLOOM_DEFAULT_COMMIT_MESSAGES.any? { |r| r.match commit_message }
      STDERR.puts commit_message if ENV['DEBUG']
      non_default = true
      puts "#{reponame}\tCommits on #{ref} are non-default!"
      break
    end
  end
  next if non_default
end

ament_cmake-release.git

@ros-discourse
Copy link

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/how-to-build-full-ros2-deb-packages-for-debian-distribution-which-are-built-from-source-code/35969/5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
rolling Issue/PR is for the ROS 2 Rolling distribution
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants