sros2 package Quality level status

[mikaelarguedas]

Discussed in the working group meeting, this is for tracking the state of sros2 with respect to Quality Levels defined in REP-2004

An example of quality declaration document for a Python package https://github.com/ament/ament_index/blob/master/ament_index_python/QUALITY_DECLARATION.md

We will likely need to do a second pass taking into account what is in the Developer Guide as it states more precisely what policies are applied for some of these items

1 Version Policy:

• 1.i Must have a version policy (e.g. semver)
• 1.ii Must be at a stable version (e.g. for semver that means version >= 1.0.0)
• 1.iii Must have a strictly declared public API
  • marking this one as checked as there are currently only non-public modules in sros2. Although we don't use "impl" or "detail" namespaces as suggested in the document and don't explicitly export any API in the __init__.py
• 1.iv Must have a policy for API stability
• 1.v Must have a policy for ABI stability
• 1.vi Must have a policy that keeps API and ABI stability within a released ROS distribution

2 Change Control Process:

• 2.i Must have all code changes occur through a change request (e.g. pull request, merge request, etc.)
• 2.ii Must have confirmation of contributor origin (e.g. DCO, CLA, etc.)
• 2.iii Must have peer review policy for all change requests (e.g. require one or more reviewers)
• 2.iv Must have Continuous Integration (CI) policy for all change requests
• 2.v Must have documentation policy for all change requests

3 Documentation:

• 3.i Must have documentation for each "feature" (e.g. for rclcpp: create a node, publish a message, spin, etc.)

• 3.ii Must have documentation for each item in the public API (e.g. functions, classes, etc.)

• 3.iii Must have a declared license or set of licenses

• 3.iv Must state copyrights within the project and attribute all authors

• 3.v Must have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements

  • 3.v.a Must have a section in the repository's README which contains the "quality declaration" or links to it
  • 3.v.b Should register with a centralized list of 'Level N' packages, if one exists, to allow for peer review of the claim (counting REP-2005 as such list)
  • 3.v.c Must reference any 'Level N' lists the package belongs to, and/or any other peer review processes undergone

4 Testing:

• 4.i Must have system tests which cover all items in the "feature" documentation

• 4.ii Must have system, integration, and/or unit tests which cover all of the public API

• 4.iii Code Coverage:

  • 4.iii.a Must have code coverage tracking for the package
  • 4.iii.b Must have and enforce a code coverage policy for new changes
  • not required but stating something like "above 95%" as a minimum would be good too (to match developer guide)

• 4.iv Performance:

  • TBD, not sure it makes sense for a non-runtime cli to try to comply with the performance section
  • 4.iv.a Must have performance tests (exceptions allowed if they don't make sense to have)
  • 4.iv.b Must have a performance regression policy (i.e. blocking either changes or releases on unexpected performance regressions)

• 4.v Linters and Static Analysis:

  • 4.v.a Must have a code style and enforce it
  • 4.v.b Must use static analysis tools where applicable
  • although we could integrate a formatter and or tighter analysis like enforcing mypy or else

5 Dependencies:

• 5.i Must not have direct runtime "ROS" dependencies which are not at the same level as the package in question ('Level N'), but...
  • ros2cli does not declare quality levels so this one is blocked at the moment
• 5.ii May have optional direct runtime "ROS" dependencies which are not 'Level N', e.g. tracing or debugging features that can be disabled
• 5.iii Must have justification for why each direct runtime "non-ROS" dependency is equivalent to a 'Level N' package in terms of quality

6 Platform Support:

• Must support all target platforms for the package's ecosystem.
  For ROS 2 this means supporting all tier 1 platforms, as defined in REP-2000

7 Security

• Must have a declared Vulnerability Disclosure Policy and adhere to a response schedule for addressing security vulnerabilities

[mikaelarguedas]

This is a visual aid to see where we stand compared to quality levels.
This is just to visually represent the list in the first comment. List in first comment is the source of truth

	SROS2	Level 1	Level 2	Level 3	Level 4
1.i	✔️	x	x	x	o
1.ii		x	x	x
1.iii	✔️	x	x	o
1.iv	✔️	x	x	x
1.v	✔️	x	x	x
1.vi	✔️	x	x	o
2.i	✔️	x	x	x	o
2.ii	✔️	x	x
2.iii	✔️	x
2.iv	✔️	x	x	x
2.v		x
3.i		x	x
3.ii		x
3.iii	✔️	x	x	x	x
3.iv	✔️	x	x	x	x
3.v		x	x
3.v.a		x	x	x
3.v.b	✔️	o	o
3.v.c		x	x	x
4.i		x	x	o	o
4.ii		x
4.iii.a	✔️	x	x
4.iii.b	✔️	x
4.iv.a		x
4.iv.b		x
4.v.a	✔️	x	x
4.v.b	✔️	x	x
5.i		x	x
5.ii
5.iii		x	x
6.i	✔️	x	x	x	o
7.i	✔️	x	x	o

[ros-discourse]

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-package-documentation/14569/1

[ros-discourse]

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/quality-levels-for-ros2cli-and-rclpy/14573/1

[kyrofa]

There are currently no plans for either ros2cli or rclpy to implement REP 2004. As a result, while this list gives us insight into areas we can and should definitely improve, I don't think we'll be able to claim quality level 2.

[mikaelarguedas]

This is unfortunate, while we could have made a case that rclpy should not be a hard dependency for sros2, ros2cli definitely is a core foundation for this package.
As you saids this will prevent from claiming quality level 2 or lower.

We could start by making sure we fulfill all level 3 requirements and move on to have all the boxes ticks for level 2 but the dependencies. We can then see what the situation is for the dependencies at that time. WDYT?

[dirk-thomas]

while we could have made a case that rclpy should not be a hard dependency for sros2, ros2cli definitely is a core foundation for this package.

rclpy is a dependency of ros2cli so you will recursively anyway depend on both.

[ros-discourse]

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-2020-11-19/17570/1