Record service requests

[kunaltyagi]

Description

In ROS1, recording services was not implemented. However, actions were implemented on top of messages.

This made it possible to debug/enhance/test a system by recording just messages and replaying them. In ROS2, server client handshake is done over services. Would it be possible to implement a rosbag record of the sent goals?

Related Issues

ROS1: ros/ros_comm#250

Completion Criteria

Successful trigger of action by following this sequence:

1. ros2 run action_tutorials_py fibonacci_action_server
2. ros2 bag record -a --include-hidden # or any other flag
3. ros2 run action_tutorials_py fibonacci_action_client
4. Close all nodes
5. ros2 run action_tutorials_py fibonacci_action_server
6. ros2 bag play rosbag2_
7. Wait to see if server responds

This sequence with ROS1 works if the goal messages are recorded. If this also works with ROS2, we can consider this feature as "ready"

Implementation Notes / Suggestions

On the interface, we can have 3 different behaviors, all of which make sense from a certain perspective:

1. ros2 bag record --include-actions or ros2 bag record --include-services to allow replaying of service requests (most sensible, keeps concerns separate)
2. Make it default behavior to record action goal/cancellations but not services (same behavior from ROS1)
3. Extend the existing ros2 bag record --include-hidden-topics (better if not done this way)

Since the services and topics are implemented similarly at a RMW level, it should be possible to record the data sent via services.

On the other hand, just because all current RMW implementations use the same mechanism for topic and service communication, this might not hold true in future.

Testing Notes / Suggestions

The integrated testing for this would be quite simple:

1. start service/action server
2. start recording
3. call service/action
4. stop recording and play the bag
5. expect a call for the service/action

I don't know how it'd be unit tested

[emersonknapp]

Given that this is a feature request - I would appreciate if you could reformat using the feature request template instead of the bug report template - it will be easier to understand.

[Karsten1987]

@kunaltyagi while it's a valid feature request, it actually goes beyond the scope of rosbag2. For this to happen, we actually need support in ROS2 for serialized service (and action) calls. Unfortunately, up to today, we only have serialized messages.

[emersonknapp]

Are ROS 2 Services implemented under the hood using Topics? If so, these don't appear in the ros2 topic list --include-hidden-topics output, so they're "very hidden"

[clalancette]

Are ROS 2 Services implemented under the hood using Topics?

Correct, assuming a DDS RMW. All three of our in-tree RMWs use a pair of topics (prefixed with rq and rr, respectively) for the call and response side of the service.

If so, these don't appear in the ros2 topic list --include-hidden-topics output, so they're "very hidden"

Yes, ros2 topic doesn't even consider them as part of the message network because they don't start with the correct prefix (rt/).

That said, I don't think that any of this is a requirement imposed by ROS 2, more of a convention. The DDS RMWs all do it since the DDS spec does not have a definition for an RPC-like call (as far as I understand). You can imagine a different RMW that does have support for true RPC doing something different.

[kunaltyagi]

@emersonknapp Updated the format of issue.

For this to happen, we actually need support in ROS2 for serialized service (and action) calls

Aren't services and actions a superset of the message format? If a message can be (de)serialized, why is doing the same with service request/response and actions an issue (provided they are visible)?

Is this a constraint?

@clalancette Are the prefixes exposed by some mechanism? Or should it be requested at a rclcpp/rclpy level?

[emersonknapp]

Aren't services and actions a superset of the message format?

Yes - but the definition format and the network mechanism / format are not necessarily the same thing.

The more important point, I think is, per @clalancette comment - Services are not necessarily a Topic under the hood by definition of the ROS 2 specification. Though, in the case of all DDS RMW implementations, they are implemented using DDS topics, and could theoretically then be exposed in a way that DDS could record. This might be worth figuring out - given that we understand that we couldn't guarantee the mechanism would work for all possible RMW implementations.

[Karsten1987]

well, DDS topics yes or no but I think this doesn't really solve the task at hand here. The functionality has to be exposed through an RMW interface and at this point it's RMW agnostic. We can't just have a big "if DDS, then record services by hacking topics" within rosbag2 :D

What @clalancette said is certainly correct, but I believe there's more to it. On the ROS2 core stack we prefix all services with rq/rr, but even the actual DDS implementation is free to do whatever with the topic. In the case of RTI connext, I believe they added a magic number (gid) to the response topic to uniquely identify a service server and the respective client.

[Barry-Xu-2018]

I am considering the requirement of recorded data and the feasibility of realization.
From the perspective of use case,

Service/Client

• Record data for debugging service side
  I think this feature request can cover this use case.

• Record data for debugging client side
  Impossible to implement (from my thought)

Action

• Record data for debugging action service side
  Record sent goal.
  I think the modification for this feature can be updated to support this.
  But I don't sure whether anyone wants to get this data in practice?
  Any comments ? @emersonknapp @clalancette @Karsten1987 @kunaltyagi

• Record data for debugging action client side
  Impossible to implement (from my thought)

[fujitatomoya]

being off possibility or feasibility, we would want to support the following use cases? (at least i would do as application engineer.)

• service server development with recorded request. probably it would be nice to load the recorded request bag file to ros2 action and then we can see the status, feedback and response. (same use case can be applied to service server.)
• all recorded data of service/action client/server will be still useful to view the transmission activities later. i believe this is really useful to debug distributed system to see what they are actually transmitting when the problem observed.

[kunaltyagi]

If this feature isn't supported out-of-the-box, the community would have to create message based services and actions. Wouldn't that kind of defeat the purpose of the design process of ROS2? The community based solutions might not have the same level of guarantees that the core ros2 libraries come with.

This is a very important feature for systems actually in deployment, and I can't stress this enough: debugging, replay, simulation, etc. use this very heavily. In fact, some people and entities I know have stopped using services and only use messages to ensure the presence of record and replay features

There is neither an alternative nor steps taken to mitigate this and in absence of these, it doesn't motivate the effort required tor migrating to ros2 (assuming creating custom srv and action interface is more complicated than staying on ros1)

[Barry-Xu-2018]

I think we need to consider how to record service/action beyond the limitations of the current implementation of rosbag2.
e.g.
rosbag2 will record all communication message between service/action service and client/action client.
And then user can use tool like wireshark to check the content of messages at each record time point , which may come from the client or from the server.

This needs to further consider.

[jacobperron]

I think having a mechanism to introspect services makes sense to support as a core feature in ROS 2, i.e. having a way to see what requests and responses being sent. Since the implementation of services are specific to the RMW, I think we should add support for service introspection at the rcl layer (or above). This means not relying on the RMW to provide topics for requests and responses (or some other mechanism).

After discussing this with some people offline, here are some ideas for what we could do:

1. Add a feature to services to publish events related to requests and responses
   • Services would publish an event whenever they receive a request or send a response.
   • This could be on a /service_events topic (perhaps namespaced by the service name).
2. Optionally, clients could also publish events for when they send requests and receive responses.
3. Since it's more useful to be able to introspect the content of a service request or response, we might want to generate custom event messages based on the service IDL (.srv file).
   • So, instead of just publishing an event with (service name, stamp, ID), it would also contain the request/response data as well.
   • Note, this probably requires us to have separate service event topics for each request and response type, since we do not support multiple message types per topic.
4. Offer an option (parameter or environment variable) to toggle this feature on/off per node.
5. Provide an implementation of a node that subscribes to service request events and forwards them as service client calls.
   • This satisfies the use-case of playing back services from a rosbag (e.g. the request event is replayed and then a separate node actually makes the service request).
   • Optionally, we could do the same for service responses, though this seems less useful to me.

I think we can make a design proposal in the form of a PR (either on design.ros2.org or under concepts?).

cc/ @ihasdapie @deepanshubansal01 @adityapande-1995 @wjwwood

[Alqio]

Is there any updates on this? What is the current recommended way for recording & playing back service calls? Change them to topics?

[ihasdapie]

@Alqio Current progress is being tracked on ros2/ros2#1285 & more information can be found ros-infrastructure/rep#360. I was hoping to iterate on it more (but good progress has been made without me) over the last few weeks but have been busier than expected...but in any case it should be included in next year's Iron release :)

[fujitatomoya]

@ihasdapie are you already working on this? or anybody else?

CC: @MichaelOrlov

[emersonknapp]

As far as I know, nobody is working on this. However, much service introspection features went into the Iron release, so some more features may be possible to implement now. This would be a good topic for the Rosbag2 Working Group meeting. I don't think I have any development time to dedicate to the project, but I'd be happy to discuss design.

[ihasdapie]

I was originally planning on helping wrap up the service introspection work last september but school ended up being a lot more work than anticipated. I may have some bandwidth to help out a bit in a couple of weeks in my spare time, but probably not anything significant like last year.

There is an old PR (#1063) up for adding syntactic sugar for recording service events but it likely needs to be updated since there appears to have been a number of changes to the feature since I was last active. Playback is a whole different discussion and in my opinion there is no good answer to the problem and instead some sensible default that isn't "we won't support this" should be converged upon. I'm not sure if that discussion should take place here or in REP2012.

I think that there is value in offering a playback option that sends requests or responses as recorded, despite the problem of "does replaying service responses and client requests at the same timing as a prior run really make sense?". Just because it would gives the most utility and options to the user. Anything more complex gets, well, more complex really fast and may be better resolved by the user writing up something custom for their specific use case. Or we will hear about it in the issue boards and can resolve it later :)

[Barry-Xu-2018]

@emersonknapp
I'm interested in implementing this feature. First, I will my design and hear your and others opinions.

@ihasdapie
I will reference your PR and share my thoughts. Please share your opinions.

[Barry-Xu-2018]

Rough Design

How to save record service message

Record service event topic (e.g. /add_two_ints/_service_event) is recorded. So the recording process is the same as for regular topics.

How to distinguish service event topic and hidden topic.

While reading recorded topic information, topic for service should have different process from other topics. So it's important to distinguish between them.
The format of service event topic is [/NameSpace]/ServiceName/_service_event. So It becomes easy to distinguish whether it's a service event topic or a hidden topic.

Expand the 'record' command

-S [SerivceName1 ServiceName2 ...] or --services [SerivceName1 ServiceName2 ...]. If the user doesn't specify service names, all services are recorded.

Change output of 'Info' command

Topic for service should be listed distinctively, as below example.

Files:             123_0.mcap
Bag size:          28.2 KiB
Storage id:        mcap
Duration:          15.59s
Start:             May 19 2023 13:22:25.340 (1684473745.340)
End:               May 19 2023 13:22:40.400 (1684473760.400)
Messages:          75
Topic information: Topic: /chatter | Type: std_msgs/msg/String | Count: 15 | Serialization Format: cdr
                   Topic: /chatter | Type: std_msgs/msg/String | Count: 16 | Serialization Format: cdr
                   Topic: /events/write_split | Type: rosbag2_interfaces/msg/WriteSplitEvent | Count: 0 | Serialization Format: cdr
                   Topic: /parameter_events | Type: rcl_interfaces/msg/ParameterEvent | Count: 0 | Serialization Format: cdr
                   Topic: /rosout | Type: rcl_interfaces/msg/Log | Count: 44 | Serialization Format: cdr

>>>> The below is new added items for service  <<<<<<<
Service Events: XX  
Service information: Service: /xxx/xxx | Type: xxx/xxx/xxx | Event Count: XX | Serialization Format: XX
                     Service: /xxx/xxx | Type: xxx/xxx/xxx | Event Count: XX | Serialization Format: XX

Expand the 'play' command

Add 2 parameters.

• -S [ServiceName1 ...] or --services [ServiceName1 ...]

  Replay service event. You can use ros2 service echo to check playing service events. If there is no service name set, all recorded service events are played. User can use ros2 service echo ServiceName to check different service events. Not sure whether this is needed.

• --services-requests [ServiceName1 ...]

  Replay request with recorded sequence and interval. If there is no service name set, all recorded service request are sent.

[emersonknapp]

@Barry-Xu-2018 thanks for volunteering! I have a request - could you open a PR with a document in the https://github.com/ros2/rosbag2/blob/rolling/docs/design/ directory so that we can make comments on it? It does not need to be very long or detailed, the above looks good - it just provides us a place to give feedback and make revisions. An issue comment is not an easy way to do that.

[Barry-Xu-2018]

@emersonknapp

I have a request - could you open a PR with a document in the https://github.com/ros2/rosbag2/blob/rolling/docs/design/ directory so that we can make comments on it? It does not need to be very long or detailed, the above looks good - it just provides us a place to give feedback and make revisions. An issue comment is not an easy way to do that.

Okay. I will create a PR in https://github.com/ros2/rosbag2/blob/rolling/docs/design/.

[Barry-Xu-2018]

I created a PR #1359

Let's discuss the design in this PR. @emersonknapp @ihasdapie

[MichaelOrlov]

• Closing as done in the Implement service recording and display info about recorded services #1480