Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[20998] Record data in SQL #144

Open
wants to merge 52 commits into
base: main
Choose a base branch
from
Open

[20998] Record data in SQL #144

wants to merge 52 commits into from

Conversation

Tempate
Copy link
Contributor

@Tempate Tempate commented May 10, 2024

Code improvements:

  • Make a Serializer to serialize and deserialize TopicQoS, TypeObjects, TypeIdentifiers, and DynamicTypes.
  • Make a time_utils to convert between different time formats.
  • Make the DdsRecorder and the DdsReplayer agnostic of the output format.
  • Refactor McapLogErrorTest to improve its readability.

Configuration improvements:

  • Move cleanup-period and max-pending-samples from specs to recorder.
  • Move safety-margin to resource-limits.
  • Configure safety-margin as a string (100MB, 128GiB, etc).

Bug fixes:

  • Avoid creating a payload when replaying a message in a topic without a reader.
  • Fix assert in ROS 2 topics.

Potential TODOs:

  • Allow to record both MCAP and SQL at the same time.
  • Separate data storage calls to dedicated thread

@Tempate Tempate changed the base branch from main to stabilise/file_rotation May 10, 2024 09:10
@Tempate Tempate force-pushed the stabilise/file_rotation branch from dcc9f62 to 53d0e2a Compare May 13, 2024 07:30
@Tempate Tempate force-pushed the stabilise/file_rotation branch 2 times, most recently from ab73b8d to a9c882c Compare May 17, 2024 09:26
@Tempate Tempate changed the title Record data in SQL [20998] Record data in SQL Jun 5, 2024
@Tempate Tempate force-pushed the feature/sql branch 4 times, most recently from aca9425 to 8deea2e Compare June 13, 2024 10:50
Base automatically changed from stabilise/file_rotation to main June 17, 2024 05:35
Tempate added 22 commits July 16, 2024 12:49
Signed-off-by: tempate <[email protected]>
Signed-off-by: tempate <[email protected]>
Signed-off-by: tempate <[email protected]>
Signed-off-by: tempate <[email protected]>
Signed-off-by: tempate <[email protected]>
Signed-off-by: tempate <[email protected]>
@Tempate Tempate changed the base branch from main to integration/bump-3.0.x July 19, 2024 06:24
Signed-off-by: tempate <[email protected]>

// Open a new file to write the remaining data.
// Throw an exception if the file cannot be opened.
open_new_file_nts_(min_file_size + e.data_size_to_write());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This callback will be invoked during both MCAP and SQL recordings if disk is full. I believe it should have distinct implementations in McapWriter and SqlWriter. In the case of SQL, it is unnecessary to open a new file when the disk is full, as you can simply delete the beginning of the file.

Comment on lines +66 to +67
void deserialize(
const fastdds::dds::DynamicType::_ref_type& dynamic_type);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This deserialize should be included in the Serializator class

Base automatically changed from integration/bump-3.0.x to main September 17, 2024 08:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants