Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added depth frame support for dataset recording/loading #455

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

lyshuga
Copy link

@lyshuga lyshuga commented Sep 27, 2024

What this does

Add depth frame support for Intel RealSense cameras in ALOHA 2 setup

This PR introduces the following changes:

  1. Recording:

    • Implemented depth frame recording from Intel RealSense cameras
    • Depth frames are saved as .png images, preserving absolute values in uint16 format
  2. Loading:

    • Added depth frame loading functionality to LeRobotDataset class
    • Implemented in a similar manner to image frame loading

(Optional) 3. Encoding attempts:

  • Explored options for encoding depth frames into video format like image frames
  • Currently, .png images are used as no suitable codec was found to preserve absolute depth values without normalization
  • .raw format video was successful, but resulted in larger file sizes compared to original .png images

How it was tested

  • With ALOHA 2 setup, recorded and replayed successfully a dataset .

How to checkout & try? (for the reviewer)

In ALOHA config (lerobot/configs/robot/aloha.yaml) for every camera, you want to record depth from, set the "use_depth" parameter to "true", like in this example:

cam_right_wrist:
  _target_: lerobot.common.robot_devices.cameras.intelrealsense.IntelRealSenseCamera
  camera_index: 130322272631
  fps: 30
  width: 640
  height: 480
  **use_depth: false** #set "true" if you want to use the depth camera

And then run the recording:

python lerobot/scripts/control_robot.py record \
   --robot-path lerobot/configs/robot/aloha.yaml \
   --fps 30 \
   --root data \
   --repo-id a/aloha_depth_test \
   --push-to-hub 0 \
   --tags tutorial \
   --warmup-time-s 5 \
   --episode-time-s 30 \
   --reset-time-s 10 \
   --num-episodes 2

@lyshuga lyshuga marked this pull request as draft September 27, 2024 12:22
@lyshuga lyshuga marked this pull request as ready for review September 27, 2024 12:22
@Cadene Cadene self-requested a review September 30, 2024 12:10
@Cadene Cadene self-assigned this Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants