integrate with LeRobot #345
Comments
|
Hi there, I'm happy spending some cycles in exploring and potentially contributing to this. I've got some experience with LeRobot (participated in their last hackathon last Oct and ported it to a new robot as part of the effort which I contributed back) which should help in this endeavour.
For now, I'll take on the first task, which is in-line with my research interests. I can't commit to the other two now but I'm happy to see how far my cycles can take me. Question though, what do you mean by:
Don't quite get the meaning behind it. I may have some questions @boczekbartek concerning the rai-manipulation-demo. I guess the discord server of the Embodied AI WG is the best place for discussion? |
|
@vmayoral
( I adjusted the wording a bit in the description to make it more clear.) My idea for this was to use the pipeline of the current rai-manipulation-demo (with LLM tools that are based on moveit and groundedSAM) to generate trajectories for imitation learning in LeRobot. From training on these trajectories we could get more robust policies. Also, if you have suggestions for the trajectory for this issue please share them. We can figure out together what makes the most sense. As for the rai-manipulation-demo (and basically other questions about rai / llm agents ) - yes, please write us on Discord. Also for questions specifically about the demo you could also use this issue: #292. We'll be happy to help! |
Thanks, it does now. Looking to engage into all of this later this week. Will report back here. |
|
Hi all, Thought it took a bit longer than I was expecting initially, I was positively hoping to report a small success integrating a LeRobot policy into RAI. Unfortunately this turned frustrating Altogether, I have not been able to integrate LeRobot policies into RAI ❌ because honestly, I have not been able to get RAI to operate. No blaming, just reporting my personal experience here and hoping to retry again in the future if time allows for it. Here's my experience: Setting up the policy and tools for "integration"I built a small policy using With this operational, I tested this outside of RAI and ROS 2. A simple Python script can trigger this policy by leveraging the Integrating the policy into
|
|
Hi @vmayoral, Thank you for sharing your detailed experience and for taking the time to document the issues you encountered. I understand that you might have had Conda sourced or used it instead of Poetry. Unfortunately, we don’t support Conda and haven’t tested RAI in that environment. The framework is officially supported and tested with Poetry, and the issues you’ve described are likely linked to incompatibilities introduced by having Conda sourced during the setup. The only issue I can directly relate to is the empy error, which can occur when building the repository for the first time in a custom Python environment. This is usually caused by package overshadowing, and we resolve it as outlined here: RAI Discussion #150. To ensure that the installation process works as expected, we have continuous integration (CI) workflows in place. These CI pipelines are based on clean humble and jazzy OSRF Docker images. They test the entire installation process, verify that all packages can be imported correctly, and run a portion of RAI to confirm its functionality. This ensures that the framework is operable in the intended environment. We’ve also received feedback from the community that similar issues have occurred, and the problems were not related to the Python environment itself but to other processes such as running demos, which we are fixing on on going basis. However, having Conda sourced can introduce subtle conflicts. To clarify, did you have a Conda environment active during your setup, even if you didn’t explicitly use Conda for the installation? Regarding the integration with LeRobot, we deeply value the potential it brings to RAI, and your insights are incredibly helpful. We would love to collaborate further to address these challenges and potentially take on the work of adapting our environment to be compatible with LeRobot on our side. This integration is highly aligned with our vision, and your contribution would be instrumental in shaping how we move forward. Once again, we’re sorry for the time and effort the troubleshooting process cost you, but we greatly appreciate your patience and detailed feedback. Let us know how we can best support you moving forward. Thank you again for your efforts and kind words about RAI. We look forward to hearing back from you! PS. I was able to run |
|
@maciejmajek thanks for the prompt response! As you hint, I indeed had a To my understanding, the whole issue boils down to a python compatibility issue, which I have not been able to solve.
Is it possible to disclose such Dockerfile (or a simplified version of it) that helps others (e.g. me) reproduce the process of getting RAI up to speed? |
I see...
Yes, I will provide the Dockerfile first thing tomorrow and possibly host the built image later on. |
|
Using the Humble Docker image from #377, I was able to execute the following: # /rai
poetry shell
git clone lerobot
cd lerobot
pip install -e .
pytest # Run lerobot tests
cd ../
pytest tests # Run rai testsAlthough the Test results for ============================== 161 passed, 150 skipped, 5 warnings in 165.17s (0:02:45) ==============================Test results for =============== 1 failed, 29 passed, 1 skipped, 6 deselected, 1 xpassed, 2 warnings, 1 error in 13.65s =============== |
|
Hi @vmayoral, we have made some progress with docker setup. We tested it a bit and it seems promising. #376 Also, fyi @boczekbartek tried running RAI solely on Conda, but due to conda's nature it turned out to be impossible :( |
|
That's true, I tried Conda setup and failed. The problem seem to lay in the
For point 5:
I'm afraid to integrate |
Is your feature request related to a problem? Please describe.
Currently rai-manipulation-demo uses ros2 moveit and a fusion of segmentation with depth camera to control Panda robotic arm.
This approach is good to for reasoning and planning, but might be limited when it comes to more complex movements of the gripper. For such task, policies implemented in LeRobot can be more useful.
LeRobot currently offers:
Describe the solution you'd like
I find the integration of all LeRobot features mentioned above useful in
rai, but I think a step-by-step should be applied.Describe alternatives you've considered
Additional context
The text was updated successfully, but these errors were encountered: