[Feature] Generating scene pointcloud from meshes

[pvskand]

The gen_scene_pcd function, which generates the point cloud by sampling points from the meshes of the rigid bodies was something that was implemented in MS2 but for MS3 it's marked as a TODO.

I've already implemented it for MS3 for my research (although I use a slightly different variant) and wanted to see a PR for the same would be of interest.

cc: @StoneT2000 as we briefly talked about it on discord.

[StoneT2000]

A PR would be greatly appreciated! Thank you so much

[pvskand]

@StoneT2000 I was able to generate scene point clouds (see L1327-L1365). I wasn't aware that my previous code was only working for a singe environment i.e when num_envs=1, so I ended up fixing that for it to work for num_envs>1. See the attached rollout video of 4 environments on PickCube task with a random policy.

Current drawbacks:

1. Doesn't work for multi-robot tasks: TwoRobotPickCube-v1 and TwoRobotStackCube-v1. This is because I'm using assuming self.robot_uids is a str and not a List. (See L1339). This should be an relatively easy fix which I'll address tomorrow.

2. I am unable to get the color for geoms. So right now the colors are defaulted to Trimesh's default colors (while creating a mesh). I'm not sure how can I get the colors for the geoms in get_component_meshes (https://github.com/pvskand/ManiSkill/blob/main/mani_skill/utils/geometry/trimesh_utils.py#L12) which is called at L1335 and L1350. I couldn't find any clues on how to move forward while going through the Sapien documentation -- any leads on this end would be helpful.

PickCube.mov

[fbxiang]

For 2. I have something you may find useful for sampling colors and visual materials. It is not complete but may give you an idea on how to implement this.

[pvskand]

Thanks @fbxiang -- that would be super helpful if you can point me to the code if it's on github or if you need me to reach out to you via email or the MS discord -- I can do that as well.

[fbxiang]

Sorry somehow my link went missing.
Here it is https://github.com/fbxiang/sapien-sample-points