Some more detailed questions on scene edit

[xmskk]

Hi! I was looking into the codebase and had some questions about scene editing. I know the topic was already discussed in #4 but I had some more specific questions.

1. How to move, turn vehicles in a realistic manner?

Right now, I'm calling the checkpoint_final.pth file and editing the instances_quats and instances_trans of the rigid model. However, this is a highly manual process and I have to edit the values frame by frame. Most importantly, it is difficult to figure out what values to give for the vehicle to move more realistically. For instance, if I want to have a vehicle that was going straight to make a left turn, is there a relatively simpler way of implementing it other than trial and errors?

2. How to add another vehicle

This one I'm completely lost on. Again, as mentioned in #4 , I get that there are a lot of other attributes that I need to add in, and I've tried some ways by copying the existing rigid node instance but having trouble. Could you give me an idea of what functions I should be looking at?

3. Representation of rotating vehicles

Conceptually, as I am using the Waymo data, I really only have information on how the vehicle looks like on the side that is captured in the cameras right? Whenever I rotate a vehicle that was never rotated in the original footage, I get a blurry mess. I just wanted to know if this is normal behavior.

3-1. Adding in a complete vehicle GS to the representation

If we had the footage of a vehicle from various angles that allowed us to rotate it however we want, we can add it into whatever scene we want just like adding in a 3D model in traditional 3D rendering, right? (I believe this was mentioned in the paper as well)

Sorry for the long questions, I'm still trying to learn the whole gaussian splatting so some of the questions might seem a little dumb. But would really appreciate it if you could help me out a little.

[zzz5y]

I am also confused about these questions and looking for some advices!

[zzz5y]

hello,May I have some advice or comment on rotating vehicles
I try to rotate vehicles by edit the Rigid by matmul(instance_pose,rotation_matrix)
but it doesn't work,Thanks for your time!

[xmskk]

As I mentioned in the original question, I create a trainer by resuming from the checkpoint file. Then I get the rigid nodes by calling trainer.models["RigidNodes"]. Once I have the rigid nodes I can iterate through them to get their trans and quats (I'm unsure as to how I can specify which vehicle in a frame has which id). Rotation works by doing a quaternion multiplication to the quats. Then save the edited trans and quats back to the rigid nodes and save the checkpoint file.

[zzz5y]

As I mentioned in the original question, I create a trainer by resuming from the checkpoint file. Then I get the rigid nodes by calling trainer.models["RigidNodes"]. Once I have the rigid nodes I can iterate through them to get their trans and quats (I'm unsure as to how I can specify which vehicle in a frame has which id). Rotation works by doing a quaternion multiplication to the quats. Then save the edited trans and quats back to the rigid nodes and save the checkpoint file.

thank you for your reply and your time! thank you and i wil try this way to rotate!Thanks a lot

[zzz5y]

I meet the same trouble like you when i try to adding vehicle.Do you have and ideas?
Thank you so much!
here my add function

def add_instances(self, copy_item_id:int,model_instance_trans,model_instance_quats) -> None:
        mask = (self.point_ids[..., 0] == copy_item_id)

        copy_item_gs_dict=self.collect_gaussians_from_ids([copy_item_id])
        gs_dict=copy_item_gs_dict[copy_item_id]

        instances_trans = model_instance_trans
        instances_quats = model_instance_quats

        clone_point_ids=self.point_ids[mask]


        self._means = Parameter(torch.cat([self._means, gs_dict["_means"]], dim=0))
        self._scales = Parameter(torch.cat([self._scales, gs_dict["_scales"]], dim=0))
        self._quats = Parameter(torch.cat([self._quats, gs_dict["_quats"]], dim=0))
        self.point_ids = torch.cat([self.point_ids, clone_point_ids], dim=0)

        self.instances_fv = torch.cat([self.instances_fv, gs_dict["instances_fv"]], dim=1)


        self.instances_trans=Parameter(torch.cat([self.instances_trans, instances_trans.unsqueeze(1)], dim=1))
        self.instances_quats=Parameter(torch.cat([self.instances_quats, instances_quats], dim=1))

        self._features_dc = Parameter(torch.cat([self._features_dc,gs_dict["_features_dc"]], dim=0))
        self._features_rest = Parameter(torch.cat([self._features_rest, gs_dict["_features_rest"]], dim=0))

        self._opacities = Parameter(torch.cat([self._opacities, gs_dict["_opacities"]], dim=0))

[zzz5y]

I meet the same trouble like you when i try to adding vehicle.Do you have and ideas? Thank you so much! here my add function

def add_instances(self, copy_item_id:int,model_instance_trans,model_instance_quats) -> None:
        mask = (self.point_ids[..., 0] == copy_item_id)

        copy_item_gs_dict=self.collect_gaussians_from_ids([copy_item_id])
        gs_dict=copy_item_gs_dict[copy_item_id]

        clone_point_ids=self.point_ids[mask]

        self._means = Parameter(torch.cat([self._means, gs_dict["_means"]], dim=0))
        self._scales = Parameter(torch.cat([self._scales, gs_dict["_scales"]], dim=0))
        self._quats = Parameter(torch.cat([self._quats, gs_dict["_quats"]], dim=0))
        self.point_ids = torch.cat([self.point_ids, clone_point_ids], dim=0)

        self.instances_fv = torch.cat([self.instances_fv, gs_dict["instances_fv"]], dim=1)


        self.instances_trans=Parameter(torch.cat([self.instances_trans, model_instance_trans.unsqueeze(1)], dim=1))
        self.instances_quats=Parameter(torch.cat([self.instances_quats, model_instance_quats], dim=1))

        self._features_dc = Parameter(torch.cat([self._features_dc,gs_dict["_features_dc"]], dim=0))
        self._features_rest = Parameter(torch.cat([self._features_rest, gs_dict["_features_rest"]], dim=0))

        self._opacities = Parameter(torch.cat([self._opacities, gs_dict["_opacities"]], dim=0))

OK，when I change the point_ids at this way,I think it can work

        clone_point_ids=self.point_ids[mask]
        clone_point_ids.fill_(self.num_instances)

[zzz5y]

I meet the same trouble like you when i try to adding vehicle.Do you have and ideas? Thank you so much! here my add function

def add_instances(self, copy_item_id:int,model_instance_trans,model_instance_quats) -> None:
        mask = (self.point_ids[..., 0] == copy_item_id)

        copy_item_gs_dict=self.collect_gaussians_from_ids([copy_item_id])
        gs_dict=copy_item_gs_dict[copy_item_id]

        clone_point_ids=self.point_ids[mask]

        self._means = Parameter(torch.cat([self._means, gs_dict["_means"]], dim=0))
        self._scales = Parameter(torch.cat([self._scales, gs_dict["_scales"]], dim=0))
        self._quats = Parameter(torch.cat([self._quats, gs_dict["_quats"]], dim=0))
        self.point_ids = torch.cat([self.point_ids, clone_point_ids], dim=0)

        self.instances_fv = torch.cat([self.instances_fv, gs_dict["instances_fv"]], dim=1)


        self.instances_trans=Parameter(torch.cat([self.instances_trans, model_instance_trans.unsqueeze(1)], dim=1))
        self.instances_quats=Parameter(torch.cat([self.instances_quats, model_instance_quats], dim=1))

        self._features_dc = Parameter(torch.cat([self._features_dc,gs_dict["_features_dc"]], dim=0))
        self._features_rest = Parameter(torch.cat([self._features_rest, gs_dict["_features_rest"]], dim=0))

        self._opacities = Parameter(torch.cat([self._opacities, gs_dict["_opacities"]], dim=0))

OK，when I change the point_ids at this way,I think it can work

        clone_point_ids=self.point_ids[mask]
        clone_point_ids.fill_(self.num_instances)

but i think it's not a good way because before i add the vehicle, if I remove some vehicle ,I'm not so sure the code's work situation!

[zohairbadshah]

@zzz5y Does this method require you to continue training from a particular ckpt or you can edit the scenes after the training is complete?

[zohairbadshah]

@xmskk Have you found the answer to adding new objects(not duplicating existing?)

[MMMaverick]

@xmskk Have you found the answer to adding new objects(not duplicating existing?)

As I understand it, the principle of adding a new object is to store the Gaussian properties and position information of an object into the model of the current scene. You can try to obtain the Gaussian properties of an object from other 3DGS methods (though they might not be compatible with DriveStudio), or you can try using DriveStudio to save the Gaussian properties from another scene and then add them to the scene you want to modify.

[zohairbadshah]

Okay, also have you found a way to define new tracklets for the objects? Say i want the car to turn left instead of goung straight?

[MMMaverick]

@zohairbadshah I am also studying this field. The simplest approach is to determine the time and coordinates of the starting and ending points, and then interpolate the transformation matrix of the object at each frame. The vehicle can be preset to move at a constant speed or a variable speed.