how to generate novel view data?

[SophieZhou]

I want to do experiments to check the results when I changed the camera's pose on waymo datasets.
For example, I want to check when I let the height of camera to be upper 2 meters to generate novel view trajectories.

What should I do on waymo datasets? Could anyone give some hints, thanks.

[MMMaverick]

Hello, as mentioned in #43 , I have written an interface for the new view in the camera.py. It only requires translating the camera's extrinsic matrix upwards. Here is my code for nuscenes:

def up_poses_trajectory(
    dataset_type: str,
    per_cam_poses: Dict[int, torch.Tensor],
    original_frames: int,
    target_frames: int
) -> torch.Tensor:
    """
    Adjust the camera pose:
    1. Increase the camera height by 5 meters.

    Args:
        per_cam_poses (Dict[int, torch.Tensor]): Dictionary of camera poses.

    Returns:
        torch.Tensor: Adjusted camera pose of shape (N, 4, 4).
    """
    assert 0 in per_cam_poses.keys(), "Front center camera (ID 0) is required"
    

    current_pose = per_cam_poses[0]

   
    translation_matrix = torch.tensor([
        [1, 0, 0, 0], # x
        [0, 1, 0, -5],# z
        [0, 0, 1, 0],
        [0, 0, 0, 1]
    ], dtype=torch.float32).cuda()

   
    angle = -np.pi / 6
    rotation_matrix = torch.tensor([
        [1,  0,           0,          0],
        [0,  1, 0, 0],
        [0,  0,  1, 0],
        [0,  0,           0,          1]
    ], dtype=torch.float32).cuda()

  
    transform_matrix = torch.mm(translation_matrix, rotation_matrix)

    
    adjusted_poses = torch.stack([torch.mm(transform_matrix, current_pose[i]) for i in range(len(current_pose))])


    return interpolate_poses(adjusted_poses, target_frames)