Compatibility between Open3D camera and gsplat/Gaussian Splatting

[loryruta]

Checklist

• I have searched for similar issues.
• For Python issues, I have tested with the latest development wheel.
• I have checked the release documentation and the latest documentation (for main branch).

My Question

Hello, I'm trying to combine Open3D with an image rendered from gsplat (Gaussian Splatting).

Gaussian Splatting coordinate system is (+X, -Y, +Z), the intrinsic matrix is calculated as follows (row-major):

K = np.zeros((3, 3), dtype=np.float64)
K[0, 0] = fx * W * 0.5
K[1, 1] = fy * H * 0.5
K[2, 0] = W * 0.5
K[2, 1] = H * 0.5
K[2, 2] = 1

The extrinsic is:

inv(Rot * t)

Rendering the Gaussian Splatting pointcloud with Open3D, using the same extrinsic_matrix/intrinsic_matrix that I use in gsplat, gives the following result:

I'm rendering using:

o3d.visualization.draw(
    [pointcloud],
    width=W,
    height=H,
    field_of_view=None,
    intrinsic_matrix=intrinsic,
    extrinsic_matrix=extrinsic_params,
)

Is there anything, like a zoom, or any parameter transforming the camera matrices that I'm missing? 👀

[Wissotsky]

Heres my code to go from 3dgs cameras.json to open3d extrinsics. I suspect you just need to transpose your rotation

    # Create rotation matrix (R) from the rotation list
    R = np.array(camera['rotation']).T
    
    # Get camera position (C)
    C = np.array(camera['position'])
    
    # Calculate translation vector (t = -RC)
    t = -np.dot(R, C)
    
    # Create extrinsic matrix [R|t]
    E = np.hstack((R, t.reshape(-1, 1)))
    E = np.vstack((E, np.array([0, 0, 0, 1])))