Projecting 'j3d' onto image plane

[thohemp]

Hello,

thanks for your work. I try to project the 3D joints onto the image plane, but I don't seem to get the correct results.

First I define the camera intrinsics:

self.focal_length=600
fx,fy = self.focal_length, self.focal_length
height=512
width=512
self.camera_center = np.array([width / 2., height / 2.])

Then I take 3d joint of the index_fingertip:

index_fingertip = results['0'][0]['j3d'][7]

Translate it to camera coordinates:

index_fingertip+=results['0'][0]['cam_trans']

Finally, I calculate and print the pixel coordinates:

u = fx * (index_fingertip[0]/index_fingertip[2]) + self.camera_center[0]
v = fy * (index_fingertip[1]/index_fingertip[2]) + self.camera_center[1]
print("u: {} +  v: {}".format(u,v))

However, the results are not correct. Placing the finger tip on the left side of the image plan will output a u coordinate of 140. Similar results are reported for the y axis.
What am I missing?

[ZhengdiYu]

Sorry for the late response due to my work here :) The 2D projection is done using the predicted weak-perspective camera parameters. And it's already in outputs['pj2d'].

To visualize, I suggest that you can add 'pj2d' into the show_items_list here:

Arbitrary-Hands-3D-Reconstruction/acr/main.py

Line 100 in 3ece8dc

show_items_list = ['mesh'] # ['org_img', 'mesh', 'pj2d', 'centermap']

After that, the visualization of 2D keypoints will be done using the following code:

Arbitrary-Hands-3D-Reconstruction/acr/visualization.py

Lines 234 to 244 in 3ece8dc

	if vis_name == 'pj2d' and outputs['detection_flag']:
	kp_imgs = []
	for img_id, inds_list in enumerate(per_img_inds):
	org_img = org_imgs[img_id].copy()
	#try:
	for kp2d_vis in outputs['pj2d'][inds_list]:
	if len(kp2d_vis)>0:
	kp2d_vis = ((kp2d_vis+1)/2 * org_imgs.shape[1])
	org_img = self.vis_keypoints(org_img, kp2d_vis.detach().cpu().numpy(), skeleton=self.MANO_SKELETON)
	kp_imgs.append(np.array(org_img))
	plot_dict['pj2d'] = {'figs':kp_imgs, 'type':'image'}