Follow up for Issue #38 The best way to correct 3D pose for multi-person dataset

[KamiCalcium]

Hi,

Thanks a lot for answering my question.

You said that I don't need to run the fitting tool if I already have the 3D pose parameters. However, I don't have some all others parameters that you have in the annot_3d.npy. I have 'f', which is the focal length, 'v', the 3d pose (but the pixel-wise global one, not local one), and 'cam_t', which I'm not sure what it is (but I have all the intrinsic and extrinsic so I believe it is possible to get the cam_t, please let me know what cam_t stands for), but I don't have 'beta', which I believe is the parameter from smpl-x, and 'pose', which I believe is also from smpl-x.

So, since I don't have 'beta' and 'pose', can I still skip the fitting process? Or I have to follow 2D -> fitting parameters -> 3D this process?

Hopefully my description is clear! Thank you in advance!

[KamiCalcium]

Hi,

Also, the coordinate system converting is not very clear to me. I notice that in 'v' all the coordinates are ranging from -1 to 1, which should be the spherical coordinates. However, the coordinates I have for my dataset is global pixel-wise coordinates (basically is the same as the 'p2d' of 'annot_3d.npy' but added one more dimension for depth). How can I convert the pixel-wise coordinates system to your local coordinate system? I read the paper but it seems there are a lot of math. Is there a easy way to do so?

[Nicholasli1995]

Does that mean that I don't need to convert the coordinate to your spherical coordinates system (i.e. with parents and children joints and the bone vector)?

Asking this because in the code in annotate_3d.py I saw many of the functions are based on the local system. Don't think they still will work if I don't change my coordinate system to the spherical system.

# Convert skeleton to local coordinate system
bones_global, bones_local, local_system = to_local(skeleton)

# Convert the local coordinates to spherical coordinates
angles = to_spherical(bones_local)
angles[:, 1:] *= 180 / np.pi

to_local converts global coordinates for you. You just need to load global coordinates.

[KamiCalcium]

Does that mean that I don't need to convert the coordinate to your spherical coordinates system (i.e. with parents and children joints and the bone vector)?
Asking this because in the code in annotate_3d.py I saw many of the functions are based on the local system. Don't think they still will work if I don't change my coordinate system to the spherical system.

# Convert skeleton to local coordinate system
bones_global, bones_local, local_system = to_local(skeleton)

# Convert the local coordinates to spherical coordinates
angles = to_spherical(bones_local)
angles[:, 1:] *= 180 / np.pi

to_local converts global coordinates for you. You just need to load global coordinates.

Hi,

Yes I was thinking the same. However, when I load the your example to run annotate_3d.py, the input of to_local(), the skeleton array seems already normalized, it is ranging from -1 to 1. It does not look global. It is shown below:

skeleton
array([[-0.00447037, -0.29515359,  0.03966612],
       [-0.09092443, -0.19689626,  0.04587529],
       [ 0.33320149,  0.05571676,  0.03730629],
       [ 0.46234675,  0.49237239,  0.32432301],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [-0.06744897, -0.28761948,  0.15649853],
       [-0.13724691,  0.04282545,  0.49695113],
       [-0.58984314,  0.12903871,  0.77036283],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.05667901, -0.39703911, -0.06351026],
       [ 0.53880969, -0.35593757, -0.25191031],
       [ 0.63369099, -0.38018736, -0.28714041],
       [ 0.63394761, -0.53357349, -0.47297305],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.45455894, -0.5619275 , -0.09895604],
       [ 0.18256732, -0.70949138,  0.03808844],
       [-0.0058556 , -0.79282213,  0.30185674],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.43485199, -0.14295278, -0.31390296],
       [ 0.54566736,  0.16876501, -0.31920674],
       [ 0.75526418,  0.43258559, -0.31452744],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ]])

Should I do any conversion to make my pixel/camera coordinates to this coordinates? After looking at fit_3d.py I think I need right?

[KamiCalcium]

I tried running fit_3d.py using my dataset. I did all the changes to make it work under py3 including installing the chumpy and opendr under py3. However, for different person with different 2d joints in the same image, I got the same 'v' and 'pose' in fitted.npy and I don't know why... Same 'v' means they have the same skeleton.. It doesn't look correct to me :(

Tomorrow I will use the fitted.npy for running annotate_3d.py and see what happens.

[Nicholasli1995]

Does that mean that I don't need to convert the coordinate to your spherical coordinates system (i.e. with parents and children joints and the bone vector)?
Asking this because in the code in annotate_3d.py I saw many of the functions are based on the local system. Don't think they still will work if I don't change my coordinate system to the spherical system.

# Convert skeleton to local coordinate system
bones_global, bones_local, local_system = to_local(skeleton)

# Convert the local coordinates to spherical coordinates
angles = to_spherical(bones_local)
angles[:, 1:] *= 180 / np.pi

to_local converts global coordinates for you. You just need to load global coordinates.

Hi,

Yes I was thinking the same. However, when I load the your example to run annotate_3d.py, the input of to_local(), the skeleton array seems already normalized, it is ranging from -1 to 1. It does not look global. It is shown below:

skeleton
array([[-0.00447037, -0.29515359,  0.03966612],
       [-0.09092443, -0.19689626,  0.04587529],
       [ 0.33320149,  0.05571676,  0.03730629],
       [ 0.46234675,  0.49237239,  0.32432301],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [-0.06744897, -0.28761948,  0.15649853],
       [-0.13724691,  0.04282545,  0.49695113],
       [-0.58984314,  0.12903871,  0.77036283],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.05667901, -0.39703911, -0.06351026],
       [ 0.53880969, -0.35593757, -0.25191031],
       [ 0.63369099, -0.38018736, -0.28714041],
       [ 0.63394761, -0.53357349, -0.47297305],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.45455894, -0.5619275 , -0.09895604],
       [ 0.18256732, -0.70949138,  0.03808844],
       [-0.0058556 , -0.79282213,  0.30185674],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.43485199, -0.14295278, -0.31390296],
       [ 0.54566736,  0.16876501, -0.31920674],
       [ 0.75526418,  0.43258559, -0.31452744],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ]])

Should I do any conversion to make my pixel/camera coordinates to this coordinates? After looking at fit_3d.py I think I need right?

SMPLify produces normalized coordinates because it does not know camera parameters. Since you have them, you don't need to normalize the coordinates. Just make sure the projection of 3D joints to 2D joints are correct.

[Nicholasli1995]

I tried running fit_3d.py using my dataset. I did all the changes to make it work under py3 including installing the chumpy and opendr under py3. However, for different person with different 2d joints in the same image, I got the same 'v' and 'pose' in fitted.npy and I don't know why... Same 'v' means they have the same skeleton.. It doesn't look correct to me :(

Tomorrow I will use the fitted.npy for running annotate_3d.py and see what happens.

SMPLify works for a single person in a single image. Did you feed different 2D poses to it and run fitting for every person?