Expressive Body Capture: 3D Hands, Face, and Body from a Single Image

[Project Page] [Paper] [Supp. Mat.]

Table of Contents

• License
• Description
  • Fitting
  • Different Body Models
  • Visualizing Results
• Dependencies
• Citation
• Acknowledgments
• Contact

License

Software Copyright License for non-commercial scientific research purposes. Please read carefully the terms and conditions and any accompanying documentation before you download and/or use the SMPL-X/SMPLify-X model, data and software, (the "Model & Software"), including 3D meshes, blend weights, blend shapes, textures, software, scripts, and animations. By downloading and/or using the Model & Software (including downloading, cloning, installing, and any other use of this github repository), you acknowledge that you have read these terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Model & Software. Any infringement of the terms of this agreement will automatically terminate your rights under this License.

Disclaimer

The original images used for the figures 1 and 2 of the paper can be found in this link. The images in the paper are used under license from gettyimages.com. We have acquired the right to use them in the publication, but redistribution is not allowed. Please follow the instructions on the given link to acquire right of usage. Our results are obtained on the 483 × 724 pixels resolution of the original images.

Description

This repository contains the fitting code used for the experiments in Expressive Body Capture: 3D Hands, Face, and Body from a Single Image.

Fitting

Run the following command to execute the code:

python -m smplifyx --config cfg_files/fit_smplx.yaml 
    --data_folder DATA_FOLDER 
    --output_folder OUTPUT_FOLDER 
    --visualize="True/False"
    --model_folder MODEL_FOLDER
    --vposer_ckpt VPOSER_FOLDER
    --part_segm_fn smplx_parts_segm.pkl

where the DATA_FOLDER should contain two subfolders, images, where the images are located, and keypoints, where the OpenPose output should be stored.

Different Body Models

To fit SMPL or SMPL+H, replace the yaml configuration file with either fit_smpl.yaml or fit_smplx.yaml, i.e.:

• for SMPL:

python -m smplifyx --config cfg_files/fit_smpl.yaml 
   --data_folder DATA_FOLDER 
   --output_folder OUTPUT_FOLDER 
   --visualize="True/False"
   --model_folder MODEL_FOLDER
   --vposer_ckpt VPOSER_FOLDER

• for SMPL+H:

python -m smplifyx --config cfg_files/fit_smplh.yaml 
   --data_folder DATA_FOLDER 
   --output_folder OUTPUT_FOLDER 
   --visualize="True/False"
   --model_folder MODEL_FOLDER
   --vposer_ckpt VPOSER_FOLDER

Visualizing Results

To visualize the results produced by the method you can run the following script:

python -m smplifyx.render_results --mesh_fns OUTPUT_MESH_FOLDER

where OUTPUT_MESH_FOLDER is the folder that contains the resulting meshes.

Headless Resulting Mesh Visualization

For visualization without a display (e.g. in a docker container):

set options (yaml or command line):

visualize: False  # this is 
save_visualization: True

Run smplifyx with the pyopengl backend set to egl, e.g.:

PYOPENGL_PLATFORM=egl python -m smplifyx --config cfg_files/fit_smplx.yaml  --model_folder models

See https://pyrender.readthedocs.io/en/latest/examples/offscreen.html#offscreen-guide and https://pyrender.readthedocs.io/en/latest/install/index.html#osmesa in case of problems with pyrender in headless mode.

Dependencies

Follow the installation instructions for each of the following before using the fitting code.

1. PyTorch
2. SMPL-X
3. VPoser
4. Homogenus

Optional Dependencies

1. PyTorch Mesh self-intersection for interpenetration penalty
   • Download the per-triangle part segmentation: smplx_parts_segm.pkl
2. Trimesh for loading triangular meshes
3. Pyrender for visualization

The code has been tested with Python 3.6, CUDA 10.0, CuDNN 7.3 and PyTorch 1.0 on Ubuntu 18.04.

Citation

If you find this Model & Software useful in your research we would kindly ask you to cite:

@inproceedings{SMPL-X:2019,
  title = {Expressive Body Capture: 3D Hands, Face, and Body from a Single Image},
  author = {Pavlakos, Georgios and Choutas, Vasileios and Ghorbani, Nima and Bolkart, Timo and Osman, Ahmed A. A. and Tzionas, Dimitrios and Black, Michael J.},
  booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
  year = {2019}
}

Acknowledgments

LBFGS with Strong Wolfe Line Search

The LBFGS optimizer with Strong Wolfe Line search is taken from this Pytorch pull request. Special thanks to Du Phan for implementing this. We will update the repository once the pull request is merged.

Sample Images Credits

data/images/sample_image1.jpg

• author: Jan Jespersen
• license: https://creativecommons.org/licenses/by-nc/2.0/

data/images/sample_image2.jpg

• source: https://pixabay.com/photos/brunette-full-body-whole-person-487061/
• license: https://pixabay.com/service/license/

Contact

The code of this repository was implemented by Vassilis Choutas and Georgios Pavlakos.

For questions, please contact [email protected].

For commercial licensing (and all related questions for business applications), please contact [email protected].