corc - Computation of Radial Curves

This project contains the source code to compute the Volumetric Facial Palsy Grading published at the MIE-2022. You can find the accompanying paper here. If you use any part of the code for your own research or development, please use the provided citation at the end of the document.

Overview

The code features several processing steps which can be used independently from each other:

• Loading of different facial landmark formats:
  • 68
  • 3DFE
  • 3DFE-MVLM
  • BP4D
  • Custom landmarking system by implementing an inheriting class of corc.landmarks.Landmarks.
• Loading of different point-cloud/mesh formats:
  • .ply
  • .obj
  • .wrl
• Computation of Radial Curves using the 3D landmarks and the point clouds
• Computation of the volumetric score using the z point-wise mean difference

All calculations are done in numpy, scipy, and igraph on the CPU. We utilize multiprocessing during the extraction and processing of the Radial Curves to achieve almost real-time results.

Installation

Note: The system has been developed and extensively been tested only on Linux based systems. For Windows and MacOS based system please open an issue of problems occur. We assume you have a proper python installation on your system and using either a virtual environment or anaconda. One can create a conda environment using:

conda create -n corc-test python=3.9 pip -y

To install the corc please copy the repository into a directory of your choise.

git clone [email protected]:cvjena/corc.git

Inside the folder please run

pip install .

Now the the package should be available in your python installation via

import corc

The following sub-modules are available for usage inside the package

from corc import core, grading, landmarks, utils

Basic Usage

This is a minimal example on how to use the corc library to compute the Radial Curves from a given 3D landmark and a point-cloud file. For extended usage please refer to the doc-strings of the functions. The algorithm usage can be fine-tuned using the defined keyword arguments.

import pathlib

import numpy as np

from corc import core, grading, landmarks, utils

# we assume you use the basic 68 landmark system in this example
lm_file = pathlib.Path("path/to/your/landmarks_file.csv")
pc_file = pathlib.Path("path/to/your/pointcloud_file.obj")

# the loaded files will be in the form of a numpy array.
lm: np.ndarray = utils.load_landmarks(lm_file)
pc: np.ndarray = utils.load_pointcloud(pc_file)

# the landmarks have to forwarded to the correct Landmark class
# this class handles the correct access of the specific landmark features
# like eyes, nose-tip, mouth
radial_curves: np.ndarray = core.compute_corc(
    point_cloud=pc,
    landmarks=landmarks.Landmarks68(lm),
    n_curves=128,
    n_points=128,
)

# the computed radial curves are also in the form of the numpy array
# in this example they have the form of 128x128x3
grad: float = grading.volume_grading(radial_curves, n_curves=128, n_points=128)

Advanced Volume Computation

import pathlib

import numpy as np

from corc import core, grading, landmarks, utils

# we assume you use the basic 68 landmark system in this example
lm_file = pathlib.Path("path/to/your/landmarks_file.csv")
pc_file = pathlib.Path("path/to/your/pointcloud_file.obj")

# the loaded files will be in the form of a numpy array.
lm: np.ndarray = utils.load_landmarks(lm_file)
pc: np.ndarray = utils.load_pointcloud(pc_file)

# the landmarks have to forwarded to the correct Landmark class
# this class handles the correct access of the specific landmark features
# like eyes, nose-tip, mouth
radial_curves: np.ndarray = core.compute_corc(
    point_cloud=pc,
    landmarks=landmarks.Landmarks68(lm),
    n_curves=128,
    n_points=128,
)

curves_l, curves_r = corc.volume.split_left_right(curves, n_curves=n_curves)
segment_point, center_point = corc.volume.compute_lower_curves_center(curves_l, factor=1.0)

points_r, triangles_r = corc.volume.make_mesh(curves_r, segment_point, center_point, right=True)
points_l, triangles_l = corc.volume.make_mesh(curves_l, segment_point, center_point, right=False)

mesh_l, volume_l = corc.volume.compute_volume(points_l, triangles_l)
mesh_r, volume_r = corc.volume.compute_volume(points_r, triangles_r)

print("L Volume:", volume_l)
print("R Volume", volume_r)

Citation

If you use this work please use the following bibtext entry:

@inproceedings{Buechner22:MIE,
    type = {inproceedings},
    key = {Buechner22:MIE},
    title = {Automatic Objective Severity Grading of Peripheral Facial Palsy Using 3D Radial Curves Extracted from Point Clouds},
    author = {Tim Büchner and Sven Sickert and Gerd F. Volk and Orlando Guntinas-Lichius and Joachim Denzler},
    booktitle = {Challenges of Trustable AI and Added-Value on Health},
    series = {Studies in Health Technology and Informatics},
    volume = {294},
    year = {2022},
    note = {},
    publisher = {IOS Press},
    pages = {179-183},
    doi = {10.3233/SHTI220433},
    url = {https://ebooks.iospress.nl/volumearticle/59591},
    abstract = {Peripheral facial palsy is an illness in which a one-sided ipsilateral paralysis of the facial muscles occurs due to nerve damage. Medical experts utilize visual severity grading methods to estimate this damage. Our algorithm-based method provides an objective grading using 3D point clouds. We extract from static 3D recordings facial radial curves to measure volumetric differences between both sides of the face. We analyze five patients with chronic complete peripheral facial palsy to evaluate our method by comparing changes over several recording sessions. We show that our proposed method allows an objective assessment of facial palsy.},
    groups = {facialpalsy},
    langid = {english},
}