This repository contains sound synthesis code used by our papers:
-
Generative Modeling of Audible Shapes for Object Perception, ICCV 2017
-
Shape and Material from Sound, NIPS 2017
Project page: http://sound.csail.mit.edu/
Our code has been tested on 64-bit Ubuntu 14.04.5.
We provide auto install scripts for all required libraries except Matlab & MKL. You need to modify auto_install.sh to indicate the paths to MKL libraries, libiomp5 (often included in an MKL install), and Matlab.
Then, simply run ./auto_install.sh (requires sudo privileges). This should install all the necessary components for you to synthesize sound, including
The script relies on GCC and CMake. We have tested it with GCC 6.3.0 and CMake 3.7.1.
If you prefer to install the dependencies manually (e.g., you don't have sudo access), please refer to the manual installation section at the end of the document for instructions on customized installation and usage.
After auto installation, run (with Python 3 and FFmpeg)
python online_synth/gen_sound.py -r 1000 101 4 1 2
You will find the result in result/scene-1000/obj-101-1/mat-4-2. The generated audio should sound like this. Here, -r indicates rendering audio only (no video). The first number 1000 is the ID of the scene. Each of the following pair describes an object (its ID and material): 101 is for the floor, and 4 indicates it's made of wood; 1 is for the object (plate) and 2 is for its material (steel). You can also add in more objects by specifying different parameters.
To render the corresponding video, you need to specify the path to Blender in gen_sound.py. Then, run the same command but this time without -r
python online_synth/gen_sound.py 1000 101 4 1 2
It should now render the video as well. The generated video should look like this.
To play with more objects & materials, run get_precompute_data.sh to download our pre-computed data. All possible object/material combinations are listed in data/obj_mat_list.txt.
Object data after precomputation (6.3G): all possible object and material combinations are in data/obj_mat_list.txt. Please place the unzipped folder in data. Alternatively, get_precompute_data.sh will download the pre-compute data and place it in the right place for you.
Sound-20k (Soundtrack only, 25G): the dataset used in our ICCV 2017 paper. Metadata can be found in data/data_entry.txt.
The code is structured as two separate parts: offline pre-computation and online synthesis.
The definitions of scenes and materials are located in the config folder.
The input to offline pre-computation is a tetrahedron mesh. One can convert an ordinary triangular mesh to a tetrahedron mesh using TetGen or IsoStuffer, as described by Labelle and Shewchuk 2007.
Our pre-computation step uses two different solvers for calculating sound propagation: a direct BEM solver and an accelerated BEM FMM solver. In short, the direct method is faster when your mesh has a small number of faces, and the FMM solver is faster and more memory-efficient when the mesh has more faces. We suggest using the direct solver for meshes with less than 1000 faces.
One can run pre-computation using the direct method via pre_compute/run_precalc_bem.sh or using the FMM solver via pre_compute/run_precalc.sh. Note that you need to pass an object id followed by a material id to the script. The structure of the data folder should be data/DATASET_NAME/OBJECT_ID; all generated files would be in data/DATASET_NAME/OBJECT_ID/MATERIAL_ID. For example
./run_precalc.sh 0 1
will pre-compute object 0 with material 1 defined in config/material/material-1.cfg using the FMM solver.
The online synthesis part uses Bullet for physical simulation and a modified version of Modal Sound Synthesis for sound synthesis.
The main file is online_synth/gen_sound.py.
As described above, for video rendering, you need to install Blender and specify its path in gen_sound.py. Also FFmpeg is used to combine separate sound tracks or rendered images into a single soundtrack / video.
gen_sound.py has some options. For example, one can use -r to skip rendering and -v to skip combining rendered images and sound tracks into a single video. Please refer to the help function for a detailed explanation.
To use the script, one needs to specify the scene id, and a number of paired object id and material id, as
gen_sound.py scene-id object1-id object1-material-id object2-id object2-material-id
Note: This section is only for users who choose not to use our auto-installation script.
One can run modelsound/auto_install.sh to install all required libraries and build all the necessary binaries.
If you wish to manually install the libraries or to customize their locations, please see this document for a detailed description.
- enter
file_generators - run
./compile_cpps.sh
We solve the Helmholtz equation related to sound propagation using a direct BEM method and an accelerated version using FMM.
- For the direct BEM method, one needs to install the Nihu Matlab library and specify its path in
pre_compute/run_precalc_bem.sh. - For the FMM BEM method, one needs to install the FMMlib3d libraries and specify its path in
pre_compute/run_precalc.sh.
An auto-install script is provided in pre_compute/external/auto_install.sh. Note that you need to specify your Matlab installation path in the first line of auto_install.sh.
- enter
Bullet3/build3 - run
premake4_linux64 gmake - enter gmake, run
make App_RigidBodyFromObjExampleGui - The built binary is located in
bullet3/bin.
If you need to modify our simulation code, they are located in bullet3/modified_scripts.