Skip to content

Latest commit

 

History

History
108 lines (77 loc) · 5.02 KB

README.md

File metadata and controls

108 lines (77 loc) · 5.02 KB
Raw

Localizing Visual Sounds the Easy Way

Official codebase for EZ-VSL. EZ-VSL is a simple yet effective approach for Visual Sound Localization. Please check out the paper for full details.

Localizing Visual Sounds the Easy Way
Shentong Mo, Pedro Morgado
arXiv 2022.

EZ-VSL Illustration

Environment

To setup the environment, please simply run

pip install -r requirements.txt

Datasets

Flickr-SoundNet

Data can be downloaded from Learning to localize sound sources

VGG-Sound Source

Data can be downloaded from Localizing Visual Sounds the Hard Way

VGG-SS Unheard & Heard Test Data

Data can be downloaded from Unheard and Heard

Model Zoo

We release several models pre-trained with EZ-VSL with the hope that other researchers might also benefit from them.

Method Train Set Test Set CIoU AUC url Train Test
EZ-VSL Flickr 10k Flickr SoundNet 81.93 62.58 model script script
EZ-VSL Flickr 144k Flickr SoundNet 83.13 63.06 model script script
EZ-VSL VGG-Sound 144k Flickr SoundNet 83.94 63.60 model script script
EZ-VSL VGG-Sound 10k VGG-SS 37.18 38.75 model script script
EZ-VSL VGG-Sound 144k VGG-SS 38.85 39.54 model script script
EZ-VSL VGG-Sound Full VGG-SS 39.34 39.78 model script script
EZ-VSL Heard 110 Heard 110 37.25 38.97 model script script
EZ-VSL Heard 110 Unheard 110 39.57 39.60 model script script

Train & Test

For training an EZ-VSL model, please run

python train.py --multiprocessing_distributed \
    --train_data_path /path/to/Flickr-all/ \
    --test_data_path /path/to/Flickr-SoundNet/ \
    --test_gt_path /path/to/Flickr-SoundNet/Annotations/ \
    --experiment_name flickr_10k \
    --trainset 'flickr_10k' \
    --testset 'flickr' \
    --epochs 100 \
    --batch_size 128 \
    --init_lr 0.0001

For testing and visualization, simply run

python test.py --test_data_path /path/to/Flickr-SoundNet/ \
    --test_gt_path /path/to/Flickr-SoundNet/Annotations/ \
    --model_dir checkpoints \
    --experiment_name flickr_10k \
    --save_visualizations \
    --testset 'flickr' \
    --alpha 0.4

The training script supports the following training sets: flickr, flickr_10k, flickr_144k, vggss, vggss_10k, vggss_144k or vggss_heard.

For evaluation, it supports the following test sets: flickr, vggss, vggss_heard, vggss_unheard.

Visualizations

The test.py script saves the predicted localization maps for all test images when the flag --save_visualizations is provided. All visualizations for OGL, AVL and EZ-VSL localization maps are saved under {model_dir}/{experiment_name}/viz/. Here's some examples.

Visualizations

Citation

If you find this repository useful, please cite our paper:

@article{mo2022EZVSL,
  title={Localizing Visual Sounds the Easy Way},
  author={Mo, Shentong and Morgado, Pedro},
  journal={arXiv preprint arXiv:2203.09324},
  year={2022}
}