Skip to content
/ MB2C Public template

The code repository for "MB2C: Multimodal Bidirectional Cycle Consistency for Learning Robust Visual Neural Representations" in PyTorch

License

Notifications You must be signed in to change notification settings

leeh2213/MB2C

Repository files navigation

MB2C: Multimodal Bidirectional Cycle Consistency for Learning Robust Visual Neural Representations

[Paper πŸš€πŸš€πŸš€]

  • Authors: Yayun Wei*, Lei Cao†, Hao Li*, Yilin Dong *: Equally Contributed †: Corresponding Author This repository hosts the official PyTorch implementation and pretrained model weights for our paper, "MB2C: Multimodal Bidirectional Cycle Consistency for Learning Robust Visual Neural Representations," which has been accepted for publication in the Conference **ACM MM 2024(Poster). alt text alt text alt text alt text Our research focuses on decoding human visual representations from brain activity data. We trained our proposed method on (ThinsEEG) and (EEGCVPR40). We perform zero-shot tasks on the ThingsEEG dataset. Additionally, we conduct EEG classification and image reconstruction on both the ThingsEEG and EEGCVPR40 datasets. Please feel free to use them for further research, and if you use our code or pretrained weights, kindly cite our paper.

Prepare for datasets:

  1. The link of EEGCVPR40 raw
  2. The link of ThingsEEG raw
## Download the required datasets ##
## preprocess ThingsEEG and EEGCVPR40 datasets ##
cd scripts
bash preprocess.sh

Data path for EEGCVPR40

  • raw data: ./Data/CVPR40/Raw_data/
  • proprocessed eeg data: ./Data/CVPR40/Preprocessed_5-95Hz/

Data path for ThingsEEG

  • raw data: ./Data/Things-EEG2/Raw_data/
  • proprocessed eeg data: ./Data/Things-EEG2/Preprocessed_data_250Hz/

Please note that the raw (image and EEG) data are not included in this repo due to their size. The raw ImageNet images and brain EEG data can be downloaded from the corresponding official sources.

πŸ› οΈ Config

# create a conda environment
conda create -n mb2c python=3.8 -y
conda activate mb2c

# install pytorch
conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch

# install CLIP
pip install ftfy regex tqdm
pip install git+https://github.com/openai/CLIP.git

# install other dependencies
pip install diffuser==0.24.0
pip install transformer==4.28.0
if you can't download these 2 packages directly, you can download the source code and install them manually using the following commands:
python setup.py install

πŸŽ“ Training

For classification tasks

  • zero-shot classification (Subject dependent)
  • zero-shot classification (Subject independent i.e. leave one subjuct out)
  • classification with targets on EEGCVPR40 Dataset
cd scripts
bash train.sh  

🎨 For image reconstruction tasks

On EEGCVPR40 dataset

  • MB2C-Stylegan Here, you can refer the paper EEGStyleGAN-ADA for more details. Just replace the EEG-encoder with our EEG-encoder(MB2C).
cd image_reconstruction/MB2C_Stylegan
bash run.sh  # train a stylegan from scratch
bash metriccompute.sh  # calcualte the metrics
  • MB2C-SDXL
cd image_reconstruction/MB2C_SDXL
bash run.sh

On ThingsEEG dataset

cd image_reconstruction/MB2C_SDXL
bash run.sh

πŸ“‹ ToDo List

Here's our ongoing task list:

  • Main model code release
  • Code about classification task released
  • Code about reconstruction task released
  • More powerful generative model can combine with MB2C

πŸ’‘ Acknowledgements

Gratitude to these foundational repositories:

  1. NICE-EEG
  2. BraVL
  3. EEGStyleGAN-ADA
  4. SDXL

Many thanks to the authors of these datasets for their great works.

  1. EEGCVPR40
  2. ThingsEEG

πŸ“ž Contact

If you have any questions, please contact us via email: leehao0517@163.com or weiyayun1660@163.com

πŸ“œ Citation

If our work assists your research, kindly cite us:

@inproceedings{wei2024mb2c,
  title={MB2C: Multimodal Bidirectional Cycle Consistency for Learning Robust Visual Neural Representations},
  author={Wei, Yayun and Cao, Lei and Li, Hao and Dong, Yilin},
  booktitle={Proceedings of the 32nd ACM International Conference on Multimedia},
  pages={8992--9000},
  year={2024}
}

About

The code repository for "MB2C: Multimodal Bidirectional Cycle Consistency for Learning Robust Visual Neural Representations" in PyTorch

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published