Skip to content

Latest commit

 

History

History
89 lines (70 loc) · 4.21 KB

README.md

File metadata and controls

89 lines (70 loc) · 4.21 KB

Running ScanQA (model) on SQA3D

Data preparation for 3D scans

Since this code is based on ScanRefer and ScanQA, you can use the same 3D features. Please also refer to the ScanRefer and ScanQA data preparation.

Before we start, make sure you are now in /path/to/SQA3D/ScanQA.

  1. Use the following command to transform data into the format ScanQA model needs.
cd ../utils
python sqa_data_2_ScanQA.py
### Dataset format
```shell
"scene_id": [ScanNet scene id, e.g. "scene0000_00"],
"object_id": [],
"object_names": [],
"question_id": [...],
"question": [...],
"answers": [...],
```
  1. (Optional) You can download our preprocessed SQA_data_in_ScanQA_format in data/qa

  2. Download answer_counter.json into data/qa, this is the file that contains all answers corresponding to all questions in the dataset.

  3. Download the preprocessed GLoVE embedding and put them under data/.

  4. Link the scannet utils to the current directory

    ln -s ../assets/data/scannet data/.
  5. Download the ScanNetV2 dataset and link scans/ to data/scannet/scans/ (Please follow the ScanNet Instructions for downloading the ScanNet dataset).

  6. Pre-process ScanNet data. A folder named scannet_data/ will be generated under data/scannet/ after running the following command:

    cd data/scannet/
    python batch_load_scannet_data.py
  7. (Optional) Pre-process the multiview features from ENet.

    a. Download the ENet pretrained weights and put it under data/

    b. Download and unzip the extracted ScanNet frames under data/

    c. Change the data paths in config.py marked with TODO accordingly.

    d. Extract the ENet features:

    python scripts/compute_multiview_features.py

    e. Project ENet features from ScanNet frames to point clouds:

    python scripts/project_multiview_features.py --maxpool
  8. Change CONF.PATH.BASE in lib/config.py to the absolute path of the current diretory (ex. /home/test/workspace/SQA3D/ScanQA).

Training

  • scripts for training and the models we evaluated in the paper can be found below
    scripts for training Model in the paper
    wo3d.sh Blind test
    wos.sh ScanQA (w/o s_txt)
    full.sh ScanQA
    auxi.sh ScanQA + aux. task

Evaluation

  • Evaluation of trained ScanQA models with the val dataset:

    python scripts/test.py --ckpt <model_path>  --split <train_val_or_test> <--option>

    <model_path> corresponds to the path to the model. <--option> corresponds to the option used when training

Pretrained models

  • Pretrained models can be downloaded here. The correspondence between the models and the results in the paper is as follows
    models Model in the paper results
    wo3d.pth Blind test 43.65
    wos.pth ScanQA (w/o s_txt) 45.27
    full.pth ScanQA 46.58
    auxi.pth ScanQA + aux. task 47.20

Note that due to the slight change of codebase, the results evaluated might be slightly different from the results in the paper(no more than 0.2%).

Acknowledgements

We would like to thank ScanQA for the useful code base.