Since this code is based on ScanRefer and ScanQA, you can use the same 3D features. Please also refer to the ScanRefer and ScanQA data preparation.
Before we start, make sure you are now in /path/to/SQA3D/ScanQA
.
- Use the following command to transform data into the format ScanQA model needs.
cd ../utils
python sqa_data_2_ScanQA.py
### Dataset format
```shell
"scene_id": [ScanNet scene id, e.g. "scene0000_00"],
"object_id": [],
"object_names": [],
"question_id": [...],
"question": [...],
"answers": [...],
```
-
(Optional) You can download our preprocessed SQA_data_in_ScanQA_format in
data/qa
-
Download answer_counter.json into data/qa, this is the file that contains all answers corresponding to all questions in the dataset.
-
Download the preprocessed GLoVE embedding and put them under
data/
. -
Link the scannet utils to the current directory
ln -s ../assets/data/scannet data/.
-
Download the ScanNetV2 dataset and link
scans/
todata/scannet/scans/
(Please follow the ScanNet Instructions for downloading the ScanNet dataset). -
Pre-process ScanNet data. A folder named
scannet_data/
will be generated underdata/scannet/
after running the following command:cd data/scannet/ python batch_load_scannet_data.py
-
(Optional) Pre-process the multiview features from ENet.
a. Download the ENet pretrained weights and put it under
data/
b. Download and unzip the extracted ScanNet frames under
data/
c. Change the data paths in
config.py
marked with TODO accordingly.d. Extract the ENet features:
python scripts/compute_multiview_features.py
e. Project ENet features from ScanNet frames to point clouds:
python scripts/project_multiview_features.py --maxpool
-
Change
CONF.PATH.BASE
inlib/config.py
to the absolute path of the current diretory (ex./home/test/workspace/SQA3D/ScanQA
).
scripts for training
and the models we evaluated in the paper can be found belowscripts for training
Model in the paper wo3d.sh
Blind test
wos.sh
ScanQA (w/o s_txt)
full.sh
ScanQA
auxi.sh
ScanQA + aux. task
-
Evaluation of trained ScanQA models with the val dataset:
python scripts/test.py --ckpt <model_path> --split <train_val_or_test> <--option>
<model_path> corresponds to the path to the model. <--option> corresponds to the option used when training
- Pretrained models can be downloaded here. The correspondence between the models and the results in the paper is as follows
models Model in the paper results wo3d.pth
Blind test
43.65 wos.pth
ScanQA (w/o s_txt)
45.27 full.pth
ScanQA
46.58 auxi.pth
ScanQA + aux. task
47.20
Note that due to the slight change of codebase, the results evaluated might be slightly different from the results in the paper(no more than 0.2%).
We would like to thank ScanQA for the useful code base.