KnowZRel: Common Sense Knowledge-based Zero-Shot Relationship Retrieval for Generalized SGG

This readme file provides an overview of the codebase for the paper implementation. The download links and destination paths for the dataset, Faster RCNN model and CSKG embeddings are provided. The notebooks ZS_SGG_FasterRCNN/ZS_SGG_FasterRCNN.ipynb and ZS_SGG_CSKG/j_SG_CSKG_ZeroShot.ipynb can be used to reproduce the results. A thoroughly documented codebase/toolkit with examples/demo of the proposed method will be released soon.

Requirements

• Ubuntu 18.04
• CUDA 10.1
• Python 3.7
• PyTorch 1.4
• KGTK 0.5

Code:

• Object detection: ZS_SGG_FasterRCNN/ZS_SGG_FasterRCNN.ipynb
• Zero-shot relationship retrieval and zero-shot SGG evaluation: ZS_SGG_CSKG/j_SG_CSKG_ZeroShot.ipynb

Directories:

• ZS_SGG_FasterRCNN contains code, data and model for object detection.
• ZS_SGG_CSKG contains code, data and CSKG embeddings for zero-shot relationship retrieval for scene graph generation
• Eval_IO contains input/output data files for evaluation
• Eval_IO/0_images contains Visual Genome images.
• Eval_IO/1_det_objs contains detected objects info.
• Eval_IO/2_zs_sg contains generated scene graphs with zero-shot relationships extracted from CSKG.

Downloads and destination paths:

Datasets

• Visual Genome: Download images.zip and images2.zip and unzip all images to Eval_IO/0_images. Also download relationships.json and image_data.json and place both files in the root directory.
• GQA: https://cs.stanford.edu/people/dorarad/gqa/download.html

CSKG Embeddings:

• Download all files from this link and place them in ZS_SGG_CSKG/cskg/output directory.
• Move 'bert_nli_large_w2v_format.txt.gz' to ZS_SGG_CSKG/cskg/output/embeddings directory.
• Download numberbatch file and place it in ZS_SGG_CSKG/cskg/output/embeddings directory.

Pre-trained Faster RCNN model:

• Download model1 and model2, and place them in 'ZS_SGG_FasterRCNN/'.