GeoDesc implementation

TensorFlow implementation of GeoDesc for ECCV'18 paper "GeoDesc: Learning Local Descriptors by Integrating Geometry Constraints", Zixin Luo, Tianwei Shen, Lei Zhou, Siyu Zhu, Runze Zhang, Yao Yao, Tian Fang and Long Quan.

Update 08/04/2019

We improve the patch cropping implementation and now it gets 5 times faster.

Update 04/12/2018

Part of the training data is released in GL3D. We will constantly update dataset to help to reproduce the work. Please also cite MIRorR if you find this dataset useful for your research.

Update 14/08/2018

We have provided an example to test the matching performance of GeoDesc (examples/image_matching.py). See usage below.

Requirements

Please use Python 2.7, install NumPy, OpenCV and TensorFlow. To run the image matching example, you may also need to compile opencv_contrib to get SIFT support.

Pre-trained model

Pre-trained GeoDesc model (in TensorFlow Protobuf format) can be found here.

Model in NumPy dictionary is available here, which is more handy to be converted to other formats.

Example scripts

1. Extract features of HPatches

An example script is provided to extract features of HPatches. HPatches should be ready in its original format.

After download HPatches, you can start to evaluate GeoDesc:

git clone https://github.com/lzx551402/geodesc.git
cd geodesc/model
wget http://home.cse.ust.hk/~zluoag/data/geodesc.pb
cd ../examples
python extract_features_of_hpatches.py \
    --hpatches_root=<hpatches_benchmark>/data/hpatches-release \
    --feat_out_path=<hpatches_benchmark>/data/descriptors

After the extraction, you can use HPatches benchmarking tools to evaluate GeoDesc (on split 'full' as GeoDesc is not trained on HPatches):

cd <hpatches_benchmark>/python
python hpatches_eval.py --descr-name='geodesc' \
    --task=verification --task=matching --task=retrieval --split=full

And then display the results:

python hpatches_results.py --descr-name='geodesc' \
    --task=verification --task=matching --task=retrieval --split=full --results-dir=

2. Test image matching

As described in the paper, the matching pipeline consists of: i) detect keypoints by SIFT detector, ii) crop patches in the scale space, iii) compute features on cropped patches, and iv) match the two images. If you want to achieve the efficiency reported in the paper and use it for large-scale matching tasks, we strongly suggest you implementing the pipeline in C++ with integrated GPU-based SIFT (e.g., SIFTGPU) and GPU-based matcher (e.g., OpenCV GPU matcher). We have provided here only a prototype for research purposes.

To get started, simply run:

cd examples
python image_matching.py --cf_sift

The matching results from SIFT (top) and GeoDesc (bottom) will be displayed. Type python image_matching.py --h to view more options and test on your own images.

(Image source: Graffiti sequence in Heinly benchmark)

Training code

The ground truth patches used to train GeoDesc are under preparation.

Benchmark on HPatches, mAP

Benchmark on Heinly benchmark

Benchmark on ETH local features benchmark

Application on 3D reconstructions