Skip to content

Latest commit

 

History

History
128 lines (106 loc) · 5.85 KB

README.md

File metadata and controls

128 lines (106 loc) · 5.85 KB
Raw

Rethink Predicting the Optical Flow with the Kinetics Perspective

This repository is the official implementation of Rethink Predicting the Optical Flow with the Kinetics Perspective.

📋

✓ Code Completeness Checklist

Requirements

To install requirements (python=3.8):

conda create -n kinetics_flow python=3.8
conda install pytorch==1.13.0 torchvision==0.14.0 torchaudio==0.13.0 cudatoolkit=11.7 -c pytorch -c nvidia
pip install -r requirements.txt

Structure of Dataset

├── datasets
    ├── FlyingChairs_release
        ├── data
        └── chairs_split.txt
    ├── FlyingThings3D
        ├── frames_cleanpass
        ├── frames_finalpass
        └── optical_flow
    ├── Sintel
        ├── training
        └── test
    ├── HD1K
        ├── hd1k_input
        └── hd1k_flow_gt
    ├── KITTI
        ├── training
        └── testing
    ├── high_speed_sintel
        ├── clean_1008fps
        ├── flow_1008fps
        └── occlusions_fps
    ├── DAVIS
        └── JPEGImages

Here are the corresponding links for each dataset, and you need to download, unzip, and rename the dataset folder as given above:

We provide a simple script to create symbolic links to link local datasets (provided the names of local datasets are the same as above structure). You can change the $LOCAL_ROOT_PATH variable in script to your local root directory of all optical flow datasets or set the value by command line.

bash scripts/link_dataset.sh [${LOCAL_ROOT_PATH}]

For better comprehension of training pipeline, we declare the datasets used in each stage as following:

Stage Dataset Name
chairs FlyingChairs
things FlyingThings3D
sintel FlyingThings3D+Sintel+KITTI+HD1K
kitti KITTI
ss HighSpeedSintel+DAVIS

Details about Our Framework

Training

To train the model(s) in the paper, run:

bash scripts/train.sh

Detailed hyperparameters are provided in the corresponding configuration files

Evaluation

To evaluate our model on Sintel and KITTI training set, run:

bash scripts/evaluate.sh

Ablations

Ablation configurations provide examples of ablations:

  • without pretrained GIM
  • without WarpNet

Pre-trained Models

You can download pretrained models here:

  • Models with name suffix representing trained for evaluating corresponding dataset.
  • You can download VGG19 from pytorch.
  • You can download LoFTR outdoor model from kornia.

📋 Give a link to where/how the pretrained models can be downloaded and how they were trained (if applicable). Alternatively you can have an additional column in your results table with a link to the models.

Results

Our final model achieves the following performance on Sintel(train set) and KITTI-2015, details on the paper:

Sintel(train/clean, EPE) Sintel(train/final, EPE) KITTI-15(train, Fl-all) KITTI-15(test, Fl-all)
Ours 0.50 0.87 1.23 4.94
RAFT 0.77 1.22 1.52 5.04
GMA 0.62 1.06 1.22 5.15
GMFlow 0.76 1.11 5.17 9.32

Citation

If you found our paper helpful to your research, please consider citing:

@inproceedings{cheng_kinetics_2024,
    title={Rethink Predicting the Optical Flow with the Kinetics Perspective},
    author={Cheng, Yuhao and Zhang, Siru and Yan, Yiqiang},    
    year={2024}
}

Acknowledgments