Skip to content

Latest commit

 

History

History
219 lines (198 loc) · 35.9 KB

README.md

File metadata and controls

219 lines (198 loc) · 35.9 KB
Raw

A Survey on Rain Removal from Video and Single Image

Hong Wang, Yichen Wu, Minghan Li, Qian Zhao, and Deyu Meng

[Arxiv]

Citation

@article{wang2022survey,
  title={Survey on rain removal from videos or a single image},
  author={Wang, Hong and Wu, Yichen and Li, Minghan and Zhao, Qian and Meng, Deyu},
  journal={Science China Information Sciences},
  volume={65},
  number={1},
  pages={1--23},
  year={2022},
  publisher={Springer}
}

@article{WangA,
  title={A Survey on Rain Removal from Video and Single Image}, 
  author={Wang, Hong and Wu, Yichen and Li, Minghan and Zhao, Qian and Meng, Deyu}, 
  journal={arXiv preprint arXiv:1909.08326},
  year={2019}
}

Physical Properties of Raindrops

  • Gemometric Property
    • Terminal velocity of raindrops aloft (JAMC1969), Foote et al [PDF]
    • A new model for the equilibrium shape of raindrops (JAS1987), Beard et al. [PDF]
  • Brightness Property
    • Photometric model of a rain drop (Technical Report, Columbia University2004), Garg et al [PDF]
    • Vision and Rain (IJCV2007), Garg et al [Project][PDF]
  • Chromatic Property
    • Rain removal in video by combining temporal and chromatic properties (ICME2006), Zhang et al [Project][PDF]
  • Spatial and Temporal Propety
    • Simulation of rain in videos (TAS2003), Starik et al [PDF]
    • Pixel based temporal analysis using chromatic property for removing rain from videos (CIC2009), Liu et al [PDF]

Video Deraining Methods

  • Time Domain
    • Detection and removal of rain from videos (CVPR2004), Garg et al [Project][PDF]
    • When does camera see rain? (ICCV2005), Garg et al [Project][PDF]
    • Rain removal using kalman filter in video (ICSMA2008), Park et al [PDF]
    • Using the shape characteristics of rain to identify and remove rain from video (S+SSPR2008), Brewer et al [PDF]
    • The application of histogram on rain detection in video (JCIS2008), Zhao et al [PDF]
    • Rain or snow detection in image sequences through use of a histogram of orientation of streaks (IJCV2011), Bossu et al [PDF]
    • A probabilistic approach for detection and removal of rain from videos (IETE JR2011), Tripathi et al [PDF]
    • Video post processing: low latency spatiotemporal approach for detection and removal of rain (IET IP2012), Tripathi et al [PDF]
    • Removal of rain from videos: a review (SIVP2014), Tripathi et al [PDF]
    • Stereo video deraining and desnowing based on spatiotemporal frame warping (ICIP2014), Kim et al [PDF]
  • Frequency Domain
    • Spatio-temporal frequency analysis for removing rain and snow from videos (PACV2007), Barnum et al [Project] [PDF]
    • Analysis of rain and snow in frequency space (IJCV2010), Barnum et al [Project] [PDF]
  • Low Rank and Sparsity
    • A generalized low-rank appearance model for spatio-temporally correlated rain streaks (ICCV2013), Chen et al [PDF]
    • A rain pixel recovery algorithm for videos with highly dynamic scenes (TIP2013), Chen et al [PDF]
    • Video deraining and desnowing using temporal correlation and low-rank matrix completion (TIP2015), Kim et al [PDF] [Code]
    • Adherent raindrop modeling, detection and removal in video (TPAMI2016), You et al. [Project] [PDF]
    • Video desnowing and deraining based on matrix decomposition (CVPR2017), Ren et al [PDF] [Code]
    • A novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors (CVPR2017), Jiang et al [PDF]
    • Should We encode rain streaks in video as deterministic or stochastic? (ICCV2017), Wei et al [PDF] [Code]
    • A directional global sparse model for single image rain removal (AMM2018), Deng et al [PDF] [Code]
    • Video rain streak removal by multiscale convolutional sparse coding (CVPR2018), Li et al [Project] [PDF] [Code]
    • Fastderain: A novel video rain streak removal method using directional gradient priors (TIP2019), Jiang et al [PDF] [Code]
  • Deep Learning
    • Robust video content alignment and compensation for rain removal in a cnn framework (CVPR2018), Chen et al [PDF] [Code]
    • Erase or fill? deep joint recurrent rain removal and reconstruction in videos (CVPR2018), Liu et al. [Project] [PDF] [Code]
    • D3R-Net: dynamic routing residue recurrent network for video rain removal (TIP2018), Liu et al. [PDF]
    • Frame consistent recurrent video deraining with dual-level flow (CVPR2019), Yang et al. [Code]
    • Self-Learning Video Rain Streak Removal: When Cyclic Consistency Meets Temporal Correspondence(CVPR2020), Yang et al.[PDF] [Supplementary Materials] [Code]
    • Online Rain/Snow Removal From Surveillance Videos(TIP2021), Li et al. [PDF]
    • Triple-Level Model Inferred Collaborative Network Architecture for Video Deraining(TIP2021), Mu et al.[PDF] [Code]
    • Let’s See Clearly: Contaminant Artifact Removal for Moving Cameras(ICCV2021), Li et al. [PDF] [Results Display]
    • Self-Aligned Video Deraining With Transmission-Depth Consistency(CVPR2021), Yan et al. [PDF] [Supplementary Materials] [Code]
    • Semi-Supervised Video Deraining With Dynamical Rain Generator(CVPR2021), Yue et al. [PDF][Supplementary Materials] [Code]
    • Robust online rain removal for surveillance videos with dynamic rains(KBS2021), Yi et al. [PDF]
    • Feature-Aligned Video Raindrop Removal With Temporal Constraints(TIP2022), Yan et al. [PDF]
  • Reivew paper
    • Removal of rain from videos: a review (SIVP2014), Tripathi et al [PDF]
    • A Survey on Rain Removal from Video and Single Image (Arxiv2019), Wang et al. [PDF] [Code]

Single Image Deraining Methods

  • Filter based methods
    • Guided image filtering (ECCV2010), He et al. [Project] [PDF] [Code]
    • Removing rain and snow in a single image using guided filter (CSAE2012), Xu et al. [PDF]
    • An improved guidance image based method to remove rain and snow in a single image (CIS2012), Xu et al. [PDF]
    • Single-image deraining using an adaptive nonlocal means filter (ICIP2013), Kim et al. [PDF]
    • Single-image-based rain and snow removal using multi-guided filter (NIPS2013), Zheng et al. [PDF]
    • Single image rain and snow removal via guided L0 smoothing filter (Multimedia Tools and Application2016), Ding et al. [PDF]
  • Prior based methods
    • Automatic single-image-based rain streaks removal via image decomposition (TIP2012), Kang et al [PDF] [Code]
    • Self-learning-based rain streak removal for image/video (ISCS2012), Kang et al. [PDF]
    • Single-frame-based rain removal via image decomposition (ICA2013), Fu et al. [PDF]
    • Exploiting image structural similarity for single image rain removal (ICIP2014), Sun et al. [PDF]
    • Visual depth guided color image rain streaks removal using sparse coding (TCSVT2014), Chen et al [PDF]
    • Removing rain from a single image via discriminative sparse coding (ICCV2015), Luo et al [PDF] [Code] pwd: d229
    • Rain streak removal using layer priors (CVPR2016), Li et al [PDF] [Code]
    • Single image rain streak decomposition using layer priors (TIP2017), Li et al [PDF]
    • Error-optimized dparse representation for single image rain removal (IEEE TIE2017), Chen et al [PDF]
    • A hierarchical approach for rain or snow removing in a single color image (TIP2017), Wang et al. [PDF]
    • Joint bi-layer optimization for single-image rain streak removal (ICCV2017), Zhu et al. [PDF]
    • Convolutional sparse and low-rank codingbased rain streak removal (WCACV2017), Zhang et al [PDF]
    • Joint convolutional analysis and synthesis sparse representation for single image layer separation (CVPR2017), Gu et al [PDF] [Code]
    • Single image deraining via decorrelating the rain streaks and background scene in gradient domain (PR2018)， Du et al [PDF]
  • Deep Learning
    • Restoring an image taken through a window covered with dirt or rain (ICCV2013), Eigen et al. [Project] [PDF] [Code]
    • Attentive generative adversarial network for raindrop removal from a single image (CVPR2018), Qian et al [Project] [PDF]
    • Clearing the skies: A deep network architecture for single-image rain streaks removal (TIP2017), Fu et al. [Project] [PDF] [Code]
    • Removing rain from single images via a deep detail network (CVPR2017), Fu et al. [Project] [PDF] [Code]
    • Image de-raining using a conditional generative adversarial network (Arxiv2017), Zhang et al [PDF] [Code]
    • Deep joint rain detection and removal from a single image (CVPR2017), Yang et al.[Project] [PDF] [Code]
    • Residual guide feature fusion network for single image deraining (ACMMM2018), Fan et al. [Project] [PDF]
    • Fast single image rain removal via a deep decomposition-composition network (Arxiv2018), Li et al [Project]) [PDF] [Code]
    • Density-aware single image de-raining using a multi-stream dense network (CVPR2018), Zhang et al [PDF] [Code]
    • Recurrent squeeze-and-excitation context aggregation net for single image deraining (ECCV2018), Li et al. [PDF] [Code]
    • Rain streak removal for single image via kernel guided cnn (Arxiv2018), Wang et al [PDF]
    • Physics-based generative adversarial models for image restoration and beyond (Arxiv2018), Pan et al [PDF]
    • Learning dual convolutional neural networks for low-level vision (CVPR2018), Pan et al [Project] [PDF] [Code]
    • Non-locally enhanced encoder-decoder network for single image de-raining (ACMMM2018), Li et al [PDF] [Code]
    • Single image rain removal via a deep decomposition-composition network (CVIU2019), Li et al.
    • Unsupervised single image deraining with self-supervised constraints (ICIP2019), Jin et al [PDF]
    • Residual multiscale based single image deraining (BMVC2019), Zheng et al.
    • Erl-net: Entangled representation learning for single image de-raining (ICCV2019), Wang et al. [code]
    • Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining (CVPR2019), Rajeev Yasarla et al.[Code]
    • Heavy rain image restoration: Integrating physics model and conditional adversarial learning (CVPR2019), Li et al.[Code]
    • Progressive image deraining networks: A better and simpler baseline (CVPR2019), Ren et al [PDF] [Code]
    • Spatial attentive single-image deraining with a high quality real rain dataset (CVPR2019), Wang et al [Project] [PDF] [Code]
    • Lightweight pyramid networks for image deraining (TNNLS2019), Fu et al [PDF] [Code]
    • Joint rain detection and removal from a single image with contextualized deep networks (TPAMI2019), Yang et al [PDF] [Code]
    • Scale-free single image deraining via visibility-enhanced recurrent wavelet learning (TIP2019), Yang et al.[PDF]
    • Towards scale-free rain streak removal via selfsupervised fractal band learning (AAAI2020), Yang et al.[Code]
    • Structural Residual Learning for Single Image Rain Removal(Arxiv2020), Wang et al. [PDF]
    • All in One Bad Weather Removal Using Architectural Search (CVPR2020), Li et al.[PDF]
    • Syn2Real Transfer Learning for Image Deraining Using Gaussian Processes(CVPR2020), Rajeev Yasarla et al. [Code]
    • Multi-Scale Progressive Fusion Network for Single Image Deraining(CVPR2020), Jiang et al. [Code]
    • Detail-recovery Image Deraining via Context Aggregation Networks(CVPR2020), Deng et al.[Code]
    • Variational image deraining(WACV2020), Du et al.[PDF]
    • DerainCycleGAN: Rain Attentive CycleGAN for Single Image Deraining and Rainmaking(TIP2021), Wei et al.[PDF][Code]
    • Unsupervised Image Deraining: Optimization Model Driven Deep CNN(ACM MM2021), Yu et al.[PDF] [Code]
    • Multi-Level Memory Compensation Network for Rain Removal via Divide-and-Conquer Strategy(JSTSP2021), Jiang et al. [PDF]
    • Robust Representation Learning with Feedback for Single Image Deraining (CVPR2021), Chen et al. [PDF] [Code]
    • Memory Oriented Transfer Learning for Semi-Supervised Image Deraining (CVPR2021), Huang et al. [PDF] [Code]
    • Image De-raining via Continual Learning (CVPR2021), Zhou et al. [PDF]
    • Improving De-raining Generalization via Neural Reorganization (ICCV2021), Xiao et al. [PDF]
    • Learning Multiple Adverse Weather Removal via Two-stage Knowledge Learning and Multi-contrastive Regularization: Toward a Unified Model (CVPR2022), Chen et al. [PDF] [Code]
    • GIQE: Generic Image Quality Enhancement via Nth Order Iterative Degradation (CVPR2022), Pranjay Shyam et al. [PDF] [Supplementary Materials]
    • Dual Heterogeneous Complementary Networks for Single Image Deraining (CVPR2022), Yuuto Nanba et al. [PDF]
    • TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions (CVPR2022), Jeya Maria Jose Valanarasu et al. [PDF] [Code]
    • Unpaired Deep Image Deraining Using Dual Contrastive Learning (CVPR2022), Chen et al. [Project] [PDF]
    • Unsupervised Deraining: Where Contrastive Learning Meets Self-similarity (CVPR2022), Ye et al. [PDF]
    • Single Image Deraining Network with Rain Embedding Consistency and Layered LSTM (WACV2022), Li et al. [PDF]
    • FLUID: Few-Shot Self-Supervised Image Deraining (WACV2022), Shyam Nandan Rai et al. [PDF] [Code]
    • Single image rain removal using recurrent scale-guide networks (Neurocomputing2022), Wang et al. [PDF]
    • Non-local channel aggregation network for single image rain removal (Neurocomputing2022), Su et al. [PDF]
  • Joint Model-driven and Data-driven
    • Deep Layer Prior Optimization for Single Image Rain Streaks Removal (ICASSP2018), Liu et al [PDF]
    • Learning bilevel layer priors for single image rain streaks removal (SP Letters 2018), Mu et al [PDF]
    • Semi-supervised transfer learning for image rain removal (CVPR2019), Wei et al [PDF] [Code]
    • Knowledge-driven deep unrolling for robust image layer separation (TNNLS2019), Liu et al.
    • A Model-driven Deep Neural Network for Single Image Rain Removal (CVPR2020), Wang et al. [PDF] [Supp] [Code]
    • Unpaired Learning for Deep Image Deraining with Rain Direction Regularizer (ICCV2021), Liu et al. [Project] [PDF]
    • Structure-Preserving Deraining with Residue Channel Prior Guidance (ICCV2021), Yi et al. [PDF] [Code]
    • Multi-Decoding Deraining Network and Quasi-Sparsity Based Training (CVPR2021), Wang et al [PDF]
    • Towards Robust Rain Removal Against Adversarial Attacks: A Comprehensive Benchmark Analysis and Beyond (CVPR2022), Yu et al. [PDF] [Code]
  • Reivew paper
    • Single image deraining: A comprehensive benchmark analysis(CVPR2019), Li et al.[PDF] [Code]
    • A Survey on Rain Removal from Video and Single Image (Arxiv2019), Wang et al. [PDF] [Code]
    • Single image deraining: From model-based to data-driven and beyond(TPAMI2020), Yang et al.[Code]

Datasets and Discriptions

*We note that:

i. RainTrainL/Rain100L and RainTrainH/Rain100H are synthesized by Yang Wenhan. Rain12600/Rain1400 is from Fu Xueyang and Rain12 is from Li Yu.

ii. In video experiment, the rain-removed results of the deep learning method are provided by the author Yang Wenhan. Really thanks!

iii. In single image experiment, we seperately retrain all the recent state-of-the-art methods via the three training datasets: RainTrainL(200 input/clean image pairs), RainTrainH(1800 pairs), and Rain12600(12600 pairs), and then evaluate their rain removal performance based on the correponding test datasets: Rain100L(100 pairs), Rain100H(100 pairs), and Rain1400(1400 pairs). Besides, the trained model obtained by RainTrainL is adpoted to predict rain-removed results of Rain12(12 pairs). Moreover, we utilize the Internet-Data(147 input images) and SPA-Data(1000 pairs) to compare the generalization ability.

iiii. In single image experiment, when training the semi-supervised method--SIRR, we always utilize Internet-Data as unsupervised samples.

Image Quality Metrics

*Please note that all quantitative results in our survey paper are computed based on Y channel.

Contact

If you have any question, please feel free to concat Hong Wang (Email: [email protected]).