Note: We provide a comprehensive survey on various image synthesis tasks under limited data, to help the community track the evolution of this field.
Image Synthesis under Limited Data: A Survey and Taxonomy
Mengping Yang, Zhe Wang*
https://arxiv.org/abs/2307.16879
(* denotes corresponding author)
This repository collects the studies on image synthesis under limited data. According to the problem definition, requirements, training, and testing schemes, image synthesis under limited data can be roughly categoried into four groups: namely Data-efficient generative models, Few-shot generative adaptation, Few-shot image generation, and One-shot image generation.
Overall, this paper collection is organized as follows. If you find some work is missing, feel free to raise an issue or create a pull request. We appreciate contributions in any form.
- Data-efficient generative models
- Few-shot generative adaptation
- Few-shot image generation
- One-shot image generation
- Other related awesome surveys
Problem Definition: Data efficient generative models refer to the scenario where a generative model is trained on a limited amount of training data, such as 100 images from 100-shot datasets[DiffAug] or 1316 images from the MetFace[StyleGAN-ADA], to produce diverse and plausible images that follow the given distribution. However, several issues such as model overfitting and memorization are prone to occur when training a generative model on limited samples
- DiffAug: Differentiable augmentation for data-efficient gan training
NeurIPS 2020 / Code - ADA: Training generative adversarial networks with limited data
NeurIPS 2020 / Code - APA: Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data
NeurIPS 2021 / Code - ContraD: Training gans with stronger augmentations via contrastive discriminator
ICLR 2021 / Code - On data augmentation for gan training
TIP 2021 / Code - Masked generative adversarial networks are data-efficient generation learners
NeurIPS 2022 / Code - Adaptive Feature Interpolation for Low-Shot Image Generation
ECCV 2022 / Code
- Regularizing generative adversarial networks under limited data
CVPR 2021 / Code - Consistency Regularization for Generative Adversarial Networks
ICLR 2020 / Code - Improved consistency regularization for gans
AAAI 2021 / Code - ProtoGAN: Towards high diversity and fidelity image synthesis under limited data
Information Sciences 2023 / Code - Feature statistics mixing regularization for generative adversarial networks
CVPR 2022 / Code - Data-efficient instance generation from instance discrimination
NeurIPS 2021 / Code - DigGAN: Discriminator gradIent Gap Regularization for GAN Training with Limited Data
NeurIPS 2022 / Code - FakeCLR: Exploring contrastive learning for solving latent discontinuity in data-efficient GANs
ECCV 2022 / Code - Few-shot image generation with mixup-based distance learning
ECCV 2022 / Code - Improving GAN Training via Feature Space Shrinkage
CVPR 2023 / Code
- Towards faster and stabilized gan training for high-fidelity few-shot image synthesis
ICLR 2021 / Code - Prototype memory and attention mechanisms for few shot image generation
ICLR 2022 / Code - FreGAN: Exploiting Frequency Components for Training GANs under Limited Data
NeurIPS 2022 / Code - AutoInfo GAN: Toward a better image synthesis GAN framework for high-fidelity few-shot datasets via NAS and contrastive learning
KBS 2023 / Code - GenCo: generative co-training for generative adversarial networks with limited data
AAAI 2022 / Code - Data-efficient gan training beyond (just) augmentations: A lottery ticket perspective
NeurIPS 2021 / Code - Improving gans with a dynamic discriminator
NeurIPS 2022 / Code - Re-GAN: Data-Efficient GANs Training via Architectural Reconfiguration
CVPR 2023 / Code
- Projected gans converge faster
NeurIPS 2021 / Code - Ensembling off-the-shelf models for gan training
CVPR 2022 / Code - KD-DLGAN: Data Limited Image Generation via Knowledge Distillation
CVPR 2023 / Code - Data instance prior (disp) in generative adversarial networks
WACV 2022 / Code - Stylegan-xl: Scaling stylegan to large diverse datasets
SIGGRAPH 2022 / Code
Problem Definition: Akin to transfer learning, the goal of few-shot generative adaptation is to transfer the knowledge of pre-trained generative models from large-scale source domains (e.g., FFHQ) to target domains with limited data (e.g., 10-shot images of baby faces) in a fast and efficient manner. Ideally, the adapted generative model should 1) inherent the attributes of the source generative models that are invariant to the distribution shift, such as the overall structure, synthesis diversity, and semantic variances of generated images, and 2) capture the internal distribution of the target domain to synthesize novel samples following the target distribution. However, the limited amount of training data available for adaptation may cause the model to overfit, leading to model degradation. Additionally, when the domain gaps between the source domain and the target domain are significant, negative transfer may occur, resulting in unrealistic generation. Furthermore, inappropriate knowledge transfer may also lead to a deterioration in synthesis performance.
- Transferring gans: generating images from limited data
ECCV 2018 / Code - Image generation from small datasets via batch statistics adaptation
ICCV 2019 / Code - On leveraging pretrained GANs for generation with limited data
ICML 2020 / Code - Few-shot Image Generation with Elastic Weight Consolidation
NeurIPS 2020 / Code - Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs
CVPR WorkShop 2020 / Code - Few-shot adaptation of generative adversarial networks
ArXiv 2020 / Code - Fine-tuning Diffusion Models with Limited Data
NeurIPS WorkShop 2020
- Minegan: effective knowledge transfer from gans to target domains with few images
CVPR 2020 / Code - One-shot generative domain adaptation
ICCV 2023 / Code - Domain Re-Modulation for Few-Shot Generative Domain Adaptation
ArXiv 2023 / Code - Few-shot image generation with diffusion models
ArXiv 2022
- Few-shot image generation via cross-domain correspondence
CVPR 2021 / Code - Few shot generative model adaption via relaxed spatial structural alignment
CVPR 2022 / Code - Dynamic Weighted Semantic Correspondence for Few-Shot Image Generative Adaptation
ACM MM 2022 - A closer look at few-shot image generation
CVPR 2022 / Code - Towards diverse and faithful one-shot adaption of generative adversarial networks
NeurIPS 2022 / Code - Generalized One-shot Domain Adaptation of Generative Adversarial Networks
NeurIPS 2022 / Code - Few Shot Generative Domain Adaptation via Inference-Stage Latent Learning in GANs
NeurIPS Workshop 2022 / Code - WeditGAN: Few-shot Image Generation via Latent Space Relocation
ArXiv 2023 - Smoothness Similarity Regularization for Few-Shot GAN Adaptation
ArXiv 2023 - Diffusion guided domain adaptation of image generators
ArXiv 2022 / Code
- Few-shot image generation via adaptation-aware kernel modulation
NeurIPS 2022 / Code - Exploring incompatible knowledge transfer in few-shot image generation
CVPR 2023 / Code
**Problem Definition:**Following the prior philosophy of few-shot learning, few-shot image generation is formulated to synthesize diverse and photorealistic images for a new category given
- FIGR: Few-shot image generation with reptile
ArXiv 2019 / Code - DAWSON: A domain adaptive few shot generation framework
ArXiv 2020 - Fast adaptive meta-learning for few-shot image generation
TMM 2021 / Code - Few-shot image generation based on contrastive meta-learning generative adversarial network
The Visual Computer 2022
- Data augmentation generative adversarial networks
ArXiv 2017 / Code - Delta{GAN}: Towards diverse few-shot image generation with sample-specific delta
ECCV 2022 / Code - Few Shot Image Generation via Implicit Autoencoding of Support Sets
NeurIPS Workshop 2021 - Few-shot Image Generation Using Discrete Content Representation
ACM MM 2022 - Learning To Memorize Feature Hallucination for One-Shot Image Generation
CVPR 2022
- Matchinggan: Matching-Based Few-Shot Image Generation
ICME 2020 / Code - Few-shot generative modelling with generative matching networks
AISTATS 2018 - F2GAN: Fusing-and-Filling GAN for Few-shot Image Generation
ACM MM 2020 / Code - Lofgan: Fusing local representations for few-shot image generation
ICCV 2021 / Code - WaveGAN: Frequency-Aware GAN for High-Fidelity Few-Shot Image Generation
ECCV 2022 / Code - AMMGAN: adaptive multi-scale modulation generative adversarial network for few-shot image generation
Applied Intelligence 2023 - EqGAN: Feature Equalization Fusion for Few-shot Image Generation
ArXiv 2023
- Attribute Group Editing for Reliable Few-shot Image Generation
CVPR 2022 / Code - Stable Attribute Group Editing for Reliable Few-shot Image Generation
ArXiv 2023 / Code - The Euclidean Space is Evil: Hyperbolic Attribute Editing for Few-shot Image Generation
ArXiv 2022 / Code - Where Is My Spot? Few-Shot Image Generation via Latent Subspace Optimization
CVPR 2023 / Code
- Few-shot diffusion models
ArXiv 2022
**Problem Definition:**One-shot image generation refers to the task of training a generative model to produce novel and diverse images using only a single reference image, without the use of any pre-trained generative models for knowledge transfer. This task is of significant importance as it demonstrates the potential application of generative models in practical domains where collecting large-scale training samples is not feasible. Consequently, the model is expected to capture the internal distribution of the training image and generate diverse images that share the same internal distribution as the reference image, which is an extremely challenging task. Intuitively, synthesizing images directly from only one image presents a risk of low-variation generation, as the model may simply replicate the given sample. However, existing methods address this issue by modeling the internal statistics of patches within the training image, allowing the model to capture the information of the target distribution.
- Singan: Learning a generative model from a single natural image
ICCV 2019 / Code - SA-SinGAN: self-attention for single-image generation adversarial networks
Machine Vision and Applications 2021 - ExSinGAN: Learning an explainable generative model from a single image
ArXiv 2021 / Code - Recurrent SinGAN: Towards scale-agnostic single image GANs
Proceedings of the 2021 5th International Conference on Electronic Information Technology and Computer Engineering 2021 - Improved techniques for training single-image gans
WACV 2021 / Code - One-shot gan: Learning to generate samples from single images and videos
CVPR 2021 / Code - Petsgan: Rethinking priors for single image generation
AAAI 2022 / Code - TcGAN: Semantic-Aware and Structure-Preserved GANs with Individual Vision Transformer for Fast Arbitrary One-Shot Image Generation
ArXiv 2023
- Sindiffusion: Learning a diffusion model from a single natural image
ArXiv 2022 / Code - Sinddm: A single image denoising diffusion model
ICML 2023 / Code - SinFusion: Training Diffusion Models on a Single Image or Video
ICML 2023 / Code
- Drop the gan: In defense of patches nearest neighbors as single image generative models
CVPR 2022 / Code - A Patch-Based Algorithm for Diverse and High Fidelity Single Image Generation
ICIP 2022 - Generating natural images with direct patch distributions matching
ECCV 2022 / Code
GAN Inversion:
- GAN Inversion: A Survey
TPAMI 2022 / Code
3D-aware GANs:
- Awesome Text-to-Image Diffusion papers
Code - A Survey on 3D-aware Image Synthesis
ArXiv 2022 / Code
Diffusion Models:
- Diffusion Models: A Comprehensive Survey of Methods and Applications
ArXiv 2022 / Code - A Survey on Generative Diffusion Model
ArXiv 2022 / Code - Awesome Video-generation/editing diffusion papers
Code - Awesome Text-to-Image diffusion papers
Code
We would continuously update the relevant papers and resources, stay tuned!
@article{yang2023image,
title={Image Synthesis under Limited Data: A Survey and Taxonomy},
author={Mengping Yang and Zhe Wang},
year={2023},
eprint={2307.16879},
archivePrefix={arXiv}
}