This demonstration page presents the generations from 50 randomly selected prompts from the AudioCaps test set.
+We present four audio sources: the consistency model fine-tuned with CLAP, + the consistency model without CLAP-fine-tuning, the diffusion baseline model, and the ground truth.
+The diffusion baseline queries the neural network 400 times per audio clip, + while the consistency models query a same-sized network only one time.
+ +| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
| Consistency model | ++ |
| Consistency model + CLAP-FT | ++ |
| Diffusion baseline (TANGO) | ++ |
| Ground truth | ++ |
Rain and thunder
+ + +A loud bang followed by an engine idling loudly
+ + +A man speaking while water runs in the background
+ + +An electric motor runs then a person speaks
+ + +A helicopter engine operating while wind blows heavily into a microphone
+ + +A sewing machine sews followed by a man talking
+ + +A woman talks briefly as several goats bleat including one that has high pitched bleats. A crunch is followed by a man speaking
+ + +High pressure liquid spraying as a radio plays in the background
+ + +Male speech and then scraping
+ + +Mechanical rotation and then a loud click occurs
+ + +A loud bang followed by an engine idling loudly
+ + +Humming from a large engine
+ + +A motor vehicle engine is revving
+ + +A bus engine driving in the distance then nearby followed by compressed air releasing while a woman and a child talk in the distance
+ + +A woman speaks, and a motor vehicle revs its engine
+ + +A vehicle accelerating then driving by as gusts of wind blow and leaves rustle in the distance
+ + +A car engine idling then starts to rev shortly after
+ + +Rain and thunder
+ + +A man talking followed by a camera muffling and footsteps shuffling then wood lightly clanking
+ + +An electric motor runs then a person speaks
+ + +A helicopter engine operating while wind blows heavily into a microphone
+ + +Mechanical rotation and then a loud click occurs
+ + +A machine motor running as a man is speaking followed by rapid buzzing
+ + +A vehicle accelerating then driving by as gusts of wind blow and leaves rustle in the distance
+ + +Train passing followed by short honk
+ + +A woman speaks, and a motor vehicle revs its engine
+ + +Several puppies yapping
+ + +A person gulping followed by glass tapping then liquid shaking in a container proceeded by liquid pouring before plastic thumps on paper
+ + +A nearby insect buzzes with nearby vibrations
+ + +A bus engine driving in the distance then nearby followed by compressed air releasing while a woman and a child talk in the distance
+ + +A bus engine driving in the distance then nearby followed by compressed air releasing while a woman and a child talk in the distance
+ + +High pressure liquid spraying as a radio plays in the background
+ + +A loud bang followed by an engine idling loudly
+ + +Mechanical rotation and then a loud click occurs
+ + +A motor vehicle engine is revving
+ + +A woman speaks, and a motor vehicle revs its engine
+ + +An electric motor runs then a person speaks
+ + +A man speaking while water runs in the background
+ + +Man talking in the wind and someone yells in the background while an engine makes squealing and air puffing sounds
+ + +A person gulping followed by glass tapping then liquid shaking in a container proceeded by liquid pouring before plastic thumps on paper
+ + +Male speech and then scraping
+ + +Mechanical rotation and then a loud click occurs
+ + +Several puppies yapping
+ + +Train passing followed by short honk
+ + +An baby laughing
+ + +Humming from a large engine
+ + +An baby laughing
+ + +A man speaking while water runs in the background
+ + +A man talking followed by a camera muffling and footsteps shuffling then wood lightly clanking
+ + +A horse gallops then trot on grass as gusts of wind blow and thunderclaps in the distance
+ + +A sewing machine sews followed by a man talking
+ + +An baby laughing
+ + +A horse gallops then trot on grass as gusts of wind blow and thunderclaps in the distance
+ + +Train passing followed by short honk
+ + +A man speaking while water runs in the background
+ + +Several puppies yapping
+ + +Several puppies yapping
+ + +A person gulping followed by glass tapping then liquid shaking in a container proceeded by liquid pouring before plastic thumps on paper
+ + +A woman talks briefly as several goats bleat including one that has high pitched bleats. A crunch is followed by a man speaking
+ + +Rain and thunder
+ + +Humming from a large engine
+ + +A car engine idling then starts to rev shortly after
+ + +High pressure liquid spraying as a radio plays in the background
+ + +A woman speaks, and a motor vehicle revs its engine
+ + +A nearby insect buzzes with nearby vibrations
+ + +Train passing followed by short honk
+ + +Rain and thunder
+ + +A bus engine driving in the distance then nearby followed by compressed air releasing while a woman and a child talk in the distance
+ + +Male speech and then scraping
+ + +An electric motor runs then a person speaks
+ + +A machine motor running as a man is speaking followed by rapid buzzing
+ + +A vehicle accelerating then driving by as gusts of wind blow and leaves rustle in the distance
+ + +A machine motor running as a man is speaking followed by rapid buzzing
+ + +A car engine idling then starts to rev shortly after
+ + +A helicopter engine operating while wind blows heavily into a microphone
+ + +A man talking followed by a camera muffling and footsteps shuffling then wood lightly clanking
+ + +A vehicle accelerating then driving by as gusts of wind blow and leaves rustle in the distance
+ + +A motor vehicle engine is revving
+ + +High pressure liquid spraying as a radio plays in the background
+ + +Man talking in the wind and someone yells in the background while an engine makes squealing and air puffing sounds
+ + +A woman talks briefly as several goats bleat including one that has high pitched bleats. A crunch is followed by a man speaking
+ + +A sewing machine sews followed by a man talking
+ + +A machine motor running as a man is speaking followed by rapid buzzing
+ + +A loud bang followed by an engine idling loudly
+ + +Man talking in the wind and someone yells in the background while an engine makes squealing and air puffing sounds
+ + +Male speech and then scraping
+ + +An baby laughing
+ + +A nearby insect buzzes with nearby vibrations
+ + +A horse gallops then trot on grass as gusts of wind blow and thunderclaps in the distance
+ + +Humming from a large engine
+ + +A nearby insect buzzes with nearby vibrations
+ + +A motor vehicle engine is revving
+ + +A car engine idling then starts to rev shortly after
+ + +A helicopter engine operating while wind blows heavily into a microphone
+ + +A horse gallops then trot on grass as gusts of wind blow and thunderclaps in the distance
+ + +A man talking followed by a camera muffling and footsteps shuffling then wood lightly clanking
+ + +A sewing machine sews followed by a man talking
+ + +A person gulping followed by glass tapping then liquid shaking in a container proceeded by liquid pouring before plastic thumps on paper
+ + +Man talking in the wind and someone yells in the background while an engine makes squealing and air puffing sounds
+ + +A woman talks briefly as several goats bleat including one that has high pitched bleats. A crunch is followed by a man speaking
+ + ++ Diffusion models power a vast majority of the text-to-audio generation methods. + Unfortunately, diffusion models suffer from a slow inference speed due to iteratively querying the + underlying denoising network, thus unsuitable for applications with time or computational constraints. + This work modifies the recently proposed "consistency distillation" framework to train text-to-audio + models that only require a single neural network query, accelerating the generation hundreds of times. +
++ By incorporating classifier-free guidance into the distillation framework, our models retain + diffusion models' impressive generation quality and diversity. Furthermore, the non-recurrent + differentiable structure resulting from the distillation allows fine-tuning with novel loss functions. + We use the CLAP loss as an example, confirming that end-to-end fine-tuning + further boosts the generation quality. +
+@article{bai2023accelerating,
+ author = {Yatong Bai, Trung Dang, Dung Tran, Kazuhito Koishida, Somayeh Sojoudi},
+ title = {Accelerating Diffusion-Based Text-to-Audio Generation with Consistency Distillation},
+ journal={arXiv preprint arXiv:2309.10740},
+ year = {2023}
+}
+ + For any questions regarding our work, please email + yatong_bai@berkeley.edu. + We are more than happy to help with implenting our method and verifying our results. +
+ */
+ font-family: inherit; /* Ensures font consistency */
+ font-size: .9em;
+ white-space: pre-wrap; /* CSS3 */
+ white-space: -moz-pre-wrap; /* Firefox */
+ white-space: -pre-wrap; /* Opera <7 */
+ white-space: -o-pre-wrap; /* Opera 7 */
+ word-wrap: break-word; /* IE */
+}
+
+.copy-icon {
+ position: absolute;
+ top: 10px;
+ right: 10px;
+ width: 20px; /* or adjust as needed */
+ color: #000; /* Color of the icon */
+ background-color: transparent; /* No fill */
+}
+
+.bibtex:hover {
+ background-color: #e8e8e8; /* subtle hover effect */
+}
+
+.bibtex:hover .copy-icon {
+ color: #007BFF; /* Change color on hover for better user feedback */
+}