Skip to content

Commit

Permalink
Added diversity page
Browse files Browse the repository at this point in the history
  • Loading branch information
Bai-YT committed Nov 22, 2023
1 parent 3a88897 commit f8dfc56
Show file tree
Hide file tree
Showing 206 changed files with 1,770 additions and 7 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
*.DS_Store
replace.py
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ The webpage includes a [demo page](https://consistency-tta.github.io/demo.html)
The training and inference code will be added soon.


### Main experiment result
### Main Experiment results

Our method reduce the computation of the core step of diffusion-based text-to-audio generation by a factor of 400, while observing minimal performance degradation in terms of Fréchet Audio Distance (FAD), Fréchet Distance (FD), KL Divergence, and CLAP Scores.

Expand All @@ -20,7 +20,7 @@ Our method reduce the computation of the core step of diffusion-based text-to-au
[This benchmark](https://paperswithcode.com/sota/audio-generation-on-audiocaps) demonstrates how our single-step models stack up with previous methods, most of which mostly require hundreds of generation steps.


### Cite our work (BibTeX)
### Cite Our Work (BibTeX)

```bibtex
@article{bai2023accelerating,
Expand Down
Binary file added compare_seed/seed_0/output_0.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_1.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_10.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_11.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_12.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_13.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_14.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_15.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_16.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_17.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_18.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_19.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_2.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_20.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_21.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_22.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_23.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_24.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_25.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_26.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_27.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_28.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_29.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_3.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_30.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_31.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_32.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_33.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_34.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_35.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_36.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_37.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_38.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_39.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_4.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_40.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_41.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_42.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_43.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_44.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_45.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_46.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_47.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_48.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_49.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_5.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_6.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_7.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_8.wav
Binary file not shown.
Binary file added compare_seed/seed_0/output_9.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_0.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_1.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_10.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_11.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_12.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_13.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_14.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_15.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_16.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_17.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_18.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_19.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_2.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_20.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_21.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_22.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_23.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_24.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_25.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_26.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_27.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_28.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_29.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_3.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_30.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_31.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_32.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_33.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_34.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_35.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_36.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_37.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_38.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_39.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_4.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_40.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_41.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_42.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_43.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_44.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_45.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_46.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_47.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_48.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_49.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_5.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_6.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_7.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_8.wav
Binary file not shown.
Binary file added compare_seed/seed_20230817/output_9.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/all_mels.pt
Binary file not shown.
Binary file added compare_seed/seed_25252/output_0.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_1.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_10.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_11.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_12.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_13.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_14.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_15.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_16.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_17.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_18.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_19.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_2.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_20.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_21.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_22.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_23.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_24.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_25.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_26.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_27.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_28.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_29.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_3.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_30.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_31.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_32.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_33.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_34.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_35.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_36.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_37.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_38.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_39.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_4.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_40.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_41.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_42.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_43.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_44.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_45.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_46.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_47.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_48.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_49.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_5.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_6.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_7.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_8.wav
Binary file not shown.
Binary file added compare_seed/seed_25252/output_9.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_0.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_1.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_10.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_11.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_12.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_13.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_14.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_15.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_16.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_17.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_18.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_19.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_2.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_20.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_21.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_22.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_23.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_24.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_25.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_26.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_27.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_28.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_29.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_3.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_30.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_31.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_32.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_33.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_34.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_35.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_36.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_37.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_38.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_39.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_4.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_40.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_41.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_42.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_43.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_44.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_45.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_46.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_47.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_48.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_49.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_5.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_6.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_7.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_8.wav
Binary file not shown.
Binary file added compare_seed/seed_45510/output_9.wav
Binary file not shown.
10 changes: 5 additions & 5 deletions demo.html
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ <h3><b>Accelerating Diffusion-Based Text-to-Audio Generation with Consistency Di
the consistency model without CLAP-fine-tuning, the diffusion baseline model, and the ground truth.</p>
<p><b>The diffusion baseline queries the neural network 400 times per audio clip,
while the consistency models query a same-sized network only one time.</b></p>
<p>Since the models are not trained on speech data, we do not expect them to produce meaningful speeaches.</p>
<p>Since the models are not trained on speech data, we do not expect them to produce meaningful speeches.</p>

<hr>
<h3>Prompt 0</h3>
Expand Down Expand Up @@ -415,7 +415,7 @@ <h4>A telephone ringing with loud echo.</h4>

<hr>
<h3>Prompt 11</h3>
<h4>Released air hissing followed by a popping explosion then a metal ding persists as a person is laughing and a man is talking..</h4>
<h4>Released air hissing followed by a popping explosion then a metal ding persists as a person is laughing and a man is talking.</h4>
<table>
<tr>
<td class="demo-data">Consistency model</td>
Expand Down Expand Up @@ -483,7 +483,7 @@ <h4>Constant hissing with mean having conversation.</h4>

<hr>
<h3>Prompt 13</h3>
<h4>A missile launching followed by an explosion and metal screeching as a motor hums in the background..</h4>
<h4>A missile launching followed by an explosion and metal screeching as a motor hums in the background.</h4>
<table>
<tr>
<td class="demo-data">Consistency model</td>
Expand Down Expand Up @@ -993,7 +993,7 @@ <h4>A person speaks with distant humming and nearby clinking.</h4>

<hr>
<h3>Prompt 28</h3>
<h4>A dog whimpering followed by laughing and barking..</h4>
<h4>A dog whimpering followed by laughing and barking.</h4>
<table>
<tr>
<td class="demo-data">Consistency model</td>
Expand Down Expand Up @@ -1571,7 +1571,7 @@ <h4>A jackhammer drilling and vibrating continuously.</h4>

<hr>
<h3>Prompt 45</h3>
<h4>A train is passing by and sound its whistle..</h4>
<h4>A train is passing by and sound its whistle.</h4>
<table>
<tr>
<td class="demo-data">Consistency model</td>
Expand Down
1,752 changes: 1,752 additions & 0 deletions diversity.html

Large diffs are not rendered by default.

10 changes: 10 additions & 0 deletions index.html
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,16 @@ <h2>Main Experiment Results</h2>
</p>
</section>

<section class="section">
<h2>Generation Diversity</h2>
<p>
Consistency models demonstrate non-trivial generation diversity, as do diffusion models.
In <a href="diversity.html">this page</a>, we present 50 groups of generations from
four different random seeds to demonstrate this diversity, showing that our method
combines the diversity of diffusion models and the efficiency of single-step models.
</p>
</section>

<section class="section">
<h2>BibTeX</h2>
<div id="bibtex1" class="bibtex" onclick="copyToClipboard('bibtex1')">
Expand Down

0 comments on commit f8dfc56

Please sign in to comment.