Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mochi docs #9934

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Mochi docs #9934

Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
dd65ae5
update
DN6 Nov 15, 2024
ee28596
update
DN6 Nov 15, 2024
9ba4aaf
Merge branch 'main' into mochi-docs
sayakpaul Nov 16, 2024
fc7d50b
Merge branch 'main' into mochi-docs
sayakpaul Nov 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 45 additions & 1 deletion docs/source/en/api/pipelines/mochi.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# limitations under the License.
-->

# Mochi
# Mochi 1 Preview

[Mochi 1 Preview](https://huggingface.co/genmo/mochi-1-preview) from Genmo.

Expand All @@ -25,6 +25,50 @@ Make sure to check out the Schedulers [guide](../../using-diffusers/schedulers.m

Copy link

@Ednaordinary Ednaordinary Nov 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(One line above this) Only FlowMatchEulerDiscreteScheduler has invert_sigmas, so anything else wouldn't work as of now as I understand it

</Tip>

## Generating videos with Mochi-1 Preview

The following example will download the full precision `mochi-1-preview` weights and produce the highest quality results but will require at least 42GB VRAM to run.

```python
import torch
from diffusers import MochiPipeline
from diffusers.utils import export_to_video

pipe = MochiPipeline.from_pretrained("genmo/mochi-1-preview")

# Enable memory savings
pipe.enable_model_cpu_offload()
pipe.enable_vae_tiling()

prompt = "Close-up of a chameleon's eye, with its scaly skin changing color. Ultra high resolution 4k."

with torch.autocast("cuda", torch.bfloat16, cache_enabled=False):
frames = pipe(prompt, num_frames=84).frames[0]
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

num_frames=84 should be num_frames=85, (14 * 6 + 1) like mentioned here


export_to_video(frames, "mochi.mp4", fps=30)
```

## Using a lower precision variant to save memory

The following example will use the `bfloat16` variant of the model and requires 22GB VRAM to run. There is a slight drop in the quality of the generated video as a result.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should bf16 be used? fp16 seems to have better precision, unless the autocasted weights compute in fp32 and somehow provide better precision?

Copy link
Collaborator Author

@DN6 DN6 Nov 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We followed the official repo, which cast to bf16.
https://github.com/genmoai/models/blob/1a96b5df0c83018fcbe7aebbbb1b34781e179921/src/genmo/mochi_preview/pipelines.py#L344

I haven't fully compared bf16 vs fp16 for full downcasting though. Are you seeing much better results with fp16?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am indeed, though I'm also doing mixed weight precision between float8 and f16. It could be beneficial to test further.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can keep it to bfloat16 for now (as done in this PR) following the official recommendations. Then @Ednaordinary you could maybe help us opening a PR to the docs including your findings? WDYT?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, that works for the moment


```python
import torch
from diffusers import MochiPipeline
from diffusers.utils import export_to_video

pipe = MochiPipeline.from_pretrained("genmo/mochi-1-preview", variant="bf16", torch_dtype=torch.bfloat16)

# Enable memory savings
pipe.enable_model_cpu_offload()
pipe.enable_vae_tiling()

prompt = "Close-up of a chameleon's eye, with its scaly skin changing color. Ultra high resolution 4k."
frames = pipe(prompt, num_frames=84).frames[0]
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same thing, num_frames=85


export_to_video(frames, "mochi.mp4", fps=30)
```

## MochiPipeline

[[autodoc]] MochiPipeline
Expand Down
Loading