Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Lightricks LTX-Video #9995

Open
hlky opened this issue Nov 22, 2024 · 3 comments
Open

Support Lightricks LTX-Video #9995

hlky opened this issue Nov 22, 2024 · 3 comments

Comments

@hlky
Copy link
Contributor

hlky commented Nov 22, 2024

Lightricks LTX-Video

LTX-Video is the first DiT-based video generation model capable of generating high-quality videos in real-time. It produces 24 FPS videos at a 768x512 resolution faster than they can be watched. Trained on a large-scale dataset of diverse videos, the model generates high-resolution videos with realistic and varied content. We provide a model for both text-to-video as well as image+text-to-video usecases

Model
Code

Integration should be fairly straight forward as the original code is based on Diffusers, some changes will be required like rewriting usage of einops.rearrange.

@sayakpaul
Copy link
Member

Assigning this to @a-r-r-o-w and @DN6 as they expressed interest?

@sayakpaul
Copy link
Member

Of course, this will be super cool to have especially because fine-tuning would be relatively cheaper and will also potentially benefit from many optimization stuff we're planning to add very soon.

I wonder why does https://huggingface.co/Lightricks/LTX-Video/tree/main not contain any text encoder.

@hlky
Copy link
Contributor Author

hlky commented Nov 23, 2024

I wonder why does https://huggingface.co/Lightricks/LTX-Video/tree/main not contain any text encoder.

Uses T5-XXL so probably just to avoid the large upload. Original code gets it from PixArt-alpha/PixArt-XL-2-1024-MS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants