Support Lightricks LTX-Video

[hlky]

Lightricks LTX-Video

LTX-Video is the first DiT-based video generation model capable of generating high-quality videos in real-time. It produces 24 FPS videos at a 768x512 resolution faster than they can be watched. Trained on a large-scale dataset of diverse videos, the model generates high-resolution videos with realistic and varied content. We provide a model for both text-to-video as well as image+text-to-video usecases

Model
Code

Integration should be fairly straight forward as the original code is based on Diffusers, some changes will be required like rewriting usage of einops.rearrange.

[sayakpaul]

Assigning this to @a-r-r-o-w and @DN6 as they expressed interest?

[sayakpaul]

Of course, this will be super cool to have especially because fine-tuning would be relatively cheaper and will also potentially benefit from many optimization stuff we're planning to add very soon.

I wonder why does https://huggingface.co/Lightricks/LTX-Video/tree/main not contain any text encoder.

[hlky]

I wonder why does https://huggingface.co/Lightricks/LTX-Video/tree/main not contain any text encoder.

Uses T5-XXL so probably just to avoid the large upload. Original code gets it from PixArt-alpha/PixArt-XL-2-1024-MS