Latte is a Latent Diffusion Transformer for video generation.
Uses Latte targets video generation tasks, including unconditional video generation, class-based conditional generation, prompt-based conditional generation, and text-to-video (T2V) generation.
Main capabilities Latte first extracts spatiotemporal tokens from input videos, and then uses a series of Transformer blocks to model the video distribution in latent space. To model the large number of tokens extracted from videos, Latte introduces four efficient variants from the perspective of decomposing the spatial and temporal dimensions of input videos. To improve generated video quality, this work determines the best practices for Latte through rigorous experimental analysis, covering video clip patch embedding, model variants, timestep-class information injection, temporal positional embedding, and learning strategies.
Generation task performance Unconditional generation: unconditional video generation is performed on the Taichi-HD (256 x 256), FaceForensics (256 x 256), and SkyTimelapse (256 x 256) datasets. Class-based conditional generation: given a class, Latte can generate the desired video, with results shown on the UCF101 (256 x 256) dataset. Prompt-based conditional generation: Latte is used to generate the desired video, with example prompts including "Yellow and black tropical fish dart through the sea.", "An epic tornado attacking above aglowing city at night.", "Slow pan upward of blazing oak fire in an indoor fireplace.", "a cat wearing sunglasses and working as a lifeguard at pool.", "Sunset over the sea.", "A dog in astronaut suit and sunglasses floating in space.".
Comparison with other state-of-the-art methods On UCF101, Taichi-HD, FaceForensics, and SkyTimelapse datasets, visual comparisons were conducted with other state-of-the-art methods respectively.
Comprehensive evaluation Comprehensive evaluation shows that Latte achieved state-of-the-art performance at the time on four standard video generation datasets: FaceForensics, SkyTimelapse, UCF101, and Taichi-HD. In addition, Latte was extended to the text-to-video generation (T2V) task, on which it achieved results comparable to recent T2V models. This work is considered to provide valuable insights for future research on integrating Transformers into diffusion models for video generation.
Usage and limitations The provided materials do not specify Latte's concrete installation commands, code invocation methods, online trial entry point, license agreement, or directly actionable steps; this information is not currently provided. The materials only include the research introduction, generation capability description, and example prompts, and use paper citations as a request for acknowledgment from users.
