What is Follow-Your-Canvas
Follow-Your-Canvas is a high-resolution video outpainting technology jointly launched by the Tencent Hunyuan team and institutions such as Tsinghua University. Through two core designs, distributed processing and layout alignment, it extends video content to any resolution while maintaining seamless connection with the original video and high-quality content generation. It is suitable for high-resolution video extension, can overcome GPU memory limitations, handles large-scale video outpainting tasks, and achieves optimal results across various resolution and aspect ratio configurations.
Main Features of Follow-Your-Canvas
- High-resolution output: Extends video content to any resolution, for example from 4K to 8K or higher.
- Not limited by VRAM size: Handles large-scale video outpainting tasks without being limited by VRAM size.
- Spatiotemporal consistency: During the extension process, maintains the spatial and temporal consistency of the video, ensuring that the final output video quality is consistent with the style of the original video.
- Generates rich new content: Generates new content consistent with the style of the original video in the specified extension areas, enhancing the overall visual effect.
- Large-scale video outpainting: Performs excellently in large-scale video outpainting, for example extending a 512×512 resolution video to 1152×2048 (about 9 times) resolution.
Technical Principles of Follow-Your-Canvas
- Spatial window partitioning: The video is divided into multiple spatial windows, each window independently performs content generation, and then they are seamlessly merged, making it possible to handle videos of any size and resolution without being limited by GPU memory.
- Layout Encoder: Uses the layout encoder to extract the global layout information of the source video and inject it into the generation process of each window, ensuring that the newly generated content is coordinated with the layout of the original video.
- Relative Region Embedding (RRE): Provides the relative positional relationship between the source video and the target window, further guiding the generation process of each window, generating extended content coordinated with the layout of the original video, and improving spatial and temporal consistency.
- Distributed generation: Processes the content generation of each window in parallel on multiple GPUs, and finally seamlessly merges the generated windows into the final video.
Project Address of Follow-Your-Canvas
- Project official website：follow-your-canvas.github.io
- GitHub repository：https://github.com/mayuelala/FollowYourCanvas
- arXiv technical paper：https://arxiv.org/pdf/2409.01055
Application Scenarios of Follow-Your-Canvas
- Video size adjustment: Extends a vertical video shot with a phone into a horizontal video, adapting to different playback needs, such as display on social media or viewing on a widescreen TV.
- Ultra-wide screen video generation: Extends ordinary videos into ultra-wide screen videos, providing a broader field of view, suitable for playback in cinemas or on ultra-wide monitors.
- Panoramic video production: Based on video extension technology, converts ordinary videos into panoramic videos, enhancing the viewing experience, suitable for virtual reality (VR) and 360-degree video production.
- High-resolution video output: Can handle video output up to 16K resolution, suitable for occasions requiring high-definition video, such as film production or high-quality advertising films