What is Seaweed APT
Seaweed APT is an Adversarial Post-Training model launched by ByteDance that enables one-stop generation of images and videos. Seaweed APT is based on a pretrained diffusion model and directly performs adversarial training on real data, rather than using a pretrained model as a teacher to generate targets, achieving high-quality image and video output in single-step generation. Seaweed APT introduces multiple improvements, including a generator initialized by deterministic distillation, an enhanced discriminator architecture, and an approximate R1 regularization objective, improving training stability and generation quality. Experiments show that Seaweed APT can generate 1024px images and 2-second, 1280×720, 24fps videos in a single step, and surpasses traditional multi-step diffusion models in visual fidelity.
Main features of Seaweed APT
- Single-step high-quality image generation: It can generate 1024px resolution images, and is comparable to multi-step diffusion models in visual fidelity, details, and structural integrity.
- Single-step high-resolution video generation: It achieves 1280×720, 24fps, 2-second-long video generation for the first time, significantly improving the efficiency and quality of video generation.
- Real-time generation capability: On a single H100 GPU, Seaweed APT can generate a 2-second 1280×720 24fps video within 6.03 seconds, and based on parallelization with 8 H100 GPUs, the entire generation process achieves real-time processing.
- Surpassing pretrained models: By directly performing adversarial training on real data, Seaweed APT avoids the high cost of precomputing video samples and surpasses pretrained diffusion models on certain evaluation criteria, especially in realism, details, and exposure issues.
Technical principles of Seaweed APT
- Pretrained diffusion model initialization: It uses a pretrained diffusion transformer (DiT) as initialization, avoiding the high cost of precomputing video samples, and surpasses pretrained models on certain evaluation criteria.
- Adversarial training: It trains on real data based on an adversarial training objective, directly training on real data to generate samples closer to the real distribution, improving generation quality and realism.
- Generator design: It adopts a deterministic distillation initialization generator, initialized with discrete-time consistency distillation and mean squared error loss. During training, the generator mainly focuses on single-step generation capability and always inputs the final time step T.
- Discriminator design: The discriminator is initialized with a pretrained diffusion network and operates directly in latent space. The discriminator contains 36 layers of Transformer blocks, with a total parameter count of 8 billion. By introducing new cross-attention blocks at multiple time steps, the discriminator can better distinguish real samples from generated samples.
- Approximate R1 regularization: To solve the high-order gradient computation problem of R1 regularization in large-scale Transformer models, an approximation method is introduced. Gaussian noise with small variance is added to real data, reducing the discriminator's gradient on real data and achieving an objective consistent with the original R1 regularization.
- Training details: It is first trained on images, and then trained on videos. A large number of H100 GPUs and gradient accumulation are used to reach a larger batch size, improving training stability and structural integrity. An exponential moving average (EMA) decay rate is used to stabilize the training process, and the learning rate is adjusted during training to ensure training stability.
Seaweed APT's project address
- Project official website：https://seaweed-apt.com
- arXiv technical paper：https://arxiv.org/pdf/2501.08316
Seaweed APT's application scenarios
- Video advertising production: Quickly generate high-quality, diverse-style video ads, reducing production costs and time.
- Film and television content creation: Generate video clips in specific styles for movies and TV series, improving creative efficiency and content diversity.
- Social media content: Generate personalized, high-quality videos and images, enhancing user engagement and content appeal.
- Game development: Quickly generate images of characters, scenes, and props in games, accelerating the game development process.
- Education and training: Generate educational videos and training materials, improving teaching effectiveness and training quality.