What is DiffSensei
DiffSensei was jointly launched by researchers from Peking University, Shanghai AI Laboratory, and Nanyang Technological Universitycomic generationframework capable of generating controllable black-and-white comic panels. DiffSensei integrates a diffusion-based image generator and a multimodal large language model (MLLM) to achieve precise control over the appearance and interactions of multiple characters in comics. The framework uses a masked cross-attention mechanism and an MLLM adapter to dynamically adjust character features based on text prompts, including expressions, poses, and actions, generating coherent and visually appealing comic panels. DiffSensei introduces the MangaZero dataset to support multi-character, multi-state comic generation tasks.
Main features of DiffSensei
- Customized comic generation: Generates comics based on character images and text prompts provided by users, supporting user customization of characters' appearance, expressions, and actions.
- Multi-character control: The framework supports comic generation in multi-character scenes, handling interactions and layout among characters.
- Text-compatible identity adaptation: Based on MLLM, dynamically adjusts character features according to text prompts, making characters' performances match the text descriptions.
- Precise layout control: Uses a masked cross-attention mechanism to precisely control the layout of characters and dialogue without direct pixel transmission.
- Dataset support: Introduces the MangaZero dataset, a large-scale annotated dataset designed for multi-character, multi-state comic generation tasks.
Technical principles of DiffSensei
- Integrating MLLM and diffusion models: Combines MLLM as a text-compatible identity adapter with a diffusion-based image generator to generate customized comic panels.
- Masked Cross-Attention: Copies the key and value matrices to create independent character cross-attention layers in each cross-attention layer, achieving precise control of character layout.
- Dialog Layout Encoding: Introduces a trainable embedding layer to represent dialogue layout, combining dialogue embeddings with noise latent representations to encode dialogue positions.
- MLLM as a feature adapter: MLLM takes source character features and panel captions as input, generates target character features compatible with the text, and dynamically adjusts character states.
- Multi-character feature extraction: Uses CLIP and an image encoder to extract local image features and image-level features, processing based on a feature extractor to avoid directly encoding fine-grained spatial features from reference images.
- Diffusion loss and language model loss: When training MLLM, calculates the language model loss (LM Loss) to constrain the output format, the mean squared error loss (MSE Loss) to guide target character features based on panel captions, and the diffusion loss to ensure that the edited features remain consistent with the image generator.
Project addresses of DiffSensei
- Project official website：jianzongwu.github.io/projects/diffsensei
- GitHub repository：https://github.com/jianzongwu/DiffSensei
- arXiv technical paper：https://arxiv.org/pdf/2412.07589
Application scenarios of DiffSensei
- Comic creation: Artists and comic creators generate comic pages, quickly realizing the transformation from scripts to visual narratives and improving creation efficiency.
- Personalized content generation: Users upload character images based on their own story ideas to generate personalized comic content for personal entertainment or social media sharing.
- Education and Training: In the field of education, create visual stories that match the teaching content to help students better understand and remember complex concepts.
- Film and Game Pre-production: In film production and game design, as a pre-production tool, quickly generate storyboards or concept art to facilitate early-stage ideation and visual development.
- Advertising and Marketing: Marketers create engaging comic advertisements to attract target audiences in novel ways.