What is EmoLLM
EmoLLM is a large language model focused on mental health support, providing users with emotional counseling and psychological support through multimodal emotional understanding. It combines multiple data forms such as text, images, and video, and based on advanced multi-perspective visual projection technology, captures emotional cues from different angles to more comprehensively understand the user's emotional state. EmoLLM is instruction-tuned based on a variety of open-source large language models and supports emotional tasks such as emotion recognition, intent understanding, humor detection, and hate detection.
Main Features of EmoLLM
- Understanding Users: Through dialogue interaction, it identifies users' emotional states and psychological needs.
- Emotional Support: Provides emotional support to help users relieve stress and anxiety.
- Psychological Counseling: Combining methods such as cognitive behavioral therapy, it guides users to improve emotional management and coping strategies.
- Role-Playing: According to different users' needs, it provides dialogue experiences with multiple roles (such as psychological counselor, gentle mature woman, fatherly boyfriend, etc.).
- Personalized Counseling: Based on users' feedback and progress, it provides customized psychological counseling plans.
- Mental Health Assessment: Uses scientific tools to assess users' mental state and diagnose possible psychological problems.
- Education and Prevention: Provides mental health knowledge to help users understand how to prevent psychological problems.
- Multi-Turn Dialogue Support: Through multi-turn dialogue datasets, it provides continuous psychological counseling and support.
- Social Support System: Considering the impact of family, work, community, and cultural background on mental health, it provides guidance on social support systems.
Technical Principles of EmoLLM
- Multi-perspective Visual Projection: EmoLLM uses multi-perspective visual projection technology to capture emotional cues in visual data from multiple angles. It analyzes emotional information from a single perspective and captures relationships between object features by constructing graph-based representations. By jointly mining content information and relationship information, the model can extract features more suitable for emotional tasks.
- Emotion-Guided Prompting (EmoPrompt): EmoPrompt is a technology used to guide multimodal large language models (MLLMs) to correctly reason about emotions. By introducing task-specific examples and combining the capabilities of GPT-4V to generate accurate chains of thought (Chain-of-Thought, CoT), it ensures the model's accuracy in emotional understanding.
- Multimodal Encoding: EmoLLM integrates multiple modality encoders to handle multiple inputs such as text, images, and audio. For example, it uses the CLIP-VIT-L/14 model to process visual information, the WHISPER-BASE model to process audio signals, and a text encoder based on LLaMA2-7B to process text data.
- Instruction Fine-Tuning: EmoLLM is based on advanced instruction fine-tuning techniques, such as QLORA and full fine-tuning, to finely adjust the original language model, enabling it to better adapt to the complex emotional contexts in the mental health field.
EmoLLM Project Address
- GitHub Repository：https://github.com/yan9qu/EmoLLM
- arXiv Technical Paper：https://arxiv.org/pdf/2406.16442
Application Scenarios of EmoLLM
- Mental Health Counseling: Provides users with emotional support and advice.
- Emotion Analysis: Used for social media sentiment monitoring, mental health monitoring, etc.
- Multimodal sentiment tasks: such as sentiment recognition in images and videos.