What is HealthGPT
HealthGPT is an advanced medical vision-language model (Med-LVLM) jointly developed by multiple institutions including Zhejiang University, the University of Electronic Science and Technology of China, and Alibaba. It achieves a unified framework for medical visual understanding and generation tasks through heterogeneous knowledge adaptation technology. It adopts the innovative Heterogeneous Low-Rank Adaptation (H-LoRA) technology, storing the knowledge for visual understanding and generation tasks in independent "plugins" to avoid conflicts between tasks. HealthGPT offers two versions: HealthGPT-M3 (3.8 billion parameters) and HealthGPT-L14 (14 billion parameters), based respectively on Phi-3-mini and Phi-4 pre-trained language models. The model introduces Hierarchical Visual Perception (HVP) and a Three-Stage Learning Strategy (TLS) to optimize the learning of visual features and task adaptation capabilities.
Main Features of HealthGPT
- Medical Image Analysis and Diagnostic Assistance: HealthGPT can process various medical images (such as X-rays, CT, MRI, etc.), helping doctors interpret imaging results and providing diagnostic suggestions.
- Visual Question Answering: The model can answer related questions based on medical images, such as explaining abnormalities or lesion locations in the images.
- Medical Text Understanding and Generation: HealthGPT can process and generate medical texts, such as medical record summaries, diagnostic reports, etc., helping doctors quickly organize and record patient information.
- Multimodal Fusion: By combining visual information with textual information, HealthGPT can more comprehensively understand complex medical scenarios and provide more accurate diagnostic and treatment suggestions.
- Personalized Treatment Plan Recommendations: Based on the patient's medical history and medical images, HealthGPT can generate personalized treatment plans to assist doctors in clinical decision-making.
Technical Principles of HealthGPT
- Heterogeneous Low-Rank Adaptation (H-LoRA): HealthGPT introduces heterogeneous low-rank adaptation technology, separating the learning processes of visual understanding and generation tasks to avoid conflicts between tasks. By introducing low-rank matrices to update weights, H-LoRA maintains the model's expressive power while significantly reducing the number of parameters that need to be trained.
- Hierarchical Visual Perception (HVP): The technology separates visual detail learning from the Vision Transformer (ViT), separately handling the different visual granularity requirements of visual understanding and generation tasks. This enables the model to process complex medical image data more efficiently.
- Three-Stage Learning Strategy (TLS): HealthGPT adopts a three-stage learning strategy to gradually train the H-LoRA plugins, enabling the model to quickly adapt to various downstream medical tasks. It performs excellently under data-constrained conditions and can achieve or exceed the performance of existing state-of-the-art models on multiple metrics.
HealthGPT Project Address
- Project Official Website：https://llsuzy.github.io/HealthGPT.github.io/
- Github Repository：https://github.com/DCDmllm/HealthGPT
- HuggingFace Model Library：https://huggingface.co/lintw/HealthGPT-M3
- arXiv Technical Paper：https://arxiv.org/pdf/2502.09838
Application Scenarios of HealthGPT
- Medical Image Generation: HealthGPT can generate high-quality medical images, for example for super-resolution tasks or image reconstruction, assisting medical diagnosis and research.
- Medical Education and Research: HealthGPT can be used in medical education to help students better understand medical images and diagnostic processes. It provides a powerful tool for medical research, supporting the analysis and processing of multimodal data.
- Intelligent Health Assistant: HealthGPT can serve as an intelligent health assistant, helping users query health data and providing daily health management suggestions.