What is VisionFM
VisionFM is a multimodal, multitask visual foundation model designed specifically for general ophthalmic artificial intelligence. Through pretraining on 3.4 million ophthalmic images from 560,457 individuals, covering a wide range of ophthalmic diseases, imaging modalities, devices, and demographic data. VisionFM can handle eight common ophthalmic imaging modalities, including fundus photography, optical coherence tomography (OCT), fluorescein fundus angiography (FFA), and others, and is applied to various ophthalmic AI tasks such as ophthalmic disease recognition, disease progression prediction, disease phenotype subdivision, and systemic biomarker and disease prediction. In diagnosing 12 common ophthalmic diseases, it surpasses ophthalmologists with basic and intermediate levels, and on a large-scale ophthalmic disease diagnosis benchmark database, it surpasses strong baseline deep neural networks. VisionFM also demonstrates strong generalization ability to new ophthalmic modalities, disease spectra, and imaging devices.
Main features of VisionFM
- Disease screening and diagnosis: VisionFM can screen and diagnose a variety of ophthalmic diseases, including but not limited to diabetic retinopathy, glaucoma, age-related macular degeneration, etc.
- Disease prognosis: The model can also predict the development trends and prognosis of diseases.
- Disease phenotype subdivision: VisionFM can perform subclassification of disease phenotypes, including segmentation of lesions, vessels, and layers, as well as landmark detection.
- Systemic biomarker and disease prediction: In addition to ocular diseases, VisionFM can also predict systemic biomarkers and diseases from ocular images.
- Multimodal processing capability: VisionFM can handle eight common ophthalmic imaging modalities, including fundus photography, optical coherence tomography (OCT), and fluorescein fundus angiography (FFA).
- Modality-agnostic diagnosis: VisionFM supports modality-agnostic diagnosis, meaning a single decoder can be used to diagnose multiple ophthalmic diseases across different imaging modalities.
- Few-shot learning: VisionFM demonstrates few-shot learning capability, being able to diagnose new diseases with high accuracy even with only a small number of annotated samples.
- Strong generalization ability: The model demonstrates strong generalization ability to new ophthalmic modalities, disease spectra, and imaging devices.
- Synthetic data enhanced learning: VisionFM can also use synthetic ophthalmic imaging data to enhance its representation learning capability, thereby achieving significant performance improvements on downstream ophthalmic AI tasks.
Technical principles of VisionFM
- Large-scale pretraining: VisionFM is a deep learning-based visual foundation model that is pretrained on 3.4 million ophthalmic images from 560,457 individuals, covering a wide range of ophthalmic diseases, imaging modalities, imaging devices, and demographic data.
- Multimodal multitask learning: VisionFM can handle multiple ophthalmic imaging modalities, including fundus photography, OCT, FFA, etc., and is applied to various ophthalmic AI tasks such as disease screening, diagnosis, disease prognosis, and disease phenotype subdivision.
- Expert-level intelligence and accuracy: After pretraining, VisionFM demonstrates expert-level intelligence and accuracy in multiple ophthalmic AI applications, and its generalist intelligence surpasses primary and intermediate ophthalmologists in the joint diagnosis of 12 common ophthalmic diseases.
VisionFM project address
- Github repository：https://github.com/ABILab-CUHK/VisionFM
- arXiv technical paper：https://arxiv.org/pdf/2310.04992
Application Scenarios of VisionFM
- Ophthalmic Clinical Tasks: VisionFM can help address ophthalmic clinical tasks, especially in disease screening and diagnosis.
- Diagnosis of Multiple Ophthalmic Diseases: The model performs excellently in diagnosing and predicting multiple ophthalmic diseases, including diabetic retinopathy, glaucoma, age-related macular degeneration, and others.
- Primary Healthcare Settings: The model can play an important role in primary healthcare settings with limited imaging resources, reducing the workload of doctors.
- Areas with Low Ophthalmologist Density: VisionFM is especially useful for regions and countries with low ophthalmologist density.
- Education and Training: VisionFM can serve as a senior ophthalmologist to train junior ophthalmic practitioners, possessing extensive knowledge in ophthalmic imaging and disease diagnosis.
- Assisted Generation of Diagnostic Reports: The model can be integrated with large language models (LLM) to generate diagnostic reports, completing the closed loop of ophthalmic disease diagnosis.