What is TITAN
TITAN is a multimodal whole-slide pathology foundation model developed by a research team at Harvard Medical School. Through visual self-supervised learning and vision-language alignment pretraining, it can extract general-purpose slide representations and generate pathology reports without the need for fine-tuning or clinical labels. It used 335,645 whole-slide images (WSIs) and corresponding pathology reports, combined with 423,122 synthetic captions generated by a multimodal generative AI collaborator. TITAN performs well in a variety of clinical tasks, including linear probing, few-shot and zero-shot classification, rare cancer retrieval, cross-modal retrieval, and pathology report generation.
Main features of TITAN
- Generate pathology reports: TITAN can generate pathology reports with generalization capability in resource-limited clinical scenarios, such as rare disease retrieval and cancer prognosis.
- Multi-task performance: TITAN demonstrates superior performance on a variety of clinical tasks, such as linear probing, few-shot and zero-shot classification, rare cancer retrieval and cross-modal retrieval, as well as pathology report generation.
- Extract general-purpose slide representations: TITAN can extract general-purpose slide representations suitable for a variety of pathology tasks, providing a powerful tool for pathology research and clinical diagnosis.
- Retrieve similar slides and reports: TITAN performs excellently in rare cancer retrieval and cross-modal retrieval tasks, and can effectively retrieve similar slides and reports to assist clinical diagnostic decision-making.
- Reduce misdiagnosis and inter-observer variability: TITAN has important potential in clinical diagnostic workflows, and can assist pathologists and oncologists in retrieving similar slides and reports, reducing misdiagnosis and inter-observer variability.
Technical principles of TITAN
- Self-supervised learning and vision-language alignment: TITAN is pretrained through visual self-supervised learning and vision-language alignment, and can extract general-purpose slide representations without any fine-tuning or clinical labels.
- Pretraining strategy: TITAN's pretraining consists of three distinct stages, ensuring that the final generated slide-level representations can leverage both visual and language supervision signals while capturing tissue morphological semantics at the ROI level as well as the WSI level.
- Stage 1 (vision-only pretraining): Pretraining is conducted on an internal dataset named Mass-340K, which contains 335,645 whole-slide images (WSIs) and 182,862 medical reports.
- Stage 2 (region-of-interest and synthetic caption alignment): TITANV is pretrained using 423,122 pairs of 8K x 8K regions of interest and their synthetic captions generated by PathChat, enabling the model to capture region-level morphological information.
- Stage 3 (whole-slide image and pathology report alignment): Further pretraining is conducted using 182,862 pairs of whole-slide images and their pathology reports to obtain the final model TITAN, enabling it to handle high-level descriptions at the slide level.
- Model design: TITAN is based on the Vision Transformer (ViT) architecture. The slide encoder uses pre-extracted image patch features, arranged in a two-dimensional feature grid to preserve spatial context. By increasing the image patch size, the input sequence length is effectively reduced. To address the issue of irregular sizes and shapes of whole-slide images, region cropping and data augmentation methods are adopted.
- Language capability endowment: By comparing the second and third stages of pretraining of the title generator (CoCa), aligning the slide representations with synthetic titles and pathology reports respectively, and fine-tuning the slide encoder, text encoder, and multimodal decoder, the model acquires language capabilities, including generating pathology reports, zero-shot classification, and cross-modal retrieval, etc.
TITAN's project address
- Github repository：https://github.com/mahmoodlab/TITAN
- HuggingFace model library：https://huggingface.co/MahmoodLab/TITAN
- arXiv technical paper：https://arxiv.org/pdf/2411.19666
TITAN's application scenarios
- Pathology research and clinical practice: Through visual self-supervised learning and vision-language alignment pretraining, TITAN can extract general slide representations and generate pathology reports, providing more effective tools for pathology research and clinical practice.
- Clinical scenarios with limited resources: TITAN is particularly suitable for clinical scenarios with limited resources, such as rare disease retrieval and cancer prognosis, and can generate pathology reports with generalization ability.
- Clinical diagnostic workflow: TITAN can assist pathologists and oncologists in retrieving similar slides and reports, reducing misdiagnosis and inter-observer variability.
- Diverse clinical tasks: TITAN performs excellently on a variety of clinical tasks, including linear probing, few-shot and zero-shot classification, rare cancer retrieval and cross-modal retrieval, as well as pathology report generation.
- Pathology report generation: TITAN can generate high-quality pathology reports without any fine-tuning and clinical labels, under resource-constrained conditions.
- Cross-modal retrieval: TITAN performs excellently in rare cancer retrieval and cross-modal retrieval tasks, can effectively retrieve similar slides and reports, and assist clinical diagnostic decision-making.