AnyModal is a flexible multimodal language model framework for PyTorch, designed to build multimodal language models in a modular way. Its core idea is to bridge pre-trained vision (or other modality) encoders with large language models (LLMs) through a pluggable projector architecture, so that users can train multimodal models with just one configuration file and a small amount of code.
[Main Capabilities]
- Provides 5 projector architectures: Linear, MLP, Q-Former (BLIP-2), Perceiver Resampler (Flamingo), C-Abstractor (Honeybee).
- Pluggable encoders: supports ViT, SigLip, CLIP, DINOv2, and any HuggingFace vision model works out of the box.
- Configuration-driven: all hyperparameters can be configured through YAML, and command-line overrides (CLI overrides) are supported for experiments.
- Integrated HuggingFace Trainer: supports learning rate scheduling, gradient clipping, mixed precision, checkpoint saving, and wandb/tensorboard logging.
- Standardized packaging: supports pip installation, the API has type annotations, and it has passed CI tests.
- Supports PEFT/LoRA: optionally enable LoRA on the vision encoder and language model.
[Usage] The materials list three usage options:
- Configuration-driven (recommended)
- Python API
- HuggingFace Trainer
The projector architecture is the key Trainable component, responsible for mapping the encoded modality features into the LLM's embedding space, and users can choose according to their needs.
[Pre-trained Models] Pre-trained models are provided on HuggingFace, for example: Image-Captioning-Llama-3.2-1B: ViT + Llama 3.2-1B, trained on Flickr30k.
[Extensibility] Supports custom projectors (Custom Projector) to extend AnyModal.
[Applicable Scenarios and Limitations] This framework is suitable for research and engineering scenarios that require quickly building, training, and experimenting with multimodal language models in PyTorch, and is especially suitable for users who want to compare different architecture combinations by changing the projector or encoder.
The materials do not provide specific installation commands or code example details, supported platforms, charging information, pricing, performance data, or usage steps; these contents are not provided for now.
[License] MIT License, see LICENSE for details.
