What is SmolVLA
SmolVLA is an open-source lightweight Vision-Language-Action (VLA) model open-sourced by Hugging Face , designed for cost-effective robots. With 450 million parameters, the model is compact, can run on a CPU, can be trained on a single consumer-grade GPU, and can be deployed on a MacBook. SmolVLA is trained entirely on open-source datasets, with the dataset labeled "lerobot".
SmolVLA's Main Features
-
Multimodal Input Processing: SmolVLA can process multiple types of input, including multiple images, language instructions, and the robot's state information. Image features are extracted through a visual encoder, language instructions are tokenized and fed into the decoder, and sensorimotor states are projected onto a token through a linear layer, aligning with the token dimension of the language model.
-
Action Sequence Generation: The model includes an action expert module, which is a lightweight Transformer that can generate future robot action sequence chunks based on the output of the Vision-Language Model (VLM). It is trained using flow matching technology, learning action generation by guiding noisy samples to regress to the true data distribution, achieving high-precision real-time control.
-
Efficient Inference and Asynchronous Execution: SmolVLA introduces an asynchronous inference stack that separates action execution from perception and prediction, enabling faster and more responsive control, allowing the robot to respond more quickly in rapidly changing environments, improving response speed and task throughput.
SmolVLA's Technical Principles
-
Vision-Language Model (VLM): SmolVLA uses SmolVLM2 as its VLM backbone. The model is optimized to handle multi-image input. It includes a SigLIP visual encoder and a SmolLM2 language decoder. Image tokens are extracted through the visual encoder, language instructions are tokenized and fed directly into the decoder, and sensorimotor states are projected onto a token through a linear layer, aligning with the token dimension of the language model. The decoder layers process the concatenated image, language, and state tokens, and the resulting features are then passed to the action expert.
-
Action Expert: The action expert is a lightweight Transformer (about 100 million parameters) that generates future robot action sequence chunks based on the output of the VLM. The action expert is trained using flow matching technology, learning action generation by guiding noisy samples to regress to the true data distribution, achieving high-precision real-time control.
-
Visual Token Reduction: To improve efficiency, SmolVLA limits the number of visual tokens per image frame to 64, greatly reducing processing costs.
-
Layer Skipping to Accelerate Inference: SmolVLA skips half of the layers in the VLM for computation, effectively halving the computational cost while maintaining good performance.
-
Interleaved Attention Layers: Unlike traditional VLA architectures, SmolVLA alternates between cross-attention (CA) and self-attention (SA) layers. This improves the efficiency of multimodal information integration and speeds up inference.
- Asynchronous Inference: SmolVLA introduces an asynchronous inference strategy that allows the robot's "hands" and "eyes" to work independently. Under this strategy, the robot can execute the current action while already starting to process new observations and predict the next set of actions, eliminating inference latency and increasing control frequency.
SmolVLA's Project Address
- HuggingFace Model Library：https://huggingface.co/lerobot/smolvla_base
- arXiv Technical Paper：https://arxiv.org/pdf/2506.01844
SmolVLA's Application Scenarios
- Object Grasping and Placing: SmolVLA can control a robotic arm to complete complex grasping and placement tasks. For example, on an industrial production line, a robot needs to accurately grasp parts based on visual input and language instructions and place them in specified positions.
- Household chores: SmolVLA can be applied to home service robots to help complete various household chores. For example, a robot can, based on natural language instructions, identify and clear clutter in a room, or place items in specified positions.
-
Cargo handling: In logistics warehouses, SmolVLA can control robots to complete cargo handling tasks. A robot can identify the position and shape of cargo based on visual input and, combined with language instructions, generate an optimal handling path and action sequence, improving the efficiency and accuracy of cargo handling.
- Robotics education: SmolVLA can serve as a tool for robotics education, helping students and researchers better understand and develop robotics technology.