What is RDT
RDT (Robotics Diffusion Transformer) is the world's largest diffusion foundation model for dual-arm robot manipulation tasks, launched by the TSAIL team at Tsinghua University's Institute for AI. RDT has a billion parameters and can autonomously complete complex tasks such as bartending and dog walking without human control. RDT is based on imitation learning from human actions, demonstrating strong generalization ability and manipulation precision, and can handle unseen objects and scenarios. The Tsinghua team has open-sourced RDT's code, models, and training datasets to advance the development and application of robotics technology.
RDT's Main Features
- Dual-Arm Coordinated Manipulation: Directs the robot's two arms to work in coordination to complete complex physical tasks, such as bartending and dog walking.
- Autonomous Task Execution: Autonomously completes entirely new tasks that have not been seen before, without direct human control.
- High-Precision Manipulation: RDT can perform precise operations, such as controlling a robot dog to walk in a straight line, which requires extremely high manipulation precision.
- Language Instruction Understanding: Understands and follows natural language instructions from humans, and executes the corresponding actions.
- Few-Shot Learning: RDT has strong learning ability and can learn new skills from only a small number of demonstrations.
RDT's Technical Principles
- Multimodal Input Encoding: RDT combines three modalities—language, vision, and action—and processes inputs based on different encoding methods.
- Action Encoding: A multilayer perceptron (MLP) with Fourier features.
- Image Encoding: Based on aligned SigLIP.
- Language Encoding: Uses the T5-XXL language model.
- Transformer Backbone Network: RDT uses Transformer as its backbone network, with key modifications for robot manipulation.
- QKNorm and RMSNorm: Mitigate extreme value issues caused by sensor failures.
- Nonlinear MLP Decoder: Enhances the ability to approximate nonlinear dynamics.
- Alternating Injection: Balances image and text modalities to prevent information overwhelming.
- Pretraining and Fine-Tuning: RDT is pretrained on large-scale embodied datasets to gain generalization, and fine-tuned on high-quality dual-arm fine-tuning datasets to enhance dual-arm manipulation capabilities.
- Unified Action Space: Builds a unified action space to unify the formats of different robot data, allowing the model to learn shared physical laws from different data.
- Generalization and Manipulation Precision Testing: Designs challenging tasks to evaluate RDT's generalization ability and manipulation precision, ensuring its effectiveness in practical applications.
RDT's Project Addresses
- Project Website：rdt-robotics.github.io/rdt-robotics
- GitHub Repository：https://github.com/thu-ml/RoboticsDiffusionTransformer
- HuggingFace Model Library：https://huggingface.co/robotics-diffusion-transformer/rdt-1b
- arXiv Technical Paper：https://arxiv.org/pdf/2410.07864
RDT's Application Scenarios
- Food and Beverage Service: Used in tasks such as automated bartending, cooking, and serving, improving the efficiency and innovation of the food and beverage service industry.
- Home Assistant: In home environments, performs household tasks such as cleaning, organizing, and laundry, and can also care for pets, such as walking dogs.
- Medical Assistance: Assist healthcare personnel with some routine nursing tasks, such as distributing medication, moving medical equipment, etc.
- Industrial Automation: In manufacturing, used for tasks such as precision assembly work, quality inspection, and material handling.
- Disaster Rescue: Carry out search and rescue missions at disaster sites, especially in environments that are difficult for humans to reach or dangerous.