What is Gen2Act
Gen2Act is a robot manipulation policy jointly launched by Google, Carnegie Mellon University, and Stanford University. It generates human videos based on motion information in predicted web data and uses the videos to guide robots in executing new tasks. The policy is based on large amounts of available web video data, avoiding the complexity of directly generating robot videos. The core of Gen2Act lies in zero-shot human video generation, combining a pretrained video generation model and a policy trained with a small amount of robot interaction data. In real-world applications, Gen2Act demonstrates strong generalization ability, can manipulate unseen object types and perform new actions, and significantly improves the success rate compared with other methods. Gen2Act supports long-horizon execution of complex tasks, such as continuously completing multi-step activities like "making coffee." This method reduces the need for large-scale robot data collection and uses a closed-loop policy to dynamically adjust and improve manipulation accuracy.
Main features of Gen2Act
- Zero-shot video generation: Gen2Act can directly use a pretrained video generation model to generate videos of humans performing tasks based on language-described tasks and scene images, without the need for task-specific fine-tuning.
- Generalization to new tasks: Guided by generated human videos, Gen2Act enables robots to perform new tasks that have not appeared in the training data, including manipulating unseen object types and performing new actions.
- Closed-loop policy execution: Combining generated videos and real-time robot observations, Gen2Act dynamically adjusts the robot's actions based on a closed-loop policy, adapting to scene changes and accurately executing tasks.
- Long-horizon task handling: Gen2Act can complete a single task and, based on chaining task sequences, execute a series of complex long-horizon tasks, such as "making coffee," which involves continuous operations across multiple steps.
- Reduced data requirements: Gen2Act requires only a small amount of robot demonstration data, greatly reducing the cost and workload of data collection.
Technical principles of Gen2Act
- Human video generation: Based on a pretrained video generation model, it zero-shot generates videos of humans performing tasks according to language-described tasks and the first-frame image of the scene.
- Video-to-action translation: Based on a closed-loop policy, it converts generated human videos into robot actions. The policy uses the visual features and point trajectories of the video to predict implicitly encoded motion information.
- Visual feature extraction: It uses a ViT encoder and a Transformer encoder to extract features from generated videos and the robot's observation history.
- Point trajectory prediction: Based on a trajectory prediction Transformer, it predicts the motion trajectories of points in the video, with an auxiliary loss to train the policy.
- Behavior cloning loss: Based on minimizing the error between predicted actions and ground-truth actions, it optimizes the policy to imitate the behavior in human videos.
Gen2Act project address
- Project official website：https://homangab.github.io/gen2act/
- arXiv technical paper：https://arxiv.org/pdf/2409.16283
Application scenarios of Gen2Act
- Home automation: In home environments, Gen2Act can control household devices, such as turning a microwave on and off, operating a coffee machine, and organizing items, helping achieve home automation.
- Industrial automation: In manufacturing, Gen2Act can perform complex assembly tasks, or replace or assist human operations in environments that require flexibility and adaptability.
- Service industry: In food and beverage or retail services, Gen2Act guides robots to complete tasks such as taking orders, serving dishes, and organizing shelves.
- Medical assistance: In the medical field, Gen2Act helps develop robots that perform delicate operations, such as assisting in surgery or delivering medical supplies.
- Disaster rescue: At disaster rescue sites, Gen2Act controls robots to carry out search and rescue tasks in unknown environments.