What is Optimus-1
Optimus-1 is launched by Harbin Institute of Technology (Shenzhen) and Pengcheng Laboratoryintelligent agentframework, which can solve the challenge of completing long-term tasks in open-world environments. The framework combines structured knowledge and multimodal experience to enable the intelligent agent to better execute complex tasks. Optimus-1 includes a hybrid multimodal memory module, composed of a hierarchical directed knowledge graph (HDKG) and an abstract multimodal experience pool (AMEP), a knowledge-guided planner and an experience-driven reflector, significantly improving long-term task performance in environments such as Minecraft, approaching human level.
Main functions of Optimus-1
- Knowledge-Guided Planner: Generates executable sequences of subgoals based on structured knowledge (HDKG), planning and executing long-term tasks.
- Experience-Driven Reflector: Evaluates the current task state based on historical experience (AMEP), and adjusts the action plan when necessary.
- Action Controller: Executes the subgoals produced by the planner and interacts with the environment to complete tasks.
- Hybrid Multimodal Memory: Combines HDKG and AMEP, stores and utilizes world knowledge and historical experience, and supports complex decision-making and adaptive learning.
- Self-Evolution: Based on the "free exploration-teacher guidance" method, it can gradually improve memory capacity and task execution capability without parameter updates.
Technical principles of Optimus-1
- Hierarchical Directed Knowledge Graph (HDKG)：
- Converts world knowledge (such as item crafting rules) into a graph structure, where nodes represent objects and edges represent relationships between objects (such as crafting relationships).
- Based on graph retrieval and topological sorting, it provides the intelligent agent with knowledge of the materials and tools required to complete tasks.
- Abstract Multimodal Experience Pool (AMEP)：
- Dynamically summarizes and stores multimodal information during task execution (such as environment, state, plan, visual frames, etc.).
- Uses the MineCLIP model to calculate the similarity between vision and subgoals, stores highly relevant visual memories, and provides reference for reflection.
- Knowledge-Guided Planner: Combines visual observations and knowledge in HDKG to generate reasonable sequences of subgoals and guide the actions of the action controller.
- Experience-Driven Reflector: Activated periodically, retrieves multimodal experience related to the current subgoal from AMEP, evaluates the current task state, and decides whether to continue execution, complete, or replan.
- Action Controller: Based on subgoals and current observations, generates the low-level actions required to interact with the environment, such as mouse and keyboard operations.
Project addresses of Optimus-1
- Project official website：cybertronagent.github.io/Optimus-1
- GitHub repository：https://github.com/JiuTian-VL/Optimus-1
- arXiv technical paper：https://arxiv.org/pdf/2408.03615
Application scenarios of Optimus-1
- Game environment automation: In complex game environments such as Minecraft, Optimus-1 automates tasks such as resource collection, building construction, and exploration, providing an immersive gaming experience.
- Virtual personal assistant: In virtual reality or augmented reality applications, it can serve as the user's personal assistant, helping manage schedules, provide information query services, and automate daily tasks.
- Home robot assistant: In smart home systems, guide home service robots to perform long-term tasks such as cleaning, security monitoring, and daily maintenance, improving the level of home automation.
- Industrial Automation and Optimization: In the field of industrial manufacturing, optimize production processes, guide robots to automatically complete complex assembly line tasks, and improve production efficiency and safety.
- Simulation Training and Education: In military training or medical education, create complex simulation environments to help trainees practice and learn decision-making and operational skills through simulation.