Cradle - A Multimodal AI Agent Framework for General Computer Control

What is Cradle

Cradle is a multimodal AI Agent framework for General Computer Control (GCC), launched by Kunlun Wanwei together with top universities and institutions such as the Beijing Academy of Artificial Intelligence, Nanyang Technological University, and Peking University. It enables AI Agents to directly control the keyboard and mouse like a human without training, without relying on any internal API, and to interact with any open- or closed-source software. Cradle is the first AI framework to date that can both play multiple commercial games and operate various software applications, and its paper, project, and code have all been open-sourced.

• Information Gathering: Extracts multimodal information from screen images and possible sounds for decision-making.
• Self-Reflection: Evaluates whether previous actions succeeded or failed, and analyzes the reasons for failure to guide subsequent actions.
• Task Inference: Infers and selects the best next task based on the current environment and historical information.
• Skill Curation: Generates and updates skills related to a given task to adapt to different computer operation needs.
• Action Planning: Generates specific operations for keyboard and mouse control, translating strategies into executable commands.

• Multimodal Input Processing: Cradle can receive and process multimodal input of screen images and audio, simulating human perception to understand computer interfaces and environments.
• Information Extraction and Understanding: Uses large multimodal models (such as GPT-4V) to recognize visual elements and text information in images, as well as instructions or feedback in audio.
• Self-Reflection Mechanism: Through the reflection module, Cradle evaluates whether previously executed actions succeeded, and analyzes the reasons for failure to provide a basis for adjusting strategies.
• Task Inference and Planning: Cradle determines the current priority task through the task inference module, and formulates new actions in the action planning module to complete the task.
• Skill Generation and Update: The skill curation module is responsible for generating new skills or updating existing skills according to the current task; skills exist in the form of code functions and can be instantiated and executed.
• Memory and Knowledge Management: Cradle has long-term and short-term memory systems that store past experiences and skills, and retrieve and apply them when needed.