What is MobA
MobA (Mobile Agent) is a new type of mobile agent launched by a team from Shanghai Jiao Tong University. It is based on multimodal large language models (MLLMs) to enhance the automated task execution capabilities of mobile devices. MobA adopts a two-level architecture: a high-level global agent (GA) is responsible for understanding user instructions, managing history records, and planning tasks; a low-level local agent (LA) executes specific actions according to the GA's plan. The system's built-in reflection module enables MobA to efficiently handle complex tasks, including those it has not encountered before.
MobA's main features
- User instruction understanding: Understands users' natural language instructions and converts them into executable tasks.
- Task planning and decomposition: The high-level global agent (GA) is responsible for decomposing complex tasks into smaller, more manageable subtasks.
- Action execution: The low-level local agent (LA) executes specific subtasks and actions according to the GA's guidance.
- Historical memory tracking: The GA tracks historical memory and considers past experience and information when planning new tasks.
- Reflection and self-optimization: The integrated reflection module supports MobA in self-evaluation after task execution and optimizing future task execution.
- Cross-application operations: Handles complex tasks involving multiple applications and enables cross-application automated operations.
MobA's technical principles
- Multimodal large language models (MLLMs): Based on MLLMs, the model can process and understand multiple types of data, such as text, images, and more.
- Two-level agent architecture: The system consists of a high-level global agent (GA) and a low-level local agent (LA), which are responsible for task planning and action execution, respectively.
- Task planning module: The planning module in the GA is responsible for decomposing user instructions into a series of subtasks and evaluating the feasibility of the tasks.
- Action module: The action module in the LA is responsible for identifying whether a task can be completed in one step and providing the corresponding action instructions.
- Memory module: The system includes a memory module used to store and update task execution traces, user preferences, and application information.
MobA's project address
- arXiv technical paper：https://arxiv.org/pdf/2410.13757
MobA's application scenarios
- Personal assistant: In daily life, as a personal assistant, it helps users manage schedules, set reminders, query information, and more.
- Smart home control: Integrated with smart home devices, it controls smart devices in the home based on voice or text instructions, such as lighting, temperature adjustment, security monitoring, and more.
- Mobile device automation: On mobile devices, it automatically executes complex operation processes, such as automatically filling out forms, managing emails, optimizing application settings, and more.
- Assistance for the elderly and people with disabilities: Provides assistance for the elderly and people with disabilities, simplifies the process of using smart devices, and improves quality of life.
- Education and learning: In the field of education, it helps students obtain information, manage learning materials, and even assist in completing assignments.