Crab

CRAB (Cross-environment Agent Benchmark for Multimodal Language Model Agents) aims to be a general benchmarking framework for multimodal language model (MLM) agents. It provides an end-to-end and easy-to-use framework for building agents, operating environments, and creating benchmarks to evaluate them, and includes three key components: cross-environment support, graph evaluator, and task generation.

Purpose CRAB is used to evaluate the performance of multimodal language model agents in cross-environment tasks, and supports building agents, operating environments, and creating benchmarks.

Main capabilities

1. Cross-environment support: CRAB supports multiple environments, ensuring that agents can seamlessly adapt and perform well across different interfaces.
2. Graph evaluator: Provides fine-grained evaluation beyond binary success rates, offering detailed analysis of agent performance, highlighting strengths and pointing out room for improvement.
3. Task generation: CRAB uses a graph-based method to automatically create tasks. By combining multiple subtasks into complex tasks, CRAB generates dynamic tasks close to real-world scenarios, saving the time and effort of manually creating tasks.
4. Ease of use: All agent operations (actions), observations, and benchmark evaluators are defined by Python functions. Therefore, adding a new environment to CRAB requires only a few lines of Python code. Benchmark configuration follows a declarative programming paradigm, making it very easy to reproduce any experimental environment.

CRAB Benchmark-v0 The CRAB Benchmark-v0 developed based on the CRAB framework includes 120 tasks under 2 environments (Ubuntu and Android), and was tested using 6 different multimodal language models under 3 different communication settings. The results are based on CRAB Benchmark v0 released on October 18, 2024, with a total of 120 tasks.

Example tasks

• Open "Slack" in Ubuntu, navigate to the "multi-modal-benchmark" channel, summarize the last two messages, and then send them in the "Messages" app on an Android phone to the first contact in the list. Setting: OpenAI GPT-4o + multi-agent by function.
• Open the "Tasks" app on Android, view the first incomplete task, and then perform the task according to its description.
• Open the "Calendar" app on Android, summarize all of today's schedules. Then, in In Ubuntu, use the terminal and Vim to create a Markdown file at "/home/crab/assets/plan.md", with each event as a checkbox bullet item. Settings: OpenAI GPT-4o + single agent.
• Please open the X app on my phone, search for CAMEL-AI.org, view the latest posts, summarize them, and then send the summary from my PC via Slack to Tianqi Xu.

The demo video was edited for a better viewing experience. In actual execution, there are tens of seconds of waiting time between each step.

Framework Comparison Compare CRAB with existing GUI agents and benchmarks. The key features of each framework include:

• Interactive environment: Indicates the presence of an interactive environment or a static dataset.
• Multimodal observation: Specifies whether vision-based observations (e.g., screenshots) are provided.
• Cross-platform: Indicates support for multiple operating systems or platforms.
• Evaluation: Evaluation metrics are categorized as: Goal-based: Checks the environment state only based on the final goal. Trajectory-based: Compares the agent's action trajectory with a golden action sequence. Multiple: Varies by task. Graph-based: A directed acyclic graph with each node as an intermediate checkpoint.
• Task construction: Task construction methods include: Handcrafted: Handcrafted by humans. LLM-inspired: Uses an LLM to generate task drafts but they are still verified and annotated by humans. Template: Generated by filling in blanks in a task template. Subtask composition: Combines multiple subtasks to construct tasks and evaluators.

Experimental Results

• Performance differences among models: GPT-4o has the highest overall success rate and completion rate. Gemini 1.5 Pro and Claude 3 Opus struggle more and are almost unable to complete tasks. Considering models with JSON output, GPT-4o's performance declines compared with the version with function calling enabled, mainly due to its higher invalid action rate. Among open-source models, Pixtral-12B, with far fewer parameters, achieves a better CR than LLaVA-ov-72B, demonstrating its efficiency. Although open-source models can generally understand screenshots and correctly generate step-by-step plans, they often fail to execute the correct actions according to the plan.
• Completion rate reflects true agent capability: The completion rate (CR) calculated by the graph evaluator provides a detailed view of performance. Even though GPT-4o has the same success rate under the single-agent structure and the per-environment multi-agent structure, its completion rate differs by as much as 4.67%.
• Termination reasons indicate directions for improvement: The high percentage of reaching the step limit (RSL) across all models indicates that the agentoften runs out of steps without achieving the final goal. Gemini 1.5 Pro's high invalid action (IA) rate highlights the problem of generating actions in the correct format. The false completion (FC) rate in multi-agent settings is higher than in single-agent settings, indicating that message loss occurs during communication among multiple agents.