CRAB

CRAB: Cross-environment Agent Benchmark

CRAB is a benchmark framework designed for evaluating multimodal language model (MLM) agents across diverse environments. It offers tools for building agents, operating environments, and creating benchmarks.

Key Features:

• Cross-environment Support: Agents can adapt and perform seamlessly across different interfaces like Ubuntu and Android.
• Graph Evaluator: Provides fine-grained performance analysis beyond simple success rates.
• Task Generation: Automates task creation using a graph-based method, mimicking real-world scenarios.
• Easy-to-Use: Python-based definitions for agent operations, observations, and benchmark evaluators.

Use Cases:

• Evaluating and comparing the performance of different MLM agents.
• Developing agents that can operate across multiple platforms.
• Analyzing agent strengths and weaknesses through detailed performance metrics.
• Automating the creation of complex, real-world tasks for agent training and evaluation.

CRAB Benchmark-v0 includes 120 tasks across Ubuntu and Android environments, tested with 6 MLMs under 3 communication settings.