ClawBench

ClawBench is an open-source benchmark that evaluates browser-based AI agents by tasking them with real online activities—booking flights, ordering food, applying for jobs—on live websites. Unlike static or simulated benchmarks, ClawBench uses a two-stage scoring system: first, HTTP interception checks if the final request matches the expected URL/method; second, an LLM judge (currently deepseek-v4-pro) evaluates whether the agent actually fulfilled the instruction. The platform currently includes 283 tasks across 163 live platforms and supports multiple evaluation harnesses (claw-eval, WildClawBench, ClawMark) from a single CLI. ClawBench is designed for AI researchers, developers, and model builders who need a rigorous, real-world test of autonomous web agents. It provides a public leaderboard with metrics for both open-source and frontier closed-source models, with detailed traces for each run. The benchmark is built around "scoring rubrics" that separate deterministic interception from subjective reward, offering granular insight into agent performance. What makes ClawBench different is its commitment to live, dynamic tasks—no static snapshots or sandboxed environments. The benchmark is fully open-source, with the dataset, evaluation scripts, and traces available on GitHub and Hugging Face. It also fosters model submissions via a quickstart CLI and a Gradio Space leaderboard. With over 1,724 judge-verified runs across 13 frontier models and 283 tasks, ClawBench reveals that even the best model (claude-opus-4-7) achieves only 44.6% reward on V2, leaving a 55-point gap to human-level performance. This makes it a valuable reality check for the field.