ClawBench

ClawBench delivers what most benchmarks lack: real websites and a deterministic first-stage check. The two-stage scoring—HTTP interception followed by an LLM judge—is a thoughtful design that separates the objective 'did it hit the right endpoint' from the subjective 'did it fulfill the intent.' The judge model (deepseek-v4-pro) is named, and the rubric is public, so you can audit any run. Strengths: The 5-layer trace bundle is a goldmine. You get video, actions, agent messages, HTTP requests, and the verdict, all timestamped on a single clock. This makes debugging failures far easier than with black-box benchmarks. The dataset is large (283 tasks, 1,724 judge-verified runs) and refreshed weekly, so it stays current with website changes. The CLI is straightforward for anyone comfortable with a terminal, and the OpenRouter integration means you can test any model without a proprietary harness. Weaknesses: The command-line requirement will deter non-technical users. There's no managed API or web interface for running evaluations—you must install and run the CLI. Live websites can change, affecting reproducibility over time. The LLM judge, while named, may introduce bias; the strict rubric helps, but it's still an LLM judgment. Cost per task can vary wildly: claude-opus-4-7 costs $4.44 per task, while free models cost nothing but score near zero. This means cost-efficiency trade-offs are stark. Where it fits: AI research labs, agent developers, and model evaluation teams that need rigorous, reproducible, real-world testing. It's also great for open-source contributors who want to add tasks or harnesses. Where it doesn't: If you need a quick, managed evaluation service, or if you're a non-technical stakeholder, look elsewhere. Air-gapped environments can't use it because it hits live sites. Overall, ClawBench is a serious tool for serious agent evaluation. The transparency and depth are unmatched in the open-source space.