BALROG

BALROG fills a critical gap. Most benchmarks test single-turn knowledge or code generation, not sustained reasoning across long episodes. By embedding models in procedurally generated games like NetHack and Crafter, it forces them to plan, explore, and adapt — abilities that matter for real-world agents. The dual leaderboard (LLM and VLM) is a smart design: you can isolate whether a model's failure is visual or linguistic. The weekly update keeps it current; we've already seen modern models like Gemini 3 Pro top the charts with 58% overall progress. Pick BALROG when you need a granular, reproducible evaluation of frontier models on long-horizon tasks — especially if you're deciding between a text-only vs. multimodal agent. Skip it if you need a quick comprehension benchmark (use MMLU or HellaSwag) or a production-grade agent framework (use LangChain or AutoGPT). The main caveat: setup requires coding. You submit models via a GitHub repo, not a web form. Also, the benchmark is hard — even the best model barely reaches 58% average, so don't expect ceiling effects. Compared to SWE-bench, BALROG trades code repair for open-ended world interaction. For labs running their own evals, the open-source code and per-environment metrics are invaluable. One limitation: no integration with model hubs like Hugging Face for push-button evaluation — you have to run the pipeline locally. Still, for what it does, it's the most principled game-based agentic benchmark we've seen.