Minebench

MineBench takes an unusual and genuinely useful position in the eval space. Most benchmarks either grade a model's text or ask a judge model to score output. MineBench does neither: it demands that the model emit structured JSON block coordinates, then renders them as a Minecraft-style voxel scene you can inspect from any angle with orbit, pan, zoom, rotate, and spin controls. That removes the biggest source of noise in AI evals — the grader's interpretation — because the build either matches the prompt or it doesn't, and a human decides. Strengths. Pairwise Arena voting with Tie, Bad, and Skip options is well-designed; it avoids forcing a meaningless choice between two equally wrong builds, which keeps the Elo signal cleaner. Sandbox mode lets you isolate a single model and a single prompt when you want to poke at one capability. The prompt set is curated natural-language spatial instructions rather than synthetic tokens, so the results map onto how people actually describe the physical world. Runner-level details are handled: fixed 1,247-block grid, Faithful Pack textures, keyboard shortcuts, and a Gallery of community builds. Signing in saves builds and keeps your votes, and no API key is needed to generate. Weaknesses and honest caveats. The scope is one skill. A model that scores well here tells you something about spatial instruction-following and nothing about reasoning, coding, writing, or tool use — MineBench itself frames the output as one lens, not a scorecard. Rankings depend on the voter pool; a small or skewed set of voters can move Elo in ways that don't reflect model capability, and there's no indication of a statistical control on voter volume. Generations render raw coordinates with no post-processing, so a malformed JSON payload can read as a total failure rather than a partial one. The fixed grid also means large or fine-grained structures can't be represented. Where it fits. Use MineBench as a qualitative complement when you're choosing between frontier models for anything involving 3D layout, spatial description, or structured coordinate output — level generation, layout planning, scene composition, robotics-adjacent reasoning. It's fast, free, and visual, which makes it a good demo for a team meeting or a gut-check before a heavier eval. Where it doesn't. If you need automated, reproducible, CI-style scoring, MineBench is manual by design and won't slot into a pipeline. If you need a general model-quality score, this isn't it. If you're looking for a creative 3D building tool rather than a benchmark, you'll be disappointed.