Vibrant Labs

Vibrant Labs sits in an unusual spot: it is selling data and evaluation infrastructure at the exact moment those two things stop being separate problems. Most agent teams today buy a dataset, train, then discover the benchmark they trained against has been saturated — or worse, leaked into the training set. The lab's self-evolving benchmark track attacks that directly, using coding agents as world-builders that construct environments, tasks and verifiers and then rebuild them as models improve. That is a structural answer to benchmark saturation rather than a bigger static test set. The UED work is the other half. Environments adapt to the agent and generate tasks at the frontier of its current ability, following PAIRED and regret-based UED lineages rather than a fixed difficulty ladder. For teams training long-horizon tool-use agents, this matters because the failure mode isn't easy tasks, it's tasks at the wrong difficulty — too easy to teach anything, too hard to produce signal. Tau2-Infinity targets exactly that by harvesting tasks inside a target model's pass@k window, with verifiers and environments attached so the mined task is checkable. Strength in depth is clear: four published artifacts in roughly three months (adversarial e-commerce, Tau2-Infinity, Ecom Bench, ITSMBench), spanning both tool-use and computer-use modalities. The Ragas pedigree is genuine leverage — an evals framework used by 80% of the Fortune 100 gives the team an unusually good prior on what makes a verifier trustworthy. The honest weaknesses: this is research-stage. There is no evidence in what we could read of a self-serve dataset catalog, a standardised task schema you can just pull from, or anything resembling a production SLA. The 14K/1B evaluation figures on the homepage are unlabelled as to what they count and over what period, so don't anchor a business case to them. Scope is deliberately narrow — tool-use and computer-use agents, mostly RL data generation — so if your agent is a coding assistant or a writing tool, most of this is off-target. Where it fits: frontier labs, well-funded applied research groups, and enterprise AI platform teams building agent capability in-house who can absorb a research collaboration timeline. Where it doesn't: startups that need a fine-tuning dataset next sprint, and anyone whose evaluation requirements are already served by a static, cheap benchmark.