Polymath

Polymath sits in an unglamorous but load-bearing part of the agent stack. Most evaluation today measures whether an agent can complete one step or one function correctly; Polymath builds environments where an agent has to hold a multi-tool workflow together across a long horizon and be judged against ground truth. The pipeline it describes — Applications & Services to define objectives, Data Tasks to structure scenarios, Verifiers to score against ground truth, and Agent(s) running inside — is a sensible decomposition, because it separates the environment you train in from the standard you grade against. That separation is what makes a simulation environment reusable rather than a one-off eval script. On strengths: the Horizon-SWE benchmark (February 2026) is the most concrete public artifact, and it is narrowly and usefully scoped — end-to-end software engineering workflows in production-grade systems, multi-tool, long-horizon, explicitly beyond simple repo-wide edits. The January 2026 post on reliability and autonomy in software engineering agents traces the trajectory from autocomplete to repo-wide changes, which is a fair description of where the practical failure modes have moved. Backing from Base10, Cervin Ventures, and Y Combinator, plus stated work with leading model labs, suggests the team has access to the environments and the people needed to build this credibly. On weaknesses: the public information is thin. Beyond the homepage and those two posts, there is little to read. That is a real cost if you are the kind of buyer who wants to evaluate independently before talking to anyone. It also means you should treat any specific claim about throughput, environment coverage, or supported task domains as something to verify in a conversation, not something you can read off the site. And the value proposition depends entirely on whether your agents actually operate on long horizons — if they don't, the environment complexity is overhead you don't need. Where it fits: research labs training agents on tasks that span hours or days; enterprise platform teams trying to diagnose why a coding agent that demos well fails in production; benchmark developers who need multi-tool evaluation in production-grade systems; model providers testing agent reliability before a release. Where it doesn't: teams whose agents handle short, single-turn tasks, non-technical buyers wanting a no-code builder, and anyone shopping primarily on public leaderboard rankings. This is a partner-led engagement, so budget for a real conversation rather than a signup flow.