Appworld

AppWorld's case rests on two things its competitors mostly skip: scale of interaction and scoring of side effects. On scale, the engine ships 9 apps behind 457 APIs and 106 simulated people with realistic digital activity. A task like "order groceries for a household" doesn't resolve into one function call — it forces the agent to read notes, check a shopping app, handle payment state, and reconcile across apps. The suite is 750 tasks deliberately built so that a simple API-call sequence won't do; the paper's own framing is that existing benchmarks are "inadequate" because they only cover those simple sequences. The reported spread — GPT-4o at roughly 49% on normal tasks and 30% on challenge tasks, other models at least 16% below that — gives you a real difficulty gradient rather than a saturated leaderboard. On side effects, this is the differentiator. Programmatic evaluation runs state-based unit tests that accept multiple valid ways of completing a task while still checking for unexpected changes. If your agent succeeds but corrupts an unrelated record, AppWorld scores that. For anyone shipping an autonomous coding agent into a system with shared state, that is the failure mode that actually hurts, and almost no benchmark measures it. The costs are real and worth stating plainly. Everything runs locally: pip install appworld, then appworld install, appworld download data, appworld play. That means a Python environment, local compute, and your own agent harness — AppWorld is an environment and a grader, not a product you point at a customer workflow. The playground for interactively exploring individual task worlds is labeled "to be deployed"; the team says they are still figuring out how to host it, so the CLI and the Jupyter notebook are the working interfaces today. The world is fixed at 9 apps and 457 APIs, which is exactly what makes results comparable across papers and exactly what stops you extending it to your own internal APIs without forking. Where it fits: research groups and agent teams who need a peer-reviewed, citable, reproducible evaluation of multi-app coding agents, especially anyone whose concern is regressions and unintended writes. Where it doesn't: anyone wanting a managed evaluation service, a turnkey agent application, or coverage of open-web browsing rather than scripted app APIs. The forthcoming AppWorld-UL extends the same world into user-in-the-loop tasks — clarifying questions, confirmations, infeasible instructions — which is a meaningful addition if your agent talks to humans mid-task.