LangWatch Scenario

We've seen agent testing tools come and go, but LangWatch Scenario stands apart because it wasn't bolted on after the fact — it's built on the same tracing engine used for production observability. Every simulation run is automatically linked to a full trace, cost metrics, and prompt versions. That tight feedback loop is rare. Pick this when you're shipping agents that chain multiple tools across several turns. The per-turn judge agent catches failures mid-conversation — like the agent calling a wrong tool or going off-policy — and the replay visualizer shows exactly where it broke. For voice teams, the built-in latency metrics and noise injection are a huge time saver. Pass if you only need single-turn prompt evals or if your team can't write scenario descriptions. There's a learning curve — you define scenarios as code or YAML, not a drag-and-drop wizard. Also, the free tier (50k events/month) is generous for small teams but may require upgrading quickly. Compared to rivals: Langfuse offers similar observability but lacks the multi-turn simulation capability. Braintrust has evals but no voice or adversarial testing. LangSmith has testing but without the tight Scenario-observability bridge. LangWatch Scenario is the best fit for teams that need testing, evals, and observability in one stack. In practice, the new Langy assistant is surprisingly useful — it converts a plain-English goal into a full test plan and even drafts prompt fixes. The 3.0 release (April 2026) simplified self-hosting with ClickHouse, removing the Elasticsearch dependency. The built-in MCP server also lets you manage prompts from Claude Code. Real-world caveat: the judge agent uses your LLM for evaluation, so costs can add up on heavy simulation runs.