LangWatch Scenario
LangWatch Scenario simulates text and voice users against your AI agent, scoring every turn before production does.
Multi-turn agents fail on paths nobody wrote a test for, and Scenario is the most complete answer to that — simulation, per-turn judging, and voice in the same stack as your traces. The no-code Agent Testing v2 and Langy's PM-to-PR loop mean non-engineers can finally own specs. If all you need is single-turn scoring, you're paying for surface area a simpler eval tool covers.
Verified 2d ago · liveness 79/100 · cite: rightaichoice.com/tools/langwatch-scenario
- ML engineers testing multi-turn agents with real tool calls and MCP invocations
- QA teams automating agent regression suites and gating merges in CI/CD
- Voice AI developers simulating realtime conversations with latency and interruption checks
- Product managers who define agent behavior in plain English and review the resulting PRs
- Teams that only need single-turn prompt evaluation or output scoring
- Products with no multi-turn flow, tool use, or voice component
- Side projects that will never exceed the free tier's 3 scenarios and 2 users
We scan live Reddit threads, YouTube comments, X posts, G2 reviews and other communities — and hand you an honest verdict in under a minute.
- Honest verdict, not marketing
- Real pros & cons from real users
- Attributed quotes with receipts
3 free scans · no card needed
Skip LangWatch Scenario if you only need single-turn output scoring or plain tracing, or if a free plan with 3 scenarios, 3 simulations, and 3 evals is already more than your project will ever use.
Events past the 200k included on Growth bill at €5 per 100k, which climbs quickly once simulations run on every pull request.
Developer at €0/mo fits a solo engineer or a proof of concept that never exceeds 3 scenarios, 3 simulations, and 3 custom evals. Growth at €29 per core seat per month plus €5 per 100k events beyond 200k is the realistic tier for a team gating PRs on simulations. Above that, self-hosted or Enterprise pricing competes on control rather than seat count; observability-only alternatives such as Langfuse can be cheaper if you never run multi-turn simulations.
In short
LangWatch Scenario — LangWatch Scenario simulates text and voice users against your AI agent, scoring every turn before production does. Best for ML engineers testing multi-turn agents with real tool calls and MCP invocations, QA teams automating agent regression suites and gating merges in CI/CD, Voice AI developers simulating realtime conversations with latency and interruption checks. Free to start; paid plans from €29/mo.
What's new in LangWatch Scenario
Checked todayAcross the latest 5 updates: 3 feature updates, 1 launch and 1 changelog entry.
3.18.0 Instant Evals in the Trace Explorer, and Files in Datasets
Ask the Trace Explorer a question no filter can answer, and upload images and files into dataset cells for evaluation.
Instant Evals: Search Your Entire Production History
Ask a question of every conversation you have stored and get the matches back in minutes instead of waiting on a batch job.
Go SDK 1.0 and Search in Datasets
Go SDK reaches 1.0, the dataset editor gains search, and agents are discovered from connected providers.
3.17.0 Agent Testing v2 and Prompt Optimization in the Workbench
Agent Testing replaces Simulations, comparison runs arrive, and Langy optimizes prompts directly in the workbench.
3.16.0 New Navigation and Voice APIs support on the Gateway
The sidebar becomes a product switcher with an icon rail, and realtime voice traffic bills to a virtual key on the Gateway.
What people actually say about LangWatch Scenario — is it worth it?
We ran a structured research pass across product reviews, community discussions, and post-purchase forum threads to surface the patterns vendors won't publish themselves. Below: the recurring strengths, the hidden costs people mention most, and the cohort that consistently regrets adopting this tool.
20 mentions across 2 sources (Hacker News, YouTube) · researched Jul 3, 2026.
Average across the 2 sources that answered — each source counts once, not each post.
- +Simulates multi-turn conversations realistically using LLM-powered user simulator.
- +Each turn judged automatically with pass/fail criteria, surfacing concrete failures.
- +Open-source SDK (MIT) works with any LLM and any agent framework.
- +Integrated with LangWatch's full observability stack for traces and metrics.
- +Supports adversarial red-teaming like Crescendo escalation out of the box.
- −Extremely limited community feedback—only the team's own posts visible.
- −No independent reviews or real-world reliability data yet.
- −YouTube returned zero relevant content; low awareness outside HN.
- −Cost of running judge-agent simulations could add up at scale.
- −Setting up high-quality simulators and judges requires careful prompt work.
- • Cost of LLM API calls for user simulator and judge per simulation turn.
- • Cloud platform pricing not publicly visible.
- • Potential data storage costs for trace storage at scale.
Viability Score
How well maintained and how widely used is LangWatch Scenario? Built from what the vendor actually publishes (docs, changelog, tutorials, integrations, pricing), whether the site is live, and how much real users discuss it. How we calculate this
Last calculated: October 2026
How we score →Key Features
- LLM-powered simulated user generating realistic multi-turn text and voice conversations
- Judge agent reads the whole trace and returns pass/fail with reasoning at any turn
- Configurable success criteria written in natural language per scenario
- Tool-call, skill, and MCP server verification across long dialogues, with mocking and fixtures
- Framework-agnostic one-method adapters for LangGraph, CrewAI, Pydantic AI, Mastra, Vercel AI SDK
- Run scenarios locally in pytest or vitest and gate merges in CI/CD
- Agent Testing v2 (3.17.0): create and run scenarios in the UI without code, plus comparison runs
- Voice agent testing on ElevenLabs, OpenAI Realtime, Twilio, Pipecat, Gemini Live
- Voice latency metrics (TTFB, p50/p95) with background-noise and interruption injection
- Adversarial red-teaming: Crescendo escalation, refusal detection, backtracking
- Instant Evals: query your entire stored production history and get matches back in minutes
- Langy automated AI engineer drafts test plans, writes JudgeAgent rubrics, opens prompt-revision PRs
- Open-source Scenario SDK for Python and TypeScript
- Go SDK 1.0 with dataset search and agent discovery from connected providers
- Scripted to auto-pilot simulations: choose your level of control per turn
About LangWatch Scenario
LangWatch Scenario is the agent simulation layer inside LangWatch: you describe the behavior you want to test in plain language and an LLM-powered simulated user pushes your agent turn after turn, in text or voice. A judge agent reads the whole trace and returns pass/fail with the reasoning behind each verdict, so long multi-step conversations get asserted rather than eyeballed. Who it's for: ML engineers and QA teams shipping agents with tool calls, multi-turn reasoning, or realtime voice. Scenario sits alongside LangWatch's observability, evals, prompt registry, and gateway, so traces, tests, and prompts live in one place instead of three vendors. Framework-agnostic one-method adapters cover LangGraph, CrewAI, Pydantic AI, Mastra, the Vercel AI SDK, or plain OpenTelemetry, with whitebox or blackbox testing. The mechanics are the draw. Tool calls, skills, and MCP server invocations are traced and mockable or fixturable for deterministic runs. The same scenarios run locally in pytest/vitest and on every pull request. Voice testing spans ElevenLabs, OpenAI Realtime, Twilio, Pipecat, and Gemini Live with TTFB and p50/p95 latency plus noise and interruption injection. Adversarial runs — Crescendo escalation, refusal detection, backtracking — probe for jailbreaks and unsafe tool calls. Recent releases reshaped the product: 3.17.0 renamed Simulations to Agent Testing with no-code scenario creation and comparison runs, Instant Evals lets you query your entire stored history, the Go SDK hit 1.0 with dataset search, and Langy drafts test plans and opens prompt-revision PRs. Alternatives like Langfuse and Arize lead on pure observability and single-turn scoring; Scenario is the pick when simulation needs to sit in the same stack you already read traces in.
Behind the Verdict
Most agent teams test the happy path by hand and learn the rest from production. Scenario is built for the gap: you write the behavior you want in plain language, an LLM plays the user across many turns, and a judge reads the full trace and explains its verdict instead of handing back a 0.73. We'd reach for this when the agent takes a hundred paths to the same goal, when tool calls and MCP invocations need asserting mid-dialogue, or when voice is in the picture — the ElevenLabs / OpenAI Realtime / Twilio / Pipecat / Gemini Live coverage plus TTFB and p95 latency and interruption injection is unusually complete. Running the same scenarios in pytest, vitest, and CI is what turns this from a demo into a merge gate. Where it bites: you want a developer owning SDK work. Scenario is code-first at its core — adapters, mockable tool calls, fixtures — even with the no-code workbench layered on top. Solo builders with a handful of prompts will hit the free tier's 3 scenarios and stay there. Closest alternative is Langfuse or Arize. Both are credible on observability and single-turn scoring, and if that's your whole need, they're cheaper and less work. The counter-argument is stack sprawl: Scenario is natively integrated with tracing, evals, prompt registry, and the gateway, so you're not stitching simulation onto a tool that stops at logs. The 2026 releases matter for evaluation. Agent Testing v2 lets you create and run scenarios in the UI, comparison runs let you A/B two versions, and Instant Evals queries your entire stored history in minutes rather than a batch queue. Langy closing the loop from PM-written goal to prompt-revision pull request is the most interesting shift — it moves testing left, into the spec. Watch the bill as you scale. Growth is €29 per core seat and
Researching LangWatch Scenario? Get your full AI stack in 60 seconds.
Free, no signup — tell us your goal and get tools matched to your budget & existing stack.
Real-world workflow fit
Concrete scenarios for the personas LangWatch Scenario actually fits — and what changes day-one when you adopt it.
Describes the refund flow in plain language inside Claude Code, lets Scenario write the test, mocks the payment tool calls for deterministic runs, and runs the scenario locally with pytest before opening a PR.
Outcome: The 10-turn refund dialogue is asserted turn by turn with judge reasoning attached, instead of being eyeballed once before release.
Runs end-to-end voice simulations against ElevenLabs or OpenAI Realtime with background-noise injection and interruption injection, then reads TTFB and p95 latency per run.
Outcome: Latency regressions and barge-in failures show up in the simulation rather than in a customer call.
Writes the goal in plain English, has Langy draft the Scenario test plan and JudgeAgent rubric, then reviews the prompt-revision pull request Langy opens through the Prompt Registry.
Outcome: The PM owns the spec and the developer reviews a concrete PR instead of trading requirements in chat.
Use Cases
- Simulate a customer support agent handling complex refund requests over 10 turns
- Gate CI/CD merges on simulation pass/fail criteria for every pull request
- Red-team a voice agent for prompt injection and policy violations
- Generate a full test plan and JudgeAgent rubric from a PM's plain-English goal via Langy
- Replicate a failing production trace as a simulation and prove the fix before shipping
- Verify that the right tools are called at the right moments across long agent dialogues
- Measure TTFB and p95 latency of a voice agent with background-noise injection
- Run A/B comparison runs between two prompt variants in the workbench
Models Under the Hood
as of 2026-10-18
Limitations
- The Developer plan caps you at 3 scenarios, 3 simulations, and 3 custom evals with 2 users and 14-day data access, so anything beyond a proof of concept moves you to Growth at €29/core-seat/month.
- Growth includes 200k events, then charges €5 per 100k events; retention past the included 30 days bills at €3/GB.
- Self-hosting is available (self-host in 15 min), and the Scenario SDK is open source, while enterprise features such as SSO/RBAC, audit logs, SLAs, and custom data retention require the Custom plan.
- Writing useful scenarios still requires someone who can describe the behavior you want to test.
as of 2026-10-10
Verification history
We have re-verified LangWatch Scenario 9 times since . Each pass re-reads the vendor's own pages and re-checks every listed field against that evidence; passes where nothing had changed are marked as such.
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-checked, vendor evidence unchanged
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
Showing the 6 most recent of 9 verification passes.
Free to cite with attribution — this page re-verifies continuously.
12-month cost
Project the real annual outlay, including the implied monthly cost when only an annual tier is published.
Vendor list price only. Add-on usage, seat overages, and contract minimums are surfaced under Hidden costs & gotchas.
Plans compared
For each published LangWatch Scenario tier: who it actually fits, and what it adds vs. the previous tier. Cross-reference the cost calculator above for projected annual outlay.
Developer
€0/mo
Ideal for
A solo engineer or a proof of concept that needs to validate the simulation loop before committing budget.
What this tier adds
Free entry point at €0/mo: 50k events/month, 14-day data access, 2 users, and 3 scenarios, 3 simulations, 3 custom evals.
Growth
€29 / core-seat / month
Ideal for
A team shipping agents to production that gates pull requests on simulation and eval results.
What this tier adds
Adds unlimited simulations, evals and prompts, 200k included events, 30-day retention, unlimited lite-users, and private Slack/Teams support for €29 per core seat per month.
Enterprise
Custom
Ideal for
Regulated teams that need deployment control, SSO/RBAC, audit logs, and contractual assurance.
What this tier adds
Adds hybrid, self-hosted or on-prem deployment, custom data retention, custom SSO/RBAC, audit logs, SLAs, ISO 27001 reports, and a Forward Deployed Engineer.
Where the pricing makes sense
The company stage and team size where LangWatch Scenario's pricing actually pencils out — and where peers do it cheaper.
Developer at €0/mo fits a solo engineer or a proof of concept that never exceeds 3 scenarios, 3 simulations, and 3 custom evals. Growth at €29 per core seat per month plus €5 per 100k events beyond 200k is the realistic tier for a team gating PRs on simulations. Above that, self-hosted or Enterprise pricing competes on control rather than seat count; observability-only alternatives such as Langfuse can be cheaper if you never run multi-turn simulations.
Setup time & first value
How long it actually takes to get something useful out of LangWatch Scenario — broken out by persona, not the marketing-page minute.
A developer instrumenting an existing agent through a LangGraph, CrewAI, or Pydantic AI adapter typically reaches a first traced simulation run in an afternoon, since the adapter is a single method and scenarios can be drafted from a coding assistant. The no-code path through the UI (Agent Testing v2) is faster for a PM-led first scenario. Self-hosting the open-source core takes about 15 minutes
Switching to or from LangWatch Scenario
How to bring data in from common predecessors and how to get it back out — written for the switcher, not the buyer.
- →From hand-written pytest or vitest agent tests: keep the runner and replace scripted turns with Scenario's simulated user plus judge scoring.
- →From a single-turn eval tool such as Langfuse: export the dataset, then re-express the criteria as multi-turn scenarios with per-turn verdicts.
- →From a custom synthetic-data script: move the persona and success criteria into a Scenario definition and let the framework handle turn-taking.
- →From manual QA scripts for voice agents: port the checklist into voice scenarios with noise and interruption injection, then read TTFB and p95.
- ↗To Langfuse or Arize: keep using them for observability and single-turn scoring, and retain Scenario only if multi-turn simulation is the reason you adopted it.
- ↗To an in-house harness: the Scenario SDK is open source in Python and TypeScript, so the scenario definitions can be re-implemented against your own runner, though you rebuild the judge and visualizer.
- ↗To a generic LLM eval framework: expect to lose per-turn pass/fail on the trace and the tool-call mocking that makes long dialogues deterministic.
Integrations
Resources & Guides
- Documentationlangwatch.ai
Docs · LangWatch Scenario
Full product docs from langwatch.ai
- Documentationlangwatch.ai
Overview · LangWatch Scenario
Full product docs from langwatch.ai
- Documentationlangwatch.ai
Connect Your Agent · LangWatch Scenario
Full product docs from langwatch.ai
- Documentationlangwatch.ai
Scenarios · LangWatch Scenario
Full product docs from langwatch.ai
- Documentationlangwatch.ai
Voice Agents · LangWatch Scenario
Full product docs from langwatch.ai
- Documentationlangwatch.ai
Red Teaming · LangWatch Scenario
Full product docs from langwatch.ai
- Documentationlangwatch.ai
Llms · LangWatch Scenario
Full product docs from langwatch.ai
Tutorials & Learning
LangWatch Scenario MCP - Writing Agent Tests (simulations) automatically for you
LangWatch
LangWatch Scenarios - AI Agent Testing
LangWatch
YouTube returned 6 videos for “LangWatch Scenario”, and we withheld 4: 4 did not mention LangWatch Scenario. Showing the 2 we can prove are about LangWatch Scenario.
Official links
Tools that pair well with LangWatch Scenario
Common stack mates teams adopt alongside LangWatch Scenario, with the specific reason each pairing earns its keep.
Future AGI
Agent evaluation platform that simulates voice and chat agents, traces production, and scores hallucinations before customers hit them
Arena AI
Arena AI ranks chat models, coding agents and fullstack builds on live head-to-head human votes.
CopilotKit
Open-source React framework for agent chat, generative UI, and shared state on any AG-UI backend.
Featured Head-to-Head Comparisons
Langwatch Scenario vs Locus Robotics
Locus Robotics and LangWatch Scenario solve completely different problems. Locus is a physical warehouse automation platform for high-volume picking/packing; LangWatch is a software testing framework for AI agent conversations. A warehouse operator would choose Locus, and an AI engineer would choose LangWatch. There is no direct competition.
Langwatch Scenario vs Presto Voice
Choose Presto Voice if you operate a QSR chain and need a production-ready drive-thru voice AI that boosts revenue through upselling. Choose LangWatch Scenario if your team builds conversational agents and needs a robust testing framework to catch failures before deployment. They solve entirely different problems: one is an end-to-end operational solution, the other is a developer tool for quality assurance.
Langwatch Scenario vs Truleo
These tools serve completely different domains — Truleo is a law enforcement intelligence platform, while LangWatch Scenario is a developer tool for testing AI agents. Choose based on your field: if you're in policing, Truleo's data integration and case lead generation is unmatched; if you build conversational AI, LangWatch Scenario's multi-turn simulation and adversarial testing are essential. They are not direct competitors, but for an AI buyer, select the one aligned with your organization's purpose.
Alternatives to LangWatch Scenario
View all
Future AGI
Agent evaluation platform that simulates voice and chat agents, traces production, and scores hallucinations before customers hit them
Arena AI
Arena AI ranks chat models, coding agents and fullstack builds on live head-to-head human votes.
CopilotKit
Open-source React framework for agent chat, generative UI, and shared state on any AG-UI backend.
Frequently Asked Questions
Best-of guides
Topics
Used LangWatch Scenario? Help shape our editorial sentiment research.