LangWatch
Open-source LLMOps platform for simulating, evaluating, and observing agentic AI before production.
LangWatch is the strongest pick we've reviewed for teams whose main problem is agent reliability, not just dashboards. The Agent Testing surface — simulated users in text and voice, JudgeAgent verdicts at any turn, tool-call verification across long dialogues — is more thorough than what observability-first rivals such as LangFuse or Helicone provide, and the red-teaming (Crescendo, refusal detection) is unusual at this price. Langy turning a PM brief into tested pull requests cuts a real coordination cost. Choose LangWatch if you ship multi-step agents and want tests gating merges; choose LangFuse or Phoenix if you only need tracing and want a wider free tier.
Verified 10h ago · liveness 87/100 · cite: rightaichoice.com/tools/langwatch
- AI engineering teams shipping multi-step agentic systems
- Voice AI teams testing simulated callers before launch
- Enterprises that need pre-production eval rigor and audit trails
- Platform teams centralising LLM gateway and coding-agent spend
- Solo developers who mainly want a generous free tracing tier
- Teams that only need basic LLM observability with no testing or evals
- Buyers wanting flat usage-based pricing with no per-seat component
We scan live Reddit threads, YouTube comments, X posts, G2 reviews and other communities — and hand you an honest verdict in under a minute.
- Honest verdict, not marketing
- Real pros & cons from real users
- Attributed quotes with receipts
3 free scans · no card needed
Skip LangWatch if you only need lightweight production tracing and don't intend to run simulations or evals — you'd be paying per-seat for testing machinery you won't use, and LangFuse's free tier covers basic tracing more cheaply.
Growth includes 200k events per month, then charges €5 per 100k — a high-traffic agent can blow past that allowance within days.
At €29 per core-seat with 200k events included, Growth fits funded teams of roughly 5–20 engineers shipping agents, and gets cheaper than LangFuse Cloud Team only once you're actually using simulations and evals. Below that size the free Developer tier's 2-user cap pushes you to pay too early; above 20 seats, ask for the volume discount and compare against self-hosting the Apache 2.0 core.
In short
LangWatch — Open-source LLMOps platform for simulating, evaluating, and observing agentic AI before production. Best for AI engineering teams shipping multi-step agentic systems, Voice AI teams testing simulated callers before launch, Enterprises that need pre-production eval rigor and audit trails. Free to start; paid plans from €29/mo.
What's new in LangWatch
Checked todayAcross the latest 5 updates: 5 feature updates.
3.17.0 Agent Testing v2 and Prompt Optimization in the Workbench
Agent Testing replaces Simulations, adds comparison runs, and lets Langy optimize prompts directly in the workbench.
3.16.0 New Navigation and Voice APIs support on the Gateway
The sidebar becomes a product switcher with an icon rail, and realtime voice usage now bills to a virtual key.
3.13.0 A Governed SQL Workbench, and the Go SDK
Query your trace and eval data in a governed SQL workbench, and instrument Go applications with a new SDK.
3.11.0 Coding Agent Capture, and Gateway Billing
Capture Copilot and Claude Code sessions for token and cost visibility, and bill gateway spend with webhooks and reconciliation.
3.8.0 The AI Gateway Grows Budgets
Budgets can now be set on every dimension of gateway traffic, with live spend tracking and a redesigned key drawer.
What people actually say about LangWatch — is it worth it?
We ran a structured research pass across product reviews, community discussions, and post-purchase forum threads to surface the patterns vendors won't publish themselves. Below: the recurring strengths, the hidden costs people mention most, and the cohort that consistently regrets adopting this tool.
45 mentions across 4 sources (Hacker News, YouTube, Product Hunt, GitHub) · researched Jul 31, 2026.
Average across the 4 sources that answered — each source counts once, not each post.
- +Simulation-first testing with synthetic users is genuinely unique.
- +Per-turn JudgeAgent scoring provides granular pass/fail insights.
- +Voice agent testing with simulated callers covers a rare niche.
- +OpenTelemetry-native tracing avoids vendor lock-in.
- +Prompt management includes version control, rollouts, and A/B testing.
- −Free tier is restrictive, encouraging quick upgrades.
- −Self-hosting setup with ClickHouse is complex for smaller teams.
- −Go SDK only fully supports OpenAI; other providers lag.
- −Large number of open issues may indicate rough edges.
- −Features are sprawling, potentially overwhelming new users.
- • Self-hosting requires you to run ClickHouse and other infrastructure, which costs in ops time and server resources
- • Additional usage over plan limits may incur per-seat or per-event charges (not clearly documented)
- • Enterprise features like SSO and audit logs are only on higher tiers
Viability Score
How well maintained and how widely used is LangWatch? Built from what the vendor actually publishes (docs, changelog, tutorials, integrations, pricing), whether the site is live, and how much real users discuss it. How we calculate this
Last calculated: September 2026
How we score →Key Features
- Agent simulations with LLM-powered simulated users in text and voice
- Multi-turn conversation testing with JudgeAgent verdict at any turn
- Configurable natural-language success criteria per scenario
- Tool-call verification across long dialogues, with mockable/fixtured tools
- Voice agent testing end-to-end with ElevenLabs, OpenAI Realtime, Twilio, Pipecat, Gemini Live
- Voice latency metrics (TTFB, p50/p95) plus noise and interruption injection
- Adversarial red-teaming: Crescendo escalation, refusal detection, backtracking
- OpenTelemetry-native tracing with full GenAI spec support
- Trace views: waterfall, flame graph, topology, sequence diagram
- Cmd+K command bar for jumping to any trace, filter or view
- Online evaluations and monitors running on production traffic
- Multimodal evaluations for images and mixed media
- Built-in evals: RAGAS, hallucination, toxicity, PII, LLM-as-a-judge
- Custom evals built visually in a workflow builder
- Prompt management with versioning, tags, rollouts, A/B testing and GitHub sync
About LangWatch
LangWatch is an open-source LLMOps platform for teams shipping complex agentic AI to production. It covers four areas you can adopt separately: LLM observability, agent testing, evaluations, and prompt management — plus newer Coding Agent tracking, an AI Gateway, and AI Governance. The core idea is simulation-first testing: instead of hand-writing a handful of prompts, you run multi-turn conversations against your agent with an LLM-powered simulated user in text or voice, define success criteria in plain language, and let a JudgeAgent that reads the whole trace return a verdict. Adversarial red-teaming (Crescendo escalation, refusal detection, backtracking) probes for jailbreaks and unsafe tool calls before your users find them. Evaluations run offline over datasets or online against production traffic, with built-in scorers like RAGAS, hallucination, toxicity and PII, plus multimodal scoring for images and mixed media. Observability is OpenTelemetry-native, traces every tool call, skill and MCP server, and tracks tokens, cost and cache per span — including for coding agents like Claude Code, Codex, Copilot, Gemini CLI and opencode. Langy, the automated AI engineer launched July 2026, turns a PM's plain-English goal into a scenario test plan, runs it in parallel, scores it, and drafts prompt revisions as pull requests, which the vendor reports at a median PM-to-PR time of 14 minutes. The platform is open core: the Apache 2.0 core self-hosts via Helm or Docker Compose (no Elasticsearch since 3.0 moved to ClickHouse), while LangWatch Cloud runs the same codebase. It's aimed at AI engineering teams and enterprises that need rigorous pre-production validation, not solo developers who just want cheap tracing.
Behind the Verdict
LangWatch's real differentiator is that it treats agent testing as a continuous engineering loop rather than a QA afterthought. You describe the behaviour you want in plain language — in Claude Code or in the UI — and Scenario writes the test; the same scenarios run on your machine in pytest or vitest and again on every pull request with no separate setup. That CI parity matters, because the failure mode for agent teams is passing locally and degrading in production. The JudgeAgent is a step above single-output scoring: it expands each step of the trace, so you get reasoning behind a verdict rather than a bare number, and it can decide pass/fail at any turn of a conversation. Tool-call verification across long dialogues plus mockable or fixtured tool calls lets you get deterministic runs out of non-deterministic agents. Voice testing is genuinely end-to-end, with ElevenLabs, OpenAI Realtime, Twilio, Pipecat and Gemini Live, TTFB and p50/p95 latency metrics, and background-noise and interruption injection — a combination few competitors ship. Observability is solid but not unique: OpenTelemetry-native tracing, waterfall, flame graph, topology and sequence views, Cmd+K navigation, topic clustering and custom metric graphs. The Coding Agent capture and AI Gateway work is where it gets interesting for platform teams: one OpenAI/Anthropic-compatible endpoint with per-caller virtual keys and budgets, every request landing as a trace, and spend reconciliation through webhooks. Weaknesses are real. The free Developer tier is tight — 50k events, 14-day retention, 2 users, 3 scenarios and 3 simulations — so evaluation at scale starts at €29 per core-seat. Per-seat pricing plus usage-based events and storage means costs can stack: €5 per 100k events beyond 200k, €3 per GB beyond 30 days of retention. Governance-grade features (SSO/RBAC, audit logs, SLAs, ISO 27001 reports) are Enterprise-only, which frustrates mid-size regulated teams. Self-hosting the Apache 2.0 core is now simpler with ClickHouse instead of Elasticsearch, but it's still infrastructure you operate, and enterprise governance features live under a separate license. Fit-wise: great for AI engineering teams with 5+ people shipping agentic products, voice AI teams, and platform groups centralising gateway and coding-agent spend. Poor fit for a solo builder who wants free tracing, or a team with no simulation or eval appetite.
Researching LangWatch? Get your full AI stack in 60 seconds.
Free, no signup — tell us your goal and get tools matched to your budget & existing stack.
Real-world workflow fit
Concrete scenarios for the personas LangWatch actually fits — and what changes day-one when you adopt it.
Writes a scenario in Claude Code describing a refund dispute, runs it locally in pytest, then promotes the same scenario to CI where it gates every pull request.
Outcome: Multi-turn regressions get caught before merge instead of surfacing as production complaints, with JudgeAgent reasoning attached to each failed turn.
Writes the goal in plain English in Langy; Langy picks the simulator, generates scenarios and the JudgeAgent rubric, runs them in parallel, and drafts a prompt revision as a pull request.
Outcome: A tested prompt revision reaches a developer for review in the vendor-reported median of 14 minutes, with no YAML or test code written by the PM.
Routes every LLM call through the AI Gateway with virtual keys and per-caller budgets, and enables coding-agent capture for Claude Code and Copilot.
Outcome: One dashboard shows spend, usage and anomalies per team and per session, with gateway spend reconciled through webhooks.
Use Cases
- Run hundreds of synthetic multi-turn conversations against your support agent before it reaches customers
- Gate pull requests in CI so a prompt or agent change can't merge if eval scores regress
- Simulate voice agent calls end to end with interruption and background-noise injection, scoring latency alongside quality
- Turn a production trace into a test scenario to reproduce a bug and prove the fix
- Centralise every LLM call behind one gateway endpoint with per-caller virtual keys and budgets
- Track token spend per session and per team across Claude Code, Codex and other coding agents
- Run online monitors on production traffic and alert when quality or cost drifts
- Evaluate images and mixed media, not just text, with the same scoring surface
Models Under the Hood
as of 2026-09-14
Limitations
- Self-hosting the Apache 2.0 core now runs on ClickHouse rather than Elasticsearch, but it is still infrastructure you operate and upgrade yourself, and enterprise-only features sit under a separate license in platform/app/ee/.
- The free Developer tier is deliberately narrow: 50k events per month, 14-day data access, 2 users, and caps of 3 scenarios, 3 simulations and 3 custom evals — not enough for a team's real test suite.
- Costs beyond the plan are usage-based: €5 per 100k events over the 200k Growth allowance and €3 per GB for data kept past 30 days, on top of €29 per core-seat.
- Governance essentials — custom SSO/RBAC, audit logs, SLAs, ISO 27001 reports and custom DPAs — are Enterprise-only, so mid-size regulated teams can't get them on Growth.
- Growth seats are billed per core-seat, so wide-but-shallow adoption gets expensive.
as of 2026-09-14
Verification history
We have re-verified LangWatch 17 times since . Each pass re-reads the vendor's own pages and re-checks every listed field against that evidence; passes where nothing had changed are marked as such.
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
Showing the 6 most recent of 17 verification passes.
Free to cite with attribution — this page re-verifies continuously.
12-month cost
Project the real annual outlay, including the implied monthly cost when only an annual tier is published.
Vendor list price only. Add-on usage, seat overages, and contract minimums are surfaced under Hidden costs & gotchas.
Plans compared
For each published LangWatch tier: who it actually fits, and what it adds vs. the previous tier. Cross-reference the cost calculator above for projected annual outlay.
Developer
$0/mo
Ideal for
An individual developer or two-person team evaluating LangWatch on a real side project before committing budget.
What this tier adds
Free entry point: 50k events/month, 14-day data access, 2 users, and caps of 3 scenarios, 3 simulations and 3 custom evals.
Growth
€29 / core-seat / month
Ideal for
A team of roughly 5–20 engineers actually shipping agents to production and running tests in CI.
What this tier adds
Adds 200k included events, 30-day retention, unlimited lite-users and unlimited simulations, evals and prompts, plus private Slack/Teams support.
Enterprise
Custom
Ideal for
Regulated organisations whose security and legal reviews demand SSO, audit trails and contractual assurances.
What this tier adds
Adds hybrid/self-hosted or on-prem deployment, custom retention, custom SSO/RBAC, audit logs and SLAs, ISO 27001 reports and a Forward Deployed Engineer.
Where the pricing makes sense
The company stage and team size where LangWatch's pricing actually pencils out — and where peers do it cheaper.
At €29 per core-seat with 200k events included, Growth fits funded teams of roughly 5–20 engineers shipping agents, and gets cheaper than LangFuse Cloud Team only once you're actually using simulations and evals. Below that size the free Developer tier's 2-user cap pushes you to pay too early; above 20 seats, ask for the volume discount and compare against self-hosting the Apache 2.0 core.
Setup time & first value
How long it actually takes to get something useful out of LangWatch — broken out by persona, not the marketing-page minute.
Vendor documentation claims you can self-host in about 15 minutes and that instrumenting with OpenTelemetry takes minutes. Realistically: a developer already using a supported framework can get traces flowing the same day via the SDK or coding-assistant skills; a team wiring scenarios into CI should plan a few days; standing up self-hosted ClickHouse with Helm or Docker Compose is a half-day plus
Switching to or from LangWatch
How to bring data in from common predecessors and how to get it back out — written for the switcher, not the buyer.
- →From LangFuse: point your existing OpenTelemetry instrumentation at LangWatch and add Scenario tests for the agent paths you previously only traced.
- →From a hand-rolled eval script: move datasets into LangWatch datasets (CSV or bulk upload) and rebuild the checks as reusable evaluators.
- →From Helicone or a proxy-based tracer: move the proxy layer to the AI Gateway so requests land as traces with virtual keys and budgets.
- →From no testing at all: start by turning one recurring production failure trace into a Scenario and gating it in CI.
- ↗To LangFuse: export traces and datasets and re-instrument with the LangFuse SDK; you lose Scenario simulations and JudgeAgent verdicts.
- ↗To Phoenix or OpenTelemetry-only tooling: keep the OTel instrumentation and point the exporter at the new backend.
- ↗To an in-house stack: self-host the Apache 2.0 core indefinitely, or fork it, since the core is open source.
Integrations
Resources & Guides
- Documentationlangwatch.ai
The Complete LLMOps Platform
Accelerate your agent development lifecycle with comprehensive observability, evaluations and agent simulations. Open-source platform, with over 3k stars on GitHub.
- Resourcelangwatch.ai
LangWatch Blog: AI Agent Testing and LLM Development Insights
Expert insights on AI agent testing, LLM monitoring, DSPy optimization, and evaluation frameworks. Technical guides for AI developers and engineers.
- Documentationlangwatch.ai
Observability & Tracing
Monitor, debug, and optimize your LLM applications with comprehensive observability and tracing capabilities
- Documentationlangwatch.ai
Evaluations Overview
Ensure quality and safety for your LLM applications with experiments, online evaluation, guardrails, and evaluators.
- Documentationlangwatch.ai
Overview
Organize, version, and optimize your AI prompts with LangWatch's comprehensive prompt management system
- Documentationlangwatch.ai
Self-Hosting Overview
Deploy LangWatch on your own infrastructure for full data control
- Documentationlangwatch.ai
Overview
Search, retrieve, and share LangWatch traces via the REST API. Traces capture the full execution of your LLM pipelines including all spans, evaluations, and metadata.
- Resourcelangwatch.ai
LangWatch Blog: AI Agent Testing and LLM Development Insights
Expert insights on AI agent testing, LLM monitoring, DSPy optimization, and evaluation frameworks. Technical guides for AI developers and engineers.
Tutorials & Learning
LangWatch LLM Optimization Studio
LangWatch
Getting started with LangWatch Optimization Studio (2/4): Datasets
LangWatch
YouTube returned 6 videos for “LangWatch”, and we withheld 4: 4 could not be judged, because “LangWatch” is a single word that other videos use for other things. Showing the 2 we can prove are about LangWatch.
Official links
Popular in LLM Observability & Evals
Arize Phoenix
Open-source LLM observability and evals for building reliable agents
Frequently Asked Questions
Used LangWatch? Help shape our editorial sentiment research.