Langfuse

Langfuse's core strength is that it treats observability, prompt management, evaluation, and experiments as one product on one data model rather than four bolted-together tools. A trace you captured can be turned into a dataset, run through an experiment, scored by an LLM-as-a-judge evaluator, reviewed in an annotation queue, and alerted on via Slack or webhook — all without exporting data anywhere. The observability layer itself is detailed: hierarchical traces cover LLM and non-LLM calls including retrieval, embedding and API calls; sessions group multi-turn conversations; agent graphs visualize complex agentic flows; and timelines help you isolate latency problems. Cost and token tracking is per-user, which makes multi-tenant billing tractable. On the evaluation side, evaluators can be attached to rules and now backfill scores onto historical observations, so you don't lose signal from data collected before the evaluator existed; evaluators can be versioned, restored as drafts, created from templates for chatbots, topic detection, exact matches and coding agents, and managed through stable ID-based public APIs. Sampling is consistent across evaluators so comparisons are apples-to-apples. Recent work leans hard into agent-native tooling: the Langfuse Assistant investigates production data and takes approved actions inside the app, runs code over thousands of observations in a background sandbox, and can build datasets and dashboards on request. Coding agents get in through an MCP server, CLI 1.0 (10x+ faster invocations, agent-readable exit codes) and a SKILL.md, so prompts and traces are manageable from an IDE. Multi-modal support extends to images, audio and video in traces, and LLM-as-a-judge evaluators handle multi-modal inputs and multi-message prompts. The operational story is where it diverges from managed competitors. Self-hosting runs on Docker Compose, Kubernetes via Helm, or Terraform on AWS, GCP or Azure, all features are MIT licensed, and self-hosted deployments get a built-in interactive API reference that works even air-gapped. There's an instance switcher for juggling multiple deployments and organized feature previews for admins. The tradeoff is real: if you self-host, you're operating ClickHouse, Redis and blob storage yourself, which is a genuine platform-engineering commitment. Langfuse Cloud removes that, but every paid tier meters: Core, Pro and Enterprise include 100k units/month and then charge $8/100k units, with volume discounts available through the pricing calculator. Data retention is tiered — 30 days on Hobby, 90 days on Core, 3 years on Pro — and SSO (e.g. Okta), SSO enforcement, fine-grained RBAC and a dedicated Slack/MS Teams channel sit in the $300/mo Teams add-on on top of Pro. Audit logs, SCIM provisioning, custom rate limits and SLAs are Enterprise-only at $2,499/mo on a yearly commitment. API rate limits also vary by tier (Core general API 30 requests/min, datasets API 100 requests/min) and the Hobby