Langfuse Prompt Experiments

Langfuse is an open-source AI engineering platform designed to help teams debug, analyze, and improve their LLM applications. It integrates observability, prompt management, evaluation, and experimentation into a single workflow, enabling teams to move from prototype to production and continuously improve with real usage data. The platform captures hierarchical traces including LLM calls, tool invocations, and retrieval steps, and supports session tracking, user tracking, and agent graphs. Prompts can be versioned, deployed via labels, and tested in a playground. Evaluation includes LLM-as-a-judge, heuristic functions, and human annotation. Experiments can be run against datasets to compare prompt versions, and dashboards monitor cost, latency, and quality with alerts. Recent additions include multi-modal datasets (images, audio, video), monitors that notify Slack or webhooks, keyboard shortcuts for annotation queues, and an Ask AI feature that drafts filters from plain language. Langfuse is self-hostable or available as a cloud service, and is trusted by 19 of the Fortune 50 companies. It integrates with many frameworks (LangChain, Vercel AI SDK, LiteLLM, CrewAI, etc.) and model providers (OpenAI, Anthropic, Bedrock, etc.). The platform is built on OpenTelemetry for vendor-neutral instrumentation and offers SDKs for Python and TypeScript, with support for many languages via OTel. Compared to alternatives like Datadog or Helix, Langfuse offers a more focused LLM engineering workflow with an open-source core, avoiding vendor lock-in.