Alternatives to Tokentelemetry

LangSmith is agent and LLM observability: trace, monitor, and evaluate AI agents in cloud, BYOC, or self-hosted.

OpenJudge is an Apache 2.0 open-source AI evaluation framework shipping 50+ benchmark-validated graders for agents, LLMs, code, math, and multimodal models.

Open-source LLM observability and agent evals: trace, evaluate, and improve production AI agents with MIT-licensed tooling.

Honeycomb Query Assistant turns plain-English questions into HQL against live production telemetry.

Open-source AI observability for agent tracing, LLM-as-a-judge evals, and coding-agent cost tracking.

Token Monitor is a free desktop widget that tracks AI coding token usage, cost, and quota limits across 44+ tools and providers.

Arena AI ranks chat models, coding agents and fullstack builds on live head-to-head human votes.

Galileo is an AI observability and eval engineering platform that turns offline evals into production guardrails for agents and RAG.

Helicone is an AI gateway and LLM observability platform for routing, debugging, and cost-tracking AI app traffic.

AppWorld is a simulated-world benchmark for evaluating AI coding agents across 9 apps and 457 APIs.

OMNI is an open-source shell hook that shrinks terminal output before your coding agent reads it, so it stops paying twice for bytes

Free open-source AI usage tracking and analytics that syncs your token and cost data from Claude, Cursor, ChatGPT and other tools onto a shareable dashboard.

Accordion is a free, MIT-licensed pi extension that shows your coding agent's entire context window as blocks you can fold, unfold, and steer.

Self-hosted layer that gives your AI coding agents shared memory, managed MCP tools, and cross-model audit.

Free MIT-licensed macOS menu bar app that tracks quota windows, spend, and burn pace for 13 AI coding providers

Arize Phoenix is open-source LLM observability: trace agent steps, run evals, and self-host your traces.

Dash0 is OpenTelemetry-native observability where Agent0 investigates incidents, finds root cause, and opens fix PRs.

Open-source LLM observability and agent evaluation you can trace, evaluate, and self-host for free.

Galileo AI Evals turns offline evaluations into production guardrails that gate your AI agents in real time.

LLM evaluation, observability, and red teaming in one workspace — ship AI you can actually trust

TruLens traces AI agent runs with OpenTelemetry-native evals, showing where an agent fails and where you can cut cost.

ToolSpend tracks AI API spend across OpenAI, Anthropic, Google AI, Bedrock and Azure in one live cost dashboard.

Agent evaluation platform that simulates voice and chat agents, traces production, and scores hallucinations before customers hit them

Fiddler AI is an enterprise AI control plane for agent observability, inline guardrails, and auditable AI governance.

Open-source AI coworker workspace that builds, schedules and ships agents to Slack or Telegram

Mastra is an open-source TypeScript agent framework for durable agents, typed workflows, and built-in evals.

Maxim AI evaluates, simulates and observes AI agents — with Bifrost, its open-source LLM gateway, handling routing, budgets and guardrails.

PostHog is product analytics with session replay, feature flags, error tracking, and a context warehouse for you and your AI agents.

Open-source Python framework that replaces vibe checks with reproducible evaluation loops for RAG pipelines and AI agents.

Developer observability platform that traces, replays, and debugs AI agent runs across OpenAI, CrewAI, Autogen and 400+ LLMs