Opik (Comet)

The honest framing of Opik is that it is two products sharing a name. The first is a competent open-source tracer and evaluator — traces, annotations, LLM-as-a-judge metrics, guardrails, audit logs. That half competes directly with Langfuse and LangSmith, and the thing that actually separates it is the closing of the loop: Test Suites let you write global rules plus item-level assertions in plain text instead of hand-building an eval metric for each scenario, and Ollie then reads the failing trace, proposes a fix, applies it to your development code under version control, and writes a regression test alongside it. Most observability tools stop at 'here is what broke.' Opik tries to also do something about it, and that is the workflow worth evaluating the product on. The second product is Cost Intelligence, and it is aimed at a narrower and more current problem: engineering teams that cannot say what Claude Code and Codex are costing them per developer. The single live view over usage, MCP installs, skills, model selection, context retrieval, and configuration is the kind of internal-visibility surface that finance and engineering management both ask for, and Comet's own research posting on capturing a 400-turn Claude Code session suggests they are mining that telemetry rather than just reselling API invoices. If your AI spend line is dominated by coding agents, this is the reason to look. Where it is weaker: everything depends on instrumentation. Nothing happens until you wire in the SDK, and the eval model assumes you will author assertions or reference datasets yourself — there is no library of drop-in templates you can run unchanged. Self-hosting and the OpenTelemetry export path (now documented for TrueFoundry AI Gateway, alongside the Nebius AI Cloud listing) are real advantages for regulated teams, but they are also engineering work, and the enterprise surface around audit logs and guardrails sits with Comet's commercial offering rather than the free core. The comparison set is crowded — LangSmith, Langfuse, Arize, Braintrust, Galileo — and an honest read is that Opik is a strong pick for agent-heavy teams who want to own their data and want fixes, not a default recommendation for someone who wants zero-instrumentation managed tracing on day one.