LiteLLM

Sweet spot: a platform or infra team at a company where many product teams are calling many models. The LiteLLM Proxy solves a genuinely hard set of problems — key management, budgets, failover, observability — with one lightweight service. The alternative is building that same stack yourself; teams that try inevitably end up with a worse version of LiteLLM. Failure modes. For a solo developer or single-team project hitting only OpenAI, LiteLLM is overhead you do not need. Some provider-specific features (OpenAI's new structured outputs at launch, Anthropic's streaming tool deltas) lag the abstraction by a few weeks. If your workload is latency-critical (sub-second response targets), the extra network hop is worth measuring. What to pilot. Deploy the LiteLLM Proxy in a staging environment, point two teams' workloads at it, and turn on cost tracking. After two weeks, check: is the cost dashboard telling you something you did not already know? Did failover save an outage? If the answer to either is yes, productionise it. If neither, stay with direct provider SDKs.

LiteLLM is the glue layer that lets you call 100+ different LLM providers (OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Together, Groq, Fireworks, Ollama, and many more) through a single OpenAI-compatible API. You write OpenAI-style code once, LiteLLM routes the actual call to whichever provider you named in the model string. It ships in two forms. The Python SDK is a drop-in for openai-python that handles the routing. The LiteLLM Proxy is a standalone server (FastAPI + Postgres) that sits in front of all your model traffic — adding authentication, virtual keys, per-team budgets and rate limits, model-level fallbacks, logging to Langfuse or Helicone, and cost tracking. For any organisation that has more than one team using more than one provider, the Proxy quickly becomes indispensable. LiteLLM is open source (MIT) and there is an Enterprise tier for SSO, audit logs, and priority support. It is used in production by many mid-size-and-up organisations as the spine of their AI platform.