LMCache

LMCache is for teams that need to squeeze more throughput out of their LLM serving stack without swapping out their inference engine. It's particularly strong for workloads with long prompts, chat history, or RAG — anywhere you see repeated content across requests. When to pick it: If you're using vLLM or TGI and have a technical team that can set up a Python-based cache layer, LMCache can cut prefill time significantly. The 3-10x speedups reported on AMD MI300X or with tiered storage on GKE are real. The fact it's open-source and research-backed (CacheGen, CacheBlend) gives confidence it won't become abandonware. When to pass: If your team is non-technical or you need a one-click GUI solution, LMCache isn't ready. Setup requires Python, the CLI, and understanding of KV cache concepts. Also, if your workloads are highly dynamic with little cache reuse (e.g., unique prompts each time), the gains will be marginal. Closest alternative: vLLM's built-in prefix caching is simpler but less capable — no tiered storage, no compression, no cross-engine reuse. LMCache extends that functionality significantly. Real-world caveats: The multiprocess deployment mode is recommended and the focus of future development, but it's more complex to set up. Documentation is good but assume some tinkering. No pricing tiers — it's free and open-source, but you'll pay for the infrastructure (GPUs, storage) yourself.