LMCache
Open-source KV cache layer that stores, compresses, and reuses LLM key-value cache beyond GPU memory.
If repeated prefill is eating your GPU bill, LMCache is the most direct free fix in the open-source stack — and its presence inside Dynamo, GKE, CoreWeave, and SageMaker HyperPod deployments tells you it's been battle-tested. CacheBlend is the real differentiator for RAG, where naive prefix caching quietly misses. Budget for Python, CLI, and config work; there's no GUI and no vendor SLA.
Verified 16h ago · liveness 70/100 · cite: rightaichoice.com/tools/lmcache
- Developers serving low-latency LLM workloads on vLLM, TGI, or Nvidia Dynamo
- RAG teams whose retrieved chunks break naive prefix caching
- Platform engineers cutting GPU spend by pushing KV cache into tiered storage
- Distributed inference teams that need cache to survive worker restarts
- Teams that want a GUI, managed control plane, or vendor SLA instead of a Python library
- Operators without Python, config, or LLM serving-stack experience
- Workloads with little cache reuse, such as one-off unique queries per request
We scan live Reddit threads, YouTube comments, X posts, G2 reviews and other communities — and hand you an honest verdict in under a minute.
- Honest verdict, not marketing
- Real pros & cons from real users
- Attributed quotes with receipts
3 free scans · no card needed
Skip LMCache if your requests share almost no prompt context or retrieved content, since cache reuse is the only thing it optimises for.
Self-hosting the cache tier means you pay for the CPU memory, NVMe, or object storage it consumes, on top of the GPUs you already rent.
LMCache is free and open source, so the meaningful comparison isn't licence cost but total cost of ownership: you trade the licence fee commercial inference accelerators charge for your own CPU memory, NVMe, or object storage and the engineering time to run the tier. For teams already on vLLM, TGI, or Nvidia Dynamo, that math usually favours LMCache at scale; smaller shops with little prompt reuse get more from a managed option.
In short
LMCache — Open-source KV cache layer that stores, compresses, and reuses LLM key-value cache beyond GPU memory. Best for Developers serving low-latency LLM workloads on vLLM, TGI, or Nvidia Dynamo, RAG teams whose retrieved chunks break naive prefix caching, Platform engineers cutting GPU spend by pushing KV cache into tiered storage. Free to use.
What's new in LMCache
Checked 7 days agoAcross the latest 5 updates: 4 feature updates and 1 news mention.
Intro to KV Cache and LMCache (Part One)
The first part of an introduction to KV caching and where LMCache fits in the LLM serving path, aimed at newcomers to the layer.
Raw Block in LMCache: Building a Fast, Recoverable NVMe Tier for KV Cache
Engineering write-up on the Raw Block NVMe tier, which targets faster and recoverable KV cache storage off the GPU.
LMCache on Amazon SageMaker HyperPod: Disaggregated Prefill and Decode
Documents running LMCache on Amazon SageMaker HyperPod with disaggregated prefill and decode to enable KV cache reuse.
From Research Paper to Real Workload: KV-Cache Editing with the LMCache SDK
The LMCache SDK adds KV-cache editing, moving the technique out of research papers and into production workloads.
Encrypting KV cache at rest using LMCache
At-rest encryption for KV cache, addressing the security of cached context stored outside the GPU in CPU memory, disk, or object storage.
What people actually say about LMCache — is it worth it?
We ran a structured research pass across product reviews, community discussions, and post-purchase forum threads to surface the patterns vendors won't publish themselves. Below: the recurring strengths, the hidden costs people mention most, and the cohort that consistently regrets adopting this tool.
60 mentions across 5 sources (Hacker News, YouTube, Bluesky, GitHub, Lemmy) · researched Jul 18, 2026.
Average across the 5 sources that answered — each source counts once, not each post.
- +Reduces time-to-first-token (TTFT) by up to 8x via KV cache reuse.
- +Open-source with permissive license and active GitHub community.
- +Integrates seamlessly with vLLM and HuggingFace TGI.
- +Research-backed algorithms (CacheGen, CacheBlend) with peer-reviewed papers.
- +Supports multi-tier caching across GPU, CPU, RAM, SSD, and S3.
- −Streaming compression may be lossy, affecting output quality.
- −Security vulnerability (CVE) in KV cache hash function up to 0.4.6.
- −High number of open GitHub issues (402) indicates ongoing bugs.
- −Setup and integration require intermediate infrastructure skills.
- −KV cache calculator supports only smaller open models.
- • Potential GPU/SSD costs for multi-tier caching infrastructure
- • Time investment for integration and debugging
Viability Score
How well maintained and how widely used is LMCache? Built from what the vendor actually publishes (docs, changelog, tutorials, integrations, pricing), whether the site is live, and how much real users discuss it. How we calculate this
Last calculated: October 2026
How we score →Key Features
- KV cache reuse across requests to eliminate redundant prefill work
- CacheBlend dynamic fusion for non-prefix chunk reuse in RAG workflows
- Tiered storage across GPU memory, CPU memory, local disk, and object storage
- KV cache compression for longer contexts at higher concurrency
- Raw Block NVMe tier for fast, recoverable off-GPU KV cache storage
- At-rest encryption for KV cache stored outside the GPU
- KV cache transfer across workers, engines, and deployment modes
- In-process deployment inside vLLM, TGI, or Nvidia Dynamo
- Multiprocess standalone server with process isolation and restart survival
- LMCache SDK with KV-cache editing support
- Observability for cache behavior, storage movement, and reuse patterns
- KV cache calculator for sizing memory and deployment tiers
- Pluggable external storage backends including object storage, Redis, and node-local storage
- Documented SageMaker HyperPod deployment with disaggregated prefill and decode
About LMCache
LMCache is an open-source KV cache layer for LLM inference — the memory tier sitting between your serving engine and the rest of the cluster. It takes the key-value cache that vLLM, TGI, and Nvidia Dynamo generate during prefill and makes it reusable: persisted beyond GPU memory into CPU memory, local disk, or external backends, then reloaded on later requests so the same prefill work isn't paid twice. The project frames this as a six-part loop — store, reuse, search, compress, move, observe. What separates it from plain prefix caching is CacheBlend, which reuses cached chunks that aren't an exact prefix match — the shape RAG retrieval actually produces, where documents land at arbitrary prompt positions. Recent engineering work added a Raw Block NVMe tier for faster, recoverable off-GPU cache storage and at-rest encryption for cache leaving the GPU, plus KV-cache editing exposed through the LMCache SDK. Two deployment shapes exist with no engine rewrite: in-process (a KV library runs inside the serving engine) and multiprocess (LMCache as a standalone server with process isolation and cache surviving worker restarts). The project points to multiprocess as the recommended path. It's grown out of University of Chicago systems research, and the ecosystem around it spans Nvidia Dynamo, Google Cloud GKE, AMD Instinct MI300X, CoreWeave AI Object Storage, Redis, the PyTorch Foundation, Tensormesh, and Amazon SageMaker HyperPod. That makes it production infrastructure rather than a hobby repo, but it is still a library and a server you operate yourself, not a managed product.
Behind the Verdict
Reach for LMCache when your bill is dominated by re-processing the same long prompts. Long system prompts, multi-turn conversation history, and RAG contexts that come back request after request are exactly the shapes it was built for. Teams running vLLM, TGI, or Nvidia Dynamo get an integration path that doesn't require touching the serving engine, and the multiprocess server mode is the one we'd pick for anything beyond a single worker — cache surviving a worker restart is worth a lot in production. Pass on it if your queries are largely unique. If there's little overlap between prompts, there's little to reuse, and you've added a memory tier and an operational surface for nothing. The same applies if nobody on the team wants to own a Python library and its configuration — this is infrastructure engineering, not a toggle. The closest alternative is what your serving engine ships natively: vLLM's built-in prefix caching. That covers exact-prefix reuse with zero extra moving parts and no off-GPU tier. LMCache earns its keep past that point — non-prefix chunk reuse via CacheBlend, tiering into CPU, NVMe, and object storage, cross-worker transfer, and compression for longer contexts at higher concurrency. If you only need exact-prefix reuse on one node, the built-in option is simpler and probably enough. In practice, published gains are workload-specific. AMD's numbers on MI300X with vLLM (3–10× across Qwen3, Llama3, and Qwen-VL) are real but tied to that hardware and those models. Run your own benchmark on your prompt distribution before you quote anyone a number. The KV cache calculator helps with sizing, but sizing is not the same as proving a speedup on your traffic. One security caveat that trips people up: once cache moves off the GPU, it's just data sitting in
Researching LMCache? Get your full AI stack in 60 seconds.
Free, no signup — tell us your goal and get tools matched to your budget & existing stack.
Real-world workflow fit
Concrete scenarios for the personas LMCache actually fits — and what changes day-one when you adopt it.
Deploys the KV library in-process inside the vLLM engine, points cache storage at node-local NVMe, and lets repeated conversation prefixes reload from disk instead of re-prefilling each turn.
Outcome: Lower GPU time per turn on repeated history, with no change to the application's request path.
Enables CacheBlend so cached chunks are fused dynamically rather than requiring an exact prefix match, matching the shape of retrieved documents inserted at varying positions.
Outcome: Retrieval workloads benefit from cache reuse that prefix-only caching would have missed entirely.
Runs LMCache in multiprocess mode as a standalone server so cache storage is isolated from model execution and survives a worker restart or failure.
Outcome: Cache persists across worker churn, and the engine stays focused on model execution.
Use Cases
- Cut GPU spend on chatbots by reloading cached conversation history instead of re-prefilling it every turn.
- Speed up RAG pipelines by fusing KV caches from retrieved document chunks that aren't an exact prefix.
- Serve long system prompts repeatedly without paying full prefill cost on each request.
- Run distributed inference where cache must survive a worker restart or failure.
- Offload KV cache to CPU memory, NVMe, or object storage to free GPU memory for more concurrent requests.
- Edit KV cache directly with the SDK for workloads that need cache manipulation rather than full recompute.
- Encrypt cached context that lives outside the GPU to satisfy security review.
Models Under the Hood
as of 2026-10-10
Limitations
- LMCache is infrastructure, not a turnkey product.
- Integration means Python, CLI, and real deployment thinking inside a serving engine like vLLM, TGI, or Nvidia Dynamo — there is no managed control plane or GUI to fall back on.
- The payoff depends entirely on cache reuse: workloads dominated by one-off unique queries per request will not benefit, and cache-agnostic applications will carry operational overhead for nothing.
- Applications that require dynamic fine-tuning during inference are out of scope.
- As a community open-source project with 12K+ GitHub stars and 350+ contributors, it does not ship with a vendor SLA.
- Multiprocess mode is the project's recommended deployment path and the focus of future development, so in-process setups may see less attention over time.
as of 2026-10-11
Verification history
We have re-verified LMCache 10 times since . Each pass re-reads the vendor's own pages and re-checks every listed field against that evidence; passes where nothing had changed are marked as such.
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
Showing the 6 most recent of 10 verification passes.
Free to cite with attribution — this page re-verifies continuously.
12-month cost
Project the real annual outlay, including the implied monthly cost when only an annual tier is published.
Vendor list price only. Add-on usage, seat overages, and contract minimums are surfaced under Hidden costs & gotchas.
Plans compared
For each published LMCache tier: who it actually fits, and what it adds vs. the previous tier. Cross-reference the cost calculator above for projected annual outlay.
Free
$0/mo
Ideal for
Developers, RAG teams, and platform engineers self-hosting vLLM, TGI, or Nvidia Dynamo who can run the cache tier themselves.
What this tier adds
Starting tier: the entire KV cache layer is free and open source, including CacheBlend, tiered storage, multiprocess mode, and the SDK.
Where the pricing makes sense
The company stage and team size where LMCache's pricing actually pencils out — and where peers do it cheaper.
LMCache is free and open source, so the meaningful comparison isn't licence cost but total cost of ownership: you trade the licence fee commercial inference accelerators charge for your own CPU memory, NVMe, or object storage and the engineering time to run the tier. For teams already on vLLM, TGI, or Nvidia Dynamo, that math usually favours LMCache at scale; smaller shops with little prompt reuse get more from a managed option.
Setup time & first value
How long it actually takes to get something useful out of LMCache — broken out by persona, not the marketing-page minute.
In-process inside vLLM or TGI: typically under an hour for a developer already comfortable with the serving stack, since no architectural change is required. Multiprocess standalone server: allow a day to stand up the server, wire storage backends, and verify cache reuse across workers. CacheBlend and compression tuning for a production RAG workload: expect a benchmark cycle measured in days,
Switching to or from LMCache
How to bring data in from common predecessors and how to get it back out — written for the switcher, not the buyer.
- →From default vLLM prefix caching: enable LMCache in-process to extend reuse beyond GPU memory and exact prefixes.
- →From no cache tier: add CPU memory, local disk, or object storage as a persistence layer behind the serving engine.
- →From a per-engine cache: move to multiprocess mode so cache survives worker restarts and is shared across engines.
- ↗To default engine prefix caching: disable the LMCache layer and rely on the serving engine's built-in cache only.
- ↗To a commercial inference accelerator: keep the same engine and compare managed cache behavior against the self-hosted tier.
Integrations
Resources & Guides
Tutorials & Learning
YouTube returned 6 videos for “LMCache”, and we withheld 6: 6 could not be judged, because “LMCache” is a single word that other videos use for other things. We are showing none, because we could not prove any of them are about LMCache.
Official links
Featured Head-to-Head Comparisons
Lmcache vs Spider Cloud
Spider Cloud and LMCache solve completely different problems. Stick with Spider Cloud if you need to pull fresh web data into your AI pipeline — its Rust engine and new Browser AI commands make it unbeatable for cost-effective scraping. Choose LMCache if your bottleneck is LLM inference latency: it caches KV caches to slash response times by up to 8x, and it's free. Don't cross-shop; buy both if your stack includes both data ingestion and inference.
Lmcache vs Temporal Ai
Temporal AI and LMCache solve different problems. Choose Temporal if you need durable, fault-tolerant orchestration for AI agents and long-running workflows with human-in-the-loop. Choose LMCache if your bottleneck is LLM inference latency and cost, and you already use vLLM or TGI. For most LLM serving pipelines, LMCache is a no-brainer performance boost at zero cost.
Lmcache vs Voyage Ai
Choose Voyage AI if your priority is high-accuracy retrieval in specialized domains like finance or legal, with transparent embedding-level cost savings. Choose LMCache if you need to slash LLM inference latency and cost by reusing KV caches, especially for chatbots and RAG at scale. They solve different problems: embeddings vs. inference optimization.
Popular in GPU Cloud & Model Inference
Rain AI
Rain AI is a pre-production hardware company building energy-efficient, brain-inspired analog in-memory chips for ultra-low-power AI inference at the edge.
Recogni
Recogni's Tensordyne Napier is a rack-scale AI inference system running on logarithmic math silicon for multi-trillion-parameter MoE
Spectral Labs SGS-1
Decentralized AI inference with sub-5ms latency and verifiable compute
Frequently Asked Questions
Categories
Topics
Used LMCache? Help shape our editorial sentiment research.