← Back to feed

Coding agents have a structural inference problem that most benchmarks don’t measure. Each agent turn sends 20,000 to 150,000 input tokens to the model — and 93 to 97% of those tokens are identical to the previous turn. Without persistent KV cache across turns, the serving stack recomputes the full attention state every time. At scale, that recomputation is a dominant cost driver and the primary source of latency in multi-step agentic runs.

LMCache is an open-source KV cache management layer that sits between LLM inference (via vLLM) and a tiered storage hierarchy, eliminating that redundant work. The project has reached 10,600 GitHub stars and is now shipping production-grade AMD MI300X support alongside multi-server coordination.

The Core Numbers

In the benchmark that most closely mirrors production agentic deployments — 32 concurrent users sending 100K-token traces — LMCache compared against standard HBM-only prefix caching delivered:

Metric	LMCache vs HBM-only
Average TTFT	3.0x lower
p95 TTFT	2.1x lower
Max TTFT	2.6x lower
Completed requests	2.3x more

In a simpler cold-to-warm comparison at 2,000 tokens: the cold path takes approximately 40 seconds; a warm LMCache hit takes around 4 seconds — a 10.7x speedup.

Time-to-first-token is the user-facing metric that determines whether a coding agent feels responsive or stalled. In a 60-step agent run, compounding that per-turn latency makes the difference between a 5-minute job and a 30-minute job.

How It Works

LMCache manages KV cache across a tiered storage hierarchy ordered by access speed:

1. GPU HBM — hottest cache, stays close to the compute
2. CPU RAM — larger capacity, slower retrieval
3. NVMe SSD — cold storage for infrequently accessed context

The serving engine (vLLM) keeps doing inference; LMCache handles the memory layer separately. Hot KV cache — likely to be needed again soon — stays in HBM. Cold cache that hasn’t been accessed recently moves to CPU RAM or SSD. This separation means the cache can scale independently of GPU memory without rebuilding the inference stack.

The vLLM integration uses a LMCacheConnector that lets workers send, retrieve, and coordinate KV cache with LMCache’s multiprocess server. No architectural overhaul required — LMCache augments vLLM rather than replacing it.

AMD and Multi-Server Support

LMCache is not CUDA-only. The AMD MI300X path runs through ROCm with the LMCacheConnector V1 path and delivers comparable gains on AMD hardware. The AMD benchmark shows 3x to 10x improvement on long-document and multi-round QA workloads.

For multi-server deployments, LMCache adds P2P KV cache transfer and multi-server coordination. KV cache blocks can move across serving instances rather than being trapped in a single worker process — which matters for production clusters where a single GPU node can’t hold the full cache for a large agent fleet.

Why Agent Workloads Specifically

Standard LLM serving optimises for session-level prefix caching — the same text appears at the beginning of related prompts. LMCache goes further by handling repeated or overlapping KV blocks even when they don’t start at position zero. That’s the pattern that shows up in:

• Coding agents that load the same repository context at each planning step
• RAG systems that repeatedly inject the same retrieved documents
• Long-document workflows where the base document is referenced across multiple queries
• Multi-turn assistants with extended system prompts

In each case, the expensive attention computation was happening multiple times on identical inputs. LMCache makes that computation persistent.

The Infrastructure Context

Goldman Sachs projected this week that AI agent token use will multiply 24x by 2030. Monthly consumption could reach 120 quadrillion tokens. Inference cost per token is falling 60 to 70% per year, but that curve needs infrastructure support to materialise as actual cost reductions for agent deployments.

LMCache attacks the problem at the serving layer — the point where repeated context hits the GPU. The gain isn’t from a better model or a cheaper chip; it’s from not recomputing the same thing twice. At the trajectory of agentic workloads, that architectural discipline compounds.