← Back to feed

Georgi Gerganov, the original author of llama.cpp, merged what may be the library’s most significant memory efficiency update in years on April 18. The feature, speculative checkpointing, changes how the inference engine manages KV-cache state during speculative decoding — and the measured gains are substantial.

What Changed

Standard speculative decoding requires a full synchronisation and backup of the Key-Value cache whenever a speculative batch needs to roll back. On memory-bandwidth-constrained hardware — Apple M-series chips, consumer RTX cards — this overhead accumulates quickly and caps how far context windows can be pushed before VRAM exhaustion terminates the session.

Speculative checkpointing replaces the full-cache flush with a sparse, lightweight snapshot of only the delta changes during speculative phases. The engine maintains just what changed, not the entire cache state, which eliminates the dominant cost of the rollback path.

Benchmark Numbers

From the merge discussion:

• VRAM reduction: up to 40% during batched operations
• Token throughput gain: 15–20% on bandwidth-limited consumer hardware
• Practical threshold: 70B-parameter models with extended context that previously exhausted VRAM now complete without memory failure on standard consumer setups

The 15–20% throughput figure is on top of any speculative decoding gains already in place — the checkpoint overhead was previously blunting those gains on commodity hardware; this removes that ceiling.

Context

Speculative decoding — generating candidate tokens with a smaller draft model, then verifying with the target model — became one of the most widely adopted inference acceleration techniques across both cloud and local deployments after DeepMind published the technique. Its integration in llama.cpp carried a memory cost that made it impractical for setups below 40+ GB VRAM. Speculative checkpointing addresses that directly.

The practical significance for the local AI inference community is immediate: 70B models with 32K–64K context windows were the threshold at which most consumer hardware (RTX 4090, M3 Pro/Max) would hit memory limits before completing long sessions. That threshold has now moved meaningfully.

What It Means for Open-Weight Models

The merge lands in a window where several open-weight models at the 70B class are competitive with mid-tier API products. DeepSeek R2 (32B dense, 92.7% AIME 2025) and models in the Qwen3.6 family are the primary beneficiaries — both fit within hardware ranges where the VRAM savings have the most practical impact. The efficiency gain effectively expands the viable use-case surface for local inference without requiring new hardware.

llama.cpp now has over 104,000 GitHub stars and remains the primary inference backend for local AI deployment on consumer hardware.