← Back to feed

LLM inference has been datacenter-local since the beginning: prefill and decode had to run in the same tightly-coupled cluster because the KV cache — the temporary memory state produced by prefill — was too large to ship over standard datacenter links in time. A paper published Sunday on arXiv (2604.15039) makes the case that this constraint has broken down for the generation of hybrid-attention models now entering production.

The paper introduces a serving pattern called Prefill-as-a-Service. Long prompts — those that cross a routing threshold set experimentally at around 19,400 tokens — are sent to a remote prefill cluster. The KV cache state is then returned over a standard 100 Gbps Ethernet link to the local decode cluster, which generates tokens from there. Short prompts stay local. A bandwidth-aware scheduler manages routing to avoid saturating the link.

Why It Works Now

The enabling condition is architectural. Older dense-attention models produce KV cache that grows proportionally with sequence length and number of layers — typically too large to ship across datacenter links at the speeds inference requires. Hybrid-attention models, which mix full-attention layers with sliding-window or state-space layers, produce substantially smaller KV cache for the same context length. The paper doesn’t give a precise reduction ratio for their internal model, but the result speaks: average cross-cluster traffic sits at 13 Gbps on a 100 Gbps link with 50% of requests routed remotely.

The Setup and Results

The test uses an internal 1-trillion-parameter hybrid-attention model across a heterogeneous hardware split: H200 GPUs for the remote prefill cluster (compute-dense, suited for the attention-heavy prefill phase), H20 GPUs for local decode (memory-bandwidth-optimised, suited for token generation). This reflects real infrastructure economics — H200s are expensive and most operators have a mix of hardware generations.

Results against two baselines:

• vs. local-only serving: +54% throughput
• vs. naive heterogeneous serving (no selective offloading or scheduling): +32% throughput

The gain is not from shipping everything further — it’s from routing selectively. Requests below the 19.4K token threshold stay local and incur no network overhead. Only long-context work, where prefill cost is high enough to justify the network round trip, gets offloaded.

What This Unlocks

Three consequences matter for operators building at scale:

1. Heterogeneous fleet utilisation. Operators with mixed GPU generations — older H20s for bulk inference, newer H200s for the compute-heavy prefill phase — can use both effectively rather than keeping them siloed.

2. Long-context economics. Prefill is the majority of compute cost for long-context workloads. Offloading it to a cluster that can absorb the load, rather than over-provisioning every inference node, directly reduces per-token cost for retrieval-augmented generation, document analysis, and extended agent chains.

3. Geographic flexibility. If inter-datacenter latency for the KV cache round-trip is acceptable (the paper validates this on standard Ethernet; wide-area networking would need different analysis), operators can place prefill capacity where power and hardware are cheapest, not where the user is.

Caveats

The paper tests one model architecture on one hardware pair in a controlled research setting. The 19.4K routing threshold is experimentally derived for that model; different hybrid architectures will produce different optimal thresholds. Wide-area network applications, where latency is higher and bandwidth lower than a 100 Gbps datacenter link, are not addressed.

The deeper point: this is a systems paper, not a product announcement. The technique works today for the generation of hybrid-attention models now in production. As model architectures continue shifting away from dense attention, the constraint that made inference locality mandatory becomes progressively weaker.

Key Numbers

• Throughput gain vs. local-only: +54%
• Throughput gain vs. naive heterogeneous: +32%
• Average cross-cluster traffic: 13 Gbps (on 100 Gbps link)
• % of requests routed remote: ~50% at the tested threshold
• Routing threshold: ~19.4K tokens
• Model size: 1T parameters (internal hybrid-attention model)
• Hardware split: H200 GPUs (prefill) + H20 GPUs (decode)
• Paper: arXiv 2604.15039, “Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter”