← Back to feed

The AI memory hierarchy currently has two tiers: HBM on the GPU package for fast access, and DRAM or NVLink-connected memory for overflow. At Hot Chips 2026, a tutorial by Anurag Agarwal and Radhakrishna Giduthuri proposed a third: High Bandwidth Flash.

HBF uses the same NAND cells as SSDs but is packaged in a cube that sits alongside HBM on the processor package. The intent is to provide far more capacity than HBM at lower cost per gigabyte, while staying physically close to compute.

The catch is that HBF does not behave like memory. It behaves like an SSD on a short leash.

Software cannot address HBF with fine-grained random reads. It must use DMA to move large, aligned chunks between HBF and DRAM or HBM. The software stack also takes on SSD controller duties: write leveling, data retention management, and access alignment. That rules out drop-in deployment and requires explicit framework integration.

MoE Expert Offloading

The clearest use case is mixture-of-experts models. In a typical large MoE deployment — say, DeepSeek V4 Pro at 2.35T total parameters — most experts are inactive on any given token. Storing inactive experts in HBF and DMA-ing the active set into HBM as needed could dramatically reduce HBM pressure without requiring additional GPU cards.

vLLM is the reference implementation the tutorial used. The framework already investigates placing model weights in pinned CPU memory when host DRAM is available. HBF would offer a more capacity-dense version of that approach, bypassing the CPU memory bottleneck entirely.

KV Cache

KV cache placement in HBF is conditional. It only works well when the attention implementation is sparse — reading a subset of tokens from the top of the cache per step, leaving most of the cache cold in flash. Full-attention workloads flush too many bytes through HBF per step to make the bandwidth cost viable.

The practical complication is that KV top-k reads are scattered, while HBF prefers sequential DMA. Software would need to buffer the top-k rows into DRAM before the attention kernel runs.

Cross-Device Weight Replication

A third application targets multi-GPU inference. Large models are typically sharded across devices, and cross-device scatter-gather operations often become the bottleneck ahead of compute or HBM bandwidth. HBF enables replicating a larger fraction of model weights on each GPU, reducing inter-device traffic. DMA off flash is slower than HBM reads but cheaper than going off-chip over NVLink or InfiniBand.

When HBF Makes Financial Sense

Cost viability narrows to workloads that are not bandwidth-bound. HBF’s cost per gigabyte is competitive; its cost per GB/s is not. The math works for small models and small batch sizes. Scale up the batch or move to a dense attention pattern and the HBF bandwidth cost dominates.

Caching hot MoE experts in HBM while leaving the rest in HBF adds another layer of complexity — the cache hit rate has to be high enough that the KV equation doesn’t flip negative.

No HBF products exist yet. The Hot Chips 2026 session was entirely simulation and projection. The benchmark for whether HBF lands in real inference pipelines is whether vLLM or a similar framework ships the DMA offloading path before the next generation of HBM erases the capacity gap.

Memory Tier Comparison

Tier	Location	Bandwidth	Capacity	Access Model
HBM	On-package	~3 TB/s	80-288 GB	Random, byte-addressable
HBF	On-package	Lower	TBs	DMA, block-aligned
DRAM	Off-chip	~1 TB/s	Hundreds of GB	Random, byte-addressable