← Back to feed

At Hot Chips 2026, Samsung presented a 16 GB LPDDR5X memory package with processing-in-memory (PIM) blocks embedded directly alongside its DRAM banks. The package delivers 614 GB/s of bandwidth to its own compute units — matching the full unified memory bandwidth of Apple’s top-spec M5 Max — while only 76.8 GB/s exits through its external pins. Tom’s Hardware reported a 3.01x inference speedup over standard LPDDR5X in AI benchmarks.

Why the pin bandwidth matters

Token generation in autoregressive models is a memory-bandwidth problem, not a compute problem. Each new token requires reading every model parameter from DRAM, multiplying it once against the activation vector, and discarding it. The matrix-vector multiply (GEMV) at batch size 1 cannot reuse weights across tokens because each token depends on the preceding one. Arithmetic intensity sits near the floor, leaving most matrix hardware idle while DRAM drains.

A first-order approximation: tokens per second approximately equals memory bandwidth in GB/s divided by model size in GB. At 614 GB/s against a 20 GB model, the theoretical ceiling is roughly 30 tokens per second per package. The real number is lower once KV cache growth is included. But the constraint is the bus, not the silicon computing the multiply.

What Samsung built

The package contains 16 GB of DRAM across four dies per rank in a standard 561-ball JEDEC footprint — a drop-in replacement for ordinary LPDDR5X modules. Sixteen PIM blocks sit beside the banks. Each block contains multiply-accumulate (MAC) trees running in parallel and an ALU handling FP8 and integer precision types.

Processing runs in four steps:

1. Activation write — the host broadcasts FP8 activation data via WRPB commands into source register files across the banks
2. Weight load — PIMX_RD reads weight elements from DRAM cells and maps them across the MAC trees
3. Vector store — partial sums move to the bank, with a variable read-to-write ratio
4. Readback — the host issues sequential reads to drain the 1-kbit vector register file

Weights stay inside the package at all times. Only the much smaller activation data crosses the external bus. The host sends commands; the PIM blocks do the arithmetic.

Fifteen precision combinations are supported via MAC precision fields in a configuration register. SINT4 weights reach 2.4 TOPS per package; FP8 reaches approximately 1.2 TFLOPS per package.

The bottleneck shift

The conventional AI accelerator stack separates compute (GPU/NPU) from memory (HBM or LPDDR), connecting them through a bus that saturates under inference loads. HBM addresses part of the problem by co-packaging memory closer to the processor with wider interfaces, but the bandwidth still crosses a die-to-die boundary.

PIM moves the compute to where the data lives. The DRAM banks hold weights; the MAC units beside those banks process them without touching the external interface. Only activations (small) and commands cross the bus instead of weights (large).

Samsung is targeting the full market: servers, mobile devices, and client hardware using the same physical package. The standard JEDEC footprint means existing socket designs can accommodate it without a board redesign.

Context

Hot Chips 2026 also featured presentations on high-bandwidth flash as a third memory tier for AI inference (Samsung, SK Hynix) and NVIDIA’s CUDA roadmap for RISC-V. The PIM announcement sits alongside those as part of a broader industry push to solve inference efficiency by rearchitecting how models interact with memory rather than by scaling GPU compute further.

Key numbers

Metric	Value
Package capacity	16 GB
Internal bandwidth	614 GB/s
External pin bandwidth	76.8 GB/s
Bandwidth ratio	8x
AI inference speedup vs LPDDR5X	3.01x
SINT4 throughput	2.4 TOPS per package
FP8 throughput	~1.2 TFLOPS per package
Precision combinations	15
Package footprint	561-ball JEDEC (drop-in)