← Back to feed

KogAI has demonstrated 3,000 tokens per second per user on an 8-GPU AMD MI300X cluster, and 2,100 tokens per second on 8 NVIDIA H200s — both without speculative decoding, both at FP16 precision. For context, typical single-user inference on comparable hardware runs at 100 to 300 tokens per second. The claimed improvement is 10x to 30x.

The test model is 2 billion parameters. KogAI says the architecture scales to large frontier MoE models at similar speeds.

Why Normal Inference Is Slow

The problem is not arithmetic — it is memory access. For a single user at batch size 1, a GPU is not doing large matrix multiplications the way training does. Instead, it repeatedly fetches model weights from high-bandwidth memory (HBM) for every new token. Speed is limited by how smoothly that weight stream flows.

Conventional inference stacks break that stream constantly. Separate GPU kernels handle different model layers, intermediate results pass through memory, synchronization barriers stall the pipeline, and the CPU handles scheduling and sampling between tokens. Each of these creates a stop-and-start cadence that degrades per-user throughput.

KogAI measured grid synchronization alone as consuming 35% of token-generation time on standard stacks.

The Monokernel

The core engineering move: replace the multi-kernel pipeline with a single, persistent GPU-resident program that handles the entire decode pass including sampling. No kernel launches between tokens. No CPU round-trips for scheduling.

Three co-designed components:

• Monokernel: One persistent GPU program for the full decode step, including sampling. Eliminates kernel launch overhead and intermediate memory round-trips between tokens.
• Targeted synchronization: Grid sync replaced with per-dependency waits. Each compute unit blocks only on the data it actually needs, not on a global barrier.
• AMD-aware memory mapping: On MI300X, memory latency varies by chiplet. KogAI maps access patterns around the physical die layout to minimize cross-chiplet latency.

Their Laneformer model adds Delayed Tensor Parallelism, which lets cross-GPU communication overlap with compute instead of blocking at each transformer layer.

Key Numbers

Platform	Tokens/sec (1 user, FP16)
8x AMD MI300X	3,000
8x NVIDIA H200	2,100
Conventional stack (2B-8B models)	100-300

What This Would Mean at Scale

Frontier models running at 3,000 tokens per second would compress the economics of voice AI, real-time reasoning, and any agent loop where latency is the binding constraint. A model producing 3,000 tokens/s generates a 1,000-token response in 333ms — within a workable conversational window.

KogAI’s tech preview remains at 2B parameters. The claimed scalability to frontier MoE sizes is unverified. But the methodology is specific: treating decoding as a memory streaming problem is a distinct starting point from most inference optimization work, which targets arithmetic utilization. AMD chiplet-aware memory mapping in particular is a hardware-grounded optimization that cannot be dismissed as benchmark theater.