← Back to feed

Kimi K3 just crossed a strange local-inference threshold. WASTE, a dependency-free C inference engine from sqliteai, runs the full 2.78-trillion-parameter Kimi K3 model on a consumer laptop by streaming activated expert weights directly from NVMe.

The result is slow: roughly 0.49 to 0.54 tokens per second on a 64GB MacBook Pro. It is also full-model execution, not a distilled, pruned, or reduced variant. Kimi K3 is converted from its 1.42 TB published form into a 982 GiB container, then served with a resident trunk in RAM and a bounded cache for active experts.

The Numbers

Metric	WASTE Kimi K3 run
Model size	2.78T parameters
Converted container	982 GiB
Minimum RAM at 4K context	29.05 GiB
Tested machine	64GB MacBook Pro
Tested speed	0.49-0.54 tok/s
Resident trunk	27.28 GiB
Expert working set per token	17.0 GB

That last number is the whole story. Kimi K3 is a mixture-of-experts model. Most of its weights are idle for any given token. WASTE keeps the trunk resident, places each expert record so that one expert costs one read, and streams only the selected experts for the token being generated.

The architecture converts a memory problem into a storage-throughput problem. On internal NVMe, the engine reports 12.78 GB/s storage reads and can crawl forward at about half a token per second. Over a USB enclosure at 0.94 GB/s, the same token takes around thirteen seconds.

Not Interactive, Still Material

Half a token per second is not competitive with cloud inference. A short response takes tens of seconds. Long-form coding-agent work would be painful without aggressive batching, speculative execution, or a much faster storage path.

But dismissing the result as too slow misses the point. The interesting claim is that the frontier-scale open-weight model is reachable at all on consumer hardware. Not convenient. Not fast. Reachable.

That matters for the parts of the market where the alternative is not “use the cloud,” but “do not run the model.” Highly sensitive data, offline environments, air-gapped research systems, export-control-sensitive deployments, and personal experimentation all sit in that category. A slow local Kimi K3 is still qualitatively different from no local Kimi K3.

Why MoE Changes the Floor

Dense models punish local inference linearly: if the full model does not fit, it does not fit. MoE models create a different failure mode. Kimi K3 activates only a small slice of the parameter set per token, so the inactive weights do not need to be hot in RAM if the engine can fetch active experts quickly enough.

That is what WASTE tests. It is less a product launch than a systems claim: expert placement, aligned disk reads, direct I/O, and cache discipline can make a trillion-scale model operate under a RAM ceiling that would normally be absurd.

The cache behaviour is sharp. Below one token’s 17GB expert working set, the cache effectively does nothing. Above it, hit rates rise, but pushing too close to the machine’s physical RAM can make the OS page out the cache and destroy throughput. The best measured path on the 64GB system is not “use all available memory.” It is a narrow budget window around the point where the expert cache clears one working set without triggering system pressure.

The Open-Weight Implication

Kimi K3’s open-weight release already changed the frontier conversation by putting a 3T-class model outside a closed API. WASTE changes the local-deployment conversation by showing that the bottleneck is not only GPU memory. It is model layout, storage bandwidth, and the economics of moving inactive weights out of the hot path.

The current version is a proof of feasibility, not a replacement for hosted inference. But feasibility tends to be the first expensive step. Once a full Kimi K3 run works at 0.5 tokens per second from disk, the next questions become engineering questions: faster SSDs, better prefetch, larger local memory, smaller active sets, and improved expert-cache policy.

The direction is clear enough. Open weights are no longer just about fine-tuning or cheap API competition. They are pushing inference systems into places where trillion-scale models were supposed to be physically out of range.