← Back to feed

World models were supposed to get bigger, not smaller. TinyWorldFormer goes the other direction: an open-source project that runs five separate game-playing world models on a Raspberry Pi Pico 2W, a microcontroller that costs roughly $6 and carries 520KB of SRAM and 4MB of flash.

The core architectural decision is a looped transformer. Instead of stacking multiple distinct transformer layers, TinyWorldFormer uses a single block applied 2 to 4 times with identical shared weights. Sharing weights compresses memory dramatically without a proportional drop in accuracy: the Breakout world model requires only 26,000 parameters, while Pac-Man (a 48x48 frame, two ghosts, 249 pellets) needs 131,000. Both fit on the chip.

The numbers

Game	Params	RAM	Flash	Frame accuracy
Breakout	26k	299 KB	361 KB	99.99%
Flappy Bird	46k	227 KB	409 KB	100%
Pong	46k	244 KB	414 KB	99.45%
Doom-lite	55k	357 KB	471 KB	83.6%
Pac-Man	131k	460 KB	600 KB	86.4%

Frame accuracy is the share of held-out transitions where every cell of the predicted next frame matches the real game. Random events (ball serves, enemy respawns) are excluded because those are sampled, not predicted. Doom-lite’s 83.6% figure looks lower than the others but the game renders a 32x32 raycast 3D view, where a single misshaded wall pixel counts the whole frame as wrong.

Adding loops improves accuracy non-linearly. Doom-lite goes from 59.8% exact frames at one loop to 83.6% at two, with diminishing returns at three and four. The chip runs at two loops by default.

Architecture

The world model takes a frame and an action as input and outputs the next frame plus a reward signal. Frames are cut into square patches (2x2 cells for Breakout, 4x4 for most games, 6x6 for Pac-Man), producing 36 to 80 tokens per step.

The looped transformer block is int8 throughout: weights and activations are 8-bit integers with one float32 scale per output row. Matmuls use the RP2350’s SMLAD instruction, which processes two multiply-adds per 32-bit cycle. Normalisation, residuals, and embeddings stay in float32, since the RP2350 has a hardware floating-point unit. A frame at two loops costs 2 to 8 million multiply-adds, peaking at 8.2M for Pac-Man.

Decoding uses three rules: background cells take the most likely token independently; singletons (the ball, each ghost) get one global placement choice so there is always exactly one of each; and multi-cell sprites stamp their state from a single carrier cell rather than predicting each pixel.

The weights are compiled to Rust source as exact integer bit patterns, verified bit-for-bit against the Python training code, then built into firmware.

On-device learning

A second network, the actor, is a feature-based MLP with two 64-unit hidden layers that observes extracted game state and picks button presses. The default shipped weights are pre-trained, but at runtime they can be replaced with a fresh stream AC(lambda) agent that learns from scratch on the board itself. The RL update runs on one transition at a time with no replay buffer, no target network, and no batches.

Both networks run on core 1 of the RP2350. Core 0 handles USB output, the optional 320x240 SPI display, and the frame channel between cores. Rendering updates only cells that changed since the last frame. If core 0 is still busy when core 1 finishes a step, the frame is dropped rather than stalling the world model.

Why it matters

TinyWorldFormer is not trying to match frontier world models on quality. What it demonstrates is that a functional game-playing world model with a streaming RL agent can run on hardware that costs less than a cup of coffee, with no operating system, no cloud connection, and minimal power consumption. The looped transformer is the transferable idea: one block, reused, gives most of the benefit of depth at a fraction of the memory.

The obvious downstream targets are embedded robotics controllers and sensor processing systems where model memory is a hard constraint rather than a soft optimisation.

Code and firmware are on GitHub. All five games run in the browser via WebAssembly using the same no-import int8 engine.