← Back to feed

A 1-billion-parameter model trained for $1,472 is matching the benchmark performance of 2-7B transformer models. The results come from Sapient Intelligence’s HRM-Text, an open-source hierarchical recurrent model released in May 2026 that is drawing renewed attention following Chinese tech media pickup and an active pull request to integrate the architecture into HuggingFace Transformers.

The paper (arxiv:2605.20613) poses a direct challenge to scale-first pretraining orthodoxy: co-design the architecture and objective function, and compute requirements drop by two orders of magnitude.

Key Numbers

• Parameters: 1B (XL variant)
• Training tokens: 40B unique tokens
• Compute cost: $1,472 on 16 H100s, 46 hours
• GSM8K: 84.5% (standard 1B transformers: 48-70%)
• ARC-Challenge: 81.9%
• MMLU: 60.7%
• MATH: 56.2%
• DROP: 82.3%
• Token efficiency vs. baselines: 100-900x fewer training tokens than comparable 2-7B models
• Compute efficiency: 96-432x less estimated FLOPs than standard baselines

Independent verification in April 2026 confirmed the results hold under strict contamination-free evaluation splits. Sapient Intelligence ran a Clean Split condition that rules out test set leakage as a confound.

The Architecture

HRM-Text replaces the standard single transformer stack with two interleaved recurrent networks operating on different timescales, modelled on the brain’s frontoparietal loop:

• H module (high-level): Slow-updating, maintains long-horizon semantic context and strategic planning
• L module (low-level): Fast-updating, handles local computation and iterative correction

Each forward pass runs two H cycles, with three L updates inside each H cycle. This nested recurrence gives the model an effective compute depth that far exceeds its parameter count without any increase in parameters.

Two training innovations stabilise deep recurrence: MagicNorm prevents gradient pathologies across recurrent steps, and warmup deep credit assignment addresses the vanishing credit problem in long recurrence chains. A third design choice separates HRM-Text from standard pretraining: training exclusively on instruction-response pairs using a task-completion objective with PrefixLM masking. Loss is computed only on answer tokens, concentrating training signal on task execution rather than undirected text prediction.

What the Architecture Gains From Each Change

The paper isolates the contribution of each design choice on ARC-Challenge:

Configuration	ARC-C Score
Standard 1B Transformer (full-sequence, causal)	51.9%
+ Answer-only prediction	62.9%
+ PrefixLM	74.3%
+ HRM architecture	81.9%

Each change compounds. The 30-point improvement over the baseline is the sum of three separable contributions, not a single lucky trick.

Who Built It

The team spans researchers from DeepMind, DeepSeek, and xAI alongside contributors from MIT, Carnegie Mellon, Tsinghua, Cambridge, the University of Alberta, and the Tsinghua Laboratory of Brain and Intelligence. Chinese tech media has reported that a Turing Award winner joined the project, though the name has not been confirmed in English-language releases.

The full training codebase, model weights, and evaluation framework are open-sourced at github.com/sapientinc/HRM-Text. The 1B checkpoint is available at sapientinc/HRM-Text-1B on HuggingFace. A pull request to merge the architecture natively into HuggingFace Transformers is in progress.

What It Is Not

HRM-Text is an explicit proof-of-concept base model. There is no RLHF, no instruction tuning beyond the task-completion objective, and no chat fine-tuning. The team describes it as an existence proof, not a production system. The question the paper answers is whether architecture-level compute efficiency is achievable at foundation model scale. The answer is yes.

Whether HRM-Text’s design scales to the parameter counts needed for production chat models is the next open question. The team is working on it.