← Back to feed

Ai2 announced on October 7 that its paper “Retrofitting language models to operate over bytes” has been published in Nature, the highest-impact journal in science. The work, which underpins the Bolmo family of byte-level language models Ai2 introduced in December 2025, comes with new model checkpoints released alongside the publication.

Publication in Nature is not routine for applied AI research. The journal’s editorial bar favours work with broad scientific significance and rigorous methodology. The decision to publish a language model architecture paper there reflects both the technical novelty of the approach and its potential implications across linguistics and computational science.

The Tokenisation Problem

Every major language model in production today uses a tokeniser: a preprocessing step that converts raw text into a sequence of subword pieces drawn from a fixed vocabulary. Models like GPT-4, Claude, and Gemini were all trained on tokenised inputs. The tokeniser is so fundamental to the transformer architecture that most discussions of model capability treat it as infrastructure, not a choice.

Tokenisation creates a specific class of problem: languages and writing systems that are poorly represented in the training vocabulary get worse performance. Languages with rich morphology, rare scripts, or non-Latin character sets produce fragmented token sequences — many tokens per concept — while English-heavy training vocabularies handle the same semantic content with far fewer. This bias is embedded in the model’s weights and is not addressable through fine-tuning or prompt engineering.

Byteification is a method for retrofitting an existing subword-tokenised model to operate on raw bytes instead. Rather than rebuilding a model from scratch with a byte-level architecture — which would require enormous compute — the approach modifies an already-trained model’s input and output layers to accept bytes directly. The underlying weights, which encode semantic and syntactic knowledge, are preserved.

What Ai2 Demonstrated

The Bolmo models — first published in December 2025, now backed by the Nature paper — showed that retrofitted byte-level models could match or exceed the performance of their tokenised counterparts on standard benchmarks while handling character-level tasks (spelling, morphology, transliteration) and low-resource languages substantially better.

The core result is that you do not need to choose between semantic capability and character-level fidelity. The byteification approach gets both: the knowledge embedded in a mature model’s weights plus the flexibility of byte-level processing.

New checkpoints released with the Nature publication extend the original Bolmo results. Ai2 has not disclosed the specific benchmark figures for the new checkpoints but the release indicates ongoing development rather than a one-time research artefact.

Why This Matters Beyond Linguistics

The significance of the byte-level framing extends past multilingual coverage. Tokenisers introduce subtle artefacts in any domain where character-level structure matters: genomic sequences, structured file formats, low-level programming languages, and mathematical notation all suffer from tokenisation that does not align with their natural granularity.

A model that processes bytes directly has no tokenisation boundary to violate. This does not guarantee better performance on all tasks — byte-level models must learn the higher-level structure that tokenisers provide for free — but the byteification approach avoids relearning from scratch by importing that structure from the retrofitted base model.

The COLM 2026 conference, where the paper was also presented, treats this as one of the more technically significant architecture results of the year. Nature’s decision to publish it confirms that the academic community outside of AI research shares that assessment.

The Retrofit Strategy

The broader implication of byteification as a method is that the technique generalises. If you can retrofit a tokenised model to handle bytes, the same approach may apply to other modality-level constraints baked into pretrained models — fixed image patch sizes, fixed audio frame lengths, or other architectural assumptions made during training that limit what the model can process at inference time.

Ai2 is an open research organisation; the Bolmo checkpoints and the byteification code are publicly available. The Nature publication makes the methodology citable and reproducible at a standard that applied AI labs do not typically achieve for architectural work of this type.