← Back to feed

Labs deciding how much RL compute to spend on a given checkpoint have historically operated without a reliable forecasting tool. A new paper says that tool exists, and it’s the pretraining validation loss already sitting on disk.

The work — “Understanding Reasoning from Pretraining to Post-Training” (arXiv:2607.16097) — reproduces the full modern LLM training pipeline in chess: pretraining 5M–1B parameter models on Lichess games, SFT on synthetic reasoning traces, then GRPO on puzzles with verifiable rewards. The controlled environment enables 36 pretraining–RL combinations that would be prohibitively expensive to sweep on frontier text models.

The Two Predictive Signals

Signal 1: Pretraining validation loss predicts post-RL pass@1. Pearson correlation rises from |ρ| = 0.93 at the start of RL to 0.99 as reference compute increases. The relationship tightens as RL scales — meaning the forecast gets more accurate exactly when the stakes are highest.

Signal 2: RL gain per 10× of compute increases with log pretraining tokens (r = +0.84). Models trained on more tokens see higher marginal returns from RL. This explains why labs that cut pretraining short to speed up a release get disproportionately less out of their RLHF or GRPO budget.

The Compute Split

The paper derives a compute-optimal RL fraction:

• ~20% of total compute should go to RL at 50M parameters
• ~28% at 680M parameters

Both are minority shares. The implication is that RL is not a substitute for pretraining investment — it is a sharpening pass that amplifies what pretraining already built. Spending more than ~28% on RL before the model is large enough to benefit is sub-optimal by the paper’s derived frontier.

Why Pass@1 Improves But Pass@16 Doesn’t

The mechanism is the part most worth understanding.

RL is not uniform sharpening. On the hardest puzzles, it generates correct moves at rates that were near-zero after SFT. That part looks like genuine capability gain.

Simultaneously, RL makes already-preferred wrong moves more likely. Errors the model was weakly inclined toward pre-RL become more confident and frequent errors post-RL. The result: pass@1 improves because the correct answer gets a higher rank, but pass@16 doesn’t because the wrong-answer modes are also amplified — and they still show up in the sample.

The authors call this wrong-mode amplification, and they identify it as the more tractable research target: suppressing the wrong modes, rather than continuing to sharpen the right ones harder.

Practical Value

The finding compresses to a decision rule: measure your pretraining validation loss, check where you land on the predictive curve, set your RL budget accordingly. The paper claims you can estimate post-RL pass@1 before committing the compute — an ROI calculation that was not previously possible.

The results replicate on a 1B model trained on math text, suggesting the relationship generalises beyond the chess setting.