← Back to feed

A Microsoft Research paper published today on arXiv offers a direct challenge to the assumption that harder tasks require more expensive reasoning models. The paper, “Reason Wide, Not Deep: Amortizing the Reasoning Premium into Distilled Skills,” finds that a significant fraction of what reasoning models do at inference time can be replaced by a small skill file derived from past trajectories — at a fraction of the cost.

The paper tests a specific alternative: collect 35 to 50 past agent trajectories, run a coding agent to extract recurring failure patterns, and turn those patterns into a short markdown skill file appended to a non-reasoning model’s system prompt. No reasoning traces required in the distillation step.

The Numbers

On GPT-5.4-mini, the distilled skills approach recovered between 55% and 100% or more of the performance gap between non-reasoning and reasoning modes across four agent benchmarks, while using 2.9 to 4.5 times fewer output tokens than running in reasoning mode.

On two benchmarks — ALFWorld and τ²-retail — the skilled non-reasoning model did not just narrow the gap. It outperformed reasoning mode outright.

On telecom and SpreadsheetBench, reasoning mode retained its advantage. The paper explains the split clearly: tasks with repeated procedural patterns can be amortised into a skill file because the same failure modes recur. Tasks with high instance-specific dependencies — where each run requires fresh search over unique context — still need reasoning. The skill file cannot anticipate what it has not seen before.

Critically, the distiller itself did not need reasoning traces. Skills built only from cheap non-reasoning rollouts were competitive across all four domains.

What This Means

The paper is not arguing that reasoning models are overrated. It is arguing that reasoning mode is expensive in the wrong places — specifically, on tasks that repeat. The cost of generating skills is paid once across 35 to 50 runs. The savings on every subsequent run accumulate.

For agentic systems that handle high-volume, procedurally similar tasks — customer support ticket handling, code review pipelines, repeated data extraction workflows — the economics are significant. Paying 3 to 4 times per token for a reasoning-capable model, every time, for a task type that has been seen hundreds of times before, is a design choice the paper suggests is unnecessary.

The failure mode the paper is trying to fix is what the authors call over-spending on depth when the task calls for breadth. Reasoning mode goes deep on each inference. Distilled skills encode breadth from prior runs and apply it cheaply.

Limitations

The paper’s results are on GPT-5.4-mini, not frontier-tier models. The gap between non-reasoning and reasoning modes is smaller for very capable non-reasoning models, which could limit how much recovery is achievable. The approach also assumes task repetition — one-off novel tasks provide no prior trajectories to learn from, making skill distillation inapplicable.

The skill file quality depends on the quality of the coding agent doing the extraction. If the extractor misidentifies which failure patterns are recurring vs incidental, the resulting skill may not transfer.

Practical Split

The paper proposes a direct decision rule: identify which tasks in a workflow have recurring failure patterns, distill once, and route those tasks to a non-reasoning model with the skill file attached. Reserve reasoning mode for tasks that are genuinely novel or highly instance-dependent on each run. The τ²-retail and ALFWorld cases suggest the division is wider than practitioners typically assume.

The arxiv paper is at arxiv.org/abs/2608.07885.