← Back to feed

Pretraining got a free corpus. Agentic posttraining does not have one, and the attrition rates from the best public attempts show why.

David Hall of Open Athena makes the case in a post titled “There is no posttraining data commons”. Pretraining consumes documents: Wikipedia, arXiv, Stack Exchange, Project Gutenberg and Common Crawl, which holds roughly 10 petabytes of archived web data. GPT-3 was about 60% Common Crawl-derived. Open models such as Nemotron and Open Athena’s own still draw most of their pretraining data from that commons. Posttraining consumes tasks, and a usable reinforcement-learning task needs a world, an intent, a set of tools and a grader.

The yield problem

Software is the best case, because version control, CI, unit tests and containers already preserve worlds and graders. The numbers are still harsh.

• The original SWE-bench crawled a bit over 90,000 pull requests from 12 Python repositories. Only about 1 in 8 mapped cleanly to the task frame, leaving 11,407. About 80% of those could not be built into executable tasks, leaving 2,294.
• OpenAI then had 93 experienced Python developers review 1,699 randomly selected SWE-bench tasks. More than two thirds were flagged for removal. The surviving pool became the 500 tasks of SWE-bench Verified, and a later audit of a problematic subset found roughly another sixth of the 500 were also broken.
• In one 5,000-task Java subset of TaskTrove, more than 90% had to be discarded during repair because tests did not parse, checked nothing, or depended on missing context and external services.
• In another Python subset, almost 98% of inspected verifier bundles could not start their tests because dependencies were missing.
• A Vals audit found two thirds of the 2,698 coding environments in Xiaomi’s MiMo v2.6 public release still contained reference fixes recoverable from Git. In one demonstrated case an agent copied the fix and passed all five hidden tests.

Hall’s conclusion: converting repositories into training tasks looks “less like curation and more like manufacturing”, because the information needed may never have been recorded.

Outside software, nothing exists

For law, medicine, sales or science there is not even a partial public task record. Frontier labs pay professionals to fit their work into the RL frame and build environments, rubrics, verifiers and simulators, which Hall identifies as the reason a posttraining data industry is growing quickly. He points to the construction of GDPval as one model for doing this deliberately.

Why it matters for open weights

Hall’s argument is about power. If only a few companies hold the tasks and environments needed to train frontier agents, they decide which kinds of work AI can do. His proposal is a deliberately built commons, seeded from open-source repositories and from traces left in documents with tracked changes, logs and procedures, with agents helping reconstruct tasks from those traces. He concedes it is harder than the pretraining case, where publishers were producing training-shaped material for unrelated reasons.

The post also lands as a data point on the benchmark side. The same task-quality problem sits underneath every coding benchmark built from repository data.

Key numbers

• GitHub PRs crawled for SWE-bench: about 90,000; final tasks: 2,294
• SWE-bench Verified: 93 reviewers, 1,699 tasks screened, 500 kept
• TaskTrove Java subset: more than 90% of 5,000 tasks discarded
• MiMo v2.6 environments with recoverable fixes: about two thirds of 2,698