How a poisoning attack works

Models trust their data. If an attacker can get content into a training set, a fine-tuning file, or a retrieval database, the system may treat it as truth. The attack happens early and the harm appears later, which makes the two hard to connect. Poisoned models can also score normally on standard tests.

How much poison does it take?

Less than people assumed. These are the best-known research results, and the setup of each one matters.

Worked example: why 250 documents surprised researchers

Security researchers often expressed poisoning as a percentage of training data. Under that view, a bigger dataset should demand proportionally more poison. In October 2025, researchers from Anthropic, the UK AI Security Institute, and the Alan Turing Institute tested that assumption on 72 models.

1. They built each poisoned document from up to 1,000 characters of real text, the trigger <SUDO>, and 400 to 900 random tokens.
2. They trained models of 600M, 2B, 7B, and 13B parameters with 100, 250, or 500 poisoned documents.
3. 100 documents did not reliably create a backdoor. 250 or more did.
4. On the largest setup, 250 documents, about 420,000 tokens, were roughly 0.00016% of the training tokens.

\frac{420{,}000}{0.00016\%} = \frac{4.2\times 10^{5}}{1.6\times 10^{-6}} \approx 2.6\times 10^{11}\ \text{tokens}

The takeaway was that success depended on the absolute number of poisoned documents, not their share of the data. The authors were careful about limits, though. Their backdoor only produced gibberish, which they said is unlikely to pose significant risk in frontier models, and they could not say whether the trend holds above 13B parameters or for subtler behaviors.

Getting poison in is the hard part: Writing 250 bad documents is easy. Getting them through a lab's data selection and filtering is much harder. The study's authors say that controlling whether specific data gets included is often the real bottleneck for attackers.

Practical attack routes

Poisoning used as a defense: Nightshade's authors propose it as a last resort for creators against scrapers that ignore opt-outs and do-not-crawl directives. The same technique can serve protest or sabotage, depending on who uses it and why.

Defenses

No single control stops poisoning. OWASP's guidance is layered:

• Track provenance: record where data came from and how it changed, using tools such as ML-BOMs and data version control.
• Verify content: pin hashes for URL-based datasets, so changed files are detected. The Carlini paper recommends low-overhead defenses of this kind.
• Vet sources: review data vendors, and use task-specific datasets for fine-tuning.
• Filter and detect: use anomaly detection on incoming data, and monitor training loss and behavior for oddities.
• Test with triggers: red team the model, and check outputs against trusted sources.
• Restrict write access: limit who can change knowledge bases and memory, and keep user-supplied data separate.
• Load models safely: prefer safer file formats over pickle, and verify what you download.

Output checks can help too. In the Nature Medicine study, a screen based on biomedical knowledge graphs captured 91.9% of harmful content. That is a promising result in one domain, not a general fix, and the PoisonedRAG authors found the defenses they evaluated insufficient.

Honest caveats

• Headlines overstate the risk to frontier models. "250 documents can poison any AI" drops the narrow backdoor, the size limits, and the filtering problem.
• Research setups favor the attacker. Lab experiments usually guarantee that poison enters training. Real attackers do not get that guarantee.
• Real-world evidence is thin. Public, confirmed poisoning of a major production LLM is hard to find, and that may reflect either rarity or poor detection.
• Small and fine-tuned models are a different case. Fine-tuning sets are much smaller, so each bad example carries more weight.

Frequently asked questions

What is the difference between data poisoning and prompt injection? Data poisoning corrupts what a system learns from or retrieves. Prompt injection manipulates it with crafted input during use. A poisoned RAG document blurs the line, since stored content delivers instructions at inference time.

Can ordinary people poison ChatGPT by chatting with it? Not in any simple way. Consumer feedback does not flow straight into a model's weights, and providers filter training data. Memory features are a separate risk for your own account.

Does Nightshade really work? It worked in the authors' experiments on text-to-image models. How well it holds up against every current pipeline is a separate question, so don't treat it as guaranteed protection.

Can a poisoned model be fixed? Sometimes, but it is hard. Removing a backdoor is uncertain, and retraining from clean data is the most reliable option when you can identify the bad data.

Do benchmarks detect poisoning? Not reliably. In the Nature Medicine study, poisoned models matched clean ones on the open benchmarks commonly used for medical LLMs.