Synthetic Data
Synthetic data is artificially generated training data — produced by algorithms, simulations, or other AI models — used to train, fine-tune, or evaluate machine learning systems. It is a standard ingredient of modern AI training, especially for small models and post-training: Microsoft's Phi-3 report describes a mix of web data and LLM-generated data. Large models are a different story. Meta's Llama 3 paper describes a filtered web-data pipeline, and most labs do not publish a source-by-source breakdown of their training data. Synthetic data also enables training in domains where real data is sensitive (medical, legal, financial) or costly to label.
AI-generated training data that teaches models without real-world data collection.
Category: Model Training & Optimization
How AI labs use synthetic data
One widely used strategy is teacher-student generation: a strong 'teacher' model generates responses to a large corpus of prompts, those responses are filtered for quality, and the results train a smaller 'student' model. Microsoft's Phi series is the best-documented case. Its Phi-3 report describes a first training phase built mostly on web sources and a second that shifts toward LLM-generated data for reasoning and niche skills. This is not a general description of open models: the Llama 3 paper uses Llama 2 to label web documents for quality classifiers rather than generating its corpus, and the Gemma 3 report describes knowledge distillation without giving a source-by-source data breakdown.
| Use case | Synthetic data type | Key benefit | Key risk |
|---|---|---|---|
| Instruction tuning | LLM-generated (instruction, response) pairs | Millions of examples at low cost | Model may inherit teacher's biases and errors |
| Code generation | Synthetically generated code + test cases | Precise correctness labels from test execution | Distribution shift if real code differs from synthetic patterns |
| Medical AI | Synthetic patient records from generative models | Privacy compliance; rare condition simulation | May not capture real clinical distribution accurately |
| Autonomous driving | Simulation-generated sensor data | Rare/dangerous scenarios without real accidents | Simulation-to-real gap: real sensors differ from simulated |
| RLHF preference data | Model-generated preference pairs rated by AI judge | Scales to millions of pairs vs thousands from humans | AI judge biases propagate to trained model |
Risks and limitations of synthetic data
- Model collapse: If models are trained on AI-generated data, which is used to train new models, which generate data for the next generation — statistical variance is progressively lost. The training distribution degenerates toward the mean. This is the 'model collapse' phenomenon documented by Shumailov et al. (2024) and remains an active concern for the field.
- Propagated errors: Synthetic data from a teacher model inherits that model's factual errors, biases, and blind spots. If the teacher hallucinates a fact and this error appears in synthetic training data, the student model learns the error as ground truth.
- Diversity loss: Synthetic generation tends toward the most probable outputs — creative, unusual, or edge-case examples are underrepresented. Models trained on synthetic data may be less robust on unusual inputs.
- Legal grey area: Generating synthetic data using a commercial model (GPT-4, Claude) and using it to train a competing model may violate terms of service. Most major AI providers explicitly prohibit using their outputs to train competing models.
The quality filtering solution: The usual mitigation for synthetic data risks is aggressive quality filtering: generating more synthetic data than you need, then keeping only what passes quality classifiers, reward models, or execution-based verification (for code). The same idea applies to real data. Meta's Llama 3 paper filters web documents with classifiers trained on Llama 2 labels.
Practice questions
- What is model collapse and why is it an existential risk for AI development pipelines? (Answer: Model collapse: when a model trained on AI-generated data is used to generate more training data for the next model, statistical diversity progressively decreases. Each generation loses subtle patterns in the real distribution, eventually converging to a narrow, low-diversity distribution. Shumailov et al. (2023) demonstrated this formally. It is 'existential' for AI pipelines that rely on synthetic data: without intervention, each generation of model is worse than the last at capturing real-world diversity.)
- What quality filtering techniques distinguish high-quality synthetic data from noise? (Answer: (1) LLM-as-judge: use GPT-4 or Claude to rate generated responses on accuracy, helpfulness, and instruction following — keep top 10–20%. (2) Reward model scoring: train a preference model and filter by reward score. (3) Consistency checks: generate multiple responses to the same prompt and keep only those where responses agree on factual claims. (4) Perplexity filtering: remove extremely high-perplexity (incoherent) or extremely low-perplexity (generic) responses. (5) Diversity filtering: remove near-duplicates using embedding similarity.)
- What is the 'Phi hypothesis' — why do small Phi models punch above their weight class? (Answer: Microsoft's Phi series (Phi-1, Phi-2, Phi-3) trains small models (1.3B–3.8B) on carefully curated 'textbook quality' synthetic data — GPT-4-generated educational content covering reasoning, science, and mathematics in clear, pedagogical style. Hypothesis: most of the performance gap between small and large models is training data quality, not model capacity. Clean, educational synthetic data teaches reasoning more efficiently than raw web text. Phi-3-mini (3.8B) matches GPT-3.5 on many benchmarks.)
- How does synthetic data generation differ for code vs natural language? (Answer: Code synthesis: generate code problems with known solutions, run the code, verify correctness automatically through test execution — verifiable ground truth without humans. Error types are objective (syntax errors, test failures). Natural language synthesis: correctness is subjective, requires human or LLM judgment. Code synthetic data can be generated and filtered entirely automatically at scale; NL data requires more sophisticated quality filtering. This is why code models (Codex, AlphaCode) scaled to much larger synthetic datasets earlier.)
- What is self-play in synthetic data generation and how does it work for reasoning models? (Answer: Self-play: the model plays against itself to generate training data. For reasoning: the model generates a problem, attempts to solve it, checks correctness (via code execution or formal verification), and uses successful and failed attempts to generate contrastive training pairs. DeepSeek-R1 is a related case rather than a pure example: it was trained with reinforcement learning (GRPO) on problems with verifiable answers, so correct reasoning paths earn reward and wrong ones do not. The model progressively improves by learning from its own errors.)