Few topics in AI research have generated more panic per finding than synthetic data. In July 2024, a Nature paper warning that "AI models collapse when trained on recursively generated data" landed like a thunderclap: the internet was filling with AI-written content, models were training on the internet, and suddenly the whole enterprise looked like it might eat its own tail.

Meanwhile, synthetic data quietly became one of the most productive tools in model training: instruction datasets, reasoning traces, textbooks written by AI for AI. Many of today's strongest models lean on it. So which story is true? Both, it turns out. The last two years of research have drawn a sharper map of where synthetic data helps, where it hurts, and what separates fuel from poison.

The panic: what "model collapse" actually demonstrated#

The canonical collapse study is Shumailov et al., published in Nature in 2024 (preprint: "The Curse of Recursion," arXiv:2305.17493). The setup was deliberately harsh: take a generative model, fine-tune it on its own outputs, then repeat the process generation after generation — replacing the real training data with synthetic data each round.

The result was real and unsettling. Models progressively lost information about the tails of the data distribution — rare events, unusual cases, minority patterns — before eventually degenerating into repetitive, low-variance gibberish. In one reported experiment, a language model that began training on Wikipedia text was, by the ninth generation, producing lists of non-existent "jackrabbits" when asked about English church towers. The mechanism is statistical, not mysterious: each generation fits a slightly smoothed, slightly averaged version of reality, and the texture gets sanded off a little more each time.

Alemohammad et al.'s "Self-Consuming Generative Models Go MAD" (ICLR 2024) confirmed the same pattern in image models: without fresh real data injected each generation, quality degrades after just a few autophagous loops.

Crucially, though, this is a statement about a specific training workflow: recursive, self-consuming, real-data-replacing. It was never evidence that all synthetic data is toxic. The question the field then asked was the right one: is collapse the default, or a special case you can design around?

The correction: collapse is containable, not inevitable#

The sharpest answer came from Kazdan, Schaeffer, Donoho, and colleagues in "Collapse or Thrive?" (arXiv:2410.16713, revised 2025). They tested three training workflows across three model settings, and the contrast is instructive:

  • Replace: every generation trains only on the previous generation's outputs. This collapses, confirming Shumailov — every time.
  • Accumulate: each generation's synthetic data is added alongside the real data, so the training set keeps growing. Here, models stayed stable — test losses did not diverge, even as the proportion of real data eventually became small.
  • Fixed-size accumulation: real and synthetic data accumulate together, but each generation trains on a fixed-size subset. This produced slow, gradual degradation — not an explosion.

The headline of the paper is its thesis: the danger is real but manageable. How you use the data matters more than whether some of it is synthetic. Single-round augmentation — generate once, mix with real data, train — is far safer than recursive iteration.

The scale evidence: Meta's 1,000-model experiment#

Theory is useful, but synthetic data skeptics had a fair objection: most collapse studies used toy setups or small models. Enter the largest empirical study of the question to date — "Demystifying Synthetic Data in LLM Pre-training," from Meta FAIR (Kang et al., arXiv:2510.01631, presented at EMNLP 2025). The team trained over 1,000 LLMs using more than 100,000 GPU hours under a unified protocol, comparing natural web data against two synthetic types (rephrased text and generated textbooks) and their mixtures.

The findings are nuanced in exactly the way that makes both the doomers and the boosters partly right:

  • Rephrased synthetic data alone (rewriting web text in different words) was not better than natural web text. The shortcut of "just rephrase everything" doesn't work.
  • But mixing roughly one-third rephrased synthetic with two-thirds natural web text sped up training 5–10x (measured as compute to reach the same validation loss) at larger data budgets. That's a massive efficiency win.
  • Textbook-style synthetic data alone — fully generated explanatory content — produced notably higher downstream loss, especially at smaller data budgets. This is the regime where collapse-like patterns appeared.
  • The optimal synthetic ratio converged to ~30% for rephrased synthetic data, depending on model size and data budget.
  • Surprisingly, larger generator models did not necessarily produce better training data than ~8B-parameter generators. Bigger teacher ≠ better data.
  • On model collapse directly: single-round (n=1) training on rephrased synthetic data showed no degradation at foreseeable scales, while textbook-style pure-generated mixtures showed collapse patterns. Mixed evidence — conditional, again, on what kind of synthetic data and how it's used.

This is about as close to a ground truth as the field has: synthetic data is not a free lunch and not a poison pill. It's an ingredient with a dosage.

Where synthetic data is already winning#

The collapse debate sometimes obscures how decisively synthetic data has already proven itself in the workflows nobody is arguing about:

  • Instruction tuning. The instruction-following revolution leaned on synthetic demonstrations. Self-Instruct generated 52,000 examples from 175 seed tasks; Stanford's Alpaca fine-tuned a 7B Llama on 52,000 examples generated with roughly $600 of API calls — and launched the open-source chatbot wave of early 2023.
  • Reasoning traces. Microsoft's Orca trained smaller models on GPT-4-generated explanations of how to reason through problems, not just answers — a landmark in distilling reasoning capability downward. DeepSeek-R1 (2025) pushed this further, using reinforcement-learning-discovered reasoning traces as synthetic training data.
  • Targeted augmentation. Rephrasing, back-translation, code self-repair loops, math problems with verifiable answers — synthetic data is strongest where outputs can be checked. When a generated math solution can be executed and verified, quality filtering is cheap and the data is genuinely additive.
  • Underrepresented domains. For low-resource languages, niche scientific topics, or edge-case scenarios, synthetic generation is often the only way to get enough training examples. The alternative isn't better data — it's no data.

The pattern across all of these: synthetic data works best as augmentation of a real-data core, generated once (not recursively), with quality filtering — deduplication, correctness verification, diversity scoring — applied before training. Precisely the workflow the collapse research says is safe.

The real risks that remain#

None of this means the panic was baseless. The research identifies several genuine, ongoing risks that good practice must manage:

  • Tail knowledge erodes first. Synthetic data systematically underrepresents the long tail — rare facts, minority viewpoints, edge cases — partly because sampling itself (temperature, top-p) concentrates probability mass on the common. One 2025 study found synthetic mixtures introduce systematic distributional discrepancies exactly here, and Dohmatob et al.'s theory work argues even small synthetic fractions can cap scaling in kernel-regression settings. For applications that need the unusual-but-correct answer, this is the risk to watch.
  • Diversity decays with recursion. Each synthetic generation is a little less varied than the last. Single-round is safe; multi-generation pipelines without fresh real data are where the photocopy-of-a-photocopy effect bites.
  • Benchmark contamination. If the teacher model saw benchmark test sets, synthetic data can leak benchmark-specific knowledge into the student — inflating scores without real capability gains. Evaluations built on synthetic-heavy data need extra decontamination.
  • Hallucinated supervision. Synthetic data containing hallucinated answers can increase hallucinations in fine-tuned models. A teacher's confident errors become a student's curriculum.
  • Safety blind spots. One unsettling finding: fine-tuning on synthetic data can strip safety guardrails while preserving output quality — unlike human-data fine-tuning, where both tend to degrade together. The virtue of synthetic data (it preserves capability) becomes a safety-amplification risk.

There's also a mundane but important caveat: many model providers' terms of service prohibit using their outputs to train competing models.

The takeaway#

The synthetic-data debate resolved into something less dramatic and more useful than either side predicted. The evidence converges on a consistent picture:

  1. Recursive, real-data-replacing synthetic training collapses. This is settled — treat it as a law.
  2. Single-round synthetic data, mixed with real data at roughly a third, helps. Meta's large-scale study puts a number on it: 5–10x training speedup, ~30% optimal mix, no degradation.
  3. The generator's size matters less than the recipe. Rephrased real text beats fully generated textbooks; an 8B generator can beat a giant one.
  4. Tails and diversity are the canaries. If synthetic pipelines aren't monitored for long-tail coverage and output diversity, the damage is quiet — headline metrics can keep rising while the rare and unusual quietly vanish.

So: fuel, carefully measured and mixed with real data. Poison, when a model's only diet is its own exhaust. The labs that understand the difference are already training on it — and the ones that don't will find out which kind they bought.