How diffusion models actually make images: noise, schedules, and conditioning

Every time an image materializes from a text prompt, you're watching one trick: a neural network takes a frame of pure static and, step by step, carves a picture out of it. Stable Diffusion, DALL-E 2, Midjourney, Imagen — they're all variations on the diffusion model.

The idea fits in a sentence: learn to destroy images with noise, then learn to run that destruction in reverse. The details — noise schedules, latent space, conditioning, guidance — are where quality comes from. Here's the full picture, minus the heavy formalisms.

The core trick: learn destruction, then reverse it#

A diffusion model is built from two processes that mirror each other.

The forward process (diffusion) takes a real image and adds Gaussian noise over ~1,000 steps, following a recipe called the noise schedule. By the last step, it's indistinguishable from pure noise. This process is fixed — no network involved — and there's a closed-form formula for jumping to any noise level directly.

The reverse process (denoising) is the learned part: a neural network predicts the noise added at each step, subtracts it, and recovers a slightly cleaner image. String enough steps together — starting from fresh random noise — and a brand-new image appears.

The elegant twist: the network never learns "what a cat looks like" directly. It learns how to remove noise from a corrupted picture of anything — a well-posed, stable task, and a big reason diffusion models train reliably, with none of the adversarial instability or mode collapse that plagued GANs.

The noise schedule: the recipe for corruption#

The noise schedule, usually written as β₁ … β_T, decides how much noise goes in at each step. Get this recipe wrong and the model struggles; get it right and training hums along.

The original DDPM paper (Ho, Jain, and Abbeel, 2020) used a linear schedule: β ramps from 0.0001 to 0.02 over 1,000 steps, starting as a whisper of noise and ending much louder. Later, "Improved DDPM" (Nichol and Dhariwal, 2021) introduced a cosine schedule, which corrupts more gently at the start and more aggressively in the middle — the first steps should barely disturb the image while the middle does the heavy work.

Why does the schedule matter so much? It sets what the network studies. Training corrupts a real image to a random noise level in one shot — xt = √(ᾱt)·x₀ + √(1−ᾱt)·ε — then asks the network to predict the noise ε, with mean-squared-error loss. The schedule's cumulative effect (ᾱt) fixes the signal-to-noise ratio at each timestep, shaping the whole curriculum.

The denoiser: usually a U-Net, increasingly a transformer#

The workhorse network is a U-Net: an encoder that compresses the noisy image down to a bottleneck, a decoder that rebuilds it, and skip connections that ferry fine details across. In DDPM, the U-Net is augmented with:

• Time embeddings — a sinusoidal encoding of the timestep tells the network which noise level it's looking at, injected at every block.
• Self-attention at low resolutions, so the model coordinates globally ("the left eye should match the right").

In the last few years, the field has been drifting from U-Nets to transformers. The Diffusion Transformer (DiT, Peebles and Xie, 2023) showed a plain transformer denoiser scales beautifully with size, and newer generators like Stable Diffusion 3 and FLUX use transformer backbones with flow matching, a close cousin of diffusion. Either way, the job is the same: given a noisy image, a timestep, and a condition, predict the noise.

Latent space: why nobody diffuses raw pixels anymore#

Running 20–50 denoising steps through a big network on a 1024×1024×3 image is brutally expensive. The breakthrough that made consumer-grade generation practical was latent diffusion (Rombach et al., 2022 — the paper behind Stable Diffusion): first compress the image with a pre-trained autoencoder (a VAE), run diffusion in the compressed latent space, then decode back to pixels. Diffusion had been doing two jobs at once in pixel space — perceptual compression (stripping high-frequency texture the eye ignores) and semantic generation — and splitting them is far cheaper.

For Stable Diffusion's v1 line, the VAE compresses a 512×512×3 image to a 64×64×4 latent — roughly a 48× reduction in values. Denoising that small latent is dramatically cheaper, which is why high-resolution text-to-image runs on a laptop GPU instead of a datacenter. The autoencoder and the diffusion model are trained in separate stages, and the autoencoder can be reused across tasks.

There's a real tradeoff: compress too aggressively and small text, hands, and fine geometry suffer (every AI-art user has seen mangled fingers); compress too little and you're back to pixel prices.

Conditioning: making the model listen to your prompt#

An unconditional diffusion model generates something from the data distribution — but you want that thing you asked for. Conditioning is how the prompt gets in.

In latent diffusion models, text conditioning typically flows through cross-attention: the prompt is encoded into embeddings (by CLIP, T5, or similar), and the denoiser's attention layers let image regions "look at" the relevant prompt tokens. This is how "a red hat on the left, a blue umbrella on the right" gets spatially grounded. Spatially aligned conditions — edge maps, depth maps, masks — can instead be concatenated directly onto the noisy latent, the approach behind ControlNet-style control.

But conditioning alone is weak. A conditioned model often produces plausible images that only loosely match the prompt. That's where guidance comes in.

Classifier-free guidance: the steering knob in every UI#

Classifier-free guidance (CFG), from Ho and Salimans (2021/2022), is the two-line trick that makes text-to-image actually follow the prompt — and it's the "guidance scale" slider in every image-generation UI.

The idea: train the model to be both conditional and unconditional by randomly dropping the condition (about 10–20% of training steps) and feeding a null token instead. At sampling time, run the model twice per step — once with the prompt, once without — and extrapolate:

guided prediction = unconditional prediction + scale × (conditional − unconditional)

The difference between the two predictions is a direction in noise space that points toward "more of this condition." The guidance scale trades diversity for fidelity:

• Scale ≈ 1: standard conditional generation — free and diverse, may ignore the prompt.
• Scale 7–8: the sweet spot for text-to-image — strong prompt adherence without breaking the image.
• Much higher: the model follows the prompt harder, but images get oversaturated, waxy, and repetitive.

Negative prompts slot into the same framework: condition the "unconditional" prediction on what to avoid, steering generation away from it.

The classic alternative, classifier guidance (Dhariwal and Nichol, 2021), used gradients from a separate classifier trained on noisy images. CFG replaced it because it needs no extra model, and it's now near-universal across image, video, and audio diffusion models.

Faster sampling: from 1,000 steps to a handful#

The original DDPM needed 1,000 sequential network evaluations per image. Speeding this up became a whole research thread: DDIM (Song et al., 2020) made sampling deterministic so steps can be skipped (~10–50× faster, and it enables smooth interpolations); DPM-Solver and UniPC are specialized ODE solvers reaching 10–20 or even 5–10 steps; distillation trains student models to jump multiple steps at once, pushing toward one-step generation.

Modern UIs default to 20–50 steps — past a point, more steps buy imperceptible improvements. Early steps decide composition and layout; middle steps resolve structure; late steps add texture and fine detail. That's also why changing the seed tends to alter details more than composition: the initial noise fixes the rough layout first.

What to remember#

The architecture behind every image generator is a pipeline: prompt → text encoder → cross-attention-conditioned denoiser → latent space → decoder → image, wrapped in a sampling loop with a guidance scale. Noise in, meaning out — and now you know where each knob lives.

---

Sources consulted: the original DDPM paper (Ho et al., 2020, arXiv:2006.11239); Latent Diffusion Models (Rombach et al., 2022, arXiv:2112.10752); Classifier-Free Diffusion Guidance (Ho & Salimans, arXiv:2207.12598); plus university course notes on diffusion architectures and conditioning/guidance for cross-verification.