Atlasof World Models
Start

T-06 · Generative Foundations: Latent Variables, Diffusion & Flow

  1. Mission · Sonu, 10

    Generative Foundations: Latent Variables, Diffusion & Flow

    There are a handful of very different recipes for turning nothing into something realistic, each with its own trade-offs.

    5 idea cards, 3 questions and a recap. About 6 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 5

    Ask four artists to paint the same chair. Each one has a different way of working.

    The first artist sketches a rough shape in a few lines. Then she fills in the details from that sketch. The rough sketch is like a secret code for the chair.

  3. Idea 2 of 5

    Keep in your head Ask four artists to paint the same chair.

    The second artist paints fast and bold. A grumpy critic stands beside him. The critic says "fake" or "real" for every painting. The artist keeps trying until the critic is fooled.

    The third artist paints one stroke at a time, always left to right. Each new stroke depends on the strokes before it. She never goes back.

    Try this Close the screen. Tell someone at home what this card says, in your own words.

  4. Idea 3 of 5

    Keep in your head The second artist paints fast and bold.

    The fourth artist starts with a page full of messy scribbles. Then she cleans them up a tiny bit at a time. Slowly a chair appears out of the mess.

    All four end with a chair. But each way is good at something different.

    The sketch artist is quick and tidy. The critic's artist makes sharp, bold pictures. He and the critic argue a lot, though.

  5. Idea 4 of 5

    Keep in your head The fourth artist starts with a page full of messy scribbles.

    The stroke-by-stroke artist is careful about order. The scribble cleaner makes the most beautiful pictures. She is also the slowest.

    Computers that make new pictures, sounds or stories use these same four ways. Scientists call them generative models. Each one turns nothing into something that looks real.

    Try this Draw this card as one small picture. Use three lines or fewer.

  6. Idea 5 of 5

    Keep in your head The stroke-by-stroke artist is careful about order.

    World models borrow these ways to imagine what happens next. A machine that imagines tomorrow's weather picks the way that fits the job.

    Try this at home with your family. Each person draws a cat using one of the four ways.

    One sketches first. One draws while someone else says "fake" or "real." One draws only left to right. One starts from scribbles and erases.

    Then line up the four cats. Which way was fastest? Which cat looks best?

  7. Words to keep

    Try this Cover the list. Say each word and what it means, then check.

  8. Quick check 1 of 3

    1A VAE's decoder learns to ignore the latent entirely. What is this called?

  9. Quick check 2 of 3

    2Which family usually costs the most at inference time for one high-quality sample?

  10. Quick check 3 of 3

    3An auditor asks how likely a generated scenario was. Which family cannot answer directly?

  11. Recap

    What you just learned

    1. Ask four artists to paint the same chair.
    2. The second artist paints fast and bold.
    3. The fourth artist starts with a page full of messy scribbles.
    4. The stroke-by-stroke artist is careful about order.
    5. World models borrow these ways to imagine what happens next.

  1. Mission · Senior AI Architect

    Generative Foundations: Latent Variables, Diffusion & Flow

    There are a handful of very different recipes for turning nothing into something realistic, each with its own trade-offs.

    7 idea cards, 3 questions and a recap. About 8 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 7

    Four families learn to generate. They differ on four properties. Two are sample quality and training stability. The other two are likelihood guarantees and inference speed.

    No family wins on all four.

    Variational autoencoders learn a latent variable and a decoder. Kingma and Welling (2013) made them trainable with the reparameterization trick. Training maximizes a lower bound on likelihood, the ELBO. Ha and Schmidhuber's 2018 world model used a VAE as its vision module.

  3. Idea 2 of 7

    Keep in your head Four families learn to generate.

    The known VAE failure is posterior collapse. A strong decoder learns to ignore the latent entirely. The KL term falls toward zero.

    Bowman et al. (2016) met this with text VAEs. They answered it with KL annealing and word dropout on the decoder.

    GANs (Goodfellow et al., 2014) replace explicit likelihood with a game. A generator tries to fool a discriminator.

    Try this Find "posterior collapse" on this card. Define it in one line without looking.

  4. Idea 3 of 7

    Keep in your head The known VAE failure is posterior collapse.

    Samples are often sharp. Training is often unstable. There is no likelihood to check.

    Autoregressive models factor a joint distribution into a chain of conditionals. They generate token by token and train with teacher forcing. They compose naturally with sequential decisions. They also inherit compounding error over long rollouts.

  5. Idea 4 of 7

    Keep in your head Samples are often sharp.

    Diffusion and flow models learn to reverse a gradual noising process. Flow matching instead follows a continuous path from noise to data.

    Training is stable and sample quality is strong. Ho et al. (2020) reported an FID of 3.17 on CIFAR-10 with DDPM.

    The cost sits at sampling time. DDPM ran 1,000 denoising steps per sample. That is why diffusion leads in visual fidelity and lags in real-time use. T-07 covers this family in depth.

    Try this Name one system you have shipped where this would break first.

  6. Idea 5 of 7

    Keep in your head Diffusion and flow models learn to reverse a gradual noising process.

    Within a world model, the backbone sets the character of its imagination. Autoregressive backbones fit step-by-step decision making. Diffusion and flow backbones lead wherever visual fidelity is the priority.

    Here is the meter version, from a toolkit lab of mine. I generated 4,000 daily load curves of 96 points each.

  7. Idea 6 of 7

    Keep in your head Within a world model, the backbone sets the character of its imagination.

    Three hidden knobs made them. The knobs were base load, the hour of the peak and its height. An autoencoder squeezed each curve into 3 numbers and redrew it.

    Reconstruction error was 0.00096 on curves that run from about 0.3 to 3.0. One latent tracked base load with a correlation of 0.82. Another tracked peak time at 0.84. Peak height came through weakly, at 0.46 at best.

    Try this Write down the one number on this card you would put on a dashboard.

  8. Idea 7 of 7

    Keep in your head Three hidden knobs made them.

    Nobody told the network what the knobs were. Compression keeps what the loss pays for. A VAE on the same curves would add sampling. A planner could then draw plausible load days for a study.

    What to measure depends on the family. For VAEs, track KL per latent dimension to catch collapse. For GANs, check diversity, since mode collapse hides behind sharp samples. For any backbone, test samples on the decision they feed, beside image or curve scores.

    I would recommend to name the property the application needs before choosing. Pick the backbone from that list.

  9. Words to keep

    Try this Cover the list. Say each word and what it means, then check.

  10. Quick check 1 of 3

    1A VAE's decoder learns to ignore the latent entirely. What is this called?

  11. Quick check 2 of 3

    2Which family usually costs the most at inference time for one high-quality sample?

  12. Quick check 3 of 3

    3An auditor asks how likely a generated scenario was. Which family cannot answer directly?

  13. Recap

    What you just learned

    1. Four families learn to generate.
    2. The known VAE failure is posterior collapse.
    3. Samples are often sharp.
    4. Diffusion and flow models learn to reverse a gradual noising process.
    5. Within a world model, the backbone sets the character of its imagination.
    6. Three hidden knobs made them.
    7. Nobody told the network what the knobs were.

  1. Mission · My CIO

    Generative Foundations: Latent Variables, Diffusion & Flow

    There are a handful of very different recipes for turning nothing into something realistic, each with its own trade-offs.

    4 idea cards, 3 questions and a recap. About 5 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 4

    None of the four generative families is strictly best. Each is a different point on a trade-off. Two of the properties in play are sample quality and training stability. The other two are likelihood guarantees and inference speed.

    A team choosing a world model backbone should name the property the job needs. Defaulting to whichever architecture is most publicized this year is the common mistake.

  3. Idea 2 of 4

    Keep in your head None of the four generative families is strictly best.

    The money shows up in two places. Training cost is the first. GANs can take many restarts before they train well. Diffusion and flow models train more predictably.

    Inference cost is the second. It repeats every day.

    Diffusion's lead in visual quality comes with a real speed cost. Classic diffusion sampling ran many steps per output. For any real-time or interactive use, that cost multiplies by every call.

    Try this Find "diffusion" on this card. Define it in one line without looking.

  4. Idea 3 of 4

    Keep in your head The money shows up in two places.

    Likelihood matters for risk work. A VAE or an autoregressive model can say how probable an outcome is. A GAN cannot. If a regulator or an auditor asks how likely a scenario was, that difference matters.

    For a utility the uses split cleanly. Offline planning studies can afford slow, high-quality samples. A dispatch or control loop needs speed and a stable answer.

  5. Idea 4 of 4

    Keep in your head Likelihood matters for risk work.

    Synthetic load days for testing need diversity. They also need proof they are realistic.

    I would recommend to write the needed property into the requirements before any vendor demo. That keeps the decision tied to the job rather than the headline.

    The risk of skipping that step is quiet. A beautiful generator gets bought for a task that needed honest probabilities or low latency. The gap shows up months later in cost or in trust.

    The question I would ask is this. Which property does this use case need most? What did we give up to get it?

    Try this Say what this card would cost you if a vendor got it wrong.

  6. Words to keep

    Try this Cover the list. Say each word and what it means, then check.

  7. Quick check 1 of 3

    1A VAE's decoder learns to ignore the latent entirely. What is this called?

  8. Quick check 2 of 3

    2Which family usually costs the most at inference time for one high-quality sample?

  9. Quick check 3 of 3

    3An auditor asks how likely a generated scenario was. Which family cannot answer directly?

  10. Recap

    What you just learned

    1. None of the four generative families is strictly best.
    2. The money shows up in two places.
    3. Likelihood matters for risk work.
    4. Synthetic load days for testing need diversity.

Search the atlas