Generative AI

What generative AI is

Type a sentence and a model writes an essay, draws a photograph that was never taken, or produces a video clip complete with dialogue and sound effects generated alongside the pictures. Different products, one shared idea: learn what real data looks like, then produce new examples that could have come from the same source.

What is a generative model?

Most machine learning you meet first is discriminative: given an input, predict a label. Is this email spam? Is there a pedestrian in the frame? Mathematically, it learns p(label | x), the probability of a label given the input.

A generative model learns about the inputs themselves, p(x): which images are plausible, which sentences are likely, which sounds are speech. Once you have that, you can do things a classifier cannot. You can sample new data, fill in missing parts, continue a sequence, or notice when something looks unlike anything seen before.

How is that harder?

A spam filter can ignore almost everything about an email except the few clues that separate spam from the rest. A model that writes emails has to get everything right at once: spelling, grammar, tone, facts, formatting. For images the gap is even larger. A 512 by 512 colour image is 786,432 numbers, and the model has to learn how all of them relate to each other.

That is why generative AI arrived later than image classification, and why it needed very large models and datasets. It is also why generative models are now so general: a model that has learned what text looks like has absorbed a lot about the world the text describes.

Why does it matter?

Generative models now write code, draft documents, design molecules, generate training data for other models, dub films and narrate audiobooks. They are also the source of new risks: fabricated facts, synthetic media used to deceive, and questions about who owns the data they learned from. This lesson surveys how the main families of generative models work, with real tiny versions you can run, and ends with those risks and costs.

Key takeaways

• Discriminative models learn p(label | x); generative models learn p(x) and can sample new data from it.
• Generating is harder than classifying because every part of the output has to be right at once.
• Text, image, video and audio generators are variations on the same idea: learn a distribution, then sample.

Learn a distribution, then sample

Every generative model, from a 1948 letter counter to a video model, does two things. It estimates a probability distribution from examples, and it draws new samples from that distribution. The hard part is getting the estimate right: too faithful and it copies, too loose and it produces mush.

What does it mean to learn a distribution?

Imagine the training data as dots scattered through a space. A distribution says how likely each location is: high where dots are dense, low where they are absent. Learning a distribution means building a function that assigns that likelihood everywhere, including in the gaps between the dots you saw. Filling those gaps sensibly is what we call generalisation.

How does the simplest version work?

The simplest estimator is a kernel density estimate. Put a small Gaussian bump on every training point and add them up. Sampling from it is easy: pick a random training point, then nudge it by random noise with spread h. The single dial h, the bandwidth, decides everything.

Sampling from a Gaussian kernel density estimate. The x_i are training points.

With tiny smoothing nearly every sample is a near-copy of a training point: perfect fidelity, zero novelty. With heavy smoothing the samples fill the space around the spiral and its shape is lost. The useful setting is in between, where samples follow the spiral’s arms but land in places no training point occupied.

Here is why real generative models need neural networks. A kernel density estimate only knows “near a training point”. In a space with hundreds of thousands of dimensions, like images, every training point is far from every other, and nudging one with noise gives you a noisy copy of an existing photo, not a new one. Neural generative models learn the structure of the data (edges, textures, objects, grammar) so that they can place new samples in the gaps sensibly.

Why does it matter?

The tension you just saw between copying and generalising is not an academic detail. When a large model copies, it can leak private data or reproduce copyrighted work. When it generalises too loosely, it produces plausible nonsense. Every architecture in this lesson is a different strategy for getting that balance right in very high dimensions.

Key takeaways

• A generative model estimates a probability distribution from examples and then draws new samples from it.
• Too little smoothing copies the training data; too much loses its structure. Generalisation lives in between.
• In high dimensions, simple estimators fail, so modern models use neural networks to learn the data’s structure.

Autoregressive generation

You cannot write down a probability for every possible paragraph. There are too many. The autoregressive trick breaks the problem into pieces small enough to learn: predict one token, append it, predict the next.

What is autoregressive generation?

The chain rule of probability says the probability of a whole sequence equals the product of the probability of each element given everything before it:

Any distribution over sequences can be written this way. The model only ever has to answer one question: what comes next?

To generate, sample x₁, then sample x₂ given x₁, and so on. The idea is old. Claude Shannon’s 1948 paper that founded information theory generated text this way, using tables of how often each letter followed the previous one or two letters, and showed that the output grew more English-like as the context lengthened.

With no context the model knows only letter frequencies and produces gibberish with the right mix of e’s and spaces. At 2 to 3 characters it makes word-shaped strings, some real words and some invented. At 6 to 7 it writes real phrases, but watch the “copied” figure climb towards 100 percent: most long contexts appeared only once in the sonnets, so the model has exactly one option and replays the text. That is the memorisation problem from the previous section, showing up in a new form.

Temperature reshapes each distribution before sampling. Below 1, the likeliest characters get even likelier and output becomes safe and repetitive. Above 1, rare options get boosted and output becomes varied, then chaotic.

How do language models scale this up?

A large language model is this exact loop with two upgrades. The lookup table becomes a transformer that computes p(next token | all previous tokens) from learned patterns rather than exact counts, so it can handle contexts it has never seen. And the context stretches from 7 characters to hundreds of thousands of tokens. The same sampling controls, temperature, top-k and top-p, sit at the end.

Anything that can be turned into a sequence of tokens can be generated this way: code, audio compressed into tokens, even images cut into patches. OpenAI’s 2025 image generator in GPT-4o is described as autoregressive, unlike its earlier diffusion-based DALL·E.

Why does it matter?

One simple recipe, predict the next token, turned out to scale further than anyone expected, and it is the backbone of every chatbot. It has built-in weaknesses too. Generation is sequential, so long outputs are slow. And each token is committed before the next is chosen, so an early mistake can snowball, which is one root of confident errors.

Key takeaways

• Autoregressive models generate one token at a time, each conditioned on everything before it.
• Longer contexts give more fluent output but sparser evidence, and sparse evidence leads to copying.
• Temperature trades predictability for variety; LLMs are the same loop with a transformer instead of a count table.

GANs: learning by competition

In 2014 Ian Goodfellow proposed teaching a network to create images by pitting it against a second network whose only job is to catch fakes. For the next several years this contest produced the most realistic synthetic images anyone had seen.

What is a generative adversarial network?

A generative adversarial network (GAN) has two parts. The generator turns a random vector into an image. The discriminator looks at an image and outputs the probability that it is real. They train together, each improving against the other, like a forger and an art expert.

How does the training work?

The discriminator D is trained to label real images 1 and generated images 0. The generator G is trained to make D output 1 on its fakes. Together they play a minimax game:

D tries to maximise this; G tries to minimise it. At the ideal equilibrium, G’s samples match the data and D can only guess 50/50.

In practice the balance is fragile. If the discriminator wins too easily, the generator gets no useful signal. If the generator finds a few outputs that reliably fool the current discriminator, it can keep producing only those, a failure called mode collapse. Much of the GAN literature is tricks for keeping the game stable.

Why does it matter?

GANs showed that neural networks could generate photorealistic images, and by around 2019 they produced faces of people who do not exist that most viewers could not tell from photos. They generate in a single forward pass, which is fast. Diffusion models have since overtaken them for general image generation because they train more stably and cover the variety of the data better, but the adversarial idea lives on: the autoencoders inside latent diffusion systems are trained with a discriminator to keep their reconstructions sharp.

Key takeaways

• A GAN trains a generator to fool a discriminator that is simultaneously learning to spot fakes.
• Training is a delicate two-player game prone to instability and mode collapse.
• GANs generate in one fast pass and pioneered photorealism; diffusion has since taken the lead for most image generation.

VAEs: a compressed space

What if you could describe any face with 100 numbers, and slide smoothly from one face to another by changing them? Variational autoencoders learn exactly that kind of compact, continuous space, and that idea turned out to be the key that made modern image generators affordable.

What is a variational autoencoder?

An autoencoder squeezes an input through a narrow bottleneck and tries to rebuild it. The variational autoencoder (VAE) of Kingma and Welling adds a twist: the encoder outputs a small probability distribution (a mean and a spread) rather than a single point, and training pushes all those distributions to sit inside a standard normal cloud.

How does it learn?

The VAE’s loss has two parts that pull against each other:

• Reconstruction: the decoded image should match the original.
• Regularisation (a KL divergence): each encoded distribution should stay close to a standard normal, so the latent space has no holes.

Because the latent space is filled in, any random point decodes into something plausible, and walking in a straight line between two points morphs one image into another. The weakness is sharpness: VAEs hedge between possibilities, and averaged guesses look blurry.

Why does it matter?

On their own, VAEs are rarely the best generator. Their lasting contribution is the compressed space. Latent diffusion, the method behind Stable Diffusion, uses an autoencoder to compress each image into a latent about 48 times smaller, runs the expensive generation in that small space, then decodes once at the end. Most image and video generators today follow that pattern.

Key takeaways

• A VAE encodes data into a smooth, low-dimensional latent space and decodes points in that space back into data.
• Its loss balances faithful reconstruction against keeping the latent space well filled, which tends to blur samples.
• Compressing into a latent space first is what makes modern image and video generation affordable.

Diffusion models

Destroying a picture is easy: keep adding static until nothing is left. Diffusion models learn to run that process backwards. Start from pure noise, remove a little at a time, and an image emerges. Almost every leading image, video and audio generator works this way.

What is a diffusion model?

The idea came from Sohl-Dickstein and colleagues in 2015, borrowing from the physics of diffusion. It became practical when Ho, Jain and Abbeel’s 2020 denoising diffusion probabilistic models (DDPM) matched the image quality of GANs with a much simpler training recipe. There are two processes:

• Forward (fixed, no learning): over T steps, blend the data with a little Gaussian noise each time until it is pure noise.
• Reverse (learned): a network looks at a noisy input and the step number, and predicts the noise. Subtract a bit of it, add a little fresh randomness, repeat.

The forward process has a closed form: jump to any step t directly. ᾱ_t shrinks from 1 towards 0 as t grows.

Training is refreshingly simple: take a real example, pick a random step, add the matching amount of noise, and ask the network to predict that noise with a squared-error loss. No adversary, no balancing act.

The forward process turns the spiral into a featureless Gaussian blob by step 60. In reverse, the blob contracts and the spiral arms condense out of it. With the exact denoiser, look at the near-copies figure: the samples land on the training points themselves. That is not a bug in the demo. For a finite dataset, the perfect denoiser reproduces the training set, the same memorisation you met with the kernel density estimate.

So why do real diffusion models create new images? Because a neural network cannot represent that perfect function and ends up with a smoother approximation, like the Smoothed option here, which fills in along the spiral. Kadkhodaie and colleagues showed that with small training sets models memorise, while with enough data two models trained on completely separate images generate nearly the same outputs: they have learned the distribution, not the examples.

From a spiral to Stable Diffusion

Four further ideas turn this into the text-to-image systems you know:

Why does it matter?

Diffusion’s stable training and good coverage of the data made it the default for images, then video and much of audio. Its main cost is speed: tens of denoising steps, each a full pass through a large network. Much current research distils those steps down to a handful, and newer models trained with flow matching take straighter paths from noise to data so that fewer steps are needed.

Key takeaways

• Diffusion models learn to reverse a fixed process that gradually turns data into Gaussian noise.
• The ideal denoiser for a small dataset reproduces the training set; generalisation comes from approximating it with enough data.
• Latent spaces, text conditioning, guidance and transformers turn the basic recipe into modern image and video generators.

The landscape in 2026

In early 2024 a minute of convincing AI video was a research preview. By 2025 leading video models produced clips with synchronised dialogue and sound. The techniques in this lesson are now products used by hundreds of millions of people, and the list of leading models changes every few months.

What exists today?

A snapshot of the main modalities as of September 2026. Treat the example names as a moving target; the approaches change more slowly.

Text and code

Autoregressive transformers predicting the next token

Claude, GPT, Gemini, Llama, Qwen, DeepSeek

Images

Latent diffusion and flow-matching transformers; some natively autoregressive models

Midjourney, Flux, Stable Diffusion, Gemini and OpenAI image models

Video

Diffusion transformers over space-time patches, increasingly with audio generated jointly

Veo, Kling, Seedance, Runway

Speech and music

Autoregressive models over audio tokens, and diffusion or flow models over spectrograms

Text-to-speech and voice cloning, Suno and Udio for music

How did we get here so fast?

The key developments stack on each other. Latent diffusion (2022) made high-resolution image generation cheap enough to release openly. Diffusion transformers made it scale like language models. Video models applied the same recipe to patches that extend through time. Google’s Veo 3, announced in May 2025, generated audio together with video, and joint audio quickly became expected. The market moves just as fast: OpenAI previewed Sora in February 2024, launched Sora 2 in September 2025, and closed the Sora app in April 2026, while video models from Chinese labs such as Kuaishou’s Kling and ByteDance’s Seedance compete at the top.

The boundaries between modalities are also dissolving. Chat models now read images and audio and produce images and speech themselves, so a single system can hold a conversation that moves between text, pictures and voice.

Why does it matter?

For anyone building with these tools, the practical lesson is to depend on capabilities, not specific models. Architectures converge (transformers everywhere, diffusion or flow for continuous media, next-token prediction for discrete), while product names and rankings churn. Understanding the underlying methods lets you judge a new release on its merits.

Key takeaways

• Text is dominated by autoregressive transformers; images, video and much audio by diffusion or flow transformers.
• Video generation went from research previews in 2024 to clips with synchronised audio in 2025.
• Model rankings change every few months; the underlying methods are far more stable.

Risks and costs

Generative models are trained to be plausible, not truthful, on data that belonged to other people, at a real cost in energy. Each of those facts has consequences that are being worked out in courts, newsrooms and power grids right now.

Why do generative models make things up?

A generative model samples what is likely under its learned distribution. A fluent, confident, false sentence can be very likely. Language models call this hallucination: an invented citation, a wrong date stated with certainty. Image models do the same with the wrong number of fingers or unreadable signage. Retrieval, tool use and training models to say “I don’t know” reduce the problem but do not eliminate it.

Who owns what the model learned from?

Models learn from vast scraped datasets that include copyrighted work, and they can sometimes reproduce it. Carlini and colleagues extracted over a thousand training images from diffusion models, including photos of real people and trademarked logos. That is the memorisation you saw in the demos, at scale.

Courts are drawing the first lines. In the UK, the High Court largely rejected Getty Images’ copyright claims against Stability AI in November 2025, finding only limited trade mark infringement. In the US, a 2025 ruling in Bartz v. Anthropic found that training on lawfully bought books could be fair use but building a library from pirated copies was not, and the case ended in a $1.5 billion settlement, about $3,000 per book. Many other lawsuits remain open, and the law differs by country.

How do we tell real from generated?

Realistic synthetic voices, photos and video make impersonation and fraud cheap. There are three main responses, none sufficient alone:

Detectors that guess whether an image or essay is AI-generated from its content alone are unreliable, and falsely accusing someone of using AI causes real harm. Verifying the source is still the most robust defence.

What does it cost?

Generation is expensive compared with classification. Luccioni, Jernite and Strubell measured energy across common AI tasks and found general-purpose generative models to be orders of magnitude costlier per query than task-specific ones, with image generation the most energy-hungry task they tested. Efficiency is improving quickly though: Google reported that its median Gemini text prompt used 0.24 watt-hours in May 2025, 33 times less than a year before, roughly what a television uses in nine seconds. The total still grows as usage explodes, which is why data-centre power has become an energy-policy issue.

Key takeaways

• Generative models optimise for plausibility, so fluent falsehoods and visual glitches are built-in risks.
• Memorisation makes copyright and privacy real legal questions, and courts in the UK and US gave their first answers in 2025.
• Watermarks, provenance and regulation each help against deepfakes; none is sufficient alone. Generation also costs real energy.

Check your understanding

Seven scenarios where knowing how generative models work changes the decision.

A bank wants to flag fraudulent transactions and has millions of labelled examples. A colleague proposes a generative model of transactions instead of a classifier “because generative AI is more advanced”. What is the best response?

References

Papers, rulings and reports cited in this lesson. Facts about products and cases are current as of September 2026.

To go deeper on text generation, continue with Large Language Models; for images, try the Diffusion Lab.

Sources cited

1. [1]

   Veo: Google DeepMind’s video generation model(opens in a new tab)

   Google DeepMind, 2025

   Veo 3, announced in May 2025, generates video with synchronised dialogue, sound effects and ambient audio.

2. [2]

   A Mathematical Theory of Communication(opens in a new tab)

   Shannon, 1948

   Bell System Technical Journal. Generated increasingly English-like text from letter and word n-gram statistics.

3. [3]

   Addendum to GPT-4o System Card: Native image generation(opens in a new tab)

   OpenAI, 2025

   Describes 4o image generation as an autoregressive model, unlike the diffusion-based DALL·E.

4. [4]

   Generative Adversarial Networks(opens in a new tab)

   Goodfellow et al., 2014

   NeurIPS 2014. A generator and a discriminator trained against each other.

5. [5]

   Auto-Encoding Variational Bayes(opens in a new tab)

   Kingma & Welling, 2013

   The variational autoencoder and the reparameterisation trick.

6. [6]

   Deep Unsupervised Learning using Nonequilibrium Thermodynamics(opens in a new tab)

   Sohl-Dickstein, Weiss, Maheswaranathan & Ganguli, 2015

   ICML 2015. First diffusion model: destroy structure with noise, learn to reverse it.

7. [7]

   Denoising Diffusion Probabilistic Models(opens in a new tab)

   Ho, Jain & Abbeel, 2020

   NeurIPS 2020. Predict the added noise with a simple squared-error loss; made diffusion competitive with GANs.

8. [8]

   Generalization in diffusion models arises from geometry-adaptive harmonic representations(opens in a new tab)

   Kadkhodaie, Guth, Simoncelli & Mallat, 2024

   ICLR 2024. Small training sets are memorised; with enough data, models trained on disjoint sets generate nearly the same images.

9. [9]

   High-Resolution Image Synthesis with Latent Diffusion Models(opens in a new tab)

   Rombach, Blattmann, Lorenz, Esser & Ommer, 2022

   CVPR 2022. Diffusion in an autoencoder’s compressed latent space; the basis of Stable Diffusion.

10. [10]

    Classifier-Free Diffusion Guidance(opens in a new tab)

    Ho & Salimans, 2022

    Trade diversity for prompt adherence by extrapolating from an unconditional prediction.

11. [11]

    Scalable Diffusion Models with Transformers(opens in a new tab)

    Peebles & Xie, 2023

    ICCV 2023. Diffusion transformers (DiT) replace the U-Net and scale predictably with compute.

12. [12]

    Sora (text-to-video model)(opens in a new tab)

    Wikipedia contributors, 2026

    History of OpenAI’s Sora: preview in February 2024, Sora 2 in September 2025, app closed in April 2026.

13. [13]

    Extracting Training Data from Diffusion Models(opens in a new tab)

    Carlini et al., 2023

    USENIX Security 2023. Extracted over a thousand training images, including photos of people and trademarked logos.

14. [14]

    Getty Images v Stability AI [2025] EWHC 2863 (Ch)(opens in a new tab)

    High Court of England and Wales, 2025

    The first major UK ruling on generative AI and copyright. The secondary copyright claim failed; limited trade mark findings.

15. [15]

    What authors need to know about the Anthropic settlement(opens in a new tab)

    Authors Guild, 2025

    Bartz v. Anthropic: a $1.5 billion class settlement over books downloaded from pirate libraries, about $3,000 per work.

16. [16]

    Scalable watermarking for identifying large language model outputs(opens in a new tab)

    Dathathri et al., 2024

    Nature 634. SynthID-Text, a watermark deployed in Google’s Gemini that survives normal use but can be weakened by heavy editing.

17. [17]

    Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 50(opens in a new tab)

    European Union, 2024

    Transparency duties: synthetic content must be machine-detectable, and deepfakes must be disclosed.

18. [18]

    Power Hungry Processing: Watts Driving the Cost of AI Deployment?(opens in a new tab)

    Luccioni, Jernite & Strubell, 2024

    FAccT 2024. Measured energy per 1,000 inferences across tasks; image generation was the most energy-intensive.

19. [19]

    Measuring the environmental impact of delivering AI at Google Scale(opens in a new tab)

    Elsworth et al., 2025

    Google’s median Gemini Apps text prompt used 0.24 Wh in May 2025, 33 times less energy than a year earlier.