World models: the physics-learning machines behind video AI
Video generators were built to make pretty pictures that move. In doing so, they started learning something much stranger: an implicit physics of how the world works. Here's what world models are, how they learn, and why they matter far beyond video.
Watch a modern video generator long enough and something odd shows up. Objects occluded behind a wall reappear instead of vanishing. A ball thrown across a room follows a plausible arc. A character walking around a table stays in the same shape when the camera circles them. Nobody programmed in gravity, occlusion, or 3D consistency. Nobody wrote down Newton's laws. The model just watched enough video, and physics — or something shaped like physics — fell out.
That something is called a world model: a system that learns how the world works well enough to predict what happens next. Here's what's actually going on.
The original idea: learn a dream, then act inside it#
The term "world model" in its modern AI sense comes from a 2018 paper by David Ha and Jürgen Schmidhuber, simply titled World Models. The setup was humble: video-game environments like CarRacing and VizDoom. Their insight was to split the problem in two.
First, train a neural network to compress observations into a compact representation and predict how that representation evolves over time — a learned dynamics model, or, loosely, a dream of the environment. Then, instead of training a game-playing agent on the real game, train it inside the dream: let it imagine thousands of futures, plan and evolve its strategy there, and transfer the resulting policy back to the real environment.
It worked. An agent could be trained almost entirely in its own hallucinated simulation and still perform in reality. The world model didn't need to be a perfect physics engine — it needed to be predictive enough that strategies learned inside it transferred outside it.
That distinction still frames everything happening today: world models are not simulators built from first principles. They are learned approximations of how things behave, judged by whether they predict well enough to act on.
The video era: physics as a phenomenon of scale#
In February 2024, OpenAI published its Sora technical report with a striking title: "Video generation models as world simulators." Sora is a diffusion transformer that denoises sequences of spacetime patches — compressed latent chunks of video, analogous to a language model's tokens.
The evidence OpenAI pointed to is worth spelling out, because it's specific:
- 3D consistency. As the camera shifts and rotates, people and scene elements move coherently through three-dimensional space.
- Long-range coherence and object permanence. The model can persist people, animals, and objects even when they are occluded or leave the frame.
- Interacting with the world. A painter can leave new strokes on a canvas that persist over time; a man can eat a burger and leave bite marks.
- Simulating digital worlds. Sora could render and even partially control a Minecraft-like environment zero-shot.
Crucially, OpenAI stressed that these capabilities emerged "without any explicit inductive biases for 3D, objects, etc. — they are purely phenomena of scale." No one taught Sora about space; space fell out of predicting patches of video really, really well.
But the report was equally honest about the flip side. Sora, as a simulator, is flaky: it does not accurately model glass shattering, eating food does not always produce the right object-state changes, and long videos drift into incoherence with objects appearing spontaneously. Implicit physics is still approximate physics — good enough to fool the eye for a minute, not good enough to trust.
Two schools: predict the pixels, or skip them#
Not everyone thinks the answer is "scale up the video generator." Yann LeCun — who has argued since 2022 that Joint Embedding Predictive Architectures (JEPAs) are the path to what he calls advanced machine intelligence — thinks generative video models are solving the wrong problem. Predicting every pixel is wasteful: the exact flutter of every leaf on a tree is unpredictable noise, and trying to reconstruct it burns capacity that should go to understanding.
His alternative, built by Meta's FAIR team as V-JEPA, is a non-generative world model. Mask out large regions of a video — in space and time, so the task can't be solved by copying the frame next door — and train a predictor to fill in the blanks not as pixels, but as abstract descriptions in a learned representation space. Because the model is free to discard unpredictable detail, Meta reported training and sample efficiency improvements of 1.5x to 6x over generative baselines.
The follow-up, V-JEPA 2 (2025), scaled this idea up into what Meta explicitly calls a self-supervised video world model, pushing it toward action-conditioned prediction — the missing link for using the model not just to understand video, but to plan inside it.
The debate, in a nutshell:
| Generative camp (Sora, Genie, Cosmos) | Predictive camp (V-JEPA, JEPA) | |
|---|---|---|
| Predicts | Future pixels/frames | Future abstract representations |
| Strength | Stunning detail, direct simulation | Efficiency, discards noise, task-flexible |
| Weakness | Expensive, hallucinates, brittle physics | No direct generation, shorter horizons |
| Bet | Physics emerges from pixel scale | Understanding emerges from abstraction |
LeCun's framing is worth quoting from the V-JEPA announcement: the JEPA predictor is "an early physical world model" — you don't have to see everything happening in the frame for it to tell you, conceptually, what's going on. The ambition is to turn that predictor into a planner.
The interactive turn: worlds you can step into#
Prediction alone, though, isn't enough for an agent. An agent needs to act — to take an action and see what the world does in response. That's the gap DeepMind's Genie 2 closed in December 2024. Described by DeepMind as a "foundation world model," Genie 2 generates action-controllable 3D environments from a single image: a human or an AI agent presses keys, and the model simulates the next frame.
Under the hood, it's an autoregressive latent diffusion model trained on a large video dataset — but the emergent behaviors are what matter: gravity, water and smoke effects, reflections and lighting, object interactions (bursting balloons, opening doors), characters that animate correctly, even other agents whose behavior the model can predict. It remembers parts of the world that leave the frame and renders them correctly when they return. And from the same starting frame, it can roll out different futures under different actions — generating counterfactuals, which is exactly what a training agent needs to ask "what if I did that instead?"
DeepMind showed SIMA, their instructable 3D game agent, following instructions in Genie-2-generated worlds it had never seen — a bid to solve the data bottleneck for training general embodied agents safely.
NVIDIA's Cosmos platform (2025) pushes the same idea toward industry: open world-foundation models aimed at physical AI — robots and autonomous vehicles that need to simulate and predict real-world dynamics before acting in them.
Why it matters beyond video#
The real story, though, is about simulation — how machines learn to act in the world without breaking things.
- Robotics. World models give robots physical intuition before they ever touch a lab bench, so they can plan safely.
- Autonomous driving. Driving world models simulate the long tail of rare events under different driving actions — scenarios too uncommon to collect enough real data for.
- Agent training. Unlimited generated worlds attack the oldest problem in reinforcement learning: environments that are too narrow, too repetitive, or too dangerous.
- Scientific modeling. The same techniques are being applied to weather and climate (Microsoft's Aurora), where data-learned "implicit physics" complements explicit numerical models.
The common thread: anywhere you need to answer "what happens if I do this?" without paying the cost of actually doing it, a learned world model is a candidate answer.
The honest caveats#
World models in 2026 are powerful demos with real limits. The implicit kind — Sora, Genie, Cosmos — still fail at precise physical reasoning: fluid dynamics, fracture, exact cause-and-effect chains. Their physics is statistical, learned from the correlations in training video, and video is a biased teacher: it over-represents common situations and under-represents the rare, dangerous, or merely unfilmed. A model that has never seen a particular failure mode can't predict it.
The predictive camp's models, meanwhile, still work on short horizons — V-JEPA shines on clips of a few seconds — and turning a good representation into a reliable long-horizon planner is unsolved. And a model that has only ever watched the world is like a physicist who has read every textbook but never touched an object: whether passive observation can ground reliable physical understanding is the live bet of the next few years.
The takeaway#
A world model is a machine that has learned enough about how the world behaves to imagine what happens next. Video generators arrived there by accident — scale turned next-frame prediction into something that looks like physics. The JEPA line of research arrived there by design, betting that abstraction beats pixels. And systems like Genie 2 are closing the loop by making those dreams interactive: worlds you can poke, agents you can train inside.
The trajectory is clear: from watching the world, to modeling it, to acting inside the model. The physics-learning machines are real; making them trustworthy is the next act.