Lesson 1 of 7 Joint embedding world models, from the ground up 2 min
Why predicting pixels fails
A model that must draw the future wastes its effort on texture, so joint embedding models predict a summary of the future instead.
Sonu, picture yourself at the window before a storm. Dark clouds roll in over the park. What happens next?
You would say rain is coming. The slide will get wet. The dog next door will run inside.
You would never try to paint every raindrop. Nobody knows where each drop will land. The best weather station in the world does not know either.
Your guess is still right. Rain, wet slide, dog inside. You predicted the parts that matter.
Some computer programs learn by painting the future. They watch a video. Then they draw the next picture, one tiny dot at a time.
Think about the raindrops again. The program cannot know where each drop lands. So it gets marked wrong over and over.
It finds a sneaky way to lose fewer marks. It paints a grey smudge. A smudge is a little bit right everywhere. It is useful nowhere.
Scientists saw this happen. Their fix was simple to say. Let the computer guess in short notes of its own. "Rain soon, slide wet, dog inside."
Notes like that can be right even when each drop is a surprise. The computer spends its effort on the dog and the clouds. It stops wasting effort on things nobody could know.
I like this idea because it matches how people think. When I picture tomorrow, I picture what will happen. I do not picture every leaf.
That is the first idea in this course. Guess the gist and skip the noise.
Try this at home. Roll a ball across the floor toward a chair. Before it stops, say in five words what will happen. Then mark the exact spot where you think it stops.
Which guess came true more often? Which one would help you more?
Every world model answers one design question first. In what space does the prediction live?
The generative answer is the raw signal. Show the model a context x and train it to output the target y. For images and video, y is pixels. For a feeder, y might be every meter read in the next hour.
The trouble starts when the future is uncertain. Many targets are consistent with the same context. A squared-error loss is minimised by the average of those targets. In pixels, the average of many plausible futures is a blur.
Capacity goes the same way. Most of the bits in a frame are texture and sensor noise. A pixel loss pays for every one of them. Few of them change a decision.
LeCun's 2022 position paper proposed the alternative, the joint embedding predictive architecture. Two encoders map x and y into representations. A predictor maps the representation of x to the representation of y. A latent variable can carry what x cannot settle.
The objective, in one line:
generative: minimise || D(E(x)) − y ||² joint embedding: minimise || P(E(x)) − E′(y) ||²
The second loss lives in a space the model chooses. The encoder may drop detail that no context could predict. That freedom is the point.
It also opens a trap. The easiest predictable representation is a constant. Lesson 2 deals with that.
The papers agree in one setting. The encoder is frozen and only a probe is trained.
- MAE (He et al., 2021) masked 75 percent of patches and reconstructed pixels. A ViT-Huge reached 87.8 percent on ImageNet-1K after fine-tuning.
- I-JEPA (Assran et al., 2023) compared frozen linear probes. MAE ViT-H/14 reached 77.2 percent after 1600 epochs. I-JEPA ViT-H/14 reached 79.3 percent after 300 epochs.
- The same paper swapped its targets. On 1 percent of ImageNet labels, representation targets gave 66.9 percent. Pixel targets gave 40.7 percent. The runs used 500 and 800 epochs, so the match is loose.
- V-JEPA (Bardes et al., 2024) ran the swap on video. Feature targets beat pixel targets on Kinetics-400 by 73.7 to 68.6. The gap on Something-Something-v2 was 0.2 points.
Read those numbers carefully. Pixel models fine-tune well. The gap shows up when features must work frozen, with little labelled data. That is the regime most world models live in.
The meter version makes it concrete.
A household read every 15 minutes holds a kettle, a dryer and a dog door. None of that is predictable from yesterday.
The feeder peak at 6 pm is predictable. It decides whether a transformer runs hot.
What I would measure on any new world model:
- a frozen linear probe on the signal the decision needs
- the same probe at equal training compute for a pixel baseline
- multi-step forecasts, checked for the blur that signals averaging
If the downstream task is generation, reconstruct. If the task is a decision, predict in representation space.
Most of the money in a world model goes into compute. The first question is what that compute buys.
A model that predicts raw signals pays to model noise. In video that is texture and glare. In our world it is every kettle and dryer behind a meter. The bill grows with detail that no decision uses.
A joint embedding model predicts a compact summary instead. The published image results point to lower cost. I-JEPA reported training a ViT-Huge on 16 A100 GPUs in under 72 hours. The paper called that over ten times as efficient as its pixel baseline.
There is a cost to the summary. Nobody can look at it. A pixel forecast can be shown to an operator. A representation is a list of numbers.
That changes how we audit. We need probes that read the summary back into terms people trust. Feeder peak, outage risk, a bill that looks wrong.
The second risk is quiet failure. A summary model can learn to say the same thing about everything. Its loss then looks perfect.
Lesson 2 covers this. The controls are cheap.
The decision is where to place the next pilot. A generative model fits when the output is a picture or a document. A representation model fits when the output is a decision.
For grid operations and billing, the output is almost always a decision. I would fund the representation route first. I would also require frozen-probe evidence on our own data before any scale-up.
Which of our forecasting projects is paying to predict detail we never act on?
Quick check
Key terms
Further reading
- A Path Towards Autonomous Machine Intelligence (version 0.9.2) Yann LeCun, 2022 The position paper that proposed JEPA and H-JEPA as non-generative predictive world models.
- Masked Autoencoders Are Scalable Vision Learners Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick, 2021 The strongest case for pixel reconstruction. It is also the baseline the JEPA papers compare against.
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, Nicolas Ballas, 2023 Holds the pixel versus representation target ablation on 1 percent of ImageNet labels.
- Revisiting Feature Prediction for Learning Visual Representations from Video Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, Nicolas Ballas, 2024 Runs the same pixel versus feature comparison on video with frozen probes.