Courses · 7 lessons
Joint embedding world models, from the ground up
I wrote this course to show how a model can predict the world without drawing it. Each lesson starts with a plain picture. It ends with the mechanism, the evidence and a decision from the grid.
Start with lesson 1 Each lesson is told three ways. Pick your reader at the top of the page.
-
Why predicting pixels fails
A model that must draw the future wastes its effort on texture, so joint embedding models predict a summary of the future instead.
-
Collapse and its cures
A model that only has to make two summaries agree can cheat by giving everything the same summary. Three families of fixes stop it.
-
Information maximisation
A good embedding keeps every dimension busy and saying something new. VICReg and SIGReg are two ways to enforce that.
-
I-JEPA, predicting the hidden parts of an image
I-JEPA sees one large block of an image and predicts the summaries of several hidden blocks, with no hand-made augmentations.
-
LeJEPA, one regulariser in place of many tricks
LeJEPA drops the teacher and the schedules. One regulariser keeps collapse away by shaping embeddings into a round Gaussian cloud.
-
V-JEPA, learning from video
V-JEPA hides large tubes of space and time in a video and predicts their summaries. The result shows signs of intuitive physics.
-
V-JEPA 2 and planning with a world model
V-JEPA 2 adds actions to a video world model and plans robot moves by searching for actions whose predicted outcome matches a goal image.
