Lesson 6 of 7 Joint embedding world models, from the ground up 2 min
V-JEPA, learning from video
V-JEPA hides large tubes of space and time in a video and predicts their summaries. The result shows signs of intuitive physics.
Sonu, do you remember playing peekaboo with your baby cousin? You hid behind your hands, then popped out. She laughed every time.
Now try a trick on her, in your head. Roll a toy car across the floor behind a big book. She watches it go behind.
What does she expect? She expects the car to roll out the other side. If it does, she stays calm.
Now imagine the car never comes out. Someone secretly lifted it away.
Her eyes go wide. She stares at the book. She is surprised.
That stare tells us something. Even a baby has a small model of how things move.
Things keep going. Things do not vanish. When the world breaks those rules, she notices.
A computer model called V-JEPA learns from videos in a similar way. During practice, big parts of each video are hidden. They stay hidden for the whole clip, like a car going behind a book.
The model must guess what is happening in those hidden parts. It guesses in short notes, the way you would. "The car is still rolling. It is behind the book now."
After watching a huge pile of ordinary videos, something interesting happened. Scientists showed it short movies where things broke the rules. A ball that vanished. A block that changed shape.
The model's guesses went badly wrong at those moments. Its notes did not match what it saw. In a way, it was surprised, just like your cousin.
Nobody taught it the rules. It picked them up from watching.
It is not perfect. It still gets confused when one thing bumps into another. Babies learn that later too.
Try this at home. Roll a ball behind a cushion while a friend watches. Ask them to say where it will come out. Then secretly stop it.
Watch their face. What did they expect to see?
V-JEPA (Bardes et al., 2024) asks one question. Can feature prediction alone train a strong video encoder? It uses no pretrained image encoder, no text, no negatives and no pixel reconstruction.
The input is 16 frames sampled with a stride of 4. Each token is a tubelet of 2 frames by 16 by 16 pixels. A 224-pixel clip gives 1,568 tokens.
Masking moves into time. A spatial block is repeated across every frame, so the hidden region is a tube. Two mask types are mixed.
- Short-range masks join 8 blocks, each covering 15 percent of a frame.
- Long-range masks join 2 blocks, each covering 70 percent of a frame.
On average about 90 percent of the clip is hidden. Tubes matter because video is redundant. A block hidden in one frame can often be copied from its neighbour. A tube removes that shortcut.
The encoder sees the visible tokens. A narrow predictor of 12 blocks and width 384 fills in mask tokens. The target encoder is an exponential moving average, with momentum rising from 0.998 to 1.0. The loss is an L1 distance, which the authors found more stable.
loss = || P(E(x), Δy) − sg(Ē(y)) ||₁
Pretraining used VideoMix2M, about 2 million public videos.
Evaluation is mostly frozen. An attentive probe uses one learned query and cross-attention over the frozen features. That probe beat average pooling by 17.0 points on Kinetics-400 in the paper's own test.
Reported results for the largest model, a ViT-H/16:
- 81.9 percent on Kinetics-400
- 72.2 percent on Something-Something-v2, which rewards motion understanding
- 77.9 percent on ImageNet-1K, from a video-only model
Against pixel-prediction video models at similar size, V-JEPA used fewer samples. It led on Kinetics, Something-Something and AVA. It trailed OmniMAE on ImageNet, 74.8 to 75.1. With 5 percent of Kinetics labels, it scored 67.0 against VideoMAE's 62.3.
What does the representation capture? Garrido et al. (2025) tested intuitive physics with a violation-of-expectation setup.
Surprise is the L1 error between predicted and observed representations. Each pair holds one possible and one impossible video. The model must flag the impossible one.
- IntPhys, 98 percent pairwise accuracy
- GRASP, 66 percent
- InfLevel-lab, 62 percent
Object permanence, continuity and shape constancy were clearly above untrained networks. Solidity and collision were not.
A pixel-prediction model and two multimodal language models stayed close to chance. One ablation used about one week of unique video. It still kept over 70 percent on the properties in that test.
The field version suggests itself. A fixed camera watches a substation yard or a ridge line in fire season. Surprise in representation space could flag events nobody labelled. The paper does not test that, so it stays a hypothesis.
What I would measure:
- frozen attentive-probe accuracy on a motion task and an appearance task
- surprise curves on a few staged impossible events
- label efficiency at 5 and 50 percent of labels
Video is the largest unlabeled dataset most utilities own. Wildfire cameras, substation cameras and drone patrols record every day. Almost none of it is labelled.
V-JEPA shows a way to learn from that pile without labels. The encoder trains on raw clips. Small probes then teach it specific tasks with few examples. In the paper, it held up best when labels were scarce.
The physics result changes what we might ask of video models. A model trained only to predict hidden parts of video became surprised by impossible events. A ball vanishing, a shape changing. That surprise is a number we can track.
For us, that suggests an anomaly signal that needs no rulebook. A camera that learns what normal looks like could flag the unusual. Smoke on a ridge or a door left open are candidates.
I want to be careful here. The papers test toy physics in lab clips. None of them looked at a utility site.
The limits are documented. The model was not reliably surprised by collisions or by objects passing through solids. Those are exactly the events a safety team cares about. Any pilot would need its own test set with staged events.
The cost side is moderate. Pretraining at paper scale is a large job. Reusing the released weights and training only a probe is small.
The decision is whether to run a frozen-encoder pilot on one camera network. It would cost a few weeks of engineering and a modest GPU budget. Success means fewer false alarms than our current motion rules.
Which camera feed has the most footage and the fewest labels today?
Quick check
Key terms
Further reading
- Revisiting Feature Prediction for Learning Visual Representations from Video Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, Nicolas Ballas, 2024 The V-JEPA paper. Tube masking, attentive probes and the pixel versus feature comparison.
- Intuitive physics understanding emerges from self-supervised pretraining on natural videos Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, Yann LeCun, 2025 The violation-of-expectation tests, with per-property results and the limits.
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, Nicolas Ballas, 2023 The image method V-JEPA extends into time.
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning Mido Assran, Adrien Bardes, David Fan, Quentin Garrido and others, 2025 The scaled successor, read next in Lesson 7.