Lesson 4 of 7 Joint embedding world models, from the ground up 2 min
I-JEPA, predicting the hidden parts of an image
I-JEPA sees one large block of an image and predicts the summaries of several hidden blocks, with no hand-made augmentations.
Sonu, remember the jigsaw puzzle of the lighthouse? The one with the pieces missing?
When a piece was missing, you did not panic.
You looked at the pieces around the hole. Then you said something like this. "That piece has the top of the red stripe. And a bit of sky."
You could not draw the exact piece. You did not know every speck of paint on it. But you knew what it was about. That was enough to find it in the box.
A computer model called I-JEPA learns in the same way.
It looks at a photo with a few patches covered up. It is not allowed to peek. Then it guesses what each hidden patch is about, in its own short notes.
A second copy of the model looks at the whole photo, with nothing covered. It writes its own notes for those same patches. The first copy gets a better score the closer its guesses come.
The second copy changes very slowly. It is like a calm older cousin who checks your homework. That steadiness stops the lazy trick from Lesson 2.
The covered patches are big on purpose. A tiny patch would only hold a few dots of colour. A big patch holds a real thing, like a stripe or a window. Guessing real things is how the model learns about the world.
After a lot of practice, the notes get good. Scientists can turn them back into rough pictures. The pictures keep the shape of the thing and its position. They forget the tiny details.
I like this because it is how I fill gaps too. I guess the idea and let the dots go.
Try this at home. Take a picture book. Cover two parts of one page with sticky notes. Ask someone to say what is under each note.
Did they guess the idea right? Did they guess the colours?
I-JEPA (Assran et al., 2023) is the first JEPA trained at scale on images. It needs no hand-crafted augmentations. The training signal comes from masking alone.
The setup has three networks.
- A context encoder sees only the visible context patches. It is a Vision Transformer.
- A target encoder sees the whole image. Its weights are an exponential moving average of the context encoder. The momentum starts at 0.996 and rises linearly to 1.0.
- A predictor maps context features to target features. It is a narrow Vision Transformer of width 384.
The predictor gets the context features plus mask tokens. Each mask token is one shared learned vector with a positional embedding. The position tells the predictor which patch to predict.
Masking is the heart of the method.
Each image gets four target blocks. Each covers 15 to 20 percent of the image. Aspect ratios run from 0.75 to 1.5.
The context is one block covering 85 to 100 percent of the image. Any context patch inside a target is removed.
Targets are cut from the output of the target encoder, never from its input. So every target feature has seen the full image.
The loss is a plain average squared distance over target patches.
loss = (1/M) Σᵢ Σⱼ∈Bᵢ || ŝⱼ − sⱼ ||²
The paper's abstract states the design rule. Targets must be large enough to be semantic. The context must be spread out enough to be informative. Small, scattered targets would teach texture again.
Collapse is held off by the second cure from Lesson 2. The target receives no gradient and trails the context encoder.
What does it learn? The authors froze the encoder and predictor. They trained a separate decoder to render predictor outputs as images.
The renders kept object parts in the right pose. The back of a bird and the top of a car are examples. They dropped precise low-level detail and background. The paper reads this as the predictor capturing positional uncertainty.
How is it evaluated? Mostly frozen, with a linear probe.
- ImageNet-1K linear probe. ViT-H/14 reached 79.3 percent after 300 epochs. ViT-H/16 at 448 pixels reached 81.1.
- ImageNet with 1 percent of labels. ViT-H/14 reached 73.3 percent and the 448 model 77.3.
- Clevr. Counting scored 86.7, below MAE at 90.5. Depth scored 72.4, equal to MAE.
- Compute. ViT-H/14 trained on 16 A100 GPUs in under 72 hours.
The paper reports about 7 percent slower iterations than MAE. It also reports roughly five times fewer iterations to converge.
The counting result matters. Tasks that need fine local detail favour pixel targets a little. Tasks that need meaning favour I-JEPA.
A field example helps. Think of drone photos of distribution poles.
Hide the crossarm block. A good predictor says "crossarm, three insulators, one leaning". It does not need the grain of the wood.
What I would measure:
- a frozen probe on the label that matters, such as pole defect class
- the same probe on a fine-detail task, to see what was dropped
- the target block scale, swept once, since the paper ties quality to it
I-JEPA is the image version of the idea. Its cost profile is the headline. The paper trained its largest model on 16 GPUs in under three days. The introduction puts it under 1,200 GPU hours.
For a utility, that changes what is in reach. Training an image encoder on our own unlabeled photos now fits a project budget. Drone inspection imagery, substation photos and meter-face images all qualify.
The second benefit is fewer design choices. Earlier methods needed teams to hand-pick image distortions, such as crops and colour shifts. Each choice quietly decided what the model ignored. I-JEPA learns from hiding blocks alone.
The risk is in fine detail. The paper's own counting benchmark scored below the pixel-based method it compared against. A cracked insulator is a small, local feature. A model that favours meaning over detail may underweight it.
So the evaluation plan matters more than the training run. We would test on two kinds of labels. One is semantic, such as asset type. One is fine-grained, such as hairline damage.
There is also a governance point. The summaries are not human-readable. The authors used a separate decoder to show what the model kept. A similar render would help our reviewers trust the model.
The decision is whether to fund a small pilot on our own imagery. It would train on unlabeled photos and test with a few hundred labels.
If it matches our current supervised models on asset type, the labelling savings are real. If it misses fine defects, we know where a pixel model still belongs.
Which inspection label would prove this pilot worth scaling?
Quick check
Key terms
Further reading
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, Nicolas Ballas, 2023 The I-JEPA paper. Masking design, predictor, evaluation and the decoder renders.
- Masked Autoencoders Are Scalable Vision Learners Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick, 2021 The pixel baseline I-JEPA compares against on probes, Clevr and compute.
- data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, Michael Auli, 2022 The closest cousin. It predicts latent targets of the full input from a masked view.
- Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond and others, 2020 The source of the slow-moving target encoder I-JEPA uses.