Lesson 5 of 7 Joint embedding world models, from the ground up 2 min
LeJEPA, one regulariser in place of many tricks
LeJEPA drops the teacher and the schedules. One regulariser keeps collapse away by shaping embeddings into a round Gaussian cloud.
Sonu, picture your whole school out on the playground for a fire drill. The head teacher wants everyone spread out well.
She does not want one big clump by the gate. She does not want a long thin line along the fence either. She wants a round crowd. Lots of kids near the middle, fewer toward the edges, the same in every direction.
How can she check from where she stands? She cannot count every child.
So she tries something clever. She walks one straight line across the playground in a random direction. She counts the kids standing close to her line. Some are near the middle and some near the ends.
Then she picks another random direction and walks again. And again.
If every walk gives the same kind of count, the crowd is round. If one walk finds a big bunch at one end, there is a clump. She calls out and the kids shuffle until it evens out.
A computer model called LeJEPA uses this exact trick. It keeps short notes about pictures, as you know from the other lessons. Each note is a point in a big space.
The model wants those points spread out like the round crowd. No clumps and no thin lines. It checks by looking along random directions, just like the head teacher's walks.
Older models needed many helper tricks to stay spread out. They needed a slow older cousin and special rules about who copies whom. LeJEPA needs just this one check. That makes it simpler to build.
I like it because a hard problem got one plain rule.
Try this at home. Drop a handful of buttons on the floor. Lay a broom handle across them in one direction. Count the buttons close to the handle.
Turn the handle and count again. Do you get about the same number each time?
LeJEPA (Balestriero and LeCun, 2025) asks what the embedding distribution should be. Then it enforces that distribution directly. Everything else in the recipe is stripped out.
The target comes from theory. The paper argues the isotropic Gaussian minimises worst-case downstream risk.
For linear probes, anisotropy amplifies bias and variance across tasks. For k-nearest-neighbour and kernel probes, the isotropic Gaussian minimises integrated squared bias. These are statements about worst-case risk over tasks. They do not promise a given benchmark gain.
SIGReg, the regulariser, enforces that target cheaply.
- Draw random unit directions and project the batch onto each one.
- Test each one-dimensional projection against a standard normal with the Epps-Pulley statistic.
- Average across directions. Resample the directions every step.
The statistic compares characteristic functions over a fixed grid of 17 points. Cost is linear in batch size. The paper recommends 1024 directions as a start.
The full objective mixes SIGReg with a prediction term.
L = λ · SIGReg + (1 − λ) · L_pred, each averaged over views
L_pred is simple. Each view's embedding is pulled toward the mean embedding of the global views. The training uses two global crops at 224 pixels plus smaller local crops at 96. The recommended λ is 0.05.
What is gone matters as much as what is there. LeJEPA uses no stop-gradient and no teacher-student pair. It uses no hyperparameter schedulers.
The core implementation is about 50 lines. The paper reports that predictor and teacher can be removed without collapse. It also notes a teacher still gives ViTs a small boost.
What does the paper claim? On what evidence?
- 79 percent ImageNet-1K linear accuracy with a frozen ViT-H/14. Pretraining used ImageNet-1K alone.
- Experiments across more than 10 datasets and more than 60 architectures.
- Stable training up to roughly 1 billion parameters, with a 1.8 billion run mentioned.
- Training loss that tracks downstream accuracy, with a Spearman correlation near 85 percent.
- In-domain pretraining on Galaxy10, about 11,000 images, that beat frozen DINOv2 and DINOv3 transfer.
Two cautions from my reading. The paper's own few-shot table shows an I-JEPA ViT-H variant averaging higher than LeJEPA ViT-L. The models differ in size, so the comparison is loose.
The work is also recent. Outside replication is still thin.
The loss-to-accuracy correlation is the most useful claim for practice. It means you can choose checkpoints without labels.
The meter version is direct. Suppose we hold five years of interval reads for one service territory. No public model has seen that data. LeJEPA's in-domain claim says we can pretrain on it alone.
What I would measure:
- the SIGReg statistic on held-out batches
- the correlation between training loss and a held-out probe, per checkpoint
- results across three values of λ around 0.05
- the same probe with a frozen public encoder, as a baseline
LeJEPA matters to us for one reason. It makes in-house pretraining simpler and easier to monitor.
Earlier methods needed several moving parts. A teacher network, schedules and tuned settings all had to work.
Each one is a place a project can stall. LeJEPA claims one main setting and a short implementation. Fewer settings means fewer expensive retries.
The second reason is model selection. The authors report that training loss tracks downstream accuracy. If that holds on our data, we can pick the best checkpoint without labelling anything. Labelling is often the largest cost in our projects.
The third reason is domain fit. The paper shows a small, specialised image set where in-domain training beat large public models. Our data is specialised too. Interval reads, inspection photos and outage logs look nothing like web images.
The risks are real. The paper appeared in late 2025. Most numbers come from the authors alone.
One of its own tables shows an older method ahead on few-shot averages. We should treat the claims as promising and still open.
The cost of finding out is modest. A bake-off needs one dataset, one GPU node and two baselines.
- a public encoder used frozen
- a LeJEPA model trained only on our data
The decision is whether to run that bake-off before we license a large external model. A small in-house model that wins on our data is cheaper to own. It also keeps our data inside our walls.
Which of our datasets is large enough to pretrain on? Which is specific enough that public models struggle?
Quick check
Key terms
Further reading
- LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics Randall Balestriero, Yann LeCun, 2025 The paper itself. Theory for the isotropic Gaussian target, SIGReg and the experiments.
- VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning Adrien Bardes, Jean Ponce, Yann LeCun, 2022 The moment-based regulariser SIGReg goes beyond.
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, Nicolas Ballas, 2023 The teacher-based JEPA that LeJEPA compares against and simplifies.
- Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin, 2021 The source of the global and local multi-crop views LeJEPA reuses.