Lesson 3 of 7 Joint embedding world models, from the ground up 2 min
Information maximisation
A good embedding keeps every dimension busy and saying something new. VICReg and SIGReg are two ways to enforce that.
Sonu, think about the weather chart on our fridge. It has a row for each day of the week.
Say it has four columns. Temperature in Celsius, temperature in Fahrenheit, rain and wind.
Look at the two temperature columns. When one goes up, the other goes up. They always agree.
One of them is wasted space. It tells you nothing the other did not.
Now say there was a fifth column called "Is it Earth?" Every row says yes. That column is wasted too. A column that never changes cannot help you tell days apart.
So a good chart follows two rules. Every column must change from day to day. No two columns may tell the same story.
Follow both rules and every column earns its place. The chart holds the most it can for its size.
Computers keep short notes about things they see. Each note is a list of numbers, like a row on our chart. Scientists want those notes to follow the same two rules.
First, each number must change from one picture to the next. Second, no two numbers may copy each other.
There is a third rule too. Two photos of the same thing should get notes that are close together.
Put all three together and the notes are useful. They stay steady for the same thing. They still tell different things apart.
Newer work adds one more wish. Look at all the notes together. They should spread out like a round, soft cloud. No clumps and no long thin lines.
I like this rule because I can check it with a pencil.
Try this at home. Pick five friends. Write three questions you could ask about each one. Fill in the answers in a little chart.
Is there a question where everyone gave the same answer? Are there two questions that always match? Cross those out. What is left?
Lesson 2 treated collapse as a failure to avoid. This lesson treats the goal positively. An embedding should carry as much information about its input as its size allows.
Entropy is the natural measure. For a Gaussian embedding, entropy grows with the log determinant of its covariance.
That determinant shrinks when any dimension loses its spread. It also shrinks when two dimensions become correlated. So two batch statistics act as a usable proxy for entropy.
LeCun's 2022 position paper lists this aim among its contributions. It calls for non-contrastive training whose representations are both informative and predictable.
VICReg (Bardes, Ponce and LeCun, 2022) writes the proxy as three terms. They act on two batches of embeddings, Z and Z′.
- Invariance is the mean squared distance between paired embeddings.
- Variance is a hinge on each dimension's standard deviation. It penalises any dimension whose spread falls below a target of 1.
- Covariance sums the squared off-diagonal entries of the covariance matrix. It divides by the dimension.
loss = λ·s(Z, Z′) + μ·[v(Z) + v(Z′)] + ν·[c(Z) + c(Z′)]
The paper's defaults are λ = 25, μ = 25 and ν = 1. The expander has three layers of width 8192. A ResNet-50 reached 73.2 percent linear top-1 on ImageNet after 1000 epochs.
The method needs no negatives, no memory bank and no stop-gradient. The two branches need not share weights. Shared weights gave 68.6 percent at 100 epochs, separate weights 66.5.
Why does covariance matter? Without it, variance alone is satisfied by dimensions that all carry one signal. The embedding then has full spread and very low rank. That is dimensional collapse in disguise.
The covariance term spends a small weight to push each dimension toward something new.
The meter version makes the point. Picture a 64-number fingerprint per customer. If twenty numbers all track temperature sensitivity, nineteen are wasted. If one number is the same for every meter, it is wasted too.
LeJEPA (Balestriero and LeCun, 2025) changes what is enforced. VICReg constrains the first two moments. A distribution can pass both and still be lumpy. SIGReg asks for the full shape of an isotropic Gaussian instead.
It does this with slices. Project the batch onto many random unit directions. Test each one-dimensional projection against a standard normal with the Epps-Pulley statistic. Average the statistics and resample directions every step.
The paper argues moment-matching tests are unstable. It argues sorting-based tests are hard to differentiate. Its chosen test has bounded gradients and runs in linear time in batch size. The paper recommends 1024 slices as a starting point.
What I would measure, in order:
- the standard deviation of each dimension across a batch
- the energy in the off-diagonal covariance
- the effective rank of the covariance spectrum
- for SIGReg runs, the average slice statistic over training
A healthy run keeps all four steady while the prediction loss falls.
This lesson sounds technical, but the business question is simple. Are we paying for model capacity we never use?
An embedding is a fixed-size summary. Every number in it costs storage and compute. If half the numbers repeat each other, we pay twice for the same signal. If some numbers never change, we pay for nothing.
The methods here put a price on that waste during training. VICReg keeps every dimension active and different from the others. LeJEPA's SIGReg goes further and shapes the whole cloud of summaries. Both need no labelled data.
The practical consequence is smaller and better summaries. A meter fingerprint that uses its full width carries more distinctions per byte. Theft patterns, solar backfeed and heating type have room to show up.
The risk is subtle. A model can look rich on paper and still be thin inside. Ask how many of its numbers carry independent information. The answer is a single figure called effective rank.
VICReg's own results are a useful anchor. It matched the leading methods of its time without negatives or special tricks. Simpler training is easier to reproduce and audit.
The decision is whether to request effective rank and spread figures in every model review. They cost minutes to compute. They also tell us whether a smaller embedding would serve as well.
I would also ask teams to report results at two embedding sizes. If the smaller one holds its probe score, we keep it and cut cost.
What is the effective rank of the embeddings our current models produce?
Quick check
Key terms
Further reading
- VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning Adrien Bardes, Jean Ponce, Yann LeCun, 2022 The three-term loss, its defaults and the ablations on weight sharing and batch norm.
- LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics Randall Balestriero, Yann LeCun, 2025 SIGReg. Random one-dimensional slices tested against a standard normal.
- A Path Towards Autonomous Machine Intelligence (version 0.9.2) Yann LeCun, 2022 Frames non-contrastive training as making representations informative and predictable.
- Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin, 2021 A contrast. DINO avoids collapse through the teacher's output, with no explicit spread term.