Lesson 2 of 7 Joint embedding world models, from the ground up 2 min
Collapse and its cures
A model that only has to make two summaries agree can cheat by giving everything the same summary. Three families of fixes stop it.
Sonu, let us play a labelling game with the kitchen jars.
I take two photos of the same jar. One is close up. One is from across the room. You see one photo and your friend sees the other.
Each of you writes a two-word label. If your labels match, you both win a sticker.
Rice jar, rice jar. Sticker. Sugar jar, sugar jar. Sticker.
It feels like hard work. It should be.
Now imagine a lazy player. She writes "kitchen thing" on every photo, every time. Her labels always match. She wins every sticker.
But her labels are useless. You could never find the sugar with them. Scientists call this collapse. Everything gets squashed into one answer.
A computer learning this game finds the lazy trick fast. So people invented rules to stop it.
Rule one. You also lose a sticker when two different jars get the same label. Now "kitchen thing" loses most of the time.
Rule two. You copy from an older cousin who changes her labels very slowly. She cannot suddenly switch to "kitchen thing" for everything. So you have a steady target to aim at.
Rule three. Across the whole shelf, your labels must use lots of different words. If every label looks alike, you lose.
Each rule stops the lazy trick in its own way. The best computer models use at least one of them. Some use two.
I find this the most useful idea in the whole course. A score that looks perfect can hide a player who stopped trying.
Try this at home. Pick six things in the kitchen. Give each a two-word label. No two labels may share a word.
Then ask someone to find each thing from its label alone. Which labels worked best? Which word would you change?
A joint embedding model trains two encoders to agree. Agreement alone has a trivial optimum. Map every input to the same vector and the prediction error is zero.
This is complete collapse. A milder form is dimensional collapse. The embeddings vary, but only along a few directions of a large space.
A collapsed encoder looks healthy on the loss curve. That is why it is dangerous. Three families of cures have shipped, each with a different mechanism.
- Contrastive methods add a repelling force
SimCLR (Chen et al., 2020) builds two augmented views of each image. Its loss pulls the pair together and pushes away every other image in the batch. In words, the positive pair must score higher than all the negatives.
loss = −log( exp(sim(z, z⁺)/τ) / Σₖ exp(sim(z, zₖ)/τ) )
A linear classifier on SimCLR features reached 76.5 percent top-1 on ImageNet. That matched a supervised ResNet-50. The price is many negatives, which means large batches. The augmentations also decide what the model learns to ignore.
- Distillation methods use asymmetry
BYOL (Grill et al., 2020) uses an online network and a target network. The online network predicts the target's embedding of another view. The target gets no gradient.
Its weights are a slow moving average of the online weights. There are no negatives at all. A ResNet-50 reached 74.3 percent top-1 under linear evaluation.
DINO (Caron et al., 2021) adds two operations on the teacher's output. Centering subtracts a running mean. Sharpening uses a low teacher temperature.
The paper's ablation is instructive. Centering alone drives the output toward uniform. Sharpening alone lets one dimension dominate.
Together, with a momentum teacher moving from 0.996 to 1, they avoid collapse. DINO reported 80.1 percent linear top-1 with a ViT-Base.
- Regularisation methods police the batch statistics
VICReg (Bardes, Ponce and LeCun, 2022) keeps agreement as one term. It then requires each embedding dimension to keep its spread. It also pushes dimensions to be uncorrelated. Lesson 3 covers it in detail.
One VICReg result shows how fragile the asymmetric tricks are. SimSiam without batch norm fell to 35.1 percent at 100 epochs. Adding VICReg's variance term brought it back to 67.3 percent.
Which family should you pick? Contrastive methods are the most direct but cost batch size. Distillation is cheap per step but depends on tuned schedules. Regularisation makes the anti-collapse pressure explicit and measurable.
The meter world has a twin of collapse. A stuck meter repeats the same read every month. It passes every month-to-month consistency rule. Nothing in the agreement check flags it.
What I would track on every run:
- the standard deviation of each embedding dimension across a batch
- the effective rank of the embedding covariance
- a k-nearest-neighbour probe on held-out labels, every few epochs
A falling loss with a falling spread is the signature of collapse.
Collapse is the failure that looks like success. The training loss drops to near zero. The dashboards turn green. The model has learned nothing.
That should worry anyone who signs off on model results. Our usual acceptance test is a loss curve or a training accuracy figure. For self-supervised models, those numbers can be perfect and meaningless at once.
The fixes are well understood. The published methods fall into three families.
One pushes different inputs apart. One trains against a slow-moving copy of the model. One forces the summaries to stay spread out. All three ship in public code.
Each has its own cost profile.
- Contrastive training needs large batches, which means more GPU memory per step.
- The slow-copy approach is cheaper per step but sensitive to its schedules.
- The spread-out approach adds a cheap check on every batch.
The real cost is in governance. A vendor or an internal team can hand over a collapsed encoder. The report can still look good. Nobody will notice until a downstream model fails in production.
I would make two checks standard for any representation model we accept. First, a measure of how spread out the embeddings are. Second, a simple probe on a business signal. Separating normal and estimated meter reads is one.
Both checks take minutes to run. Neither needs labels beyond a small held-out set.
The decision is whether to write these into our acceptance criteria now. Doing so before the first pilot is far cheaper than doing it after an incident.
Who on our side would own the spread and probe checks?
Quick check
Key terms
Further reading
- A Simple Framework for Contrastive Learning of Visual Representations Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton, 2020 SimCLR. The clearest contrastive recipe, with 76.5 percent linear top-1 on ImageNet.
- Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond and others, 2020 BYOL. Online and target networks, a slow moving average and no negatives.
- Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin, 2021 DINO. Its ablation of centering and sharpening shows two different ways to collapse.
- VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning Adrien Bardes, Jean Ponce, Yann LeCun, 2022 Makes the anti-collapse pressure explicit. Also shows the variance term rescuing other methods.