-
Mission · Sonu, 10
Collapse and its cures
A model that only has to make two summaries agree can cheat by giving everything the same summary. Three families of fixes stop it.
5 cards. About 3 minutes. One idea per card, so read one, then look away and say it back.
Arrow keys or swipe to move.
-
Idea · 1 of 5
Agreement has a cheap answer
Two encoders trained only to agree can map everything to one vector. The loss reaches zero. The model has learned nothing. This is collapse.
-
Try this · 2 of 5
Spot the lazy label
Write one label that would match every jar in your kitchen. Now write one that fits only the rice jar. Which label helps someone find the rice?
-
Try this · 3 of 5
Find a stuck meter
Think of a meter that reports the same read every month. List the checks it would pass. Then name one check that would catch it.
-
Idea · 4 of 5
Cure three, police the batch
VICReg keeps each dimension spread out and the dimensions uncorrelated. The anti-collapse pressure is a loss term you can read on every batch.
-
Recap · 5 of 5
What to carry forward
Agreement alone collapses. Push negatives apart, chase a slow teacher or regulate batch statistics. Always measure spread and a probe beside the loss.
-
Mission · Senior AI Architect
Collapse and its cures
A model that only has to make two summaries agree can cheat by giving everything the same summary. Three families of fixes stop it.
10 cards, 2 of them quick checks. About 5 minutes. One idea per card, so read one, then look away and say it back.
Arrow keys or swipe to move.
-
Idea · 1 of 10
Agreement has a cheap answer
Two encoders trained only to agree can map everything to one vector. The loss reaches zero. The model has learned nothing. This is collapse.
-
Idea · 2 of 10
The quiet version
Dimensional collapse is subtler. Embeddings still vary, but only along a few directions. Most of the space goes unused. The loss can look healthy throughout.
-
Quick check · 3 of 10
Quick check on collapse
Your self-supervised loss falls to almost zero in a day. What do you check first?
Say your answer, then show it
The spread of each embedding dimension. A near-zero spread means collapse.
-
Try this · 4 of 10
Find a stuck meter
Think of a meter that reports the same read every month. List the checks it would pass. Then name one check that would catch it.
-
Idea · 5 of 10
Cure one, push apart
Contrastive methods like SimCLR pull two views of one image together. They push every other image in the batch away. The cost is large batches.
-
Idea · 6 of 10
Cure two, a slow teacher
BYOL and DINO train against a copy of the model that moves slowly. The copy gets no gradient. It gives a steady target that cannot jump to a constant.
-
Try this · 7 of 10
Feel the slow average
Take the numbers 10, 2, 10, 2, 10. Keep a running value that moves one tenth of the way each step. Start at 10. Watch how little it swings.
-
Quick check · 8 of 10
Quick check on DINO
DINO keeps centering but drops sharpening. Where does the output drift?
Say your answer, then show it
Toward a uniform distribution. Drop centering instead and one dimension dominates.
-
Idea · 9 of 10
Cure three, police the batch
VICReg keeps each dimension spread out and the dimensions uncorrelated. The anti-collapse pressure is a loss term you can read on every batch.
-
Recap · 10 of 10
What to carry forward
Agreement alone collapses. Push negatives apart, chase a slow teacher or regulate batch statistics. Always measure spread and a probe beside the loss.
-
Mission · My CIO
Collapse and its cures
A model that only has to make two summaries agree can cheat by giving everything the same summary. Three families of fixes stop it.
6 cards, 1 of them quick checks. About 3 minutes. One idea per card, so read one, then look away and say it back.
Arrow keys or swipe to move.
-
Idea · 1 of 6
Agreement has a cheap answer
Two encoders trained only to agree can map everything to one vector. The loss reaches zero. The model has learned nothing. This is collapse.
-
Try this · 2 of 6
Find a stuck meter
Think of a meter that reports the same read every month. List the checks it would pass. Then name one check that would catch it.
-
Idea · 3 of 6
Cure three, police the batch
VICReg keeps each dimension spread out and the dimensions uncorrelated. The anti-collapse pressure is a loss term you can read on every batch.
-
Try this · 4 of 6
Write the acceptance rule
Draft one sentence for a model acceptance checklist. It should require an embedding spread figure and one business probe. Name who signs it.
-
Quick check · 5 of 6
Quick check on governance
A vendor shows a near-zero training loss for a new encoder. Is that evidence of quality?
Say your answer, then show it
No. It is also what a collapsed encoder shows.
-
Recap · 6 of 6
What to carry forward
Agreement alone collapses. Push negatives apart, chase a slow teacher or regulate batch statistics. Always measure spread and a probe beside the loss.