Lesson 5 of 7 · LeJEPA, one regulariser in place of many tricks
-
Mission · Sonu, 10
LeJEPA, one regulariser in place of many tricks
LeJEPA drops the teacher and the schedules. One regulariser keeps collapse away by shaping embeddings into a round Gaussian cloud.
5 cards, 2 of them quick checks. About 3 minutes. One idea per card, so read one, then look away and say it back.
Arrow keys or swipe to move.
-
Idea · 1 of 5
Pick the shape first
LeJEPA starts by asking what shape the embedding cloud should have. Its answer is a round Gaussian, the same in every direction.
-
Try this · 2 of 5
The broom test
Drop buttons on the floor and lay a broom handle across them. Count buttons close to it. Turn the handle and count again. Same count means a round spread.
-
Quick check · 3 of 5
Quick check on the target
What shape does LeJEPA push embeddings toward?
Say your answer, then show it
An isotropic Gaussian. SIGReg checks it through random one-dimensional slices.
-
Quick check · 4 of 5
Quick check on evidence
Is LeJEPA's advantage settled?
Say your answer, then show it
Not yet. Most numbers come from the authors. One of its own tables favours an I-JEPA variant.
-
Recap · 5 of 5
What to carry forward
One regulariser shapes embeddings into a round Gaussian through random slices. Fewer knobs, a loss that tracks accuracy and a strong in-domain claim.
-
Mission · Senior AI Architect
LeJEPA, one regulariser in place of many tricks
LeJEPA drops the teacher and the schedules. One regulariser keeps collapse away by shaping embeddings into a round Gaussian cloud.
9 cards, 3 of them quick checks. About 5 minutes. One idea per card, so read one, then look away and say it back.
Arrow keys or swipe to move.
-
Idea · 1 of 9
Pick the shape first
LeJEPA starts by asking what shape the embedding cloud should have. Its answer is a round Gaussian, the same in every direction.
-
Idea · 2 of 9
Why a round Gaussian
The paper argues an isotropic Gaussian minimises worst-case error for linear and nearest-neighbour probes. It is a bet across all future tasks at once.
-
Idea · 3 of 9
SIGReg in three steps
Project the batch onto random directions. Test each slice against a standard normal. Average the tests and draw new directions next step.
-
Try this · 4 of 9
Slice by hand
Write ten two-number points. Project each onto the direction (1, 1) by adding the numbers. Sort the sums. Do they look bell-shaped?
-
Quick check · 5 of 9
Quick check on the target
What shape does LeJEPA push embeddings toward?
Say your answer, then show it
An isotropic Gaussian. SIGReg checks it through random one-dimensional slices.
-
Idea · 6 of 9
What was removed
It has no teacher and no stop-gradient. One weight, λ, trades SIGReg against prediction. The paper suggests 0.05.
-
Quick check · 7 of 9
Quick check on removals
Name one part LeJEPA drops that I-JEPA needed. The slow-moving teacher, along with its stop-gradient.
-
Quick check · 8 of 9
Quick check on evidence
Is LeJEPA's advantage settled?
Say your answer, then show it
Not yet. Most numbers come from the authors. One of its own tables favours an I-JEPA variant.
-
Recap · 9 of 9
What to carry forward
One regulariser shapes embeddings into a round Gaussian through random slices. Fewer knobs, a loss that tracks accuracy and a strong in-domain claim.
-
Mission · My CIO
LeJEPA, one regulariser in place of many tricks
LeJEPA drops the teacher and the schedules. One regulariser keeps collapse away by shaping embeddings into a round Gaussian cloud.
7 cards, 2 of them quick checks. About 4 minutes. One idea per card, so read one, then look away and say it back.
Arrow keys or swipe to move.
-
Idea · 1 of 7
Pick the shape first
LeJEPA starts by asking what shape the embedding cloud should have. Its answer is a round Gaussian, the same in every direction.
-
Quick check · 2 of 7
Quick check on the target
What shape does LeJEPA push embeddings toward?
Say your answer, then show it
An isotropic Gaussian. SIGReg checks it through random one-dimensional slices.
-
Try this · 3 of 7
Count the knobs
List the settings your last model training needed. Circle the ones that caused a retry. Imagine that list shrinking to one.
-
Idea · 4 of 7
Loss you can steer by
The paper reports training loss tracking downstream accuracy, near 85 percent Spearman. That lets you choose checkpoints without labels.
-
Try this · 5 of 7
Design the bake-off
Write down one dataset, one frozen public encoder and one in-domain model. Add the probe you would use to judge them. Name the owner.
-
Quick check · 6 of 7
Quick check on evidence
Is LeJEPA's advantage settled?
Say your answer, then show it
Not yet. Most numbers come from the authors. One of its own tables favours an I-JEPA variant.
-
Recap · 7 of 7
What to carry forward
One regulariser shapes embeddings into a round Gaussian through random slices. Fewer knobs, a loss that tracks accuracy and a strong in-domain claim.