-
Mission · Sonu, 10
Information maximisation
A good embedding keeps every dimension busy and saying something new. VICReg and SIGReg are two ways to enforce that.
5 cards. About 3 minutes. One idea per card, so read one, then look away and say it back.
Arrow keys or swipe to move.
-
Idea · 1 of 5
Every number should earn its place
An embedding is a short list of numbers. The aim is to pack in as much information as the list can hold.
-
Try this · 2 of 5
Find the wasted column
Find any table with Celsius and Fahrenheit side by side. Cover one column. Did you lose any information? That column was redundant.
-
Idea · 3 of 5
Beyond two moments
VICReg checks spread and correlation. A cloud can pass both and still be lumpy. LeJEPA's SIGReg asks for the whole shape of a round Gaussian.
-
Try this · 4 of 5
Slice a cloud
Scatter rice on a plate. Look at it from four sides, low to the table. Each view is a one-dimensional slice. Does any view show a lump?
-
Recap · 5 of 5
What to carry forward
Keep every dimension moving and every dimension different. VICReg enforces spread and decorrelation. SIGReg enforces a round Gaussian shape through random slices.
-
Mission · Senior AI Architect
Information maximisation
A good embedding keeps every dimension busy and saying something new. VICReg and SIGReg are two ways to enforce that.
8 cards, 2 of them quick checks. About 4 minutes. One idea per card, so read one, then look away and say it back.
Arrow keys or swipe to move.
-
Idea · 1 of 8
Every number should earn its place
An embedding is a short list of numbers. The aim is to pack in as much information as the list can hold.
-
Idea · 2 of 8
Two ways to waste a dimension
A dimension that never changes carries nothing. Two dimensions that always move together carry one signal twice. Both shrink the embedding's entropy.
-
Idea · 3 of 8
VICReg in three terms
Invariance keeps paired views close. Variance keeps each dimension's spread above 1. Covariance pushes off-diagonal correlations toward zero. Default weights are 25, 25 and 1.
-
Try this · 4 of 8
Compute one spread
Take five meter fingerprints of three numbers each. Make up the values. Work out the spread of each column. Which column would the variance term punish?
-
Quick check · 5 of 8
Quick check on covariance
Every dimension has full spread, but all are perfectly correlated. Is the embedding healthy?
Say your answer, then show it
No. It carries one signal many times over.
-
Idea · 6 of 8
Beyond two moments
VICReg checks spread and correlation. A cloud can pass both and still be lumpy. LeJEPA's SIGReg asks for the whole shape of a round Gaussian.
-
Quick check · 7 of 8
Quick check on SIGReg
How does SIGReg test the shape of the cloud?
Say your answer, then show it
It projects onto random directions and tests each slice against a standard normal. Then it averages.
-
Recap · 8 of 8
What to carry forward
Keep every dimension moving and every dimension different. VICReg enforces spread and decorrelation. SIGReg enforces a round Gaussian shape through random slices.
-
Mission · My CIO
Information maximisation
A good embedding keeps every dimension busy and saying something new. VICReg and SIGReg are two ways to enforce that.
6 cards, 1 of them quick checks. About 3 minutes. One idea per card, so read one, then look away and say it back.
Arrow keys or swipe to move.
-
Idea · 1 of 6
Every number should earn its place
An embedding is a short list of numbers. The aim is to pack in as much information as the list can hold.
-
Idea · 2 of 6
Beyond two moments
VICReg checks spread and correlation. A cloud can pass both and still be lumpy. LeJEPA's SIGReg asks for the whole shape of a round Gaussian.
-
Idea · 3 of 6
Waste costs money
Each embedding number costs storage and compute. Redundant numbers double the bill for one signal. Effective rank tells you how many numbers carry independent information.
-
Try this · 4 of 6
Ask for two sizes
Ask one team to report its probe score at two embedding sizes. If the smaller holds up, write down what it would save.
-
Quick check · 5 of 6
Quick check on value
Which single figure tells you how many embedding numbers carry independent information?
Say your answer, then show it
Effective rank.
-
Recap · 6 of 6
What to carry forward
Keep every dimension moving and every dimension different. VICReg enforces spread and decorrelation. SIGReg enforces a round Gaussian shape through random slices.