Atlasof World Models
Start

T-05 Province II · The Ateliers The Cartographer’s Table 5 min 5 of 21

Representation Learning

How a model decides what is worth remembering from raw pixels, sound, or text, and what to throw away.

Representation Learning: what an encoder keeps, and collapse Plain words: the photo becomes a short description. With only matching pairs, everyone ends up looking the same. Architect: encoder, embedding, projection head, collapse to one vector under positives only, negatives for contrastive learning, a slow-moving teacher as in BYOL. CIO: raw data, kept signal, lost signal and a loss near zero that can mean nothing was learned. T-05 · WHAT SURVIVES spread a month, like a photo the short description freckles dropped each kid stays findable everyone looks the same copy a slow, steady friend Keep what finds you. Drop the freckles. x: 30 days x 6 blocks encoder embedding proj. head contrastive: negatives push apart collapse: one vector for all BYOL: chase an EMA target Positives alone admit a trivial answer. Something must forbid it. raw data kept signal lost signal signal kept: groups separate loss near zero, nothing learned no negatives needed, still apart Ask for a probe on a business signal beside the loss.
Signal

A pale tabletop holding a small loupe beside a neat grid of cobalt dots on white paper.

In one sentence Keep the few facts that matter – chair, door, window – and throw away the millions of pixels that do not.

Illustration of Sonu, a ten-year-old girl with a dark ponytail in a yellow shirt, holding a small notepad. Sonu, 10

Say your friend is coming to pick you up at the school gate. She has never met you. You call her and describe yourself.

You do not list every freckle. You do not count your hairs. You say something like this. Red jacket, curly hair, a blue backpack with a fox on it.

Those few words are enough. Out of hundreds of kids, she finds you. That short description is a representation. It keeps the few facts that matter and drops the rest.

A photo of you holds millions of tiny dots. Each dot has its own color number. Almost none of them help your friend.

Representation learning teaches a computer to squeeze those dots into a short description. The description still has to find you.

There is a funny way to get this wrong. Say the computer gets a gold star whenever two descriptions of the same kid match. A lazy computer learns a trick. It describes every kid as "a kid."

Now every description matches, so it earns every gold star. But your friend cannot find you anymore. Scientists call this collapse. Much of this field is about stopping it.

One fix is to also say how kids differ. Red jacket is not green jacket.

Another fix is a guessing game. Cover half of a photo of the playground. The computer describes what is hidden, like a swing. It does not have to draw every leaf.

That guessing game is the idea behind a model called JEPA. I have been studying it this autumn. It learns to describe hidden parts of the world without drawing them.

Try this at school. Pick a classmate without saying who. Describe them in five words to a friend.

Can your friend find them? Then try with three words. Which words did you keep?

Illustration of the Senior AI Architect, a person in glasses and a blue shirt holding a pen. Senior AI Architect

Representation learning decides what a model keeps from raw input. Every world model downstream inherits that choice. Three self-supervised recipes dominate. Each keeps different things.

Masked prediction hides part of the input and reconstructs it. MAE (He et al., 2021) masked 75 percent of image patches. It trained at least 3 times faster and reached 87.8 percent on ImageNet-1K with ViT-Huge. Reconstruction keeps fine detail, which helps generation.

Contrastive learning pulls two augmented views of one input together. It pushes different inputs apart.

SimCLR (Chen et al., 2020) reached 76.5 percent top-1 with a linear classifier on ImageNet. It needs many negatives and large batches. The augmentations decide what gets ignored.

Asymmetric methods drop the negatives. BYOL (Grill et al., 2020) has an online network predict a slow-moving target network. It reached 74.3 percent top-1 with a ResNet-50.

Joint-embedding predictive architectures predict the representation of a hidden region. They never predict its pixels.

I-JEPA (Assran et al., 2023) predicts target blocks from one context block. It needs no hand-crafted augmentations. It trained a ViT-Huge/14 on 16 A100 GPUs in under 72 hours. Skipping pixels saves capacity that would go to unpredictable noise.

V-JEPA 2 (Assran et al., 2025) took this to video. It pretrained on over 1 million hours of video and images. Under 62 hours of robot video then let it plan zero-shot pick-and-place on Franka arms.

LeJEPA (Balestriero and LeCun, 2025) replaces stop-gradients and teachers with one regularizer, SIGReg. It pushes embeddings toward an isotropic Gaussian with a single trade-off hyperparameter.

The failure all of these fight is collapse. Train with positives only and the encoder outputs one vector for every input.

In the course notebook I studied, the loss reached 0.000000. Embedding spread fell to 0.001 of its start. A nearest-neighbor test scored 15 percent against 10 percent for guessing.

A second detail matters in practice. Most methods train a projection head on the encoder, then throw it away. SimCLR tested a linear classifier on each output. The encoder output beat the head output by more than 10 points.

Here is the meter version. A month of smart-meter data is 30 days of 96 reads, or 2,880 numbers. An encoder can turn that into a 384-number usage fingerprint.

Trained once on a narrow task, it can serve theft detection or EV-charger detection. It can also feed rate-plan advice.

Collapse also has a meter twin. A broken meter that reports the same reading every month passes a month-to-month consistency rule.

I would recommend to measure spread or effective rank of the embeddings first. Then run a linear probe on a signal your decision depends on. A falling loss proves nothing by itself.

Illustration of My CIO, a grey-haired person in a purple jacket with arms folded. My CIO

This territory is where "garbage in, garbage out" gets decided for every downstream world model. The encoder chooses which details survive. Every planner built on top can only see what survived.

The danger is quiet. A representation can discard a safety-relevant cue because it looked like noise in training. Nothing breaks.

The model simply never sees that cue again. No later stage can recover it.

Pretrained encoders are attractive for cost reasons. One encoder, trained once, can feed many tasks. In utility work one meter fingerprint might feed theft detection and rate advice. The savings are real when the fingerprint holds the right signal.

So the diligence question changes. Benchmark numbers on public image sets say little about meter data or field photos. Ask instead whether the encoder was trained to keep the signal our domain needs.

There is a second check that costs almost nothing. A training loss near zero can mean the model learned nothing at all. This failure has a name, collapse. Teams should report a probe on a real business signal, beside the loss.

Recent work makes this cheaper. JEPA-style models learn from unlabeled data, which utilities hold in volume. Meta's V-JEPA 2 learned from over a million hours of video before any robot data.

The lesson for us is scale on unlabeled history first. A small domain step comes after.

The decision is where to invest. A shared encoder for one data type, like interval reads, can pay back across teams. It needs one owner and one test suite.

The question I would ask is plain. Which signal does our decision depend on? Have we tested that this encoder keeps it?

Quick check

3 questions. Pick one answer each. Get them all right to mark this territory as checked.

1An encoder trained only to make two views agree reaches a loss of zero. What is the most likely story?

2What does a JEPA predict for a hidden region?

3Which question matters most before reusing a pretrained encoder on meter data?

Key terms

Further reading

  1. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, Nicolas Ballas, 2023 I-JEPA. Predicts hidden blocks in representation space, with no hand-crafted augmentations.
  2. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning Mido Assran, Adrien Bardes, David Fan, Quentin Garrido and others, 2025 Over a million hours of video, then under 62 hours of robot video for zero-shot planning.
  3. A Simple Framework for Contrastive Learning of Visual Representations Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton, 2020 SimCLR. The clearest contrastive recipe. It also holds the projection-head result.
  4. Masked Autoencoders Are Scalable Vision Learners Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick, 2021 Masked prediction at 75 percent masking, the reconstruction side of the comparison.
  5. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics Randall Balestriero, Yann LeCun, 2025 One regularizer, SIGReg, in place of stop-gradients and teacher networks.
  6. Contrastive Representation Learning Lilian Weng, 2021 A thorough tutorial on the contrastive family.

Search the atlas