Representation Learning

A photograph of a room contains millions of numbers — one for the brightness of every tiny dot. Almost none of that detail matters for understanding the room: you care that there is a chair, a window, a door, not the exact shade of gray on the fortieth pixel of the wall. Representation learning is the skill of squeezing a firehose of raw detail down into the small number of facts that actually matter, so everything built afterward can work with "chair, window, door" instead of millions of pixels.

Three dominant training recipes produce useful representations without hand-labeled data. Masked prediction hides part of the input and trains a model to reconstruct it, forcing it to infer missing structure from context. Contrastive learning trains an embedding space where augmented views of the same input are pulled together and different inputs are pushed apart, without ever reconstructing pixels directly. Joint-embedding predictive architectures (JEPA-style) predict the representation of a masked or future region — using a separate, slowly-updated target encoder — rather than predicting raw pixels, which sidesteps wasting capacity on unpredictable low-level noise.

The choice among these is really a choice about what the resulting representation will be good for: reconstruction-based objectives tend to preserve fine detail useful for generation, while contrastive and joint-embedding objectives tend to preserve semantic and dynamic structure more useful for prediction and control.

This territory is where "garbage in, garbage out" gets decided for every downstream world model. A representation that discards the wrong details — say, collapsing a safety-critical visual cue because it looked like noise during training — silently caps what any planner or policy built on top of it can ever perceive. When adopting a pretrained encoder, the diligence question is not "how good are its benchmark numbers" but "was it trained to preserve the kind of signal my domain actually depends on."