CLIP: Learning Transferable Visual Models From Language Supervision

07/11/26 on LLM Papers

Paper: Radford, A., et al. (OpenAI), Learning Transferable Visual Models From Natural Language Supervision, ICML 2021. arXiv:2103.00020

Why this paper matters

CLIP changed what “understanding an image” means. Instead of labeling millions of pictures with fixed classes, OpenAI trained a model on 400 million (image, text) pairs scraped from the internet, learning to match an image to its caption. The result: a visual representation that transfers to any downstream task via natural language, with zero task-specific training. CLIP is the connective tissue of modern multimodal AI — it is the image encoder behind DALL·E, Stable Diffusion, and nearly every vision-language model. For the LLM era, it’s the bridge from text to pixels.

The core idea: contrastive language–image pretraining

Two encoders — an image tower $I(\cdot)$ and a text tower $T(\cdot)$ — embed into a shared space. For a batch of $N$ pairs, pull the matching (image, text) closer and push the $N^2 - N$ non-matching pairs apart with the InfoNCE contrastive loss:

\[\mathcal{L} = -\frac{1}{N}\sum_{i=1}^{N} \left[ \log \frac{e^{\,\mathrm{sim}(I_i, T_i)/\tau}}{\sum_{j=1}^{N} e^{\,\mathrm{sim}(I_i, T_j)/\tau}} + \log \frac{e^{\,\mathrm{sim}(T_i, I_i)/\tau}}{\sum_{j=1}^{N} e^{\,\mathrm{sim}(T_i, I_j)/\tau}} \right]\]

where $\mathrm{sim}$ is cosine similarity and $\tau$ a temperature. There is no fixed label set — “supervision” is just the caption.

Zero-shot transfer

Because text is the classifier, you never retrain. For a new task you write prompts (“a photo of a {label}”) and pick the label whose text embedding is closest to the image embedding:

image_emb = I(image)
label_embs = T([f"a photo of a {c}" for c in classes])
pred = classes[argmax(image_emb @ label_embs.T)]

Key results

Zero-shot CLIP matched a fully supervised ResNet-50 on ImageNet — without seeing ImageNet labels.
Strong transfer across OCR, action recognition, geo-localization, and more.
Became the de-facto image encoder for generative and multimodal systems.

Why it matters today

CLIP is the “text meets vision” foundation that the LLM world is built on top of. When a model “sees,” it usually does so through a CLIP-family encoder. Read it alongside the Transformer and RAG posts: it extends the representation-learning playbook from text to the joint image–text manifold, and it’s the reason “describe this image” just works.

References

Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision (CLIP). ICML 2021. arXiv:2103.00020
Vaswani et al. (2017). Attention Is All You Need. arXiv:1706.03762