Interactive, simplified visualizations of the core ideas behind the papers in
Paper Deep-Dives. Nothing here is a real model — these are
teaching sketches that make the mechanics intuitive. Click, drag, route, and watch.
From Attention Is All You Need (Transformer). Click a word to see what it attends to.
What's happening: Self-attention lets every token in a sequence "look at" all other tokens at once, and computes a set of attention weights for each (softmax-normalized, summing to 1). A higher weight means that token matters more to the current word. This replaces RNN recurrence, making parallel training and long-range dependency capture possible.
Paper: The Transformer
Weights are softmax-normalized, so they sum to 1. Try "it" — notice it leans toward "cat" and "mat".
2 - Mixture of Experts
From Mixtral of Experts. A router sends each token to the top-2 of 4 experts.
What's happening: MoE splits one big model into many "expert" sub-networks, but each token only activates a few of them (top-2 here). A lightweight router decides the routing. That means the total parameter count can be huge, while compute per forward pass grows only with the activated experts — the key to "more parameters without being slower."
Paper: Mixtral
Only a fraction of experts activate per token — that's why MoE stays cheap at inference.
3 - Scaling Laws
From Scaling Laws & Chinchilla. Drag to grow the model and its data.
What's happening: Large models follow a remarkably clean rule: as you scale up parameters (N), data (D), and compute (C) together, validation loss drops smoothly along a power law. Chinchilla went further — most models were "under-fed on data," and D and N should grow roughly 1:20 together rather than piling on parameters alone.
Papers: Scaling Laws · Chinchilla
7.0B
140B
Illustrative curves (not exact Chinchilla coefficients). The point: spend compute on both dimensions, and returns diminish.
4 - Retrieval-Augmented Generation
From RAG. Pick a question; the system retrieves the closest notes, then answers from them.
What's happening: RAG separates a model’s "memory" from an external knowledge base: it first retrieves the most relevant document snippets for the user’s question, then stitches them into the prompt so the model answers from that evidence. This cuts hallucinations and lets you update knowledge without retraining.
Paper: RAG
Similarity here is plain word overlap (Jaccard) — enough to show the retrieve-then-read loop.
5 - Chain-of-Thought
From Chain-of-Thought Prompting. Letting the model reason step by step unlocks hard problems.
What's happening: Asked for a direct answer, a model often trips up on the intermediate steps; but add a line like "let’s think step by step" and it will spell out the reasoning chain before giving the conclusion. The essence of CoT is having the model first generate "reasoning tokens" that break a hard problem into verifiable small steps — which sharply lifts accuracy on arithmetic, commonsense, and symbolic reasoning.
Paper: Chain-of-Thought
6 - Tree-of-Thoughts
From Tree of Thoughts. Explore many reasoning branches, score them, keep the best.
What's happening: CoT has only a single reasoning line — if it goes wrong, everything fails. ToT turns reasoning into a tree: multiple candidate "thoughts" branch out from the problem, each step is scored / self-evaluated, weak branches are pruned, and the strongest is expanded further, finally backtracking to the best solution. This gives the model "planning + backtracking" power, ideal for hard search problems.
Paper: Tree of Thoughts
7 - RLHF vs DPO
From InstructGPT / RLHF & DPO. A reward model ranks responses; the policy learns to match preferences.
What's happening: Making a model "helpful, useful, and safe" is about preference alignment. The recipe: take human rankings of which of two answers is better to train a "reward model," then push the policy toward higher rewards. RLHF needs two stages — reward model + PPO reinforcement learning; DPO is cleverer — it derives a classification-style loss directly from the preference pairs in one step, skipping both the reward model and the RL loop.
Papers: RLHF (original) · InstructGPT · DPO
1.2
0.3
8 - LoRA (Low-Rank Adaptation)
From LoRA. Fine-tune a tiny low-rank update instead of the whole weight matrix.
What's happening: Full fine-tuning updates the entire weight matrix W (d×d) — an enormous parameter count. LoRA’s insight is that the "change" ΔW during fine-tuning is usually low-rank: it can be approximated by the product of two thin matrices B (d×r) and A (r×d), and you only train those two (params = 2·d·r, with r tiny). The base model stays frozen; to switch tasks you just swap the small LoRA tail — fast and cheap.
Paper: LoRA
4
9 - RoPE (Rotary Position Embedding)
From RoFormer. Rotate token vectors by angle θ·position so attention depends on relative position.
What's happening: A Transformer has no built-in sense of word order. RoPE’s elegant trick: rotate each position’s query/key vector by an angle θ·position. Because the inner product of rotated vectors depends only on the "position difference" (m−n) rather than absolute position, attention naturally carries relative-position information — which is why LLaMA / Mistral / Qwen use it by default.
Paper: RoPE
2
5
10 - BPE Tokenization
From GPT-2 / GPT-3 tokenizers. Byte-Pair Encoding merges the most frequent character pairs, again and again.
What's happening: A model can’t read "words," only tokens. BPE starts from single characters and repeatedly merges the "most frequent adjacent character pair" into a new token (e.g. t+o→to, to+k→tok…) until it hits the vocabulary size. Common words become short tokens, rare words split into fragments — which is why "non-English / gibberish" gets chopped into many small pieces and costs more tokens.
Paper: GPT-3 (tokenizer)