Transformers & Attention Basics

Have you ever sat in a live class staring at "Self-Attention", "Q/K/V", "Multi-Head", nodding along the whole time -- and then realized on the way home that you didn't actually understand a word of it?

Honestly, the first time I looked at the classic Transformer architecture diagram, that was me. A dozen boxes packed together, arrows looping everywhere, a pile of matrix formulas on the side. Maximum intimidation. Then I figured something out: you don't need to derive any of those formulas to do AI engineering, but you do need an intuition solid enough to carry you through RAG, Agents, and fine-tuning later -- knowing what actually happens inside the model's head when it "reads a sentence." This chapter takes the engine apart for you, with numbers and code, but without dragging you through the paper.

---

Why bother? (What breaks if you don't)

You can absolutely call the OpenAI API without understanding Transformers. But the moment you go deeper, these questions catch up with you:

• Why is there a context window limit? Why does everything get slow and expensive once tokens pile up? → The answer is in attention's compute cost.
• Why do models suffer from "lost in the middle", dropping information from the middle of long documents? → It's tied to attention weight distribution + positional encoding.
• What step do API parameters like temperature and top_p actually tune? → The final sampling layer. Without knowing the architecture, you're just memorizing jargon.
• Fine-tuning, KV Cache, Flash Attention, quantization... which part of the pipeline does each one optimize? → Once you know the architecture, they all snap into place.

Understand this layer, and a lot of "magic" downstream turns into "oh, so that's all it is."

---

First, kill a misconception: an LLM is not a "database"

Term: Transformer

• One-line explanation: a neural network architecture that uses "attention" to let every word look at every other word in the sentence.
• Analogy: a group meeting. Before anyone (a word) speaks, they scan everyone else at the table first, then decide what to say.
• How it shows up at work: GPT, Claude, Llama -- nearly every mainstream LLM is built on it (more precisely, on its decoder half).
• Most common trap: assuming it "stores" answers like a database. What it stores is relationship weights between words; answers are computed on the fly -- which is also the root cause of confident nonsense (hallucination).

Before 2017, the workhorses for text were RNNs/LSTMs, which read like someone reciting a passage in a queue: one character at a time, and by the end they've mostly forgotten the beginning. The Transformer paper had an arrogant title -- "Attention Is All You Need" -- meaning: drop the queue entirely, just let all the words look at each other at once.

That solved two big problems in one shot: (1) it parallelizes (GPUs stay fed, training is fast); (2) words arbitrarily far apart can "lock eyes" directly, so nothing gets forgotten along the way.

---

What is attention actually "looking at"?

Take this sentence:

The cat didn't jump onto the table because it was too tired.

What does "it" refer to? A human knows instantly: the cat. How does the model know? Attention: when processing the word "it", the model scores every word in the sentence on "how much should I pay attention to you." "Cat" gets the highest score, so the meaning of "it" gets pulled toward "cat."

That's the entire intuition behind self-attention: every word, based on the current context, dynamically decides which other words to absorb information from.

Q / K / V: don't panic, it's just "looking things up"

The model computes three things for every word. The library analogy makes it click:

Term: Query / Key / Value (Q/K/V)

• One-line explanation: Query = the question I'm asking, Key = each word's "label", Value = each word's actual content.
• Analogy: a library. You take your question (Q) and match it against every book's spine label (K); for the books that match, you read in their contents (V).
• How it shows up at work: attention weight = how well Q matches K; final output = weighted sum of the Vs.
• Most common trap: thinking Q/K/V are three different kinds of words. They're not -- they're the same word multiplied by three different weight matrices, giving three "views" of it.

The one-line formula (just look, no need to memorize):

attention output = softmax( Q · Kᵀ / √d ) · V
                   └── how much attention each word gets ──┘  └ take content by weight ┘

• Q · Kᵀ: compute the match score between the "question" and every "label".
• / √d: divide by a constant to keep scores from blowing up (otherwise softmax turns all-or-nothing).
• softmax: squash the scores into weights that sum to 1.
• · V: blend the contents according to those weights.

Work one out by hand: attention over three words (with numbers)

Formulas alone don't stick, so let's hand-compute who "it" should attend to, using three words: "cat" / "tired" / "it". Suppose after one layer, the match scores between the Query of "it" and the three Keys come out like this (numbers made up, purely for demonstration):

"it" vs "cat":    raw score  8.0
"it" vs "tired":  raw score  2.0
"it" vs "it":     raw score  3.0

Step 1, scale by /√d (say d=64, so √d=8): 1.0 / 0.25 / 0.375. Step 2, softmax (squash them into weights that sum to 1), which gives roughly:

attend to "cat":    0.60   ← highest; "it" absorbs its meaning mostly from "cat"
attend to "tired":  0.16
attend to "it":     0.24

Step 3: the new representation at the position of "it" = 0.60×V_cat + 0.16×V_tired + 0.24×V_it. "It" just got pulled 60% toward "cat" -- that's coreference resolution happening in plain arithmetic. GPT runs this whole routine at every layer, for every word, on every generation step.

Don't trust my hand math? Run it in 20 lines of numpy

The numbers above came from me. You can reproduce the whole flow with numpy -- no GPU, no deep learning framework, pure matrix multiplication:

# pip install numpy
import numpy as np

def softmax(x):
    e = np.exp(x - x.max())
    return e / e.sum()

# 3 words: "cat" "tired" "it", demoed with tiny d=4 vectors (numbers made up, logic real)
q_it = np.array([1.0, 0.5, 0.2, 0.8])   # Query of "it"
K = np.array([
    [1.1, 0.4, 0.3, 0.9],   # Key of "cat" -- deliberately designed to resemble q_it
    [0.1, 0.9, 0.1, 0.0],   # Key of "tired"
    [0.3, 0.2, 0.6, 0.3],   # Key of "it"
])
V = np.array([
    [10.0, 0.0, 0.0, 0.0],  # Value of "cat" (easy-to-spot integers)
    [0.0, 10.0, 0.0, 0.0],  # Value of "tired"
    [0.0, 0.0, 10.0, 0.0],  # Value of "it"
])

scores = K @ q_it / np.sqrt(4)   # Q·Kᵀ / √d
weights = softmax(scores)
output = weights @ V             # blend Values by weight
print("attention weights:", weights.round(3))       # ≈ [0.503 0.237 0.260]
print("new representation of 'it':", output.round(2))  # ≈ [5.03 2.37 2.6  0.  ]

Run it and you'll see "cat" takes about 0.5 of the weight, and the first dimension (the "cat" component) clearly dominates the new representation of "it". The entire core of self-attention is those three lines of matrix math -- scores, softmax, weighted sum. Real models just swap d=4 for d=tens-of-thousands, 3 words for tens of thousands of tokens, one head for dozens, then stack dozens of layers. That's it. Get this running and you understand 80% of attention.

Multi-Head: same sentence, several teams each watching for different things

Multi-Head Attention: instead of sending one person to read the whole sentence, send 8 or 16 teams simultaneously, each watching a different kind of relationship -- one team tracks grammar (subject-verb-object), one tracks coreference (it = cat), one tracks tense or sentiment... Then all the teams' conclusions get concatenated and fused through a linear layer. One layer thus captures multiple kinds of relationships at once, far richer than a single head. GPT-3 has 96 heads -- not a random choice, but the parallelism of "divide and watch."

The "library / division of labor" analogy helps you understand the retrieval part, but real systems differ in one way: library books are fixed, whereas Q/K/V get recomputed and re-blended at every layer, growing more abstract the higher you go.

---

What happens inside one full Transformer block

Attention is only half of a layer. A real layer (block) looks like this, bottom to top:

        input vectors (a string of numbers per token)
              │
   ┌──────────▼───────────┐
   │  Multi-Head Attention │  ← words look at each other, absorb context
   └──────────┬───────────┘
              │  ＋ residual (add the input back unchanged) → LayerNorm
   ┌──────────▼───────────┐
   │  Feed-Forward (FFN)   │  ← each word runs through its own small two-layer net
   └──────────┬───────────┘
              │  ＋ residual → LayerNorm
              ▼
        output vectors → fed to the next layer, repeated N times

Three names you're guaranteed to hear, cleared up in one pass:

• FFN (feed-forward network): attention handles "words looking at each other"; the FFN handles "each word thinking things over on its own." It's a two-layer net that expands (e.g., 4x the dimension) then compresses back -- and a large share of the model's "knowledge" actually lives here.
• Residual connection: add the layer's input, untouched, onto its output. Why? Stack a network dozens or hundreds of layers deep and gradients vanish, information gets lost. The residual keeps a "highway" open so information and gradients can travel straight through. Without it, deep networks simply don't train.
• LayerNorm (layer normalization): pulls each layer's values back into a stable range so things don't drift off the rails. Think of it as a "voltage regulator" between layers.

GPT-3 is just this block stacked 96 times. The "large" in "large language model" is mostly layers × width.

Keep one set of numbers handy for classes and interviews: GPT-3 (175B parameters) = 96 layers × 96 heads × 12288 hidden dimensions. And the LoRA/PEFT you keep hearing about in fine-tuning mostly touches those attention projection matrices -- why "training just 1% of parameters" works will make sense when you reach the fine-tuning chapter and come back to this diagram.

---

Positional encoding: the model can't tell word order on its own

Attention has an awkward built-in flaw: it looks at everything at once, so it has no innate concept of order. To pure attention, "cat chases dog" and "dog chases cat" contain the exact same set of words -- it can't tell who comes first.

Term: Positional Encoding

• One-line explanation: attach a unique "numeric fingerprint" to each position so the model knows who sits where.
• Analogy: theater seat numbers. Same crowd of people, but only with row and seat numbers do you know who's next to whom and who's up front.
• How it shows up at work: early on (the original Transformer) used fixed sine/cosine waveforms as fingerprints; the current mainstream (e.g., RoPE, rotary positional encoding) bakes position directly into Q/K, which extrapolates better on long text.
• Most common trap: assuming the context window can be cranked as large as you like. The positional encoding design determines how much length the model handles reliably -- go too far beyond training length and quality drops. This is one of the actual battlegrounds of "long context" models.

One line: word order information comes entirely from positional encoding. Remove it and GPT degrades into a bag of words with no grammar.

---

From text to "the next word": the full pipeline

Feed "the cat is too tired" into GPT and roughly this happens:

text
  │  ① Tokenize: split into tokens (not exactly "characters")
  ▼
[the][cat][is][too][tired]
  │  ② Embedding: each token becomes a string of numbers (a vector) -- its "semantic coordinates"
  │  ③ Positional encoding: tell the model who comes before whom
  ▼
  │  ④ Stack N blocks (Attention + FFN, each with residual/LayerNorm)
  ▼
  │  ⑤ Output layer: project the final vectors into "a score (logit) per candidate token"
  │  ⑥ Sampling: pick the next token by score
  ▼
next word → append to the input → run again → loop, emitting word by word

Diagram walkthrough (follow along):

1. What this diagram shows: the full production line from text to "predict the next word", and why it's word-by-word chaining.
2. Where to look first: start with ①②③ to see "text becomes numbers + position gets added"; then ④, the core repeated dozens of times; finally ⑤⑥ explain why it "makes things up."
3. The key judgment call at work: steps ⑤⑥ only "pick the next word", so an LLM is fundamentally a supercharged word-chaining game, not an answer lookup -- which explains why it's fluent yet hallucinates.

Step ⑥ is exactly where temperature / top_p live

The output layer produces a list of candidate-token scores. How you pick one from that list is the sampling strategy -- these are the API parameters you tune every day:

So the folk wisdom "set temperature=0 for stable output" translates, architecturally, to "pick the highest-scoring word at every step."

While we're at it: token ≠ character

Back to step ① and the most-stepped-on trap: a token is neither a character nor a word. "ChatGPT" might be split into 2 tokens; one Chinese character often takes 1–2 tokens. You pay per token, and the context window is measured in tokens, so "character count" and "token count" never line up -- never estimate cost by character count.

The star of step ②: Embedding

Term: Embedding

• One-line explanation: turn each token into a string of numbers (a vector), where words with similar meanings get similar numbers.
• Analogy: give every word a spot on a "semantic coordinates" map -- "cat" and "dog" live close together; "cat" and "invoice" are ten blocks apart.
• How it shows up at work: the first step of every LLM input; vector retrieval in RAG, semantic search, and recommender systems all run on this same "distance = semantic similarity" idea.
• Most common trap: thinking embedding is a fixed lookup table and that's that. The lookup is only the starting point -- after passing through attention layers, the same word gets different vectors in different sentences ("Apple keynote" vs. "apple pie"), which is exactly where Transformers beat old-school static word vectors.

---

Hands-on code: see tokenization and attention with your own eyes

# pip install transformers torch
from transformers import AutoTokenizer, AutoModel
import torch

tok = AutoTokenizer.from_pretrained("bert-base-chinese")
model = AutoModel.from_pretrained("bert-base-chinese", output_attentions=True)

text = "那只猫没跳上桌子因为它太累了"  # "The cat didn't jump onto the table because it was too tired"
enc = tok(text, return_tensors="pt")
print("token split:", tok.convert_ids_to_tokens(enc["input_ids"][0]))
# ['[CLS]','那','只','猫','没','跳',... ,'它','太','累','了','[SEP]']  ← each character ≈ 1 token

with torch.no_grad():
    out = model(**enc)

# out.attentions: (num_layers) tensors, each shape = [batch, heads, seq, seq]
att = out.attentions[-1][0]          # last layer, drop batch dim → [heads, seq, seq]
att = att.mean(0)                    # average over all heads → [seq, seq]
tokens = tok.convert_ids_to_tokens(enc["input_ids"][0])
it_idx = tokens.index("它")          # "它" = "it"
scores = att[it_idx]                 # the "it" row = its attention weight over every word
top = torch.topk(scores, 3)
print("'it' attends most to:", [(tokens[i], round(float(scores[i]),3)) for i in top.indices])
# You'll usually see "猫" (cat) near the top -- coreference, verified with your own eyes

Code walkthrough:

1. What this does: load a real model, pull out the attention weights, and see who "it" is actually attending to.
2. Where to look first: check the four dimensions of att.shape (layers, heads, word×word) -- that's the multi-head attention matrix from earlier; then look at the final "attends most to" printout.
3. The key judgment call: different heads attend to different things, so we mean(0) to see the overall tendency; to study "which head handles coreference," skip the averaging and inspect head by head.

---

KV Cache: why the first token is slow and the rest are fast

Here's a production phenomenon you'll run into sooner or later: when calling the API, the first token takes the longest, then the rest pour out. The first time I debugged a "slow response" ticket for a client, this is exactly where I tripped -- I stared at total latency in the logs for half a day before realizing you have to split it into two metrics: time to first token and tokens per second. Behind them are two completely different phases:

• Prefill: run your entire prompt through all the layers in one pass, computing K and V for every token. The longer the prompt, the slower this is -- and it's the bulk of first-token latency.
• Decode (token-by-token generation): for each new token, in theory you'd recompute K/V for all preceding tokens -- but they haven't changed! So inference frameworks cache the computed K/V, and each step only computes the new token's own Q/K/V, then reads the cache. That's the KV Cache.

What happens without it? Do the math: a 1000-token prompt generating a 500-token reply means step 500 would recompute K/V for 1500 tokens; across the whole generation, the redundant compute is hundreds of times the cached approach. KV Cache trades memory for speed -- and the price is VRAM: for a 7B model (fp16), each token's K/V cache takes roughly 0.5 MB, so a 4096-token conversation costs about 2 GB of VRAM. Why long conversations are slow and expensive on APIs, why local deployment blows up VRAM on long context -- the bill is right here.

One line to remember: Prefill decides how fast the first token arrives, KV Cache decides how fast the rest stream out, and VRAM decides how much you can cache. These three terms will keep coming back in deployment and cost optimization.

---

Why this architecture is expensive and context-limited

Attention requires every word to look at every word: n words means n×n comparisons. Double the text, quadruple the compute (O(n²)). How vicious is that square? Plug in numbers:

1,000 tokens   →  1 million comparisons
10,000 tokens  →  100 million comparisons   (text ×10, compute ×100)
100,000 tokens →  10 billion comparisons    (text ×100, compute ×10000)

Which is why:

Everything that came later -- Flash Attention (compute attention with less memory), sparse/sliding-window attention (stop every word from looking at everything), KV Cache (cache computed K/V during generation instead of recomputing) -- is a fight against this O(n²). You don't need to implement any of it yet; knowing "the bottleneck is attention" is enough.

This is also the first principle behind RAG: rather than stuffing 100k tokens into context and diluting attention, retrieve the most relevant 2000 tokens first and feed those -- more accurate and cheaper. When you reach the RAG chapter, you'll see the entire approach is a detour around this n².

---

Where do these weights come from? Training in one minute

By now you might ask: who decided the values in those Q/K/V projection matrices and FFN parameters? The answer is almost disappointingly simple -- they were all learned by "guessing the next word":

1. Take a large chunk of real text, e.g. "The cat didn't jump onto the table because it was too tired"
2. Cover the tail, have the model predict the next token: "The cat didn't jump onto the table because it was too ___"
3. The model outputs a probability per candidate token; compare with the correct answer "tired" and compute the gap (loss)
4. Backpropagation: nudge every weight slightly toward "more likely to guess right next time"
5. Move to the next chunk of text, back to step 2 -- repeat for trillions of tokens

"Pretraining" is this loop running for months. Nobody labeled rules like "it = cat" for the model -- coreference, grammar, common sense were all picked up "incidentally," in service of guessing the next word better. That explains two things:

• Why fine-tuning works: pretraining already compressed the regularities of language into those matrices; fine-tuning just steers slightly on top of a finished foundation. LoRA dares to train only 1% of parameters because it's betting "the foundation doesn't need touching."
• Why hallucination can't be cured at the root: the training objective, start to finish, is "guess plausibly," not "state correctly." Plausible and correct are highly correlated -- but they're not the same thing.

That's it. The mathematical details of training (gradient descent, optimizers) belong to the ML fundamentals chapter; here all you need is this intuition.

---

Three kinds of Transformer: GPT is just one model on the lot

The original "Attention Is All You Need" had two halves, encoder and decoder, which later split into three lineages:

Why can GPT only look left? Because its job is "predict the next word", and during both training and inference it must not peek at the answer (the right side) -- so it uses masked attention to block it out. That's also why it naturally generates left to right, word by word.

One aside: the embedding models that handle retrieval in RAG are mostly descendants of the encoder family; the chat models you talk to daily are the decoder family. Same attention mechanism, two ways to serve it -- one for "understanding," one for "chaining."

---

⚠️ Common Pitfalls

---

🔧 Hands-on Exercises

Exercise 1: Reproduce the "it → cat" attention by hand (about 20 minutes)

1. Run the numpy snippet above as-is (local pip install numpy or fire up a Colab).
2. Edit the second row of K (the Key of "tired") to closely resemble q_it (e.g. [1.0, 0.5, 0.2, 0.8]), rerun -- watch the weight flip from "cat" toward "tired".
3. Delete the / np.sqrt(4) from the scores line, multiply all values in K by 10, rerun -- watch the softmax output turn all-or-nothing. That's why the /√d exists in the formula.
4. Going further: run the transformers BERT snippet, change the sentence to "The lawyer handed the documents to the assistant because he was running late", and see whether "he" attends to "lawyer" or "assistant" -- does the model's judgment match your intuition?

Exercise 2: Verify the "sampling layer" through API parameters (about 15 minutes)

1. Open the OpenAI Tokenizer, paste in "人工智能" and "Artificial Intelligence", count the tokens each takes, and feel how "Chinese costs more tokens."
2. With the same prompt (an open-ended one like "write a one-line slogan for these headphones" works well), generate 3 times each at temperature=0 and temperature=1.2; compare stability vs. nonsense, and map the results back to the "sampling layer" section of this chapter.
3. Take a 500-word document, place one key conclusion in the dead center vs. at the end, ask the model once for each -- observe which answer is more accurate. You've just reproduced "lost in the middle" yourself.
4. Log the token usage (usage field) of each call, cross-check against the tokenization from step 1, and verify how "billed per token" actually adds up.

---

📚 Summary

1. Transformers use attention to let every word see the whole sentence at once, replacing the RNN's queue-style processing -- fast and forgetful-proof; Q/K/V = match a question against labels, then take content by match score; multi-head means "several teams each watching different relationships"; the core is the three-step matrix routine softmax(QKᵀ/√d)·V, reproducible in 20 lines of numpy.
2. One block = attention + FFN, each with residual + LayerNorm; the residual is the "highway" that makes deep networks trainable, the FFN stores much of the model's knowledge, and "large" in LLMs is mostly layers × width (GPT-3 = 96 layers × 96 heads).
3. Attention has no innate word order; positional encoding injects it, and its design also determines how much context is reliable.
4. An LLM is fundamentally word-by-word chaining (⑤ output logits → ⑥ sampling); temperature/top_p/top_k tune exactly that sampling step; token ≠ character -- cost and context are both measured in tokens; generation splits into Prefill (decides first-token latency) + Decode (KV Cache speeds up streaming), with KV Cache trading VRAM for speed (roughly 0.5 MB/token for a 7B model).
5. Attention is O(n²) (text ×100 = compute ×10000) -- the common root of context limits, slow-and-pricey long text, and "lost in the middle", and the first principle behind RAG's "retrieve first, then feed"; GPT is decoder-only (left-context only, generates word by word), with a different job from BERT (encoder, understanding) and T5 (encoder-decoder).

---

Related Reading

Next up: ML / Deep Learning Fundamentals →

• LLM API Basics -- temperature/top_p, token billing, streaming: what this chapter's sampling layer looks like from the API side
• Context Fundamentals -- engineering the context window, picking up this chapter's positional encoding and n² root cause
• Context Degradation Patterns -- how "lost in the middle" shows up in real systems, and what to do about it
• RAG Basics -- the proper detour around n²: retrieve first, then feed
• Fine-tuning: PEFT / QLoRA -- LoRA touches exactly the attention projection matrices covered in this chapter
• Deployment & Cost Optimization -- the production ledger of KV Cache, VRAM, and first-token latency