Start with a sentence: The cat sat on the. A transformer does not see words the way you do. It sees a sequence of integers, each one an index into a vocabulary. The first job is to turn those integers into something the model can do arithmetic on.
Tokens and embeddings
Text is split into tokens — roughly word pieces. Common words are one token; rarer words are broken into fragments. Each token is mapped to a vector of numbers, typically a few thousand dimensions. At this point the model still has no idea about order; it just has a bag of vectors.
Position is added separately, either by summing a positional vector or by rotating the vectors in a way that encodes distance (the rotary trick used in most current models). Without this step, “dog bites man” and “man bites dog” would look identical.
Attention is a lookup with weights
The central operation is self-attention. Every token produces three vectors: a query, a key and a value. A token’s query is compared with the keys of all tokens before it. The comparison produces a score, the scores are normalised into weights, and the weights are used to mix the values.
In plain language: each token asks “which earlier tokens matter to me right now?” and pulls in a blend of their information. The word it in a later sentence can attend to the noun it refers to. The word not can change how the next clause is read.
Models do this many times in parallel — multi-head attention. Different heads learn different relationships: syntax in one, coreference in another, punctuation or code structure in a third. Their outputs are concatenated and projected back into the model’s working width.
The rest of the block
After attention comes a small feed-forward network, applied to each position independently. It is usually a two-layer MLP that expands the vector, applies a non-linearity and projects it back. This is where a large share of the parameters live, and where much of the model’s factual recall is thought to sit.
Both attention and the MLP are wrapped in residual connections and normalisation. Residuals let gradients flow through dozens or hundreds of layers; normalisation keeps the activations in a sane range. A modern block is therefore: normalise, attend, add; normalise, MLP, add. Repeat.
From vectors to words
The final hidden state for each position is projected onto the vocabulary. Those scores — logits — become a probability distribution over the next token. Training adjusts the weights so that the correct next token gets a higher score. Generation samples from that distribution (or takes the most likely token) and feeds the result back in.
That is the whole trick: next-token prediction, repeated. Everything that looks like reasoning, translation or code generation is learned as a by-product of getting very good at this one objective.
Why context is expensive
Attention compares each token with the tokens before it. Double the context and the work grows roughly fourfold. During generation the model also caches the keys and values of earlier tokens — the KV cache — so it does not recompute them. That cache is memory, and long conversations can consume more memory than the weights themselves.
What this explains
- Why models hallucinate: the objective rewards plausible continuations, not verified facts.
- Why long prompts cost more: more tokens means more attention work and a larger cache.
- Why scale helps: more parameters and data let the model store finer distinctions and follow longer patterns.
- Why fine-tuning is cheap compared with pretraining: it adjusts an existing representation instead of building one from scratch.
The transformer is not magic. It is a very large, very parallel pattern machine. Knowing the shape of the machine makes the headlines about context windows, attention variants and inference costs much easier to read.