LM HEAD
projecting to vocab…
01 · backbone → head → softmax

The last layer
gets the last word.

Every transformer ends the same way: a stack of blocks that understands the prompt, then one small linear layer — the LM head — that turns that understanding into a score for every word in the vocabulary. Walk the pipeline from tokens to probabilities.

Drag to orbit · scroll to zoom · click any part
· step 1 / 6


					

					

						
This model

						

					

				


				

					

						
02 · z = h · W

						
Sixteen scores,
one matmul.

						

							The head is a linear layer with no activation and usually no bias. Each column
							of its weight matrix is one word’s direction in hidden space, and each
							logit is one dot product. Slide along the columns and watch the multiply happen.
						

						
Bars below the line are negative logits · click a weight

					

					

						
 · step 1 / 6

						

						

						

					

					

						
This projection

						

					

				


				

					

						
03 · p = softmax(z / T)

						
Raw scores become
an honest bet.

						

							Logits are unbounded and mean nothing on their own. Softmax shifts, exponentiates
							and normalises them into a probability for every word — and one temperature
							knob decides whether the model plays it safe or takes risks.
						

						
Same 16 logits on every step · click a bar

					

					

						
 · step 1 / 6

						

						

						

					

					

						
This distribution

						

					

				


				

					

						
04 · d_model × vocab_size

						
A “small head” that is
the biggest matrix.

						

							Two numbers fix the head’s size: the backbone’s width and the
							tokeniser’s vocabulary. In GPT-2 their product is 31% of the whole model
							— which is why heads get tied to embeddings, and why “head”
							really means “swappable last layer”.
						

						
Frames are log scale · click a model

					

					

						
 · step 1 / 6

						

						

						

					

					

						
This head