CORTEX
warming up the attention heads…
01 · From text to vectors

Words become
numbers.

"How the weather today?" is split into tokens, each token maps to an integer ID, and each ID indexes into an embedding matrix — a continuous, 768-dimensional vector the transformer can actually compute with. The scatter below projects that 768d space down to 3 dimensions so nearby meanings can be seen as nearby points.

Drag to orbit · scroll to zoom · click a token
02 · Inside one decoder layer

Six stages,
one residual stream.

Stage 1 / 6

LayerNorm

03 · The core mechanism

Every token looks back
at what came before.

Step 1 / 7

Compute Q, K, V for each head


		

		

		

			
Compute mode

			
Recomputing K/V is wasteful

			

			

			

		

		

			
			
			

			
		

	


	
	

		

			
04 · From hidden state to next token

			
Logits become
a probability.

			

				lm_head projects the final hidden state into one logit per vocabulary
				token — 50,257 of them — then softmax turns those logits into probabilities that
				sum to 1. Shown here: the top 5 candidates.
			

			
The softmax curve is an illustrative squashing shape — real softmax is computed jointly over the whole vocabulary, not point-by-point.

		

		

		

	


	
	

		

			
05 · Training vs. inference

			
One number tells you
how wrong you were.

			

				During training, NLL measures how much probability the model assigned to the
				correct next token. Lower NLL means the model was more confident in the right
				answer.
			

			
Correct next token: "weather" (id 610)

		

		

			
Logits, log-probs, NLL

			

		

	


	
	

		

			
06 · Key terms

			
Six words worth
knowing.

			
Click any node for a concise definition and where it showed up in the tour.

			
Drag to orbit · click a term