Let's build GPT from scratch, in code

Why it matters

The classic hands-on Transformer tutorial: Karpathy opens an empty file and builds a working nano-GPT in under two hours. Self-attention, multi-head attention, residual connections and LayerNorm become runnable PyTorch. The single most effective video for de-magicking LLMs.

Key takeaways

• Core of GPT: attention lets tokens 'communicate' by relevance, then predict the next one.
• Full Transformer from scratch: embeddings → multi-head attention → feed-forward → residuals + LayerNorm → output head.
• Training: sample text batches → compute loss → backprop — all in a few hundred lines.
• The tiny Shakespeare-style model shows viscerally how intelligence grows out of prediction.
• After this, the GPT-2/GPT-3 papers read like old friends.