Let's build GPT from scratch, in code

Why it matters

The classic hands-on Transformer tutorial: Karpathy opens an empty file and, in under two hours, builds a working nano-GPT from character-level tokenization up. Self-attention, multi-head attention, residual connections and LayerNorm all become runnable PyTorch. Even without coding, watching it once demystifies what 'attention' actually computes — the single most effective video for de-magicking LLMs.

Key takeaways

• The core of GPT is one idea: attention lets tokens 'communicate' by relevance, then predict the next one.
• A full Transformer from scratch: embeddings → multi-head attention → feed-forward → residuals + LayerNorm → output head.
• Training is just sampling text batches, computing loss, backpropagating — all in a few hundred lines.
• The tiny Shakespeare-style model you train shows viscerally how 'intelligence grows out of prediction'.
• After this, the GPT-2/GPT-3 papers read like old friends.