Building makemore Part 4: Becoming a Backprop Ninja

Name: Building makemore Part 4: Becoming a Backprop Ninja
Uploaded: 2022-10-11T17:56:19.000Z
Duration: 1 h 55 min 24 s
Channel: Andrej Karpathy
Description: Tutorial on manually implementing backpropagation in neural networks, working through gradients step-by-step in a 2-layer MLP with BatchNorm without using autograd. Hands-on exercise focused on deep understanding of optimization and gradient flow.

Andrej Karpathy16 June 2026Watch on YouTube

Part of series

Ep. 2 · Building Makemore Part

View the series

What you'll learn

You learn to implement backpropagation manually by fully backpropagating through a 2-layer MLP with BatchNorm without autograd
You understand how gradients flow backwards step-by-step through cross-entropy loss, linear layers, activation functions, and BatchNorm
You develop intuition about gradient flow at the tensor level rather than individual scalars, helping you debug and optimize neural networks more effectively

Frequently asked questions

Why is it important to implement backpropagation manually instead of using autograd?

Manual implementation gives you deep understanding of how gradients flow through the network and how optimization works. This helps you modify and debug modern neural networks more confidently.

Which components of the network are backpropagated step-by-step in this tutorial?

The tutorial covers backpropagation through cross-entropy loss, the second linear layer, tanh activation, BatchNorm, the first linear layer, and the embedding table.

Is this a video you can watch passively or do you need to actively participate?

This is not a passive tutorial. You are recommended to work through the exercises yourself and pause the video when stuck to see the answers provided.

Topics

backpropagation neural networks deep learning fundamentals education

Description from the channel

We take the 2-layer MLP (with BatchNorm) from the previous video and backpropagate through it manually without using PyTorch autograd's loss.backward(): through the cross entropy loss, 2nd linear layer, tanh, batchnorm, 1st linear layer, and the embedding table. Along the way, we get a strong intuitive understanding about how gradients flow backwards through the compute graph and on the level of efficient Tensors, not just individual scalars like in micrograd. This helps build competence and intuition around how neural nets are optimized and sets you up to more confidently innovate on and debug modern neural networks. !!!!!!!!!!!! I recommend you work through the exercise yourself but work with it in tandem and whenever you are stuck unpause the video and see me give away the answer. This video is not super intended to be simply watched. The exercise is here: https://colab.research.google.com/drive/1WV2oi2fh9XXyldh02wupFQX0wh5ZC-z-?usp=sharing !!!!!!!!!!!! Links: - makemore on github: https://github.com/karpathy/makemore - jupyter notebook I built in this video: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part4_backprop.ipynb - collab notebook: https://colab.research.google.com/drive/1WV2oi2fh9XXyldh02wupFQX0wh5ZC-z-?usp=sharing - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - our Discord channel: https://discord.gg/3zy8kqD9Cp Supplementary links: - Yes you should understand backprop: https://karpathy.medium.com/yes-you-should-understand-backprop-e2f06eab496b - BatchNorm paper: https://arxiv.org/abs/1502.03167 - Bessel’s Correction: http://math.oxford.emory.edu/site/math117/besselCorrection/ - Bengio et al. 2003 MLP LM https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf Chapters: 00:00:00 intro: why you should care & fun history 00:07:26 starter code 00:13:01 exercise 1: backproping the atomic compute graph 01:05:17 brief digression: bessel’s correction in batchnorm 01:26:31 exercise 2: cross entropy loss backward pass 01:36:37 exercise 3: batch norm layer backward pass 01:50:02 exercise 4: putting it all together 01:54:24 outro