2026-10-15 · neural networks · notes

Backprop in 90 lines: what micrograd taught me

Notes from Andrej Karpathy's Neural Networks: Zero to Hero. The first video builds an entire automatic-differentiation engine from scratch, and once you've done that, PyTorch stops being magic. Here's the engine, every piece explained, and then what each video in the series added.

Everything below is code I wrote following along and then rewrote from memory to make sure it stuck. Every number shown is the output of running it.

The idea in one sentence

A neural network is a big expression. Training it means nudging every parameter in the direction that makes the loss go down. The direction is the gradient. Backpropagation is just the chain rule applied to that expression, mechanically, from the output back to the inputs.

micrograd makes that literal: each number in the expression is an object that remembers how it was made, so the chain rule can walk backward through the history.

Part 1 — a Value that remembers its parents

class Value:
    def __init__(self, data, _children=(), _op=''):
        self.data = data          # the number
        self.grad = 0.0           # dL/d(this), filled in by backward()
        self._backward = lambda: None
        self._prev = set(_children)
        self._op = _op

Two fields matter. data is the forward value. grad will hold "how much does the final loss change if this number changes" — the derivative of the loss with respect to this node. It starts at zero because nothing has been computed yet.

Part 2 — each operation knows its own local derivative

This is the whole trick. When you build out = a + b, you also attach a tiny function to out that knows how to push a gradient back to a and b.

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), '+')
        def _backward():
            self.grad  += out.grad      # d(a+b)/da = 1
            other.grad += out.grad      # d(a+b)/db = 1
        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), '*')
        def _backward():
            self.grad  += other.data * out.grad   # d(a*b)/da = b
            other.grad += self.data  * out.grad   # d(a*b)/db = a
        out._backward = _backward
        return out

    def tanh(self):
        t = math.tanh(self.data)
        out = Value(t, (self,), 'tanh')
        def _backward():
            self.grad += (1 - t**2) * out.grad    # d tanh(x)/dx = 1 - tanh²(x)
        out._backward = _backward
        return out

    def __pow__(self, k):
        out = Value(self.data ** k, (self,), f'**{k}')
        def _backward():
            self.grad += k * self.data ** (k-1) * out.grad
        out._backward = _backward
        return out

Read the _backward closures as the chain rule: (local derivative) × (gradient flowing in from above). out.grad is the upstream gradient; the local derivative is whatever calculus says for that one operation. That's it. Addition routes the gradient through unchanged. Multiplication swaps in the other operand. tanh scales by 1 − t².

Part 3 — backward() walks the graph in the right order

You can only compute a node's gradient after every node that depends on it has computed theirs. That's a topological sort: list the nodes so every node comes after all of its children, then walk the list in reverse.

    def backward(self):
        order, seen = [], set()
        def visit(v):
            if v not in seen:
                seen.add(v)
                for c in v._prev:
                    visit(c)
                order.append(v)
        visit(self)
        self.grad = 1.0               # dL/dL = 1
        for v in reversed(order):
            v._backward()

Set the output's gradient to 1 (the loss with respect to itself), then call each node's _backward from the output down. Each call fills in its parents' grads, which are then ready when their turn comes.

Part 4 — check it by hand

a = Value(2.0); b = Value(-3.0); c = Value(10.0); f = Value(-2.0)
e = a * b          # -6
d = e + c          #  4
L = d * f          # -8
L.backward()

And the numeric check — nudge a by a tiny h and see how much L moves:

h = 1e-6
(L(a=2+h) - L(a=2)) / h    # → 6.0000

This check is the habit to keep. Any time you write a new _backward, compare it against the finite difference. It catches sign errors and forgotten terms instantly.

Part 5 — a single neuron

A neuron is tanh(Σ wᵢxᵢ + b). Nothing new; just the ops above composed.

x1, x2 = Value(2.0), Value(0.0)
w1, w2 = Value(-3.0), Value(1.0)
b = Value(6.8813735870195432)        # chosen so n = 0.8814 → tanh = 0.7071
n = x1*w1 + x2*w2 + b
o = n.tanh()
o.backward()
# o = 0.7071   w1.grad = 1.0   w2.grad = 0.0   x1.grad = -1.5

Walk w1.grad by hand: do/dn = 1 − 0.7071² = 0.5; dn/dw1 = x1 = 2; product 1.0. ✓ And w2.grad = 0 because x2 = 0 — a weight attached to a zero input gets no learning signal. That's a real thing you'll see in real networks.

Part 6 — layers, an MLP, and a training loop

class Neuron:
    def __init__(self, nin, nonlin=True):
        self.w = [Value(random.uniform(-1, 1)) for _ in range(nin)]
        self.b = Value(0.0)
        self.nonlin = nonlin
    def __call__(self, x):
        act = sum((wi*xi for wi, xi in zip(self.w, x)), self.b)
        return act.tanh() if self.nonlin else act
    def parameters(self):
        return self.w + [self.b]

class Layer:
    def __init__(self, nin, nout, **kw):
        self.neurons = [Neuron(nin, **kw) for _ in range(nout)]
    def __call__(self, x):
        out = [n(x) for n in self.neurons]
        return out[0] if len(out) == 1 else out
    def parameters(self):
        return [p for n in self.neurons for p in n.parameters()]

class MLP:
    def __init__(self, nin, nouts):
        sz = [nin] + nouts
        self.layers = [Layer(sz[i], sz[i+1], nonlin=(i != len(nouts)-1))
                       for i in range(len(nouts))]
    def __call__(self, x):
        for layer in self.layers:
            x = layer(x)
        return x
    def parameters(self):
        return [p for l in self.layers for p in l.parameters()]

A 3→4→4→1 network has 41 parameters. Train it on four examples with a squared-error loss:

model = MLP(3, [4, 4, 1])
xs = [[2.0, 3.0, -1.0], [3.0, -1.0, 0.5], [0.5, 1.0, 1.0], [1.0, 1.0, -1.0]]
ys = [1.0, -1.0, -1.0, 1.0]

for step in range(50):
    ypred = [model(x) for x in xs]                          # forward
    loss = sum((yp - yt)**2 for yp, yt in zip(ypred, ys))   # loss
    for p in model.parameters(): p.grad = 0.0               # zero grads  ← don't skip
    loss.backward()                                         # backward
    for p in model.parameters(): p.data -= 0.05 * p.grad    # step

Those five lines in the loop are the entire training procedure for every neural network you've heard of. PyTorch's loss.backward() and optimizer.step() are this, vectorized and on a GPU. The minus sign in the update is the part people forget: the gradient points uphill, so you step against it.

What stuck

• Backprop is bookkeeping. There's no insight inside it — just the chain rule, applied in reverse topological order. The insight is that it's cheap: one backward pass gives you the gradient with respect to every parameter at once.
• Gradients accumulate. +=, and zero them every step. Half of my early bugs were this.
• Check against finite differences. It's the unit test for calculus.
• The non-linearity is load-bearing. Without tanh, stacked layers collapse into one linear map — a bigger affine function, which can't bend.
• Learning rate is a real decision. Too small and nothing moves; too large and the loss bounces. Later videos spend serious time on this.

The rest of the series

Each video builds on the last. These are my notes on what each one added, not summaries of the videos.

Series: Andrej Karpathy, Neural Networks: Zero to Hero (YouTube). The engine above is my own implementation following the micrograd design; notes are mine.