← All examples

Machine Learning: Neural Networks & Deep Learning

Everything in this section is one idea repeated: a network of simple units, trained by backpropagation. Convolutional nets, LSTMs and transformers are not different algorithms so much as different inductive biases bolted onto that same core — assumptions about what structure the data has, built into the architecture. So the first example builds the core by hand, derives the backward pass, and proves it correct against finite differences before any framework appears.

The section opens with a result that sets up everything after it. On the tabular data that runs through the rest of the collection, the neural network loses to gradient boosting — comfortably clearing a linear baseline, tying the random forest, and never catching XGBoost. That is not a tuning failure, it is a well-replicated regularity: tabular columns have no spatial or temporal order for a network's biases to exploit, and permuting the features changes nothing. Deep learning earns its keep precisely where structure exists to be assumed — adjacency in images, ordering in sequences, long-range dependence in language — which is what each subsequent example supplies.

A second thread runs underneath: what a framework actually buys you. Matched on optimiser and initialisation, hand-written backpropagation and PyTorch's autograd agree to 0.001 AUC. The gap you see otherwise is settings, not mathematics — and it decomposes into pieces worth knowing the size of.

The training loop, and the words for its parts. Every model below is trained the same way, and the vocabulary is worth fixing once. A forward pass sends inputs through the network to a prediction, applying an activation function at each layer — almost always the ReLU, which simply zeroes negatives and passes positives through, and whose only real virtue is that its derivative is 0 or 1 and so neither shrinks nor explodes. A loss scores that prediction: cross-entropy for classification, which is the negative log-likelihood of the correct class and therefore the same objective a logistic regression maximises. Backpropagation is the chain rule applied backwards through the layers to get the gradient of that loss with respect to every weight, and an optimiser takes a step against it — SGD a fixed fraction of the gradient, Adam a per-parameter adaptive one — scaled by the learning rate. That step is taken not on the whole dataset but on a minibatch of a few dozen rows, and one sweep through all the batches is an epoch. Nothing in the section is more complicated than that loop; the architectures differ only in what sits between input and loss.

Initialisation is not a detail. The weights have to start somewhere, and where they start changes where they end up. Xavier and He initialisation both draw the starting weights with a variance set by the number of inputs to each layer, so that the signal neither dies out nor blows up as it passes through — He is the version tuned for ReLUs. This is usually filed as an implementation footnote, and the first example below shows why it should not be: the gap between the from-scratch network and PyTorch decomposes into +0.0098 from initialisation against +0.0071 from the optimiser. The starting point mattered slightly more than the algorithm. What a framework really supplies is not better gradients — hand-written backpropagation and autograd agree here to 0.0011 — but defaults someone else has already tuned.

What each architecture assumes. The three middle examples are not three unrelated designs but three assumptions about the data, each encoded in the wiring. A convolution assumes that a pattern means the same thing wherever it appears, so it slides one small filter across the whole image and reuses those weights everywhere — weight sharing — with pooling then shrinking the map so later layers see a coarser, wider view. A recurrent network assumes order matters and carries a state forward one step at a time, which creates its characteristic failure: gradients multiplied through many steps shrink toward nothing, the vanishing gradient problem, and long-range information is lost. The LSTM’s gates are learned switches that let information pass through unchanged, which is how it keeps a memory alive. Attention abandons recurrence entirely: every position reads every other in a single step, so nothing has to survive a long chain — at the cost of having no idea what order anything came in, which is why a positional encoding has to be added back by hand.

What the network does not know. A trained network returns one number and no sense of its own reliability. The Bayesian treatment puts a distribution over the weights instead of a point estimate, but the exact posterior is hopeless at this scale, so it is approximated by variational inference: propose a simple family of distributions and pick the member closest to the true posterior, where closeness is KL divergence. Because the KL cannot be computed directly either, what is actually maximised is the ELBO — the evidence lower bound, which is the fit to the data minus a penalty for straying from the prior, and which differs from the quantity of interest by exactly the KL gap. The cheap alternative is Monte Carlo dropout: leave dropout switched on at prediction time and treat the spread across repeated forward passes as uncertainty, which is approximate variational inference in disguise and costs one line of code.