Convolutional Networks — the Payoff of Structure

Python · NumPy · PyTorch · Fashion-MNIST

Why Images Are Different

The previous example ended on a loss: a plain network does not beat gradient boosting on tabular data, because tabular columns have no structure to exploit. Images are the opposite. A 28×28 image is not 784 unordered numbers — neighbouring pixels are strongly related, features are local, and the same object can appear anywhere in the frame. A convolutional network is built around exactly those facts, and this is where the section's argument has to pay off.

Three ideas do the work. Local receptive fields — each unit looks at a small patch rather than the whole image. Weight sharing — the same filter slides across every position, so a detector learned in one corner works everywhere. Translation equivariance — shift the input and the feature map shifts with it, so a shape need not be relearned in each location. The convolution itself is two loops, built from scratch here and matched to PyTorch's F.conv2d to 10⁻⁷ (float32 precision).

(f∗k)ij=∑u∑vf i+u,  j+v kuvone small filter k, applied at every position (i,j)(f * k)_{ij} = \sum_{u}\sum_{v} f_{\,i+u,\;j+v}\, k_{uv} \qquad \text{one small filter } k \text{, applied at every position } (i,j)

The Parameter Argument, Tested

The usual claim at this point is that CNNs win because they use fewer parameters. On the numbers here that is true but weak — 207k against the MLP's 235k, a 12% difference that depends entirely on a width chosen arbitrarily. It is also slightly misleading about the CNN itself: only 4,800 of its parameters, 2.3%, are convolutional. Sixteen shared 3×3 filters cost 160 weights; a dense layer over 784 pixels costs 200,704. Almost the whole parameter count is the fully-connected head.

Two extra fits make the stronger case. First, is the dense net simply too small? Halve it — 109k parameters instead of 235k — and accuracy moves from 0.8648 to 0.8749. So 125,760 extra dense weights buy +0.0101, while the CNN's 4,800 convolutional weights buy +0.0148 over that larger MLP. In absolute terms the two gains are comparable; per parameter they differ by a factor of about 38. Capacity is not what the dense network lacks — it lacks the assumption that nearby pixels belong together, and that assumption is nearly free to encode.

modelparameterstest accuracy
linear (softmax)7,8500.835
dense MLP, halved109,3860.865
dense MLP235,1460.875+126k weights buy +0.010
CNN206,922 (4,800 convolutional)0.8904.8k conv weights buy +0.015 — ~38× more per parameter
CNN, dense head removed (global average pooling)5,1300.729filters alone are not enough

Second, and as a counterweight: can the filters carry it alone? Replace the dense head with global average pooling — 5,130 parameters, almost all convolutional — and accuracy falls to 0.729. So the convolutions are not doing the whole job. They supply features the dense head could not have learned by itself, but the head is still what classifies. The defensible claim is not that structure replaces capacity, but that a small amount of well-directed computation makes ordinary capacity far more productive.

What It Learned, and How Well It Knows It

The diagnostics behave as they should. Errors concentrate exactly where a human would put them — Shirt is the hardest class at 0.54, trading mistakes with coat, pullover and T-shirt, while sneakers and bags are near-perfect. Calibration is measured rather than eyeballed: mean confidence 0.898 against actual accuracy 0.890, an expected calibration error of 0.0085 — essentially calibrated. That is worth flagging, because softmax networks are usually described as over-confident; that reputation comes from long training runs that drive the training loss toward zero, and after four epochs this one has not got there. The learned first-layer filters turn out to be edge and texture detectors — discovered by backpropagation, not designed.

Where this sits

The convolution's weight sharing across positions is the same idea the next example applies across time: a recurrent network shares weights along a sequence, the temporal analogue of spatial sharing. Attention later replaces both. And the calibration question opened here is answered properly in the section's capstone, where MC-dropout turns any of these networks into an approximate Bayesian model — the same uncertainty theme as BART and Gaussian Processes & Splines.

Notebook

Python The convolution built in two loops and validated against PyTorch, hand-designed edge filters versus learned ones, pooling, a full CNN on Fashion-MNIST, the parameter argument tested rather than asserted, and confusion/reliability diagnostics (Python) Open →

Data

Fashion-MNIST (Xiao, Rasul & Vollgraf, 2017) — 70,000 grayscale 28×28 images in 10 clothing classes, 60,000 train / 10,000 test. Downloaded automatically by torchvision when the notebook runs, so there is no data file to distribute.

References