Model Compression Lab

About

A trained network is a pile of numbers, and most of them are stored far more precisely than they need to be. Model compression asks how far you can shrink that pile before the model's behaviour changes, and which kind of shrinking costs the least.

This lab trains a three-hidden-layer network (2 inputs, three layers of 32 tanh units, 3 outputs: 2,307 parameters, about 9 KB in 32-bit floats) to separate three classes in the plane. Then you compress it four ways: quantization (fewer bits per weight), low-rank factorization (replace a matrix by a product of two thin ones), pruning (delete small weights) and distillation (train a much smaller student to copy the teacher).

Watch two things. The decision maps show where the compressed model changes its mind (turn on disagreements to see it in red). The Pareto chart records every configuration you try, so you can see which method buys the most accuracy per byte. The lesson Fine-Tuning and Efficient Models explains how the same ideas apply to models with billions of parameters.

How it works

The teacher

A 2-32-32-32-3 multilayer perceptron with tanh activations and a softmax output, trained full-batch with Adam on 360 labelled points for 900 epochs. Test accuracy is measured on 3,000 fresh points from the same distribution. Agreement is the share of test points where the compressed model predicts the same class as the original, which matters more than accuracy when you want a drop-in replacement.

The pipeline

Enabled methods are applied in a fixed order, the same order most real toolchains use:

1. Low-rank: each 32×32 hidden matrix is replaced by its best rank-r approximation from the SVD, stored as two factors of size 32×r and r×32.
2. Pruning: weights with the smallest magnitudes are set to zero, across all prunable tensors (global) or equally in each (per layer).
3. Quantization: each stored tensor is rounded onto a uniform grid of 2^b levels (or to IEEE half precision at 16 bits).

There is no retraining after compression. Real pipelines usually fine-tune for a while afterwards (quantization-aware training, or retraining a pruned network with its mask fixed), which recovers much of the lost accuracy. What you see here is the harsher, post-training case.

Quantization

Uniform quantization stores each weight as a small integer q and one shared scale s per group, so the weight is reconstructed as w ≈ s · q (symmetric) or w ≈ s · (q − z) (asymmetric, with integer zero point z).

Symmetric or asymmetric

Symmetric uses the range [−max|w|, +max|w|] and 2^b−1−1 levels on each side, so zero is exactly representable but one code is wasted. At 2 bits that leaves only −s, 0 and +s: ternary weights. Asymmetric fits [min, max] with all 2^b levels, which helps when a tensor is lopsided, at the cost of storing a zero point.

Per-tensor or per-channel

One scale for a whole matrix means a single large weight stretches the grid for everyone, and small weights round to zero. A scale per output channel (row) adapts to each neuron's own range. The overhead is one 16-bit number per row, which is why per-channel wins decisively at 3 bits and below in this lab. LLM quantizers go further and use groups of 32 to 128 weights, each with its own scale.

What is lost

The histogram shows it directly: a smooth bell of weights collapses onto a few spikes. The rounding error per weight is at most half a step, so it shrinks by half with every extra bit (about 6 dB of signal-to-quantization-noise ratio per bit). Whether the model survives depends on how sensitive its decision boundary is to that noise.

Low-rank

Any matrix W can be written as a sum of rank-one pieces, W = Σ σᵢ uᵢ vᵢᵀ, sorted by the singular values σᵢ. Keeping the first r terms gives the best possible rank-r approximation in the least-squares sense (the Eckart–Young theorem), with squared error equal to the sum of the dropped σᵢ².

The spectrum chart shows how quickly the singular values fall. If a few are large and the rest small, the matrix is nearly low-rank and can be compressed cheaply. Trained weight matrices are usually somewhere in between: their spectra decay, but not to zero.

The connection to LoRA

LoRA uses the same shape for a different purpose. Instead of compressing W, it freezes W and learns a low-rank update BA on top of it during fine-tuning, because updates for a new task turn out to need far fewer directions than the weights themselves. The lesson shows that idea with a live SVD.

Pruning

Magnitude pruning removes the weights closest to zero on the theory that they matter least. It is crude but remarkably effective, especially when the network is retrained afterwards, as Han and colleagues showed in 2015 by pruning about 90% of the weights of AlexNet and VGG-16 without losing accuracy.

Here there is no retraining, so you see the raw damage. Two choices matter. Global pruning applies one threshold everywhere, so layers with many small weights lose more; per-layer removes the same fraction from each. And the first and last layers are tiny but every input and output flows through them: include them and watch accuracy fall off a cliff.

Note the storage rule: below roughly 1/b sparsity the one-bit mask costs more than it saves, so the lab keeps dense storage until pruning is deep enough to pay for itself.

Distillation

Distillation does not shrink the teacher. It trains a separate, much smaller student (here 2-h-3, with h = 4, 8 or 16 hidden units) to match the teacher's full probability distribution, not just the correct label. Those soft targets say which wrong answers are nearly right, which is information one-hot labels throw away.

The lab trains label-trained and distilled students in pairs from the same three random starts, for the same number of steps, and reports the mean, because a network this small lands in noticeably different places depending on where it starts. The most reliable gain comes from the transfer set: the teacher can label as many extra unlabelled points as you like, while the label-trained student only ever sees the original 360. That is how distillation is used in practice, and it is why large models are so often used to generate training data for small ones.

Be honest about what you see. On easy datasets both students do equally well. On the spiral, low temperatures (1 to 2) work best: with only three classes there is little “dark knowledge” in the wrong answers, and high temperatures flatten the targets into mush that a tiny student cannot use. Hinton and colleagues saw the same effect on MNIST: students with 300 or more units per layer did well at any temperature above 8, but when they shrank the student to 30 units, temperatures between 2.5 and 4 worked significantly better than higher or lower ones.

Try this

1. 01Find the cliffTurn on quantization and step the bits down from 16. Accuracy barely moves until a threshold, then collapses. Where is it for spiral?
2. 02Per-channel rescueAt 3 bits, switch per-tensor to per-channel. How much accuracy comes back, and how many bytes did the extra scales cost?
3. 03Rank budgetTurn on low-rank and slide the rank down. Which rank keeps 95% accuracy? Look at the spectrum: how much energy is kept at that point?
4. 04Protect the endsPrune 70% globally, then include the first and last layers. Why does a handful of weights in the first layer matter so much?
5. 05Stack themCombine rank 8, 50% pruning and 4-bit per-channel quantization. Is the combined model on the Pareto frontier?
6. 06Student versus compressed teacherTrain a distilled student with 8 hidden units and the transfer set. Compare its point on the chart with the best compressed teacher of similar size.
7. 07Easy data forgivesSwitch to blobs and repeat the 3-bit test. Simple boundaries survive far more compression than the spiral.
8. 08Where they disagreeTurn on disagreements. Compression damage appears first near the class boundaries and in regions with no training data.