Neural Network Playground

Build a network layer by layer and watch it learn a decision boundary in real time.

Intermediate interactive lab, about 20 minutes. Techniques: Backpropagation, Activations, Classification.

About

This playground trains a real neural network in your browser. Every number you see comes from actual forward passes and backpropagation over the points on screen, recomputed dozens of times a second. Nothing is pre-recorded.

The task is to learn a function of two coordinates. For the four classification datasets the network outputs the probability that a point is yellow rather than purple; the background shading is that probability evaluated at every location, so the boundary between the colours is where the model is exactly 50/50. For the two regression datasets it predicts a number between -1 and 1 instead.

The small squares in the diagram are the most useful thing on the page. Each one shows what a single neuron computes over the whole plane. Hover one to enlarge it in the output panel. You can watch how the first hidden layer only draws straight edges, and how deeper layers combine those edges into curves, rings and spirals.

What to try first

  1. Press Run on the circle. It should reach close to zero test loss in a few seconds.
  2. Switch to the spiral without changing anything. The small network struggles. Add layers and neurons until it copes.
  3. Raise the noise to 40% and the training split to 10%. Watch train and test loss drift apart.
  4. Open the Experiments tab for guided setups that each isolate one idea.

How it works

Training repeats four steps. Each pass over the whole training set is one epoch; within an epoch the points are shuffled and processed in mini-batches of the size you choose.

1. Forward pass

Every neuron takes a weighted sum of the previous layer's outputs, adds a bias, and applies its activation function. The output neuron uses a sigmoid for classification (so it reads as a probability) and no activation for regression.

aj(l) = f( Σi wji(l) ai(l-1) + bj(l) )

Layer l computes this for every neuron j, with activation f.

2. Loss

Classification uses binary cross-entropy, which heavily penalises confident mistakes: predicting 0.99 for a point that is actually class 0 costs about 4.6, while predicting 0.6 costs about 0.9. Regression uses half the squared error. The loss chart shows the mean over the training set (solid) and over the held-out test set (dashed).

L = -[ y log p + (1 - y) log(1 - p) ]

Binary cross-entropy for one point with label y and predicted probability p.

3. Backpropagation

The chain rule gives the gradient of the loss with respect to every weight in a single backward sweep. A neat fact makes the start easy: for sigmoid with cross-entropy, and for a linear output with squared error, the error signal at the output is simply p - y. Each earlier layer receives the error of the layer above, weighted by the connecting weights and scaled by its own activation's slope.

δ(l) = f′(z(l)) ⊙ (W(l+1))T δ(l+1), ∂L/∂W(l) = δ(l) (a(l-1))T

Error signal δ flowing back into layer l, and the resulting weight gradient.

4. Update

The optimiser moves each weight a little against its gradient. Plain SGD takes a step of learning rate times gradient. Momentum keeps a running velocity so it rolls through small bumps. Adam, the default here and the workhorse of modern deep learning, divides each step by a running estimate of that weight's gradient size, so every weight gets its own effective step length.

Race SGD, momentum and Adam on a loss surface in the Gradient Descent Lab

Activations and tuning

Activation functions

Without a non-linear activation, any stack of layers collapses into a single linear map, and the boundary is always a straight line. Set the activation to Linear and try the circle to see this for yourself.

TanhS-shaped, output in (-1, 1), centred on zero. Smooth boundaries. Saturates for large inputs, where its gradient vanishes.ReLUmax(0, z). Cheap and does not saturate for positive inputs, so deep networks train fast. Units that go negative everywhere stop learning ("dead" units).SigmoidOutput in (0, 1). Historically popular, but not zero-centred and saturates easily, so hidden layers learn slowly.LinearNo activation. Useful as a control experiment: depth adds nothing without a non-linearity.

Learning rate and batch size

The learning rate is the single most important knob. Too small and training crawls; too large and each step overshoots, so the loss curve jitters or explodes (the lab pauses and warns you if the loss becomes infinite). Good values depend on the optimiser: around 0.003 to 0.03 for Adam, and 0.03 to 0.3 for SGD here. Smaller batches give noisier but more frequent updates, which often helps escape poor regions.

Regularisation

Regularisation adds a penalty on large weights to the loss, trading a little training fit for simpler functions that generalise better. L2 adds λ·w²/2 per weight, shrinking every weight in proportion to its size, which smooths the boundary. L1 adds λ·|w|, a constant pull towards zero that drives unhelpful weights to exactly zero, so it performs feature selection. Biases are not penalised.

Feature engineering

The seven input squares are hand-made features. With X₁² and X₂² a circle becomes a linear problem; with X₁X₂ XOR does. Choosing good features lets a tiny model do what a deep one otherwise must learn. Deep learning's big shift was letting the hidden layers learn such features from raw inputs instead.

Experiments

Each setup isolates one idea. Loading one replaces the current settings and scrolls you back to the network; press Run (or Space) to start.

  1. 01

    A straight line cannot draw a circle

    No hidden layers and only X₁, X₂: this is logistic regression, and its boundary is a line. It plateaus near 50% accuracy.

    Now click the X₁² and X₂² input squares in the diagram. With squared features the same zero-hidden-layer model solves it at once, because a circle is linear in (x², y²).

  2. 02

    XOR, the problem that stalled the field

    A single hidden layer of 4 tanh units learns XOR from raw coordinates. Minsky and Papert showed in 1969 that a single-layer perceptron never can.

    Hover the hidden units: each one learns a tilted line, and the output combines them into the checkerboard.

  3. 03

    Depth for the spiral

    Three hidden layers (8, 8, 6) learn two interleaved spirals from raw X₁, X₂. It takes several hundred epochs; watch the boundary wind itself up.

    Hover units layer by layer: the first layer only knows lines, later layers build curves out of them.

  4. 04

    Overfitting in plain sight

    Very noisy XOR, only 10% of the points for training, every feature on and a large network. Training loss heads to zero while the dashed test loss climbs well above where it started.

    The boundary grows islands around individual noisy points. Now set regularisation to L2 with rate 0.03 and press Reset: training loss stays higher, but test loss drops several-fold.

  5. 05

    L1 finds the features that matter

    All seven features feed a small network on the circle, with L1 regularisation.

    Lines from unhelpful inputs such as sin X₁ and X₁X₂ fade to nothing as L1 pushes their weights to exactly zero. X₁² and X₂² survive.

  6. 06

    A learning rate that is too large

    Adam with learning rate 1 on the easy circle. The first few steps are so large that every tanh unit is slammed into saturation, where its gradient is almost zero.

    The loss freezes near 0.693, which is ln 2: the cost of answering 50/50 for every point. Set the rate to 0.01 and press Reset to see the same network solve it.

  7. 07

    Regression with ReLU

    Fit a smooth landscape of five bumps with ReLU units. The output is now a number, not a class, so the loss is squared error.

    ReLU networks are piecewise linear: look for faint straight creases in the heatmap where units switch on and off.

Related