LearnDeep learning
Activation functions
Compare the small curves that let neural networks bend, and see why some of them make deep networks hard to train.
Drag along the chart to move the input z. The dashed line is the slope at that point; the chart in the panel shows how much learning signal survives the stack of layers.
How it works
The bend that makes depth useful.
Without activation functions, a stack of layers would collapse into a single straight line, because lines of lines are still lines. Each function bends its input: sigmoid and tanh squash it into a fixed range, ReLU keeps positive values and zeroes negative ones, and GELU is a smooth version of ReLU used in transformers.
During training, the learning signal travels backwards and is multiplied by the slope of the activation at every layer. Here every stacked layer is assumed to have a weight of 1 and the same input, so the signal after L layers is f′(z) to the power L. The slope of sigmoid is never more than 0.25, so the signal shrinks fast: the vanishing gradient problem.
ReLU has a slope of exactly 1 for positive inputs, which is a big reason deep networks became trainable. Its catch is a slope of 0 for negative inputs, where a neuron can stop learning; Leaky ReLU keeps a small slope there (0.1 in this simulation).