Transcription
The hyperbolic tangent, its formula maps any input to a value between minus one and plus one with a smooth S-shaped curve passing through the origin. That a zero-centered output is actually a nice property, but here's the problem.
Look at the derivative or in the tails, it flattens to nearly zero. And when gradients are that small, they effectively vanish as they propagate backward through layers. This is the vanishing gradient problem.
Now, ReLU takes a radically simpler approach. It just returns zero for negative inputs and the input itself for positive ones. Compare it with tanh and the difference is striking. For positive values, the derivative is a constant one. No shrinking, no saturation.
But there's a catch. If a neuron's input is always negative, it outputs zero forever. It's dead. This is the dying ReLU problem.
Now, let's see why this matters in practice. Here's what happens when gradients flow backward through five layers. With tanh, the gradient shrinks at every step until there's almost nothing left. The early layers barely learn.
But with ReLU, the gradient stays constant. Each layer gets the same learning signal. This is exactly why ReLU became the default activation for hidden layers.