An activation function applies a nonlinear transformation to a neuron's input before passing it forward, giving neural networks the ability to learn patterns that linear models cannot represent.
An activation function is a mathematical operation applied to each neuron's weighted sum. Without it, every neural network layer would collapse to a single linear transformation, no matter how deep. Common functions include sigmoid, tanh, ReLU, and GELU.
That nonlinearity is what separates neural networks from stacked linear regression. Each nonlinear layer adds bends to the decision boundary, and enough bends can approximate arbitrarily complex shapes. The choice of activation affects gradient flow, training speed, and the kinds of patterns the network can learn reliably.
Think of it like this. Think of a light dimmer instead of an on-off switch. A linear neuron can only pass full brightness or none. An activation function shades the signal, letting the network express nuance.
Each neuron computes a weighted sum of its inputs plus a bias. The activation function transforms that sum into the neuron's output. ReLU zeros out negative values, sigmoid squashes values between 0 and 1, tanh centers outputs around zero, and GELU smooths the ReLU for transformer layers.
"More nonlinearities is always better." Too many or poorly chosen activations can slow or stall training. "ReLU is always the best choice." ReLU dominates many tasks, but sigmoid and tanh remain useful for gates, RNNs, and bounded outputs. "Activation functions are only about output range." They also affect gradient magnitude and sparsity.
Essential for expressive models, but some activations cause vanishing or exploding gradients. ReLU is fast and sparse but can deaden; sigmoid and tanh are smooth but compress gradients; GELU improves transformer training at slightly higher compute cost.