A recurrent neural network processes a sequence one step at a time. After each step it passes a hidden state forward, so what it saw earlier can affect what it does next.
That loop was the first workable answer to sequence data, and it is also the source of its problems. Processing must happen in order, and information from far back fades as it is passed along.
Think of it like this. Think of reading a novel while only being allowed to keep one page of notes. Each page you read updates the notes. By chapter thirty, whatever happened in chapter one has been overwritten many times.
How it works
- One step at a time. Each input is processed in order.
- A hidden state carries forward. It is the network's running summary of everything so far.
- The same weights are reused at every step, which is what makes it recurrent.
- Training unrolls the loop. Backpropagation through time treats the sequence as one very deep network.
Trade-offs
- Sequential by nature. Step ten needs step nine finished, so training cannot parallelise the way transformers can. This is why they lost.
- Vanishing gradients. Signal from far back shrinks as it propagates, so long-range dependencies are weak.
- Compact. They use far fewer parameters than a transformer, which still matters on small devices.
Common mistakes
- "RNNs remember everything." The hidden state is a fixed-size summary that gets overwritten continuously.
- "An RNN is an LSTM." LSTM is a gated variant built specifically to fix the plain RNN's memory problem.
- "They process in parallel." They cannot. Each step depends on the previous one.
