Long Short-Term Memory

An LSTM is a recurrent network with gates that decide what to keep, what to discard, and what to pass on. The gates are what give it usable memory across long sequences.

A plain RNN overwrites its memory at every step. An LSTM maintains a separate cell state that information can travel along largely untouched, so something from step one can still matter at step five hundred.

Think of it like this. Think of a notebook alongside your working memory. Most of what happens gets forgotten, and you deliberately write down the few things worth keeping. The gates are the decision about what gets written.

How it works

• A cell state runs through the whole sequence, modified only where a gate allows it.
• The forget gate decides what to drop from that state.
• The input gate decides what new information to add.
• The output gate decides what to expose at this step.
• Gradients survive the journey, because the cell state offers a path that does not repeatedly shrink them.

Common mistakes

• "LSTMs solved long-range memory." They improved it enormously. Very long sequences still degrade.
• "LSTMs always beat GRUs." GRUs use two gates instead of three and often match them, while training faster.
• "They are obsolete." For long sequences in language, transformers won. For small models, streaming, and modest time series, LSTMs remain a sensible choice.