Deep learning is machine learning that uses neural networks with many layers. The word deep refers to the number of layers, nothing more.
What those layers buy you is feature discovery. Older approaches needed a human to decide what mattered in the data. A deep network works that out for itself, given enough examples.
Think of it like this. Think of learning to recognise faces. You were never handed a list of measurements to check. You saw thousands of faces and your sense of what makes one recognisable formed on its own.
How it works
- Layers build on each other. Early layers find edges and simple shapes, later ones find eyes, then faces.
- Features are learned, not specified. Nobody writes the rule for what an eye looks like.
- Backpropagation assigns blame. The error at the output is traced back through every layer to work out which weights caused it.
- Scale is the price. Deep networks need far more data and compute than the methods they replaced.
Trade-offs
- Automatic features, at a cost. You save the feature engineering and you pay in data, compute, and training time.
- Hard to interpret. A model with billions of weights cannot easily tell you why it decided something, which matters in regulated settings.
- Not always the right tool. On small tabular datasets, gradient boosted trees routinely beat deep learning and train in seconds.
Common mistakes
- "Deep learning and neural networks are the same thing." The network is the structure. Deep learning is what you get when it has many layers and a lot of data.
- "Deeper is always better." Past a point extra layers add cost and overfitting rather than accuracy.
- "It is the answer to every problem." For small structured datasets, simpler models usually win.
