Transfer Learning

Transfer learning starts from a model already trained on a large general dataset and adapts it to your specific task. You inherit everything it learned and only teach it the difference.

It works because early layers learn general things. Edges, textures, grammar. Those transfer across tasks, so you only need to retrain the parts that are specific to yours.

Think of it like this. Think of hiring an experienced engineer rather than training someone from nothing. They already know how to program. What they need from you is your codebase and your conventions.

How it works

• Start from pretrained weights rather than random ones.
• Keep the early layers, which learned general features.
• Replace the final layer with one shaped for your task.
• Optionally freeze the base and train only the new layer, which is fast and needs very little data.
• Or unfreeze and fine-tune everything gently, at a much lower learning rate.

When to use it

• Almost always. Training vision or language from scratch is rarely justified now.
• Especially with little data. Hundreds of examples can be enough where scratch training would need hundreds of thousands.
• Less useful when your domain is unlike the pretraining data. Medical scans transfer less well from photographs than photographs do.

Common mistakes

• "Use the same learning rate as scratch training." Large steps destroy pretrained knowledge. Go ten to a hundred times lower.
• "Always fine-tune everything." With very little data, freezing the base and training only the head overfits far less.
• "Any pretrained model will do." The closer the pretraining domain is to yours, the more transfers.