Model interpretability is being able to understand why a model produced a particular output. It ranges from models that are readable by design to techniques that probe a black box after the fact.
It matters wherever a decision has to be justified. A loan refusal, a medical flag, a rejected claim: accuracy alone is not enough when someone is entitled to an explanation.
Think of it like this. Think of a doctor who is right almost every time and cannot say why. You would trust the record and still want reasoning before major surgery, because reasoning is what lets you catch the case where they are wrong.
Types
- Interpretable by design. Linear models and shallow decision trees, where the logic is readable directly.
- Feature importance. Which inputs mattered most overall.
- SHAP and LIME. Per-prediction explanations, showing what pushed this specific decision each way.
- Attention and saliency maps. Which parts of an input the model focused on, common in vision and language.
Trade-offs
- Accuracy against transparency. The most accurate model is often the least readable. Sometimes a slightly worse interpretable model is the right call.
- Explanations can mislead. Post-hoc methods approximate the model. A plausible explanation is not proof it is the real reason.
- Cost. SHAP on a large model is computationally expensive.
Common mistakes
- "Feature importance explains a prediction." It describes the model overall. For one decision you need a per-prediction method.
- "The explanation is the reason." Post-hoc explanations approximate. Two methods can disagree about the same prediction.
- "Attention shows what the model used." Attention weights show where it looked, which is not the same as what determined the output.
- "Only regulated industries need it." Interpretability is also how you find out your model learned a shortcut instead of the task.
