Dictionary of Applied Machine Learning

interpretability

Typeset PDF version — the authoritative form of this entry

The interpretability of a machine learning (ML) method is the extent to which a human user can comprehend its behavior. Interpretability is closely related to the notion of simulatability: the user should be able to anticipate the predictions of the method on an arbitrary test set. Interpretability differs from explainability, which concerns understanding specific predictions with the help of provided explanations.

Definition

Interpretability is often defined, informally, as the ability to explain, or to present in understandable terms, to a human (Doshi-Velez and Kim, 2017). The international terminology standard for artificial intelligence (AI) does not define interpretability (Standardization and Commission, 2022). Its closest notion is predictability, the property of an artificial intelligence system (AI system) that enables reliable assumptions by users about the predictions delivered by a machine learning (ML) method (Standardization and Commission, 2022, Sect. 3.5.8 and 5.15.7).

An ML method is interpretable for a human user if they can comprehend the computational process of the method. Interpretability requires simulatability which is the ability of a human to mentally simulate the behavior of the ML method (Chen et al., 2018; Colin et al., 2022; Doshi-Velez and Kim, 2017; Hase and Bansal, 2020; Lipton, 2018). Simulatability is a stronger requirement compared to predictability as it covers not only the input-output behavior but also the internal computational processes of an AI system (Standardization and Commission, 2022, Sect. 5.15.7).

Fig. 1 depicts a training set and a test set. The training set is used by two different methods that learn hypotheses $\learnthypothesis$ and $\learnthypothesis'$. The ML method producing the hypothesis $\learnthypothesis$ is interpretable to a human user familiar with the concept of a linear map. Since $\learnthypothesis$ is a linear map, the user can anticipate the predictions of $\learnthypothesis$ on the test set. In contrast, the ML method producing $\learnthypothesis'$ is less interpretable: the behavior of $\learnthypothesis'$ deviates from the user's expectations.

Figure 1 of the entry interpretability
Figure 1: Interpretability via simulatability: a human user provides pseudo-labels (squares) for the test set $\dataset'$ by extrapolating the linear trend of the training set $\dataset$ (circles). The predictions of the hypothesis $\learnthypothesis$ agree with these pseudo-labels, while the predictions of another hypothesis $\learnthypothesis'$ deviate from them
The AI Risk Management Framework of the US National Institute of Standards and Technology distinguishes interpretability from explainability (Standards and Technology, 2023), the property pursued by explainable artificial intelligence (XAI) (Gunning and Aha, 2019). Both notions concern the extent to which a human user can understand an ML method. In contrast to interpretability, explainability requires that an explanation is provided along with each prediction. These explanations may take the form of saliency maps or reference examples from the training set. A challenge for XAI methods is that a computed explanation can be unfaithful to the learned hypothesis and thereby mislead the user (Rudin, 2019).

Interpretability can also be obtained by decomposing a learned hypothesis into components that are themselves interpretable. This property is called decomposability (Lipton, 2018). The hypothesis $\learnthypothesis$ in Fig. 1 decomposes into a slope and an intercept: the slope states how the prediction changes with the feature. A trained deep net lacks such a decomposition, since the activation of an individual neuron typically does not correspond to a human-interpretable concept. Mechanistic interpretability aims to recover interpretable components from a trained deep net (Olah et al., 2020; Elhage et al., 2021) by decomposition techniques such as sparse coding of activations (Cunningham et al., 2024).

See also: explainability, trustworthy artificial intelligence, regularization, local interpretable model-agnostic explanations, interpretable machine learning, mechanistic interpretability, transparency.

References

  1. Doshi-Velez and Kim (2017). Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608.
  2. Standardization and Commission (2022). ISO/IEC 22989:2022 --- Information technology --- Artificial intelligence --- Artificial intelligence concepts and terminology.
  3. Chen et al. (2018). Learning to Explain: An Information-Theoretic Perspective on Model Interpretation. Proc. 35th Int. Conf. Mach. Learn..
  4. Colin et al. (2022). What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods. Adv. Neural Inf. Process. Syst..
  5. Hase and Bansal (2020). Evaluating explainable AI: Which algorithmic explanations help users predict model behavior?. Proc. 58th Annu. Meeting Assoc. Comput. Linguistics.
  6. Lipton (2018). The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue.
  7. Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0).
  8. Gunning and Aha (2019). DARPA's Explainable Artificial Intelligence (XAI) Program. AI Magazine.
  9. Rudin (2019). Stop explaining black box machine learning models for high-stakes decisions and use interpretable models instead. Nature Mach. Intell..
  10. Olah et al. (2020). Zoom In: An Introduction to Circuits. Distill.
  11. Elhage et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread.
  12. Cunningham et al. (2024). Sparse Autoencoders Find Highly Interpretable Features in Language Models. 12th Int. Conf. Learn. Representations.

Cite this entry

@misc{dictml_interpretability,
  author = {Jung, Alexander},
  title = {interpretability},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-08-06},
  url = {https://dictionaryofml.org/terms/interpretability.html}
}