Dictionary of Applied Machine Learning

interpretability

Updated on 2026-09-16

Typeset PDF version — the authoritative form of this entry

Python demo — a script that recomputes what this entry states and prints one line per check

The interpretability of a machine learning (ML) method is the extent to which a human user can comprehend the computational process it carries out. Interpretability is closely related to predictability: the user should be able to anticipate the predictions of the method on a test set. Interpretability differs from explainability, which concerns understanding specific predictions with the help of provided explanations.

Definition

P-costConsider a training set of monthly sales and a straight line fitted through it. Reading next month's prediction off that line takes a slope, an intercept, and one multiplication, and a user can carry the computation out on paper. The same prediction delivered by a deep net arrives with no account a user could follow. Interpretability is what separates the two.

A machine learning (ML) method is interpretable for a human user if they can comprehend the computational process of the method. The international terminology standard for artificial intelligence (AI) does not define interpretability (Standardization and Commission, 2022). Its closest notion is predictability, the property of an artificial intelligence system (AI system) that enables reliable assumptions by users about the predictions delivered by an ML method (Standardization and Commission, 2022, Sect. 3.5.8 and 5.15.7).

P-weakerComprehension of a computation cannot be observed directly, so interpretability is judged through predictability: the user anticipates the predictions the method delivers, and the evaluations proposed in the literature measure how often those anticipations are right (Chen et al., 2018; Colin et al., 2022; Doshi-Velez and Kim, 2017; Hase and Bansal, 2020). Predictability is weaker than interpretability: a method whose computation the user can follow is predictable to that user, but a method whose predictions the user anticipates need not be one whose computation they could carry out. The terminology standard makes the same point: a user may rely on the predictions of an AI system without being able to comprehend how the method produced them (Standardization and Commission, 2022, Sect. 5.15.7).

P-decompFig. 1 depicts a test of predictability. A training set is used by two different methods that learn hypotheses $\learnthypothesis$ and $\learnthypothesis'$. The user anticipates the predictions on a test set by visually extrapolating the linear trend of the training set. The predictions of $\learnthypothesis$ agree with these anticipations: the method producing $\learnthypothesis$ is predictable to the user. The predictions of $\learnthypothesis'$ deviate from them: the method producing $\learnthypothesis'$ fails the test of predictability and therefore cannot be one whose computation the user comprehends. The agreement alone does not establish interpretability, since the user anticipated the predictions of $\learnthypothesis$ without carrying out its computation. The method producing $\learnthypothesis$ is nevertheless also interpretable to a user familiar with the concept of a linear map: such a user could comprehend the computation of $\learnthypothesis$.

Figure 1 of the entry interpretability
Figure 1: A human user provides pseudo-labels (filled red squares) for the test set $\testset$ by visually extrapolating the linear trend of the training set $\trainset$ (filled blue circles). The predictions of the hypothesis $\learnthypothesis$ agree with these pseudo-labels, while the predictions of another hypothesis $\learnthypothesis'$ deviate from them. The user anticipates the predictions here without carrying out the computation (a slope, an intercept, and one multiplication), which comprehending the method would additionally require
Interpretability can also be obtained by decomposing a learned hypothesis into components that are themselves interpretable (Lipton, 2018). The hypothesis $\learnthypothesis$ in Fig. 1 decomposes into a slope and an intercept: the slope states how the prediction changes with the feature. A trained deep net lacks such a decomposition, since the activation of an individual neuron typically does not correspond to a human-interpretable concept. Mechanistic interpretability aims to recover interpretable components from a trained deep net (Elhage et al., 2021; Olah et al., 2020) by decomposition methods such as sparse coding of activations (Cunningham et al., 2024).

P-explainThe AI Risk Management Framework of the US National Institute of Standards and Technology distinguishes interpretability from explainability (Standards and Technology, 2023), the property pursued by explainable artificial intelligence (XAI) (Gunning and Aha, 2019). In contrast to interpretability, explainability requires that an explanation is provided along with each prediction. These explanations may take the form of saliency maps or reference examples from the training set. A challenge for XAI methods is that a computed explanation can be unfaithful to the learned hypothesis and thereby mislead the user (Rudin, 2019).

See also: explainability, explainable artificial intelligence, interpretable machine learning, trustworthy artificial intelligence, regularization, local interpretable model-agnostic explanations, mechanistic interpretability, transparency.

References

  1. Standardization and Commission (2022). ISO/IEC 22989:2022 --- Information technology --- Artificial intelligence --- Artificial intelligence concepts and terminology. www.iso.org/standard/74296.html
  2. Chen et al. (2018). Learning to Explain: An Information-Theoretic Perspective on Model Interpretation. Proc. 35th Int. Conf. Mach. Learn.. proceedings.mlr.press/v80/chen18j.html
  3. Colin et al. (2022). What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods. Adv. Neural Inf. Process. Syst.. proceedings.neurips.cc/paper_files/paper/2022/hash/13113e938f2957891c0c5e8df811dd01-Abstract-Conference.html
  4. Doshi-Velez and Kim (2017). Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608. arxiv.org/abs/1702.08608
  5. Hase and Bansal (2020). Evaluating explainable AI: Which algorithmic explanations help users predict model behavior?. Proc. 58th Annu. Meeting Assoc. Comput. Linguistics. aclanthology.org/2020.acl-main.491
  6. Lipton (2018). The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue. doi.org/10.1145/3236386.3241340
  7. Elhage et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread. transformer-circuits.pub/2021/framework/index.html
  8. Olah et al. (2020). Zoom In: An Introduction to Circuits. Distill. doi.org/10.23915/distill.00024.001
  9. Cunningham et al. (2024). Sparse Autoencoders Find Highly Interpretable Features in Language Models. 12th Int. Conf. Learn. Representations. openreview.net/forum?id=F76bwRSLeK
  10. Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). doi.org/10.6028/NIST.AI.100-1
  11. Gunning and Aha (2019). DARPA's Explainable Artificial Intelligence Program. AI Magazine. doi.org/10.1609/aimag.v40i2.2850
  12. Rudin (2019). Stop explaining black box machine learning models for high-stakes decisions and use interpretable models instead. Nature Mach. Intell.. doi.org/10.1038/s42256-019-0048-x

Cite this entry

@misc{dictml_interpretability,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {interpretability},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-21},
  url = {https://dictionaryofml.org/terms/interpretability.html}
}