Dictionary of Applied Machine Learning: 147 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

explanation

Updated on 2026-11-20

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also feature data point classification explainability explainable artificial intelligence interpretable machine learning feature importance empirical risk minimization

An explanation accompanies a prediction delivered by a machine learning (ML) method and states which properties of the data point led to that prediction. Explanations can be a textual description, a feature importance score, an approximation of a hypothesis map, or a heat map. An explanation of local or global scope can be learned as a simple hypothesis that reproduces the predictions of the learned hypothesis on a probing set of feature vectors. On the one hand, an explanation must be faithful: it must reflect the computation by which the learned hypothesis arrives at the prediction. On the other hand, it must be understandable: its user must be able to anticipate, from it alone, how the prediction changes when the features change.

Figure produced by the Python demo of the entry explanation
Figure written by the entry's Python demo explanation.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

A weather service forecasts frost for the coming night at Krems an der Donau (Austria) and adds which of the recent measurements weighed most. That addition is an explanation: it is provided alongside the prediction delivered by a machine learning (ML) method to increase its transparency for its human user.

B-trainExplanations can take different forms. For instance, they may consist of human-readable text or quantitative indicators, such as feature importance scores (Molnar, 2025; Ribeiro et al., 2016). Fig. 1 illustrates two types of explanations, both for the same learned hypothesis $\widehat{h}(x)$ of a single feature $x$. The first explanation approximates $\widehat{h}(x)$ around a specific feature value $x'$ by a linear function $g(x)$. The weight of $g(x)$, its slope, serves as a feature importance score. The second explanation type is a small set of predictions $\widehat{h}(x^{(1)}), \widehat{h}(x^{(2)}), \widehat{h}(x^{(3)})$ at selected feature values, possibly chosen by the user.

Figure 1 of the entry explanation
Figure 1: A learned hypothesis $\learnthypothesis(\feature)$ of a single feature $\feature$ (solid curve) with two different explanations. The thick gray segment is a linear approximation $g(\feature)$ of $\learnthypothesis(\feature)$ at the point $\feature'$, marked by the filled dot on the curve. The three further filled dots are the predictions $\learnthypothesis(\feature^{(1)})$, $\learnthypothesis(\feature^{(2)})$, $\learnthypothesis(\feature^{(3)})$ at the feature values $\feature^{(1)}$, $\feature^{(2)}$, $\feature^{(3)}$ marked on the horizontal axis. A dashed line drops from each of the four filled dots to its feature value on the horizontal axis
Another form of explanation is a heat map, also called an intensity map or, for images, a saliency map (Zeiler and Fergus, 2014; Montavon et al., 2018). Such a map assigns each region of an image a score for how much that region contributed to the prediction and indicates these scores by color intensities overlaid on the image itself (Selvaraju et al., 2017). For a convolutional neural network (CNN), the scores can be computed from its activations, which is what a class activation map (CAM) does; Fig. 2 shows one.

Besides their form, explanations also differ in their scope with respect to the feature space. The local linear approximation of local interpretable model-agnostic explanations (LIME) delivers a local explanation for a learned hypothesis $\widehat{h}$ around a specific feature vector ${\mathbf x} \in \mathcal{X}$. Another example of a local explanation is a counterfactual for a classifier $\widehat{h}$. For a given feature vector ${\mathbf x}$, a counterfactual states a (small) modification ${\mathbf x} \mapsto {\mathbf x}'$ such that the prediction changes, $\widehat{h}({\mathbf x}) \neq \widehat{h}({\mathbf x}')$ (Wachter et al., 2018). The Court of Justice of the European Union has named such information as an appropriate explanation of an automated decision: the extent to which a variation in the personal data would have led to a different result (Court of Justice of the European Union, 2025).

In contrast to a local explanation, a global explanation of $\widehat{h}$ approximates it across the entire feature space. Such an approximation can be obtained by training a model, such as a decision tree, to reproduce the predictions $\widehat{h}({\mathbf x}^{(r)})$ for feature vectors ${\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(m)}$ that cover the entire feature space well (Molnar, 2025). Algorithm 1 states this construction: the feature vectors form a probing set, each labeled with the prediction of $\widehat{h}$, and the explanation $g$ is learned via empirical risk minimization (ERM) over a hypothesis space $\mathcal{H}'$ of candidate explanations. The scope is fixed by the probing set alone. The same algorithm delivers the local explanation of LIME when the probing set consists of feature vectors drawn around the one ${\mathbf x}$ to be explained (Ribeiro et al., 2016, Sect. 3.2); the linear function $g$ of Fig. 1 is its output for a probing set near $x'$ and $\mathcal{H}'$ the linear model.

Algorithm 1 of the entry explanation
Algorithm 1: Explanation by a surrogate model, following Ribeiro et al. (2016, Sect. 3.2)
Explanations of (the predictions obtained from) ML methods should be (i) understandable and (ii) faithful (Ribeiro et al., 2016, Sect. 3; Lipton, 2018). Understandable means that the user can work with the explanation on their own: from it alone, they can anticipate how the prediction changes when the features change. In Algorithm 1, this requirement falls on the choice of $\mathcal{H}'$, which must consist of hypotheses the user can comprehend, such as linear maps or shallow decision trees. Faithful means that the explanation reflects the computation the learned hypothesis carries out. For the output $g$ of Algorithm 1, the training error on the probing set measures how faithfully $g$ reproduces $\widehat{h}$ there, and nowhere else. An intensity map over the pixels of an image is faithful when manipulating a few of the pixels it highlights changes the prediction, while the same manipulation of pixels it leaves dark does not (Samek et al., 2017, Sect. III-A, p. 2666).

Understandability and faithfulness can be contradictory requirements. An unfaithful explanation can be understandable while carrying no information about the computation. For some saliency methods, the map computed from an artificial neural network (ANN) with random weights is nearly the same as the map computed from the trained ANN (Adebayo et al., 2018, Sect. 3). Such a map traces the edges of the input image and carries nothing about the prediction, yet it looks like a plausible explanation. An explanation that agreed with the learned hypothesis everywhere would be that hypothesis again, which might be incomprehensible for a user, so a simpler explanation may have to give up faithfulness somewhere (Ribeiro et al., 2016, Sect. 3.2).

Fig. 2 illustrates the understandable part of an explanation for a prediction made from weather radar. The features are the hourly precipitation over a $120$ km box around Krems an der Donau, given on a grid of cells of $2$ km by $2$ km, the pixels of the radar image. The prediction is whether it will rain at Krems two hours after the hour shown. A CNN trained on these images is explained by a CAM (Selvaraju et al., 2017), shown in panel (b). A second explanation, in words, is printed below the panels, and its construction is described after the faithfulness test. These two explanations might be considered understandable by a non-expert user of a weather app. Their faithfulness, however, has to be verified by a perturbation test.

For the hour drawn in Fig. 2(a), the CNN assigns rain a probability close to one, and the CAM in panel (b) scores the bands of precipitation highest south and south-east of the town rather than on the rain already overhead. Faithfulness is tested by perturbing the radar image. Setting the precipitation to zero in the 360 cells with the highest CAM scores lowers the logit of rain, the number that the sigmoid function turns into the predicted probability, by $1.57$, while zeroing the 360 cells with the lowest scores lowers it by only $0.26$. The same ordering holds for 63 of the 72 hours of the test set, the last $30$ percent of the recorded hours, which were not used to train the CNN.

B-textFig. 2 also depicts an explanation in the form of text, the sentence printed below the panels. A fixed rule turns the CAM of panel (b) into that sentence: it collects the 360 cells the map scores highest, the same cells the perturbation test above sets to zero, and computes two quantities from them. The first is the radius around Krems that holds nine in ten of these cells, rounded up to a multiple of $5$ km. The last tenth sits in the far corners of the box and is left out, since it would stretch the radius over the whole box. The second is the compass direction in which these cells lie, seen from the center of Krems: their average position, each cell weighted by its CAM score, rounded to one of 16 compass sectors. Because the rule is fixed, the sentence is exactly as faithful as the map it summarizes, whose faithfulness the perturbation test above checks. The sentence can also be compared with the radar image in panel (a).

Figure 2 of the entry explanation
Figure 2: An explanation for a prediction made from weather radar on 15 September 2024 at 08:00 UTC. (a) The observed precipitation in mm/h over a 120 km box around Krems an der Donau (Austria), marked by the circle, on a grid of $2$ km cells; the color scale runs from white (dry) to dark blue ($16$ mm/h and above). From this image a CNN predicts rain at Krems two hours later. (b) The CAM, which scores each cell by how much it contributed to the prediction; the color scale runs from white (0) to dark red (1). The highest scores lie on the bands south and south-east of the town rather than on the rain overhead. Below: a textual explanation generated from the CAM, naming the radius and direction of the cells it scores highest. Data generated by pythondemos/explanation.py

See also: machine learning, prediction, feature, data point, classification, explainability, explainable artificial intelligence, interpretable machine learning, local interpretable model-agnostic explanations, SHapley Additive exPlanations, counterfactual, class activation map, feature importance, empirical risk minimization.

References

  1. Molnar (2025). Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. Ebook. christophm.github.io/interpretable-ml-book/
  2. Ribeiro et al. (2016). ``Why Should I Trust You?'': Explaining the Predictions of Any Classifier. Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining. doi.org/10.1145/2939672.2939778
  3. Zeiler and Fergus (2014). Visualizing and Understanding Convolutional Networks. Proc. European Conf. Comput. Vis. (ECCV). doi.org/10.1007/978-3-319-10590-1_53
  4. Montavon et al. (2018). Methods for interpreting and understanding deep neural networks. Digital Signal Processing.
  5. Selvaraju et al. (2017). Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. 2017 IEEE Int. Conf. Comput. Vis.. doi.org/10.1109/ICCV.2017.74
  6. Wachter et al. (2018). Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR. Harvard Journal of Law & Technology. doi.org/10.2139/ssrn.3063289
  7. Court of Justice of the European Union (2025). Judgment of 27 February 2025, CK v Dun & Bradstreet Austria GmbH and Magistrat der Stadt Wien, C-203/22, ECLI:EU:C:2025:117. eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:62022CJ0203
  8. Lipton (2018). The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue. doi.org/10.1145/3236386.3241340
  9. Samek et al. (2017). Evaluating the Visualization of What a Deep Neural Network Has Learned. IEEE Trans. Neural Netw. Learn. Syst.. doi.org/10.1109/TNNLS.2016.2599820
  10. Adebayo et al. (2018). Sanity Checks for Saliency Maps. Adv. Neural Inf. Process. Syst. (NeurIPS). proceedings.neurips.cc/paper_files/paper/2018/hash/294a8ed24b1ad22ec2e7efea049b8737-Abstract.html

Cite this entry

@misc{dictml_explanation,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {explanation},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-20},
  url = {https://dictionaryofml.org/terms/explanation.html}
}