Dictionary of Applied Machine Learning: 147 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here
Updated on 2026-11-20
Readiness 100/100
See also feature data point classification explainability explainable artificial intelligence interpretable machine learning feature importance empirical risk minimization
An explanation accompanies a prediction delivered by a machine learning (ML) method and states which properties of the data point led to that prediction. Explanations can be a textual description, a feature importance score, an approximation of a hypothesis map, or a heat map. An explanation of local or global scope can be learned as a simple hypothesis that reproduces the predictions of the learned hypothesis on a probing set of feature vectors. On the one hand, an explanation must be faithful: it must reflect the computation by which the learned hypothesis arrives at the prediction. On the other hand, it must be understandable: its user must be able to anticipate, from it alone, how the prediction changes when the features change.

explanation.py, which recomputes what the entry states: the script block by block · open in ColabA weather service forecasts frost for the coming night at Krems an der Donau (Austria) and adds which of the recent measurements weighed most. That addition is an explanation: it is provided alongside the prediction delivered by a machine learning (ML) method to increase its transparency for its human user.
B-trainExplanations can take different forms. For instance, they may consist of human-readable
text or quantitative indicators, such as feature importance scores
(Molnar, 2025; Ribeiro et al., 2016). Fig. 1
illustrates two types of explanations, both for the same learned hypothesis
$\widehat{h}(x)$ of a single feature $x$. The first explanation
approximates $\widehat{h}(x)$ around a specific feature value
$x'$ by a linear function $g(x)$. The weight of $g(x)$,
its slope, serves as a feature importance score. The second explanation type is a
small set of predictions $\widehat{h}(x^{(1)}),
\widehat{h}(x^{(2)}), \widehat{h}(x^{(3)})$ at selected
feature values, possibly chosen by the user.
Besides their form, explanations also differ in their scope with respect to the feature space. The local linear approximation of local interpretable model-agnostic explanations (LIME) delivers a local explanation for a learned hypothesis $\widehat{h}$ around a specific feature vector ${\mathbf x} \in \mathcal{X}$. Another example of a local explanation is a counterfactual for a classifier $\widehat{h}$. For a given feature vector ${\mathbf x}$, a counterfactual states a (small) modification ${\mathbf x} \mapsto {\mathbf x}'$ such that the prediction changes, $\widehat{h}({\mathbf x}) \neq \widehat{h}({\mathbf x}')$ (Wachter et al., 2018). The Court of Justice of the European Union has named such information as an appropriate explanation of an automated decision: the extent to which a variation in the personal data would have led to a different result (Court of Justice of the European Union, 2025).
In contrast to a local explanation, a global explanation of $\widehat{h}$
approximates it across the entire feature space. Such an approximation can be
obtained by training a model, such as a decision tree, to reproduce the
predictions $\widehat{h}({\mathbf x}^{(r)})$ for
feature vectors ${\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(m)}$ that cover
the entire feature space well (Molnar, 2025).
Algorithm 1 states this construction: the
feature vectors form a probing set, each labeled with the prediction of
$\widehat{h}$, and the explanation $g$ is learned via empirical risk minimization (ERM) over a
hypothesis space $\mathcal{H}'$ of candidate explanations. The scope is fixed by the
probing set alone. The same algorithm delivers the local explanation of
LIME when the probing set consists of feature vectors drawn around the one
${\mathbf x}$ to be explained (Ribeiro et al., 2016, Sect. 3.2); the linear
function $g$ of Fig. 1 is its output for a probing
set near $x'$ and $\mathcal{H}'$ the linear model.
Understandability and faithfulness can be contradictory requirements. An unfaithful explanation can be understandable while carrying no information about the computation. For some saliency methods, the map computed from an artificial neural network (ANN) with random weights is nearly the same as the map computed from the trained ANN (Adebayo et al., 2018, Sect. 3). Such a map traces the edges of the input image and carries nothing about the prediction, yet it looks like a plausible explanation. An explanation that agreed with the learned hypothesis everywhere would be that hypothesis again, which might be incomprehensible for a user, so a simpler explanation may have to give up faithfulness somewhere (Ribeiro et al., 2016, Sect. 3.2).
Fig. 2 illustrates the understandable part of an explanation for a prediction made from weather radar. The features are the hourly precipitation over a $120$ km box around Krems an der Donau, given on a grid of cells of $2$ km by $2$ km, the pixels of the radar image. The prediction is whether it will rain at Krems two hours after the hour shown. A CNN trained on these images is explained by a CAM (Selvaraju et al., 2017), shown in panel (b). A second explanation, in words, is printed below the panels, and its construction is described after the faithfulness test. These two explanations might be considered understandable by a non-expert user of a weather app. Their faithfulness, however, has to be verified by a perturbation test.
For the hour drawn in Fig. 2(a), the CNN assigns rain a probability close to one, and the CAM in panel (b) scores the bands of precipitation highest south and south-east of the town rather than on the rain already overhead. Faithfulness is tested by perturbing the radar image. Setting the precipitation to zero in the 360 cells with the highest CAM scores lowers the logit of rain, the number that the sigmoid function turns into the predicted probability, by $1.57$, while zeroing the 360 cells with the lowest scores lowers it by only $0.26$. The same ordering holds for 63 of the 72 hours of the test set, the last $30$ percent of the recorded hours, which were not used to train the CNN.
B-textFig. 2 also depicts an explanation in the form of text,
the sentence printed below the panels. A fixed rule turns the
CAM of panel (b) into that sentence: it collects the 360 cells
the map scores highest, the same cells the perturbation test above sets to zero, and
computes two quantities from them. The first is the radius around Krems that holds
nine in ten of these cells, rounded up to a multiple of $5$ km. The last tenth sits
in the far corners of the box and is left out, since it would stretch the radius over
the whole box. The second is the compass direction in which these cells lie, seen
from the center of Krems: their average position, each cell weighted by its
CAM score, rounded to one of 16 compass sectors. Because the
rule is fixed, the sentence is exactly as faithful as the map it summarizes, whose
faithfulness the perturbation test above checks. The sentence can also be compared
with the radar image in panel (a).
pythondemos/explanation.py
See also: machine learning, prediction, feature, data point, classification, explainability, explainable artificial intelligence, interpretable machine learning, local interpretable model-agnostic explanations, SHapley Additive exPlanations, counterfactual, class activation map, feature importance, empirical risk minimization.
@misc{dictml_explanation,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {explanation},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-20},
url = {https://dictionaryofml.org/terms/explanation.html}
}