The (subjective) explainability of a machine learning (ML) method is the
level to which a human user can anticipate the predictions
delivered by the method, based on the explanations that the
method provides along with each prediction. It is quantified
by comparing the
predictions with user-provided anticipations on a
test set, or via the conditional entropy of the
predictions given the anticipations. Explainability is
user-relative, and providing explanations raises it.
Terminology standards define explainability qualitatively, as the
property of an artificial intelligence system (AI system) to express the important factors
influencing its results in a way that humans can understand.
Definition
The (subjective) explainability of a
machine learning (ML) method is the level to which a human user can anticipate,
or simulate, the predictions delivered by the method, based
on the explanations that the method provides
(Colin et al., 2022; Jung and Nardelli, 2020). Explainability
thus includes the notion of an explanation: the method
complements each prediction with an explanation for
this specific prediction, such as the feature values
that drove it or, for image data, the relevant pixels
(see Fig. 1). A clinician
using a diagnosis system can act
on its predictions only if these are explainable: the
clinician must be able to anticipate, at least roughly, which
prediction the system delivers for a given patient record,
and the explanation, e.g., the highlighted region of a
medical image, is what enables this anticipation.
Explainability is user-relative: the same trained hypothesis
can be explainable for one user and inscrutable for another.
Figure 1: Explainability refers to explanations: the learned
hypothesis $\learnthypothesis$ delivers, for a
data point with feature vector $\featurevec$, the
prediction $\learnthypothesis(\featurevec)$ along with an
explanation for this specific prediction, such as
the most relevant features (for image data, the
relevant pixels). Both are provided to the user
Quantitative measures of explainability compare the
predictions of a trained hypothesis with the
anticipations provided by a user for the data points in a
test set (Colin et al., 2022; Zhang et al., 2024).
Alternatively, a probabilistic model for the data allows measuring
explainability via the conditional (differential) entropy of
the predictions, given the user's anticipations: the smaller
this conditional entropy, the better the user's anticipations
determine the predictions (Chen et al., 2018;
Jung and Nardelli, 2020). In practice, this conditional entropy
is unknown and must be replaced by an estimator, e.g., a
plug-in estimate computed from discretized predictions and
anticipations on a test set.
Providing explanations raises explainability.
Fig. 2 illustrates this effect for a
user who reasons in terms of linear maps and anticipates the
predictions of an opaque hypothesis (a
kernel method): without explanations, the anticipations
deviate strongly from the predictions; given a local linear
approximation of the hypothesis around each data point
(cf.\ local interpretable model-agnostic explanations (LIME)), the anticipations match the predictions
almost exactly. Methods that deliver such explanations are
discussed under explainable artificial intelligence (XAI); explainable empirical risk minimization (EERM) makes explainability
itself a design goal of training.
Figure 2: User anticipations against the predictions of an
opaque hypothesis (Gaussian kernel
ridge regression) on a test set, for a user who
reasons in terms of linear maps. Without
explanations (open squares), the anticipations deviate
strongly from the predictions; with a local linear
approximation around each data point as explanation
(filled circles), the anticipations track the diagonal of
perfect anticipation. The estimated conditional entropy (a plug-in estimate on the test set) of the
predictions given the anticipations drops from 1.22 to
0.09 bits. Data generated by
pythondemos/explainability.py
The international terminology standard for artificial intelligence (AI) defines
explainability as the property of an artificial intelligence system (AI system) to express
important factors influencing its results in a way that humans
can understand; an explanation is intended to answer the
question of why the AI system produced a result, without
arguing that the result was optimal (Standardization and Commission, 2022, Sect. 3.5.7).
The AI risk management framework of the US National Institute of
Standards and Technology ties explainability to a representation
of the mechanisms underlying the operation of an AI system,
and reserves interpretability for the meaning of the output
in the context of the designed purpose (Standards and Technology, 2023). The
anticipation-based definition above operationalizes these
qualitative notions: a user who understands the important factors
influencing the predictions can anticipate them, and
Fig. 2 shows how providing
explanations improves the anticipations.
Regulation treats explainability as an ingredient of
transparency. The EU AI Act does not define
explainability among the definitions of its Article 3; its
Article 13 instead requires that a high-risk artificial intelligence system (high-risk AI system) be
sufficiently transparent to enable deployers to interpret its
output (Parliament and Union, 2024). For individual automated decisions,
the right to explanation entitles an affected person to a
clear and meaningful account of the role that an AI system
played in the decision (Parliament and Union, 2024, Art. 86). Fulfilling
this right requires explainability techniques such as LIME
or SHapley Additive exPlanations (SHAP) (see right to explanation). Recent state law in
the United States imposes a similar duty: the Colorado Automated
Decision-Making Technology Act of 2026 requires that the deployer
provide the affected consumer, after an adverse consequential
decision (for example, on lending or employment), with a
plain-language disclosure of the role that the technology played
in the decision (Assembly, 2026).
Colin et al. (2022). What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods. Adv. Neural Inf. Process. Syst..
Jung and Nardelli (2020). An Information-Theoretic Approach to Personalized Explainable Machine Learning. IEEE Signal Process. Lett..
Parliament and Union (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance).
Assembly (2026). Senate Bill 26-189 --- Automated Decision-Making Technology.
Cite this entry
@misc{dictml_explainability,
author = {Jung, Alexander},
title = {explainability},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-08-06},
url = {https://dictionaryofml.org/terms/explainability.html}
}
