explainability.py, which recomputes what the entry states: the script block by block · open in ColabDictionary of Applied Machine Learning: 115 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here
Updated on 2026-11-08
Readiness 100/100
See also explanation interpretability explainable artificial intelligence transparency trustworthy artificial intelligence regularization
Explainability is the property of a machine learning (ML) method whose explanations let a human user anticipate the predictions it delivers. The method delivers an explanation along with every prediction, and that explanation is effective if it lets the user anticipate the prediction made for a data point; comprehending the explanation is not required for this test. Explainability is always relative to a specific user or group of users: an explanation can be effective for one and useless for another. Explainability can be measured by comparing the predictions of a learned hypothesis to the anticipations of a user before and after they are provided with an explanation.
explainability.py, which recomputes what the entry states: the script block by block · open in ColabB-explA radiologist who is shown which pixels of a
scan led an image classifier to its prediction can tell, for
the next scan, what the classifier will say. A machine learning (ML) method is
explainable for a human user if the explanations it provides let
the user anticipate the predictions it delivers
(Colin et al., 2022; Jung and Nardelli, 2020). Explainability
thus includes the notion of an explanation: each
prediction is delivered with an explanation for
this specific prediction
(see Fig. 1), such as the
feature values that contributed most to it or, for image data,
the relevant pixels. In the figure, the explanation for the
prediction at a data point with feature vector
${\mathbf x}$ is a linear map that approximates the learned
hypothesis near ${\mathbf x}$.
The definition asks for the existence of an explanation, not
for a particular one: explainability is certified by
exhibiting one explanation that works, and one
explanation the user cannot follow does not show that the
method is unexplainable.
Explainability is relative to a specific user or group of users:
the same learned hypothesis
can be explainable for one user and inscrutable for another.
B-entA second measure of this predictability uses the conditional differential entropy, which is defined for random variables (RVs). If the data point is modeled as the realization of an RV, the prediction and the anticipation of the user are RVs too, and their joint probability distribution yields the conditional differential entropy of the predictions given the anticipations (Jung and Nardelli, 2020, Sect. II). The conditional differential entropy quantifies the uncertainty that remains about the predictions once the anticipations are known, so a smaller value means that the anticipations determine the predictions more tightly (Cover and Thomas, 2006, Sect. 8.4). In practice, this conditional differential entropy is unknown and must be replaced by an estimator. One such estimator bins the predictions and the anticipations on a test set, and takes the fraction of test data points that fall into each pair of bins as the joint probability distribution. A related measure, the mutual information (MI) between a selected subset of the features and the prediction, is maximized by the explainer of Chen et al. (2018, Sect. 2.1).
B-localAn explanation raises explainability when the user can
reason with it.
Fig. 2 illustrates this for a
user who reasons in terms of linear maps and anticipates the
predictions of an opaque hypothesis (Gaussian
kernel ridge regression, a kernel method). Without explanations, the user
anticipates by a single linear map fitted to a few
predictions, and the anticipations deviate strongly from
the predictions. The explanation for a test
data point with feature value $x_{0}$ is a local
linear approximation of the hypothesis around it
(cf. local interpretable model-agnostic explanations (LIME)), a slope and an intercept that the user
evaluates at $x_{0}$. With
this explanation, the anticipations match the
predictions almost exactly. The example presupposes that
the user knows how to apply a linear approximation: the
anticipations improve only because the user can evaluate the
explanation for a data point. A user who cannot gains
nothing from it.
pythondemos/explainability.py
The international terminology standard for artificial intelligence (AI) defines explainability as the property of an artificial intelligence system (AI system) to express important factors influencing its results in a way that humans can understand. An explanation is intended to answer the question of why the AI system produced a result, without arguing that the result was optimal (International Organization for Standardization and International Electrotechnical Commission, 2022, Sect. 3.5.7). The AI Risk Management Framework of the US National Institute of Standards and Technology (NIST) ties explainability to a representation of the mechanisms underlying the operation of an AI system. NIST reserves interpretability for the meaning of the output in the context of the designed purpose (National Institute of Standards and Technology, 2023). The two documents thus divide the terms differently. The expressed factors influencing a result, which the terminology standard places under explainability, are close to the NIST notion of interpretability; the NIST notion of explainability is instead close to interpretability as comprehension of the computation of the method.
Regulation treats explainability as a component of transparency. The EU AI Act requires that high-risk artificial intelligence systems (high-risk AI systems) are sufficiently transparent to enable deployers to interpret a system's output and use it appropriately (European Parliament and Council of the European Union, 2024, Art. 13(1)). For individual automated decisions, the right to explanation entitles an affected person to a clear and meaningful account of the role that an AI system played in the decision (European Parliament and Council of the European Union, 2024, Art. 86). The Colorado Automated Decision-Making Technology Act of 2026 requires that the deployer give the affected consumer a plain-language description of the decision and of the role the technology played in it, together with the means to request further information (Colorado General Assembly, 2026, Sect. 6-1-1704(3)). What counts as an automated decision is drawn widely: the Court of Justice of the European Union held that a credit information agency computing a person's ability to meet future payments already makes one, when a third party draws strongly on that value (Court of Justice of the European Union, 2023).
See also: explanation, interpretability, explainable artificial intelligence, explainable empirical risk minimization, local interpretable model-agnostic explanations, transparency, right to explanation, trustworthy artificial intelligence, regularization.
@misc{dictml_explainability,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {explainability},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
url = {https://dictionaryofml.org/terms/explainability.html}
}