\documentclass{article} \usepackage[utf8]{inputenc} \usepackage{amsmath,amssymb} \usepackage{algorithm,algpseudocode} \usepackage{bm} \usepackage{hyperref} \usepackage{float} \usepackage{subcaption} \usepackage{enumitem} \usepackage{xcolor} \definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0} \definecolor{aaltoBlue}{RGB}{0, 51, 102} \definecolor{pinegreen}{cmyk}{0.92,0,0.59,0.25} \definecolor{royalblue}{cmyk}{1,0.50,0,0} \definecolor{lavander}{cmyk}{0,0.48,0,0} \definecolor{violet}{cmyk}{0.79,0.88,0,0} \definecolor{lightblue}{RGB}{173, 216, 230} \usepackage{tikz} \usepackage{pgfplots} \usetikzlibrary{calc,arrows.meta,positioning,fit,shapes.geometric,shapes.misc,intersections,patterns,decorations.pathreplacing,backgrounds,matrix} \usepgfplotslibrary{fillbetween,groupplots} \setcounter{secnumdepth}{-\maxdimen} % Long unbreakable tokens (the ISO/IEC author names in citations, term names % in the title) otherwise overflow the margin by a few points; a modest % emergency stretch lets TeX loosen the paragraph instead. \emergencystretch=2em \usepackage{mleftright} % Project macros used by this entry (auto-extracted). \newcommand{\feature}{x} \newcommand{\featurevec}{\vx} \newcommand{\hypothesis}{h} \newcommand{\learnthypothesis}{\widehat{\hypothesis}} \newcommand{\truelabel}{y} \newcommand{\vx}[0]{{\bf x}} \begin{document} \section{explainability} \subsection*{Author and editors} Alexander Jung\textsuperscript{*}, Department of Computer Science, Aalto University, Espoo, Finland (ORCID \mbox{\href{https://orcid.org/0000-0001-7538-0990}{0000-0001-7538-0990}}) \\ Editors: Konstantina Olioumtsevits (ORCID \mbox{\href{https://orcid.org/0000-0003-2697-8659}{0000-0003-2697-8659}}), Ekkehard Schnoor (ORCID \mbox{\href{https://orcid.org/0000-0001-8252-1499}{0000-0001-8252-1499}}), Aalto University, Espoo, Finland \\ \textsuperscript{*}Corresponding author: \href{mailto:alex.jung@aalto.fi}{alex.jung@aalto.fi} \subsection{Abstract} Explainability is the property of a machine learning (ML) method whose explanations let a human user anticipate the predictions it delivers. The method delivers an explanation along with every prediction, and that explanation is effective if it lets the user anticipate the prediction made for a data point; comprehending the explanation is not required for this test. Explainability is always relative to a specific user or group of users: an explanation can be effective for one and useless for another. Explainability can be measured by comparing the predictions of a learned hypothesis to the anticipations of a user before and after they are provided with an explanation. \subsection{Definition} A radiologist who is shown which pixels of a scan led an image classifier to its prediction can tell, for the next scan, what the classifier will say. A machine learning (ML) method is explainable for a human user if the explanations it provides let the user anticipate the predictions it delivers (Colin et al., 2022; Jung and Nardelli, 2020). Explainability thus includes the notion of an explanation: each prediction is delivered with an explanation for this specific prediction (see Fig.~\ref{fig_explainability_delivery_dict}), such as the feature values that contributed most to it or, for image data, the relevant pixels. In the figure, the explanation for the prediction at a data point with feature vector $\featurevec$ is a linear map that approximates the learned hypothesis near $\featurevec$. The definition asks for the existence of an explanation, not for a particular one: explainability is certified by exhibiting one explanation that works, and one explanation the user cannot follow does not show that the method is unexplainable. Explainability is relative to a specific user or group of users: the same learned hypothesis can be explainable for one user and inscrutable for another. \begin{figure}[H] \begin{center} \begin{tikzpicture}[>=Stealth, box/.style={draw, rounded corners, align=center, font=\small, inner sep=6pt, minimum height=0.95cm}] \node[box] (x) {data point\\ $\featurevec$}; \node[box, right=1.2cm of x] (h) {learned hypothesis\\ $\learnthypothesis$}; \node[box, right=1.2cm of h] (y) {prediction\\ $\learnthypothesis(\featurevec)$}; \node[box, below=0.9cm of h] (g) {simple approximation\\ $g \approx \learnthypothesis$ near $\featurevec$}; \node[box, right=1.2cm of g] (e) {explanation\\ for the user}; \draw[->] (x) -- (h); \draw[->] (h) -- (y); \draw[->] (x.south) |- (g.west); \draw[->] (h) -- (g); \draw[->] (g) -- (e); \end{tikzpicture} \end{center} \caption{Explainability refers to explanations. The learned hypothesis $\learnthypothesis$ delivers the prediction $\learnthypothesis(\featurevec)$ for a data point with feature vector $\featurevec$. An explanation is constructed from a simple hypothesis $g$, e.g., a linear map, that approximates $\learnthypothesis$ near $\featurevec$. The explanation is delivered to the user along with the prediction \label{fig_explainability_delivery_dict}} \end{figure} The usefulness of an explanation can be measured by how much it enables the user to anticipate the predictions on a curated test set: the user states an anticipation for each data point in the test set. If the user comprehends the explanations, these anticipations should be close to the predictions of the learned hypothesis (Colin et al., 2022). A measure of this form is referred to as predictability (International Organization for Standardization and International Electrotechnical Commission, 2022, Sect.~5.15.7). As with interpretability, predictability is weaker than comprehension of the explanations: anticipating the predictions does not require comprehending the provided explanations. \par A second measure of this predictability uses the conditional differential entropy, which is defined for random variables (RVs). If the data point is modeled as the realization of an RV, the prediction and the anticipation of the user are RVs too, and their joint probability distribution yields the conditional differential entropy of the predictions given the anticipations (Jung and Nardelli, 2020, Sect.~II). The conditional differential entropy quantifies the uncertainty that remains about the predictions once the anticipations are known, so a smaller value means that the anticipations determine the predictions more tightly (Cover and Thomas, 2006, Sect.~8.4). In practice, this conditional differential entropy is unknown and must be replaced by an estimator. One such estimator bins the predictions and the anticipations on a test set, and takes the fraction of test data points that fall into each pair of bins as the joint probability distribution. A related measure, the mutual information (MI) between a selected subset of the features and the prediction, is maximized by the explainer of Chen et al. (2018), Sect.~2.1. \par An explanation raises explainability when the user can reason with it. Fig.~\ref{fig_explainability_dict} illustrates this for a user who reasons in terms of linear maps and anticipates the predictions of an opaque hypothesis (Gaussian kernel ridge regression, a kernel method). Without explanations, the user anticipates by a single linear map fitted to a few predictions, and the anticipations deviate strongly from the predictions. The explanation for a test data point with feature value $\feature_{0}$ is a local linear approximation of the hypothesis around it (cf.\ local interpretable model-agnostic explanations (LIME)), a slope and an intercept that the user evaluates at $\feature_{0}$. With this explanation, the anticipations match the predictions almost exactly. The example presupposes that the user knows how to apply a linear approximation: the anticipations improve only because the user can evaluate the explanation for a data point. A user who cannot gains nothing from it. \begin{figure}[H] \centering \begin{tikzpicture} \begin{groupplot}[ group style={group size=2 by 1, horizontal sep=1.6cm}, width=0.5\linewidth, height=0.5\linewidth, axis lines=left, tick label style={font=\scriptsize}, label style={font=\scriptsize, align=center}, legend style={draw=none, fill=none, font=\scriptsize}, legend cell align=left, ] \nextgroupplot[ xlabel={feature $\feature$\\(a) one explanation}, ylabel={label $\truelabel$}, legend style={at={(0.5,-0.42)}, anchor=north}] \addplot[thick, black] table[col sep=comma, x=x, y=h]{explainability_local.csv}; \addlegendentry{opaque hypothesis (solid curve)} \addplot[thin, dashed, black] table[col sep=comma, x=x, y=u_glob]{explainability_local.csv}; \addlegendentry{linear map of the user (dashed line)} \addplot[line width=2.2pt, black!45, restrict x to domain=0.45:1.7] table[col sep=comma, x=x, y=g_loc]{explainability_local.csv}; \addlegendentry{local explanation (thick gray segment)} \addplot[only marks, mark=square, mark size=2.2pt] table[col sep=comma, x=x0, y=u_no0]{explainability_point.csv}; \addlegendentry{anticipation without explanation (open square)} \addplot[only marks, mark=*, mark size=2.4pt, mark options={fill=black, draw=white, line width=0.5pt}] table[col sep=comma, x=x0, y=u_expl0]{explainability_point.csv}; \addlegendentry{anticipation with explanation (filled circle)} \nextgroupplot[ xlabel={prediction $\learnthypothesis(\feature)$\\(b) 20 test data points}, ylabel={user anticipation}, legend style={at={(0.5,-0.42)}, anchor=north}] \addplot[thin, dashed, domain=-2.5:2.5] {x}; \addlegendentry{perfect anticipation (dashed line)} \addplot[only marks, mark=square, mark size=1.8pt] table[col sep=comma, x=yhat, y=u_no]{explainability_scatter20.csv}; \addlegendentry{without explanations (open squares)} \addplot[only marks, mark=*, mark size=1.5pt] table[col sep=comma, x=yhat, y=u_expl]{explainability_scatter20.csv}; \addlegendentry{with explanations (filled circles)} \end{groupplot} \end{tikzpicture} \caption{An opaque hypothesis $\learnthypothesis$ (Gaussian kernel ridge regression, solid curve) and a user who reasons in terms of linear maps. (a) Without explanations the user anticipates by a single linear map fitted to a few predictions (dashed line), which at the test data point $\feature_{0}$ is far off (open square). The explanation for $\feature_{0}$ is a local linear approximation of $\learnthypothesis$ (thick gray segment); the user evaluates it at $\feature_{0}$ and anticipates almost exactly (filled circle). (b) The same two anticipations against the predictions for $20$ of the $40$ data points of the test set (every second one). Without explanations (open squares) they deviate strongly; with the local linear approximation as explanation (filled circles) they track the diagonal of perfect anticipation. The estimated conditional differential entropy of the predictions given the anticipations, over all $40$, drops from 1.22 to 0.09 bits. Data generated by \texttt{pythondemos/explainability.py} \label{fig_explainability_dict}} \end{figure} An explanation can also mislead. A computed explanation can be unfaithful to the learned hypothesis, so that anticipations based on it are confidently wrong (Rudin, 2019). Saliency maps, intensity maps over the pixels of an input image (see explainable artificial intelligence (XAI)), barely change for some methods when the trained deep net is replaced by one with random weights (Adebayo et al., 2018, Sect.~3). The predictability test detects such an explanation: anticipations drawn from it deviate from the predictions. \par The international terminology standard for artificial intelligence (AI) defines explainability as the property of an artificial intelligence system (AI system) to express important factors influencing its results in a way that humans can understand. An explanation is intended to answer the question of why the AI system produced a result, without arguing that the result was optimal (International Organization for Standardization and International Electrotechnical Commission, 2022, Sect.~3.5.7). The AI Risk Management Framework of the US National Institute of Standards and Technology (NIST) ties explainability to a representation of the mechanisms underlying the operation of an AI system. NIST reserves interpretability for the meaning of the output in the context of the designed purpose (National Institute of Standards and Technology, 2023). The two documents thus divide the terms differently. The expressed factors influencing a result, which the terminology standard places under explainability, are close to the NIST notion of interpretability; the NIST notion of explainability is instead close to interpretability as comprehension of the computation of the method. \par Regulation treats explainability as a component of transparency. The EU AI Act requires that high-risk artificial intelligence systems (high-risk AI systems) are sufficiently transparent to enable deployers to interpret a system's output and use it appropriately (European Parliament and Council of the European Union, 2024, Art.~13(1)). For individual automated decisions, the right to explanation entitles an affected person to a clear and meaningful account of the role that an AI system played in the decision (European Parliament and Council of the European Union, 2024, Art.~86). The Colorado Automated Decision-Making Technology Act of 2026 requires that the deployer give the affected consumer a plain-language description of the decision and of the role the technology played in it, together with the means to request further information (Colorado General Assembly, 2026, Sect.~6-1-1704(3)). What counts as an automated decision is drawn widely: the Court of Justice of the European Union held that a credit information agency computing a person's ability to meet future payments already makes one, when a third party draws strongly on that value (Court of Justice of the European Union, 2023). \subsection{Cross References} explanation, interpretability, XAI, EERM, LIME, transparency, right to explanation, trustworthy AI, regularization \subsection{References} \begin{enumerate} \item Adebayo et al. (2018). Sanity Checks for Saliency Maps. In: Adv. Neural Inf. Process. Syst. (NeurIPS), pp.~9505--9515. \item Court of Justice of the European Union (2023). Judgment of 7 December 2023, SCHUFA Holding (Scoring), C-634/21, ECLI:EU:C:2023:957. Court of Justice of the European Union, First Chamber. URL: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:62021CJ0634. \item Colorado General Assembly (2026). Senate Bill 26-189 --- Automated Decision-Making Technology. Colorado General Assembly, 2026 Regular Session, signed May 14, 2026. URL: https://leg.colorado.gov/bills/sb26-189. \item Chen et al. (2018). Learning to Explain: An Information-Theoretic Perspective on Model Interpretation. In: Proc. 35th Int. Conf. Mach. Learn., pp.~883--892. \item Colin et al. (2022). What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods. In: Adv. Neural Inf. Process. Syst., pp.~2832--2845. \item European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). Official Journal of the European Union, L series, Jul. 12. URL: https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng. \item International Organization for Standardization and International Electrotechnical Commission (2022). ISO/IEC 22989:2022 --- Information technology --- Artificial intelligence --- Artificial intelligence concepts and terminology. International Standard, 1st edition. URL: https://www.iso.org/standard/74296.html. \item Jung and Nardelli (2020). An Information-Theoretic Approach to Personalized Explainable Machine Learning. IEEE Signal Process. Lett., 27, pp.~825--829. \item National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. URL: https://doi.org/10.6028/NIST.AI.100-1. \item Cover and Thomas (2006). Elements of Information Theory. 2nd ed., Wiley, Hoboken, NJ, USA. \item Rudin (2019). Stop explaining black box machine learning models for high-stakes decisions and use interpretable models instead. Nature Mach. Intell., 1(5), pp.~206--215. \end{enumerate} \end{document}