\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb}
\usepackage{algorithm,algpseudocode}
\usepackage{bm}
\usepackage{hyperref}
\usepackage{float}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{xcolor}
\definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0}
\definecolor{aaltoBlue}{RGB}{0, 51, 102}
\definecolor{pinegreen}{cmyk}{0.92,0,0.59,0.25}
\definecolor{royalblue}{cmyk}{1,0.50,0,0}
\definecolor{lavander}{cmyk}{0,0.48,0,0}
\definecolor{violet}{cmyk}{0.79,0.88,0,0}
\definecolor{lightblue}{RGB}{173, 216, 230}
\usepackage{tikz}
\usepackage{pgfplots}
\usetikzlibrary{calc,arrows.meta,positioning,fit,shapes.geometric,shapes.misc,intersections,patterns,decorations.pathreplacing,backgrounds,matrix}
\usepgfplotslibrary{fillbetween,groupplots}
\setcounter{secnumdepth}{-\maxdimen}
% Long unbreakable tokens (the ISO/IEC author names in citations, term names
% in the title) otherwise overflow the margin by a few points; a modest
% emergency stretch lets TeX loosen the paragraph instead.
\emergencystretch=2em
\usepackage{mleftright}
% Project macros used by this entry (auto-extracted).
\DeclareMathOperator*{\argmin}{arg\,min}
\newcommand{\feature}{x}
\newcommand{\featurespace}{\mathcal{X}}
\newcommand{\featurevec}{\vx}
\newcommand{\hypospace}{\mathcal{H}}
\newcommand{\hypothesis}{h}
\newcommand{\learnthypothesis}{\widehat{\hypothesis}}
\newcommand{\lossfunc}[2]{L\left(#1,#2 \right)}
\newcommand{\pair}[2]{\left( #1,#2 \right)}
\newcommand{\sampleidx}{r}
\newcommand{\samplesize}{m}
\newcommand{\truelabel}{y}
\newcommand{\vx}[0]{{\bf x}}

\begin{document}

\section{explanation}

\subsection*{Author and editors}
Alexander Jung\textsuperscript{*}, Department of Computer Science, Aalto University, Espoo, Finland (ORCID \mbox{\href{https://orcid.org/0000-0001-7538-0990}{0000-0001-7538-0990}}) \\
Editors: Konstantina Olioumtsevits (ORCID \mbox{\href{https://orcid.org/0000-0003-2697-8659}{0000-0003-2697-8659}}), Ekkehard Schnoor (ORCID \mbox{\href{https://orcid.org/0000-0001-8252-1499}{0000-0001-8252-1499}}), Aalto University, Espoo, Finland \\
\textsuperscript{*}Corresponding author: \href{mailto:alex.jung@aalto.fi}{alex.jung@aalto.fi}



\subsection{Abstract}
An explanation accompanies a prediction delivered by a machine learning (ML) method and
		states which properties of the data point led to that prediction.
		Explanations can be a textual description, a feature importance score, an
		approximation of a hypothesis map, or a heat map. An explanation of local or
		global scope can be learned as a simple hypothesis that reproduces the
		predictions of the learned hypothesis on a probing set of
		feature vectors. On the one hand, an explanation must be faithful: it must reflect
		the computation by which the learned hypothesis arrives at the prediction.
		On the other hand, it must be understandable: its user must be able to anticipate, from
		it alone, how the prediction changes when the features change.

\subsection{Definition}
A weather service forecasts frost for the coming night at Krems an der
		Donau (Austria) and adds which of the recent measurements weighed most. That addition is
		an explanation: it is provided alongside the prediction delivered by a machine learning (ML)
		method to increase its transparency for its human user.
\par
		Explanations can take different forms. For instance, they may consist of human-readable
		text or quantitative indicators, such as feature importance scores
		(Molnar, 2025; Ribeiro et al., 2016). Fig.\ \ref{fig_explanation_dict}
		illustrates two types of explanations, both for the same learned hypothesis
		$\learnthypothesis(\feature)$ of a single feature $\feature$. The first explanation
		approximates $\learnthypothesis(\feature)$ around a specific feature value
		$\feature'$ by a linear function $g(\feature)$. The weight of $g(\feature)$,
		its slope, serves as a feature importance score. The second explanation type is a
		small set of predictions $\learnthypothesis(\feature^{(1)}),
		\learnthypothesis(\feature^{(2)}), \learnthypothesis(\feature^{(3)})$ at selected
		feature values, possibly chosen by the user.
\begin{figure}[H]
   \begin{center}
	 		\begin{tikzpicture}[x=0.5cm]
	 		\begin{axis}[
	 			axis lines=left,
	 			xmin=-3, xmax=6.5,
	 			ymin=0, ymax=4,
	 			domain=0:6,
	 			samples=100,
	 			width=10cm,
	 			height=6cm,
	 			clip=false,
	 			xlabel={feature $\feature$},
	 			ylabel={prediction $\learnthypothesis(\feature)$},
	 			xtick={-1,0,1.5,5},
	 			xticklabels={$\feature^{(1)}$,$\feature^{(2)}$,$\feature'$,$\feature^{(3)}$},
	 			ytick=\empty,
	 			label style={font=\small},
	 			tick label style={font=\small}
	 		   ]
	 		% Original nonlinear function h(x)
			\addplot[thick, domain=-2:6] {2 + sin(deg(x))};
			\node[anchor=south, font=\small] at (axis cs:5.3,2.1) {$\learnthypothesis(\feature)$};
	 		% Tangent line as local linear approximation at x = 1.5
	 		% h(3) = 2 + sin(3), h'(3) = cos(3)
	 		\addplot[black!55, line width=2pt, domain=0.5:2.5]
	 		{2 + sin(deg(1.5)) + cos(deg(1.5))*(x - 1.5)}
	 		node[pos=0.2, above] {$g(\feature)$};
	 		% Mark point of approximation
	 		\addplot[mark=*, black, only marks] coordinates {(1.5, {2 + sin(deg(1.5))})};
	 		    % Vertical dashed line (ruler) at x = 1.5
			\addplot[dashed, gray] coordinates {(1.5,0) (1.5,{2 + sin(deg(1.5))})};
	      		% add function values for finite set of featurevecs
	       		\addplot[mark=*, black, only marks] coordinates {(-1, {2 + sin(deg(-1))})};
	       		\addplot[dashed, gray] coordinates {(-1,0) (-1,{2 + sin(deg(-1))})};
	 	   	\addplot[mark=*, black, only marks] coordinates {(0, {2 + sin(deg(0))})};
	 	   	\addplot[dashed, gray] coordinates {(0,0) (0,{2 + sin(deg(0))})};
	 	   	\addplot[mark=*, black, only marks] coordinates {(5, {2 + sin(deg(5))})};
	 	   	\addplot[dashed, gray] coordinates {(5,0) (5,{2 + sin(deg(5))})};
			% 		    % Plot the two points
			% 		    % Coordinates of the two points
			% 		\pgfmathsetmacro{\xA}{-1.5}
			% 		\pgfmathsetmacro{\xB}{3}
			% 		\pgfmathsetmacro{\yA}{2 + sin(deg(\xA))}
			% 		\pgfmathsetmacro{\yB}{2 + sin(deg(\xB))}
	 		\end{axis}
			% 	\vspace*{-10mm}
			\end{tikzpicture}
  \end{center}
  \caption{A learned hypothesis $\learnthypothesis(\feature)$ of a single feature $\feature$ (solid curve) 
           with two different explanations. The thick gray segment is a linear approximation $g(\feature)$ of
	   $\learnthypothesis(\feature)$ at the point $\feature'$, marked by the filled dot on the curve. The 
           three further filled dots are the predictions $\learnthypothesis(\feature^{(1)})$, 
           $\learnthypothesis(\feature^{(2)})$, $\learnthypothesis(\feature^{(3)})$ at the feature values
	   $\feature^{(1)}$, $\feature^{(2)}$, $\feature^{(3)}$ marked on the horizontal axis. A dashed line drops from each of the four filled dots to its feature value on the horizontal axis
			\label{fig_explanation_dict}}
  \end{figure}
		Another form of explanation is a heat map, also called an intensity map or, for images,
		a saliency map (Zeiler and Fergus, 2014; Montavon et al., 2018). Such a map assigns
		each region of an image a score for how much that region contributed to the
		prediction and indicates these scores by color intensities overlaid on the image
		itself (Selvaraju et al., 2017). For a convolutional neural network (CNN), the scores can be computed from its
		activations, which is what a class activation map (CAM) does;
		Fig.~\ref{fig_explanation_radar_dict} shows one.
\par
		Besides their form, explanations also differ in their scope with respect to the
		feature space. The local linear approximation of local interpretable model-agnostic explanations (LIME) delivers a local
		explanation for a learned hypothesis $\learnthypothesis$ around a specific
		feature vector $\featurevec \in \featurespace$. Another example of a local explanation
		is a counterfactual for a classifier $\learnthypothesis$. For a given
		feature vector $\featurevec$, a counterfactual states a (small) modification
		$\featurevec \mapsto \featurevec'$ such that the prediction changes,
		$\learnthypothesis(\featurevec) \neq \learnthypothesis(\featurevec')$
		(Wachter et al., 2018). The Court of Justice of the European Union has named such
		information as an appropriate explanation of an automated decision: the extent to
		which a variation in the personal data would have led to a different result
		(Court of Justice of the European Union, 2025).
\par
		In contrast to a local explanation, a global explanation of $\learnthypothesis$
		approximates it across the entire feature space. Such an approximation can be
		obtained by training a model, such as a decision tree, to reproduce the
		predictions $\learnthypothesis(\featurevec^{(\sampleidx)})$ for
		feature vectors $\featurevec^{(1)}, \ldots, \featurevec^{(\samplesize)}$ that cover
		the entire feature space well (Molnar, 2025).
		Algorithm~\ref{alg_explanation_surrogate_dict} states this construction: the
		feature vectors form a probing set, each labeled with the prediction of
		$\learnthypothesis$, and the explanation $g$ is learned via empirical risk minimization (ERM) over a
		hypothesis space $\hypospace'$ of candidate explanations. The scope is fixed by the
		probing set alone. The same algorithm delivers the local explanation of
		LIME when the probing set consists of feature vectors drawn around the one
		$\featurevec$ to be explained (Ribeiro et al., 2016, Sect.~3.2); the linear
		function $g$ of Fig.~\ref{fig_explanation_dict} is its output for a probing
		set near $\feature'$ and $\hypospace'$ the linear model.
		\begin{algorithm}[H]
		\caption{Explanation by a surrogate model, following
			Ribeiro et al. (2016), Sect.~3.2
			\label{alg_explanation_surrogate_dict}}
		\begin{algorithmic}[1]
			\Statex \textbf{Input:} learned hypothesis $\learnthypothesis$ to be
				explained; probing set of feature vectors $\featurevec^{(1)}, \ldots,
				\featurevec^{(\samplesize)}$; hypothesis space $\hypospace'$ of understandable
				hypotheses; loss function $\lossfunc{\cdot}{\cdot}$
			\For{$\sampleidx = 1, \ldots, \samplesize$}
				\State $\truelabel^{(\sampleidx)} := \learnthypothesis(\featurevec^{(\sampleidx)})$
					\Comment{the prediction to be explained serves as label}
			\EndFor
			\State $g \in \argmin_{\hypothesis \in \hypospace'} \frac{1}{\samplesize}
				\sum_{\sampleidx=1}^{\samplesize}
				\lossfunc{\pair{\featurevec^{(\sampleidx)}}{\truelabel^{(\sampleidx)}}}{\hypothesis}$
				\Comment{ERM}
			\Statex \textbf{Output:} the explanation $g$ and its training error on the probing
				set
		\end{algorithmic}
		\end{algorithm}
		Explanations of (the predictions obtained from) ML methods should be (i)
		understandable and (ii) faithful (Ribeiro et al., 2016, Sect.~3;
		Lipton, 2018). Understandable means that the user can work with the
		explanation on their own: from it alone, they can anticipate how the
		prediction changes when the features change. In
		Algorithm~\ref{alg_explanation_surrogate_dict}, this requirement falls on the choice
		of $\hypospace'$, which must consist of hypotheses the user can comprehend,
		such as linear maps or shallow decision trees. Faithful means that the
		explanation reflects the computation the learned hypothesis carries out.
		For the output $g$ of Algorithm~\ref{alg_explanation_surrogate_dict}, the
		training error on the probing set measures how faithfully $g$ reproduces
		$\learnthypothesis$ there, and nowhere else.
		An intensity map over the pixels of an image is faithful when manipulating a few of
		the pixels it highlights changes the prediction, while the same manipulation of
		pixels it leaves dark does not (Samek et al., 2017, Sect.~III-A, p.~2666).
\par
		Understandability and faithfulness can be contradictory requirements. An unfaithful
		explanation can be understandable while carrying no information about the
		computation. For some saliency methods, the map computed from an artificial neural network (ANN) with random
		weights is nearly the same as the map computed from the trained ANN
		(Adebayo et al., 2018, Sect.~3). Such a map traces the edges of the input image and
		carries nothing about the prediction, yet it looks like a plausible
		explanation. An explanation that agreed with the learned hypothesis
		everywhere would be that hypothesis again, which might be incomprehensible for a
		user, so a simpler explanation may have to give up faithfulness somewhere
		(Ribeiro et al., 2016, Sect.~3.2).
\par
		Fig.~\ref{fig_explanation_radar_dict} illustrates the understandable part of an
		explanation for a prediction made from weather radar. The features are the
		hourly precipitation over a $120$\,km box around Krems an der Donau, given on a grid of
		cells of $2$\,km by $2$\,km, the pixels of the radar image. The prediction is
		whether it will rain at Krems two hours after the hour shown. A CNN trained on
		these images is explained by a CAM (Selvaraju et al., 2017), shown in
		panel (b). A second explanation, in words, is printed below the panels, and its
		construction is described after the faithfulness test.
		These two explanations might be considered understandable by a non-expert user of a
		weather app. Their faithfulness, however, has to be verified by a perturbation test.
\par
		For the hour drawn in Fig.~\ref{fig_explanation_radar_dict}(a), the CNN assigns
		rain a probability close to one, and the CAM in panel (b)
		scores the bands of precipitation highest south and south-east of the town rather than
		on the rain already overhead. Faithfulness is tested by perturbing the radar image.
		Setting the precipitation to zero in the 360 cells with the highest
		CAM scores lowers the logit of rain, the number that the
		sigmoid function turns into the predicted probability, by $1.57$, while zeroing the
		360 cells with the lowest scores lowers it by only $0.26$. The same ordering holds for
		63 of the 72 hours of the test set, the last $30$ percent of the recorded hours,
		which were not used to train the CNN.
\par
		Fig.~\ref{fig_explanation_radar_dict} also depicts an explanation in the form of text,
		the sentence printed below the panels. A fixed rule turns the
		CAM of panel (b) into that sentence: it collects the 360 cells
		the map scores highest, the same cells the perturbation test above sets to zero, and
		computes two quantities from them. The first is the radius around Krems that holds
		nine in ten of these cells, rounded up to a multiple of $5$\,km. The last tenth sits
		in the far corners of the box and is left out, since it would stretch the radius over
		the whole box. The second is the compass direction in which these cells lie, seen
		from the center of Krems: their average position, each cell weighted by its
		CAM score, rounded to one of 16 compass sectors. Because the
		rule is fixed, the sentence is exactly as faithful as the map it summarizes, whose
		faithfulness the perturbation test above checks. The sentence can also be compared
		with the radar image in panel (a).
\begin{figure}[H]
   \begin{center}
  \begin{tikzpicture}[
		/pgfplots/colormap={rain}{color=(white); color=(blue!15); color=(cyan!60!blue);
		color=(blue!85!black)},
		/pgfplots/colormap={camscore}{color=(white); color=(orange!40);
		color=(red!80!black)}]
			\begin{axis}[name=rad, width=0.38\columnwidth, height=0.38\columnwidth, axis equal image, xlabel={west--east (cells of 2\,km)\\(a) precipitation (mm/h)}, ylabel={south--north (cells of 2\,km)}, label style={font=\small, align=center}, tick label style={font=\small}, colormap name=rain, point meta min=0, point meta max=16, colorbar, colorbar style={font=\small, width=0.25cm, ytick={0,4,8,12,16}, yticklabels={0,4,8,12,$\geq 16$}}, enlargelimits=false, axis on top]
				\addplot[matrix plot*, point meta=explicit, mesh/cols=60, mesh/ordering=rowwise] table[col sep=comma, x=x, y=y, meta=rr] {explanation_frame.csv}; \addplot[only marks, mark=*, mark size=2.2pt, white, mark options={draw=black, line width=0.5pt}] coordinates {(30,30)}; \draw[black, thin] (axis cs:31.2,31.2) -- (axis cs:36,38); \node[font=\scriptsize, anchor=south west, text=black, inner sep=1pt] at (axis cs:36,38) {Krems};
			\end{axis}
			\begin{axis}[name=cam, at={($(rad.east)+(2.3cm,0)$)}, anchor=west, width=0.38\columnwidth, height=0.38\columnwidth, axis equal image, xlabel={west--east (cells of 2\,km)\\(b) CAM score}, yticklabels={}, label style={font=\small, align=center}, tick label style={font=\small}, colormap name=camscore, point meta min=0, point meta max=1, colorbar, colorbar style={font=\small, width=0.25cm}, enlargelimits=false, axis on top]
				\addplot[matrix plot*, point meta=explicit, mesh/cols=60, mesh/ordering=rowwise] table[col sep=comma, x=x, y=y, meta=cam] {explanation_frame.csv}; \addplot[only marks, mark=*, mark size=2.2pt, white, mark options={draw=black, line width=0.5pt}] coordinates {(30,30)}; \draw[black, thin] (axis cs:31.2,31.2) -- (axis cs:36,38); \node[font=\scriptsize, anchor=south west, text=black, inner sep=1pt] at (axis cs:36,38) {Krems}; \end{axis}
			\node[anchor=north west, text width=0.88\columnwidth, align=left, font=\small]
				at ($(rad.south west)+(-0.9cm,-1.75cm)$)
				{Textual explanation: \emph{``Rain is likely at Krems in two hours since precipitation has been recorded within a radius of 60\,km, mainly to the south-south-east of the town.''}};
			\end{tikzpicture}
			\end{center}
  \caption{An explanation for a prediction made from weather radar on 15 September 2024 at 08:00 UTC. 
          (a) The observed precipitation in mm/h over a 120\,km box around Krems an der Donau (Austria), 
               marked by the circle, on a grid of $2$\,km cells; the color scale runs from white (dry) to 
              dark blue ($16$\,mm/h and above). From this image a CNN predicts rain at Krems two hours later.
	  (b) The CAM, which scores each cell by how much it contributed to the prediction; 
              the color scale runs from white (0) to dark red (1). The highest scores lie on the bands south and 
              south-east of the town rather than on the rain overhead. Below: a textual explanation generated from
	    the CAM, naming the radius and direction of the cells it scores highest. Data generated by \texttt{pythondemos/explanation.py}
			\label{fig_explanation_radar_dict}}
  \end{figure}

\subsection{Cross References}
ML, prediction, feature, data point, classification, explainability, XAI, interpretable ML, LIME, SHAP, counterfactual, CAM, feature importance, ERM

\subsection{References}
\begin{enumerate}
  \item Adebayo et al. (2018). Sanity Checks for Saliency Maps. In: Adv. Neural Inf. Process. Syst. (NeurIPS), pp.~9505--9515.
  \item Court of Justice of the European Union (2025). Judgment of 27 February 2025, CK v Dun \& Bradstreet Austria GmbH and Magistrat der Stadt Wien, C-203/22, ECLI:EU:C:2025:117. Court of Justice of the European Union, First Chamber. URL: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:62022CJ0203.
  \item Selvaraju et al. (2017). Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In: 2017 IEEE Int. Conf. Comput. Vis., pp.~618--626.
  \item Lipton (2018). The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3), pp.~31--57.
  \item Molnar (2025). Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. 3rd ed., Ebook.
  \item Montavon et al. (2018). Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73, pp.~1--15.
  \item Ribeiro et al. (2016). ``Why Should I Trust You?'': Explaining the Predictions of Any Classifier. In: Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, pp.~1135--1144.
  \item Samek et al. (2017). Evaluating the Visualization of What a Deep Neural Network Has Learned. IEEE Trans. Neural Netw. Learn. Syst., 28(11), pp.~2660--2673.
  \item Wachter et al. (2018). Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR. Harvard Journal of Law \& Technology, 31(2), pp.~841--887.
  \item Zeiler and Fergus (2014). Visualizing and Understanding Convolutional Networks. In: Proc. European Conf. Comput. Vis. (ECCV), pp.~818--833.
\end{enumerate}

\end{document}
