\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb}
\usepackage{algorithm,algpseudocode}
\usepackage{bm}
\usepackage{hyperref}
\usepackage{float}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{xcolor}
\definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0}
\definecolor{aaltoBlue}{RGB}{0, 51, 102}
\definecolor{pinegreen}{cmyk}{0.92,0,0.59,0.25}
\definecolor{royalblue}{cmyk}{1,0.50,0,0}
\definecolor{lavander}{cmyk}{0,0.48,0,0}
\definecolor{violet}{cmyk}{0.79,0.88,0,0}
\definecolor{lightblue}{RGB}{173, 216, 230}
\usepackage{tikz}
\usepackage{pgfplots}
\usetikzlibrary{calc,arrows.meta,positioning,fit,shapes.geometric,shapes.misc,intersections,patterns,decorations.pathreplacing,backgrounds,matrix}
\usepgfplotslibrary{fillbetween,groupplots}
\setcounter{secnumdepth}{-\maxdimen}
% Long unbreakable tokens (the ISO/IEC author names in citations, term names
% in the title) otherwise overflow the margin by a few points; a modest
% emergency stretch lets TeX loosen the paragraph instead.
\emergencystretch=2em
\usepackage{mleftright}
% Project macros used by this entry (auto-extracted).
\newcommand{\concept}{\mathcal{C}}
\newcommand{\defeq}{:=}
\newcommand{\featurevec}{\vx}
\newcommand{\hypothesis}{h}
\newcommand{\innerprod}[2]{\langle #1, #2 \rangle}
\newcommand{\learnthypothesis}{\widehat{\hypothesis}}
\newcommand{\norm}[1]{\Vert  {#1} \Vert}
\newcommand{\vv}[0]{{\bf v}}
\newcommand{\vw}[0]{{\bf w}}
\newcommand{\vx}[0]{{\bf x}}
\newcommand{\vz}[0]{{\bf z}}
\newcommand{\weights}{\vw}

\begin{document}

\section{concept activation vector (CAV)}

\subsection*{Author and editors}
Alexander Jung\textsuperscript{*}, Department of Computer Science, Aalto University, Espoo, Finland (ORCID \mbox{\href{https://orcid.org/0000-0001-7538-0990}{0000-0001-7538-0990}}) \\
Editors: Konstantina Olioumtsevits (ORCID \mbox{\href{https://orcid.org/0000-0003-2697-8659}{0000-0003-2697-8659}}), Ekkehard Schnoor (ORCID \mbox{\href{https://orcid.org/0000-0001-8252-1499}{0000-0001-8252-1499}}), Aalto University, Espoo, Finland \\
\textsuperscript{*}Corresponding author: \href{mailto:alex.jung@aalto.fi}{alex.jung@aalto.fi}



\subsection{Abstract}
A concept activation vector (CAV) is a direction in the
		activations of one hidden layer of a deep net
		that stands for a concept named by a user. The user supplies
		data points that carry the concept and data points that
		do not, and a linear classifier is trained to tell the two apart from
		their activations. The CAV is the direction perpendicular to
		the hyperplane that separates them, pointing to the concept
		side. Moving the activations of a data point a little
		along the CAV changes the score the deep net assigns to a
		label; the rate of that change is the sensitivity of the
		prediction to the concept. The fraction of data points
		with a positive sensitivity is the TCAV score. That score belongs
		to one layer and one label; it aggregates the
		per-data point sensitivities and thereby describes a whole
		class of data points rather than a single prediction.

\subsection{Definition}
A deep net predicts from five consecutive days of
		weather at a station whether it rains on the sixth
		day. Whether it uses the
		notion of a heat wave, five hot days in a row, to arrive at that
		prediction cannot be decided by inspecting a single
		neuron. A heat wave is not a feature the network was
		built with, since its features are the daily temperatures
		and rain amounts. Nor need it be represented by any one
		neuron (Kim et al., 2018, Sect.~2.3). A concept is a property
		that a user can name and exemplify but that the network was not built
		to compute, such as a heat wave. The concept is specified
		by examples instead: five-day windows that form a heat wave, the
		positive examples, and windows that do not, the negative examples
		(Kim et al., 2018, Sect.~3.1). A CAV turns such a
		concept into a direction in the activations of one hidden
		layer, the direction along which the positive examples lie
		apart from the negative ones.
\par
		Consider such a deep net, with several hidden layers,
		trained to predict the label of a data point from its
		feature vector; the trained network is the learned
		hypothesis $\learnthypothesis$. Fix one hidden layer
		and write $\vz = f(\featurevec)$ for its activations, with
		$f$ the function computed by the layers up to that
		point. The deep net is applied to the positive and to the
		negative examples of a concept $\concept$. A binary linear classifier
		$g(\vz)$, whose features are those activations, is
		trained to tell the two apart. Its decision boundary is a
		hyperplane; the CAV for $\concept$ is the normal vector
		$\widehat{\weights}$ of that hyperplane
		(Kim et al., 2018, Sect.~3.2).
\par
		A hyperplane has
		two normal vectors, one pointing to each side. The CAV is the
		one pointing towards the side that $g(\vz)$ assigns to the concept
		class. Training $g(\vz)$ with the positive examples as the positive
		class delivers this orientation, and when the two sets of
		activations are linearly separable, it is the side of the
		positive examples. In the terminology of
		mechanistic interpretability, this direction defines a
		feature of the trained deep net, the projection of
		the activations onto it: a human-interpretable concept
		represented by a direction in the space of activations.
\par
		Fig.~\ref{fig_cav_plane_dict} draws the CAV of the heat-wave
		concept for a deep net with two hidden layers of 16
		neurons, trained on the daily weather at Krems an der Donau
		of the training years 2000--2018 and tested on 2019--2024. A
		data point is a window of five days, its feature vector
		holds the daily maximum and minimum temperature and the rain amount
		of each day, and its label says whether at least 1\,mm of
		rain fell on the sixth day. The concept is specified by 40 heat
		waves, windows with a daily maximum of at least
		$30\,^{\circ}$C on all five days, and 40 windows that are not,
		drawn from the windows of the record. The plane drawn passes
		through the mean activation of the first hidden layer
		and is spanned by the CAV and the direction orthogonal to it
		along which the activations vary most. The concept
		hyperplane therefore meets the plane in a vertical line, and
		the CAV is its horizontal axis.
\par
		Two classification problems appear in the figure. The first is the concept classification that $g(\vz)$ solves on the activations. The second is the task the deep net is trained for, whose decision boundary on the plane is the solid line: rain is predicted to its right, where every heat wave lies. The CAV depends on the examples that specify the concept: training $g(\vz)$ on all 9127 windows of the record instead turns it by 26 degrees.
		\begin{figure}[H]
			\begin{center}
			\begin{tikzpicture}
			\begin{axis}[width=0.72\columnwidth, height=0.72\columnwidth,
				axis lines=left, xlabel={activation along the CAV},
				ylabel={activation along the leading orthogonal direction},
				label style={font=\small}, tick label style={font=\small},
				xmin=-1.35, xmax=2.05, ymin=-1.9, ymax=1.4,
				legend style={draw=none, fill=none, font=\scriptsize,
					at={(0.5,-0.16)}, anchor=north, legend columns=2}]
				\addplot[only marks, mark=*, mark size=1.2pt, gray]
					table[col sep=comma, x=z1, y=z2]
					{cav_windows20.csv};
				\addlegendentry{five-day window of the test years (gray dot)}
				\addplot[thick, no marks]
					table[col sep=comma, x=z1, y=z2]
					{cav_boundary.csv};
				\addlegendentry{decision boundary of the deep net (solid line)}
				\addplot[thick, dashed, no marks]
					table[col sep=comma, x=z1, y=z2]
					{cav_cavline.csv};
				\addlegendentry{concept hyperplane (dashed line)}
				\addplot[scatter, only marks, mark size=2.2pt,
					scatter src=explicit symbolic,
					scatter/classes={
						k1={mark=*, draw=black, fill=black},
						k0={mark=o, draw=black}}]
					table[col sep=comma, x=z1, y=z2, meta=grp]
					{cav_points20.csv};
				\addlegendentry{heat-wave example (filled circle)}
				\addlegendentry{non-heat-wave example (open circle)}
				\addplot[-{Stealth[length=6pt]}, very thick, forget plot]
					table[col sep=comma, x=z1, y=z2]
					{cav_arrow.csv};
				\node[font=\small, anchor=south] at (axis cs:1.4,-0.19) {CAV};
				\node[font=\small] at (axis cs:0.35,-1.6) {rain predicted};
				\node[font=\small] at (axis cs:-0.75,0.75) {no rain predicted};
			\end{axis}
			\end{tikzpicture}
			\end{center}
		\caption{The heat-wave CAV of a deep net that predicts rain
			at Krems an der Donau from the five preceding days, drawn in
			the plane through the mean activation of its first hidden
			layer spanned by the CAV (horizontal axis) and the
			direction orthogonal to it along which the
			activations vary most. Gray dots are $20$ of the $400$
			five-day windows of the test years drawn at random. Filled
			circles are $10$ of the $40$ windows supplied as heat-wave
			examples and open circles $10$ of the $40$ supplied as
			non-heat-wave examples, every fourth of each; the dashed line is
			the decision boundary of the linear classifier trained on all
			$80$, and the CAV is its normal vector. The solid line is the
			decision boundary of the deep net on this plane, with
			rain predicted to its right. Data generated by
			\texttt{pythondemos/cav.py}
			\label{fig_cav_plane_dict}}
		\end{figure}
		Like any explanation, a CAV must be understandable and
		faithful (see explanation). It is understandable by
		construction: the user supplies the examples that define the concept.
		Faithfulness is a separate question, because the concept is a
		direction in the activations whether or not the deep net
		responds to it. Testing with CAVs (TCAV) decides it.
\par
		Write the trained deep net as a concatenation of two
		functions, the activations $\vz = f(\featurevec)$ of
		the chosen hidden layer and a score $s(\vz)$ that the
		remaining layers assign to one label. The conceptual
		sensitivity of a data point is the directional derivative
		of that score along the unit CAV
		$\vv \defeq \widehat{\weights} / \norm{\widehat{\weights}}$, which
		for differentiable $s$ equals the inner product of the
		gradient of $s$ with $\vv$
		(Rudin, 1976, Example~9.18),
		\begin{equation}
			\label{equ_cav_sensitivity}
			S(\featurevec) \defeq \innerprod{\nabla s\big(f(\featurevec)\big)}{\vv}
			\text{.}
		\end{equation}
		It measures how much the score for that label moves when the
		activations move a little towards the concept
		(Kim et al., 2018, Sect.~3.3). The orientation
		convention gives the sign its meaning: $S(\featurevec) > 0$ says
		the score rises towards the concept, and the opposite
		normal vector would flip every sensitivity.
\par
		The sensitivity \eqref{equ_cav_sensitivity} delivers one number per
		data point. TCAV collects these numbers over one label:
		the TCAV score is the fraction of the data points carrying that
		label for which $S(\featurevec) > 0$
		(Kim et al., 2018, Sect.~3.4). A random set of
		non-concept data points also yields a CAV, so a single such
		fraction is not evidence alone. A random concept is a set of data points drawn at random and labeled as if they carried a concept. It is as likely to raise the score as to lower it, so its TCAV score, the fraction of positive sensitivities, is one half on average. The CAV is therefore refitted against many fresh draws of non-concept data points, and the resulting TCAV scores are compared with those of random concepts: a concept the deep net responds to scores consistently away from one half, a random one does not (Kim et al., 2018, Sect.~3.5).
\par
		The explanation delivered is a single number attached to a
		concept, a label and a layer. Unlike the sensitivity
		$S(\featurevec)$, which is defined for a single data point, it
		describes a whole class of data points rather than a single
		prediction (Kim et al., 2018, Sect.~2.1).
\par
		For the deep net of Fig.~\ref{fig_cav_plane_dict}, moving
		along the heat-wave CAV raises the rain score at every test window
		followed by rain. A cold wave is five days with a minimum of at
		most $-5\,^{\circ}$C, and moving along its CAV lowers that score at
		every one of those windows. The record agrees: rain follows 30\% of the heat waves and 16\% of the cold waves, against 23\% of all windows.
\par
		Training the linear classifier $g(\vz)$ and evaluating the sensitivity
		$S(\featurevec)$ both read quantities inside the deep net: the
		activations $f(\featurevec)$ of the chosen layer, and
		the gradient $\nabla s(\vz)$ of the score with respect to
		them. A CAV is therefore unavailable for a system that returns
		predictions for submitted feature vectors and nothing
		else, such as a deep net served behind an interface or
		supplied by a vendor. An explanation produced by local interpretable model-agnostic explanations (LIME),
		by contrast, needs predictions and nothing else: its local
		approximation is fitted to predictions obtained by querying
		the learned hypothesis $\learnthypothesis$
		(Ribeiro et al., 2016, Sect.~3). A second consequence of reading the
		internals is that a CAV is not a property of $\learnthypothesis$
		alone: the split into $f$ and $s$ is a choice of layer, and
		two deep nets that deliver the same $\learnthypothesis$
		through different layers can have different CAVs.
\par
		Both functions a CAV involves are defined on the space of
		activations: $g(\vz)$ approximates concept membership there,
		and the sensitivity $S(\featurevec)$ is a directional
		derivative of the score $s$ on that same space. A CAV therefore
		concerns the interpretability of the deep net rather
		than its explainability; mechanistic interpretability
		studies such directions.

\subsection{Cross References}
deep net, feature, linear classifier, trustworthy AI, interpretability, mechanistic interpretability, transparency, explanation, explainability, neuron, decision boundary, hyperplane, LIME

\subsection{References}
\begin{enumerate}
  \item Ribeiro et al. (2016). ``Why Should I Trust You?'': Explaining the Predictions of Any Classifier. In: Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, pp.~1135--1144.
  \item Rudin (1976). Principles of Mathematical Analysis. 3rd ed., McGraw-Hill, New York, NY, USA.
  \item Kim et al. (2018). Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In: Proc. 35th Int. Conf. Mach. Learn., pp.~2668--2677.
\end{enumerate}

\end{document}
