\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb}
\usepackage{algorithm,algpseudocode}
\usepackage{bm}
\usepackage{hyperref}
\usepackage{float}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{xcolor}
\definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0}
\definecolor{aaltoBlue}{RGB}{0, 51, 102}
\definecolor{pinegreen}{cmyk}{0.92,0,0.59,0.25}
\definecolor{royalblue}{cmyk}{1,0.50,0,0}
\definecolor{lavander}{cmyk}{0,0.48,0,0}
\definecolor{violet}{cmyk}{0.79,0.88,0,0}
\definecolor{lightblue}{RGB}{173, 216, 230}
\usepackage{tikz}
\usepackage{pgfplots}
\usetikzlibrary{calc,arrows.meta,positioning,fit,shapes.geometric,shapes.misc,intersections,patterns,decorations.pathreplacing,backgrounds,matrix}
\usepgfplotslibrary{fillbetween,groupplots}
\setcounter{secnumdepth}{-\maxdimen}
% Long unbreakable tokens (the ISO/IEC author names in citations, term names
% in the title) otherwise overflow the margin by a few points; a modest
% emergency stretch lets TeX loosen the paragraph instead.
\emergencystretch=2em
\usepackage{mleftright}
% Project macros used by this entry (auto-extracted).
\newcommand{\dataset}{\mathcal{D}}
\newcommand{\eigval}[1]{\lambda_{#1}}
\newcommand{\featuredim}{d}
\newcommand{\featureidx}{j}
\newcommand{\featurespace}{\mathcal{X}}
\newcommand{\featuretrafovec}{{\bf \Phi}}
\newcommand{\featurevec}{\vx}
\newcommand{\innerprod}[2]{\langle #1, #2 \rangle}
\newcommand{\mQ}{\mathbf{Q}}
\newcommand{\mR}[0]{{\bf R}}
\newcommand{\mW}[0]{{\bf W}}
\newcommand{\normgeneric}[2]{\mleft\lVert #1 \mright\rVert_{#2}}
\newcommand{\reals}{\mathbb{R}}
\newcommand{\sampleidx}{r}
\newcommand{\samplesize}{m}
\newcommand{\simplex}[1]{\Delta^{#1}}
\newcommand{\va}[0]{{\bf a}}
\newcommand{\vocabulary}{\mathcal{V}}
\newcommand{\vw}[0]{{\bf w}}
\newcommand{\vx}[0]{{\bf x}}
\newcommand{\vz}[0]{{\bf z}}
\newcommand{\weights}{\vw}

\begin{document}

\section{decoder}

\subsection*{Author and editors}
Alexander Jung\textsuperscript{*}, Department of Computer Science, Aalto University, Espoo, Finland (ORCID \mbox{\href{https://orcid.org/0000-0001-7538-0990}{0000-0001-7538-0990}}) \\
Editors: Konstantina Olioumtsevits (ORCID \mbox{\href{https://orcid.org/0000-0003-2697-8659}{0000-0003-2697-8659}}), Ekkehard Schnoor (ORCID \mbox{\href{https://orcid.org/0000-0001-8252-1499}{0000-0001-8252-1499}}), Aalto University, Espoo, Finland \\
\textsuperscript{*}Corresponding author: \href{mailto:alex.jung@aalto.fi}{alex.jung@aalto.fi}



\subsection{Abstract}
A decoder is a map that turns the code of a data point,
		the short vector an encoder computes from it, back
		into the space the data point itself lives in. The
		encoder and the decoder are learned together by empirical risk minimization (ERM),
		with a loss function that measures the reconstruction error
		between a data point and what the decoder returns for its
		code. In principal component analysis (PCA) both maps are linear
		and the decoder is the transpose of the encoder, so
		the reconstruction is the orthogonal projection of the
		feature vector onto the subspace spanned by the retained
		eigenvectors, and the average squared reconstruction error
		is the sum of the discarded eigenvalues. An
		autoencoder replaces both linear maps by artificial neural networks (ANNs),
		and its decoder is often discarded once the code is wanted for a
		later learning task. A decoder may return a probability distribution
		instead of a point: the decoder of a variational autoencoder (VAE)
		is a conditional probability distribution over feature vectors given a
		latent variable. The output step of a large language model
		(LLM) is a decoder as well, turning a vector into a
		probability distribution over the vocabulary, which is what lets text
		be generated one token at a time.

\subsection{Definition}
The weather station at Krems an der Donau
		(Austria) records the lowest and the highest air temperature of
		each day, so a day is a point of the plane with these two
		features as its coordinates. Storing one number per
		day instead of two is possible, but that number is of no use
		unless some map turns it back into a pair of temperatures. A
		decoder is such a map.
\par
		A decoder is a map $\featuretrafovec^{*}: \featurespace'
		\rightarrow \featurespace$ from a space $\featurespace'$ of codes
		back to the feature space $\featurespace$ that holds the
		feature vector $\featurevec$ of a data point. Its
		counterpart is the encoder $\featuretrafovec: \featurespace
		\rightarrow \featurespace'$, which computes the code $\vz =
		\featuretrafovec(\featurevec)$, a vector of $\featuredim'$
		numbers against the $\featuredim$ numbers of the feature vector,
		with $\featuredim' < \featuredim$. The two maps form an
		autoencoder, and they are learned together by empirical risk minimization (ERM) with
		a loss function that measures the reconstruction error
		$\featurevec -
		\featuretrafovec^{*}(\featuretrafovec(\featurevec))$
		(Goodfellow et al., 2016, Sect.~14.1). No label enters
		this learning task: what the decoder has to reproduce is the
		data point itself. What the decoder returns need not be a
		point of $\featurespace$: it may be a probability distribution, over
		$\featurespace$ or over a vocabulary of tokens.
		Fig.~\ref{fig_decoder_kinds_dict} shows the three shapes the
		decoder takes in principal component analysis (PCA), in a variational autoencoder (VAE), and in a large language model (LLM).
		\begin{figure}[H]
		\centering
		\begin{tikzpicture}[>=Latex, thick, font=\small,
			box/.style={draw, rounded corners, inner sep=5pt}]
			\node (z) at (0,0) {$\vz \in \featurespace'$};
			\node[box, right=0.7cm of z] (dec) {decoder $\featuretrafovec^{*}$};
			\node[right=1.5cm of dec, yshift=1.3cm, align=left] (a)
				{a feature vector $\widehat{\featurevec} \in \featurespace$\\
				 {\scriptsize PCA, autoencoder}};
			\node[right=1.5cm of dec, align=left] (b)
				{a probability distribution over $\featurespace$\\
				 {\scriptsize VAE}};
			\node[right=1.5cm of dec, yshift=-1.3cm, align=left] (c)
				{a probability distribution over $\vocabulary$\\
				 {\scriptsize LLM}};
			\draw[->] (z) -- (dec);
			\draw[->] (dec.east) -- (a.west);
			\draw[->] (dec.east) -- (b.west);
			\draw[->] (dec.east) -- (c.west);
		\end{tikzpicture}
		\caption{The three shapes a decoder takes. The arrows leave the
			decoder for a feature vector $\widehat{\featurevec}$, which
			is what PCA and an autoencoder return, for a
			probability distribution over the feature space, which is what the
			decoder of a VAE returns, and for a probability distribution over
			the vocabulary $\vocabulary$, which is the output step
			of an LLM
			\label{fig_decoder_kinds_dict}}
		\end{figure}
		PCA is the linear case (Hotelling, 1933). Its
		encoder is a matrix $\mW \in \reals^{\featuredim'
		\times \featuredim}$ with orthonormal rows, so the code of a
		feature vector is $\vz = \mW \featurevec$, and its decoder is
		the transpose $\mW^{\top}$ of that matrix
		(Hastie et al., 2009, Sect.~14.5.1, Eq.~(14.52)).
		The reconstruction $\mW^{\top} \mW \featurevec$ differs from
		$\featurevec$ by a vector orthogonal to the rows of $\mW$,
		and no other point of the subspace they span lies closer to
		$\featurevec$: the reconstruction is the
		orthogonal projection of $\featurevec$ onto that
		subspace (Hastie et al., 2009, Sect.~14.5.1).
		The rows of $\mW$ are the eigenvectors of the
		sample covariance matrix $\widehat{\mQ}$ belonging to its $\featuredim'$
		largest eigenvalues (Bishop, 2006, Sect.~12.1.2).
\par
		The eigenvalues of $\widehat{\mQ}$, in decreasing order
		$\eigval{1} \geq \cdots \geq \eigval{\featuredim} \geq 0$, state
		what every code length costs. For a dataset $\dataset =
		\{\featurevec^{(1)}, \ldots, \featurevec^{(\samplesize)}\}$ of
		feature vectors with sample mean zero, the average squared
		reconstruction error is the sum of the eigenvalues the code
		drops (Bishop, 2006, Sect.~12.1.2, Eq.~(12.18)),
		\begin{equation}
			\label{equ_decoder_eigenvalues_dict}
			\frac{1}{\samplesize} \sum_{\sampleidx=1}^{\samplesize}
			\normgeneric{\featurevec^{(\sampleidx)} - \mW^{\top} \mW
			\featurevec^{(\sampleidx)}}{2}^{2} =
			\sum_{\featureidx=\featuredim'+1}^{\featuredim}
			\eigval{\featureidx} \text{.}
		\end{equation}
		The identity \eqref{equ_decoder_eigenvalues_dict} holds at every
		code length $\featuredim' = 1, \ldots, \featuredim$. The archive at
		Krems holds eight measurements per day: the lowest, highest and
		mean air temperature, the precipitation, the sunshine duration, the
		humidity, the air pressure, and the wind speed. Each of the eight
		is scaled to unit sample variance, because they are measured
		in different units. A code of two numbers then leaves $2.91$ of the
		total $8$, so the decoder recovers $64$ percent of the total
		sample variance. Fig.~\ref{fig_decoder_krems_dict} draws the
		decoder of a code of one number for the two temperatures, and both
		sides of the identity \eqref{equ_decoder_eigenvalues_dict} at
		every code length.
		\begin{figure}[H]
		\centering
		\begin{tikzpicture}
		\begin{groupplot}[
			group style={group size=2 by 1, horizontal sep=1.9cm},
			width=0.44\linewidth, height=0.42\linewidth,
			axis lines=left,
			tick label style={font=\scriptsize},
			label style={font=\scriptsize, align=center},
			legend style={draw=none, fill=none, font=\scriptsize,
				at={(0.5,-0.52)}, anchor=north},
			legend cell align=left,
		]
			\nextgroupplot[
				xlabel={lowest temperature, centered\\(a) $20$ days and their reconstructions},
				ylabel={highest temperature, centered},
				xmin=-20, xmax=20, ymin=-28, ymax=34]
			\addplot[gray!70, dotted, thin, forget plot, -,
				quiver={u=\thisrow{r1}-\thisrow{x1}, v=\thisrow{r2}-\thisrow{x2}}]
				table[col sep=comma, x=x1, y=x2]{decoder_points_plot.csv};
			\addplot[black, thick]
				table[col sep=comma, x=x1, y=x2]{decoder_direction.csv};
			\addlegendentry{retained direction (solid line)}
			\addplot[only marks, mark=o, mark size=1.8pt, black]
				table[col sep=comma, x=x1, y=x2]{decoder_points_plot.csv};
			\addlegendentry{day (open circles)}
			\addplot[only marks, mark=square*, mark size=1.5pt, black]
				table[col sep=comma, x=r1, y=r2]{decoder_points_plot.csv};
			\addlegendentry{reconstruction (filled squares)}
			\nextgroupplot[
				xlabel={code length $\featuredim'$\\(b) error on the eight measurements},
				ylabel={reconstruction error},
				xtick={1,...,8}, ymin=-0.3, ymax=5.2]
			\addplot[black, thick]
				table[col sep=comma, x=dprime, y=err]{decoder_errord.csv};
			\addlegendentry{error \eqref{equ_decoder_eigenvalues_dict} (solid curve)}
			\addplot[only marks, mark=x, mark size=3.2pt, thick, black]
				table[col sep=comma, x=dprime, y=dropped]{decoder_errord.csv};
			\addlegendentry{discarded eigenvalues (crosses)}
		\end{groupplot}
		\end{tikzpicture}
		\caption{(a) Twenty of the $366$ days of $2024$ at Krems, every
			$19$th, each a data point with the lowest and the highest
			temperature of the day as its two features, centered
			(open circles), the reconstruction of each from a code of one
			number (filled squares), and the retained direction (solid
			line); the dotted segments are the error the decoder leaves,
			which averages $7.94$ squared degrees over all $366$ days and
			equals the discarded eigenvalue. (b) The average squared
			reconstruction error \eqref{equ_decoder_eigenvalues_dict} on
			the eight measurements of a day, each scaled to unit
			sample variance, against the code length $\featuredim'$
			(solid curve), together with the sum
			$\sum_{\featureidx = \featuredim'+1}^{\featuredim}
			\eigval{\featureidx}$ of the discarded eigenvalues
			(crosses). Data generated by
			\texttt{pythondemos/decoder.py}
			\label{fig_decoder_krems_dict}}
		\end{figure}
		An autoencoder keeps the reconstruction error and replaces
		both linear maps by artificial neural networks (ANNs), so its decoder is a network
		that turns the code back into a feature vector
		(Goodfellow et al., 2016, Sect.~14.1). A
		denoising autoencoder changes what the decoder is asked for: the
		encoder reads a corrupted feature vector and the decoder
		has to return the clean one
		(Goodfellow et al., 2016, Sect.~14.5). After training,
		the decoder is often discarded and the code kept, so that a later
		learning task learns a hypothesis on the code rather
		than on the raw features.
\par
		A decoder need not return a point. The decoder of a VAE is a
		conditional probability distribution $p_{\weights}(\featurevec \mid \vz)$ over
		feature vectors given a latent variable $\vz$, with
		model parameters $\weights$: given a code it returns a
		probability distribution over the feature vectors that could have
		produced it (Kingma and Welling, 2014, Sect.~2.1). Drawing $\vz$
		from a fixed probability distribution and then a feature vector from the
		decoder makes the pair a generative model. The two readings
		meet where that conditional probability distribution is Gaussian with mean
		$\mR \vz$, for a matrix $\mR \in \reals^{\featuredim \times
		\featuredim'}$, and a spread that does not depend on $\vz$. Its
		log-likelihood function is then a decreasing affine function of the
		average squared reconstruction error, so the matrix $\mR$
		that maximizes the one minimizes the other.
\par
		The step that turns a vector into text is a decoder too. An
		LLM computes a vector $\vz \in \reals^{\featuredim}$
		from the tokens read so far. The decoder maps it to a
		probability distribution over the vocabulary $\vocabulary$ by a
		linear map followed by the softmax function
		(Vaswani et al., 2017, Sect.~3.4). The rows of that
		linear map are $\va^{(1)}, \ldots, \va^{(|\vocabulary|)} \in
		\reals^{\featuredim}$, and the decoder returns
		\begin{equation}
			\label{equ_decoder_vocabulary_dict}
			\big( \featuretrafovec^{*}(\vz) \big)_{v} =
			\frac{\exp \big( \innerprod{\va^{(v)}}{\vz} \big)}
			{\sum_{v'=1}^{|\vocabulary|}
			\exp \big( \innerprod{\va^{(v')}}{\vz} \big)}
			\quad \text{for } v = 1, \ldots, |\vocabulary| \text{.}
		\end{equation}
		Every value the quotient \eqref{equ_decoder_vocabulary_dict}
		assigns is nonnegative, and the values sum to one over $v = 1,
		\ldots, |\vocabulary|$, so the decoder returns a point of the
		probability simplex $\simplex{|\vocabulary|-1}$. Drawing one
		token from that probability distribution, appending it to the
		tokens read so far, and repeating is how text is generated
		one token at a time (Vaswani et al., 2017, Sect.~3):
		without the decoder, the vector $\vz$ names no token
		at all.
\par
		A vocabulary of eight tokens is already enough to
		check the decoder \eqref{equ_decoder_vocabulary_dict} numerically.
		Cutting the highest temperature of each day at Krems into eight
		bins of about $45$ days each, and choosing the rows $\va^{(v)}$ by
		maximum likelihood, reproduces the observed frequencies of the
		next token to within $0.0014$. A sequence of
		$20{,}001$ tokens generated one at a time then matches the
		observed shares of the eight tokens to within $0.009$.
\par
		The reconstructions of a linear decoder lie in a subspace of
		dimension $\featuredim'$, whatever the dataset looks like.
		The identity \eqref{equ_decoder_eigenvalues_dict} says what that
		costs on average, and nothing about where the reconstructions
		land. Read back in the recorded units, the reconstruction of a
		day from a code of two numbers gives $128$ of the $366$ days a
		negative precipitation, down to $-7.3$ millimeters, and $17$ days
		a negative sunshine duration. No day of the record takes a
		negative value in either measurement. A linear decoder learned
		with the squared error loss recovers the subspace PCA
		finds, so only a nonlinear decoder places a reconstruction off a
		subspace (Goodfellow et al., 2016, Sect.~14.1), which
		is why the decoder of an autoencoder is an ANN and
		not a matrix.
\par
		What the linear decoder buys is a closed form: its
		eigenvectors come from the sample covariance matrix with no
		training run, and its eigenvalues state the error of
		every code length in advance. Curving the reconstruction set does
		not always lower the error: with the same code of one number, the
		average of the days in each of eight bins leaves $5.10$ on days it
		was not learned from, against the $5.07$ of the linear decoder.
\par
		A decoder that returns a probability distribution rather than a point buys
		a probability for every reconstruction and the ability to
		generate, since a feature vector or a token can be drawn
		from what the decoder returns. The price is a probabilistic model
		that has to be learned and can be wrong, and a support fixed in
		advance. The Gaussian decoder above commits to a spread that does
		not depend on the code. The vocabulary of eight
		tokens resolves the maximum daytime temperature of a day
		only to its bin: $1.4$ degrees in root mean square, and $10.3$
		degrees for the coldest token, which holds every day at or
		below $6.7$ degrees. No sequence of these eight
		tokens tells two days of one bin apart.

\subsection{Cross References}
encoder, autoencoder, denoising autoencoder, VAE, latent variable, conditional probability distribution, probabilistic model, PCA, feature learning, dimensionality reduction, feature transformation, ERM, squared error loss, orthogonal projection, subspace, eigenvector, generative model, softmax function, probability simplex, vocabulary, LLM

\subsection{References}
\begin{enumerate}
  \item Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media, New York, NY, USA.
  \item Goodfellow et al. (2016). Deep Learning. MIT Press, Cambridge, MA, USA.
  \item Hotelling (1933). Analysis of a Complex of Statistical Variables into Principal Components. Journal of Educational Psychology, 24(6), pp.~417--441.
  \item Kingma and Welling (2014). Auto-Encoding Variational Bayes. In: 2nd International Conference on Learning Representations (ICLR).
  \item Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed., Springer Science+Business Media, New York, NY, USA.
  \item Vaswani et al. (2017). Attention is all you need. In: Adv. Neural Inf. Process. Syst., pp.~5998--6008.
\end{enumerate}

\end{document}
