\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb}
\usepackage{algorithm,algpseudocode}
\usepackage{bm}
\usepackage{hyperref}
\usepackage{float}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{xcolor}
\definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0}
\definecolor{aaltoBlue}{RGB}{0, 51, 102}
\definecolor{pinegreen}{cmyk}{0.92,0,0.59,0.25}
\definecolor{royalblue}{cmyk}{1,0.50,0,0}
\definecolor{lavander}{cmyk}{0,0.48,0,0}
\definecolor{violet}{cmyk}{0.79,0.88,0,0}
\definecolor{lightblue}{RGB}{173, 216, 230}
\usepackage{tikz}
\usepackage{pgfplots}
\usetikzlibrary{calc,arrows.meta,positioning,fit,shapes.geometric,shapes.misc,intersections,patterns,decorations.pathreplacing,backgrounds,matrix}
\usepgfplotslibrary{fillbetween,groupplots}
\setcounter{secnumdepth}{-\maxdimen}
% Long unbreakable tokens (the ISO/IEC author names in citations, term names
% in the title) otherwise overflow the margin by a few points; a modest
% emergency stretch lets TeX loosen the paragraph instead.
\emergencystretch=2em
\usepackage{mleftright}
% Project macros used by this entry (auto-extracted).
\newcommand{\dataset}{\mathcal{D}}
\newcommand{\defeq}{:=}
\newcommand{\featuredim}{d}
\newcommand{\featuretrafovec}{{\bf \Phi}}
\newcommand{\featurevec}{\vx}
\newcommand{\lossfunc}[2]{L\left(#1,#2 \right)}
\newcommand{\mE}{\mathbf{E}}
\newcommand{\mW}[0]{{\bf W}}
\newcommand{\normgeneric}[2]{\mleft\lVert #1 \mright\rVert_{#2}}
\newcommand{\reals}{\mathbb{R}}
\newcommand{\sampleidx}{r}
\newcommand{\samplesize}{m}
\newcommand{\ve}[0]{{\bf e}}
\newcommand{\vocabulary}{\mathcal{V}}
\newcommand{\vx}[0]{{\bf x}}
\newcommand{\vz}[0]{{\bf z}}

\begin{document}

\section{encoder}

\subsection*{Author and editors}
Alexander Jung\textsuperscript{*}, Department of Computer Science, Aalto University, Espoo, Finland (ORCID \mbox{\href{https://orcid.org/0000-0001-7538-0990}{0000-0001-7538-0990}}) \\
Editors: Konstantina Olioumtsevits (ORCID \mbox{\href{https://orcid.org/0000-0003-2697-8659}{0000-0003-2697-8659}}), Ekkehard Schnoor (ORCID \mbox{\href{https://orcid.org/0000-0001-8252-1499}{0000-0001-8252-1499}}), Aalto University, Espoo, Finland \\
\textsuperscript{*}Corresponding author: \href{mailto:alex.jung@aalto.fi}{alex.jung@aalto.fi}



\subsection{Abstract}
An encoder is a map that sends the raw feature vector of a
		data point to a shorter vector, a code of $\featuredim'$
		numbers with $\featuredim'$ below the number $\featuredim$ of raw
		features. Its counterpart is the decoder, which maps a
		code back to a vector of the original length, and the two are
		judged together: the decoder applied to the code of a
		data point should return approximately the feature vector
		the encoder started from. The average loss that remains, the
		reconstruction error, is what training minimizes.
		Principal component analysis (PCA) uses a linear map as the encoder, a matrix
		whose rows are the eigenvectors of the sample covariance matrix
		belonging to its largest eigenvalues. An autoencoder
		learns the encoder jointly with the decoder, so it can be a
		deep net rather than a linear map. A large language model (LLM) begins
		with the token-to-vector encoder, which carries every token
		of a vocabulary to one row of an embedding
		matrix. The code length decides what the decoder can
		rebuild, and the code kept in place of the raw feature vectors
		is what makes an encoder a feature transformation for a later
		learning task. What a shorter code drops is beyond every
		decoder, so a code learned to reconstruct well need not
		serve that learning task well.

\subsection{Definition}
The weather station at Krems an der Donau
		(Austria) records eight numbers for each day: the lowest, the
		highest, and the mean air temperature, the precipitation, the
		sunshine duration, the humidity, the air pressure, and the wind
		speed. Two numbers computed from the eight are enough to tell a day
		in January from a day in July, and a second map rebuilds the eight
		from the two with a small error. The map that delivers the two
		numbers is an encoder. It sends the raw feature vector
		$\featurevec \in \reals^{\featuredim}$ of a data point to a
		shorter vector $\vz \in \reals^{\featuredim'}$, its
		\emph{code},
		$$\featuretrafovec: \reals^{\featuredim} \rightarrow
		\reals^{\featuredim'}: \featurevec \mapsto \vz \defeq
		\featuretrafovec(\featurevec) \text{,}$$
		with a code length $\featuredim'$ below the number $\featuredim$ of
		raw features. Its counterpart is the decoder
		$\featuretrafovec^{*}: \reals^{\featuredim'} \rightarrow
		\reals^{\featuredim}$, which maps a code back to a vector with
		$\featuredim$ entries (Goodfellow et al., 2016, Ch.~14).
\par
		Encoders differ in the hypothesis space they are picked from.
		Principal component analysis (PCA) picks a linear map: the encoder is a matrix
		$\mW \in \reals^{\featuredim' \times \featuredim}$ with $\vz = \mW
		\featurevec$, whose rows are the eigenvectors of the
		sample covariance matrix of the feature vectors belonging to its
		$\featuredim'$ largest eigenvalues, and the decoder is
		the transpose $\mW^{\top}$
		(Bishop, 2006, Sect.~12.1.2; Hotelling, 1933). An
		autoencoder picks the encoder from a hypothesis space of
		deep nets and learns it jointly with the decoder, so
		that neither map need be linear
		(Goodfellow et al., 2016, Sect.~14.1). A large language model (LLM) begins with
		the token-to-vector step, an encoder on text. Every token of
		a vocabulary $\vocabulary$ carries one row of an
		embedding matrix $\mE \in \reals^{|\vocabulary| \times
		\featuredim'}$. Picking the row of the token with index $v$
		is the linear map $\vz = \mE^{\top} \ve^{(v)}$ applied to the
		one-hot vector $\ve^{(v)} \in \reals^{|\vocabulary|}$ of that
		token (Vaswani et al., 2017, Sect.~3.4).
\par
		What makes a code useful is that the decoder can rebuild the
		data point from it. The reconstruction error of an encoder on
		a dataset $\dataset = \{\featurevec^{(1)}, \ldots,
		\featurevec^{(\samplesize)}\}$ of sample size $\samplesize$ is
		the average loss between a feature vector and what the
		decoder returns for its code,
		\begin{equation}
			\label{equ_encoder_reconstruction_dict}
			\frac{1}{\samplesize} \sum_{\sampleidx=1}^{\samplesize}
			\lossfunc{\featurevec^{(\sampleidx)}}{\featuretrafovec^{*}
			\big( \featuretrafovec \big( \featurevec^{(\sampleidx)} \big)
			\big)} \text{.}
		\end{equation}
		A common loss function is the squared error loss
		$\normgeneric{\featurevec -
		\featuretrafovec^{*}(\featuretrafovec(\featurevec))}{2}^{2}$, and
		minimizing the reconstruction error
		\eqref{equ_encoder_reconstruction_dict} over a hypothesis space of
		encoders and a second one of decoders is empirical risk minimization (ERM). No
		labels enter it: what the decoder has to reproduce is
		the data point itself, so the training of an encoder is
		a form of self-supervised learning.
\par
		Each of the eight measurements of a day at Krems is scaled to zero
		sample mean and unit sample variance, so that they carry
		comparable weight. Over the $366$ days of $2024$, the PCA encoder with
		$\featuredim' = 2$ then leaves a reconstruction error of $2.91$ out
		of the total variance $8.00$, and the eight reconstructed
		measurements of 1 January 2024 differ from the scaled originals by
		at most $0.80$.
		Fig.~\ref{fig_encoder_dict}(a) places $21$ of the days at their two
		code numbers, split at the median of the maximum
		temperature, and Fig.~\ref{fig_encoder_dict}(b) compares the
		reconstruction error of this encoder with what a random encoder
		with orthonormal rows leaves as the code grows.
		\begin{figure}[H]
		\centering
		\begin{tikzpicture}
		\begin{groupplot}[
			group style={group size=2 by 1, horizontal sep=1.7cm},
			width=0.5\linewidth, height=0.44\linewidth,
			axis lines=left,
			tick label style={font=\scriptsize},
			label style={font=\scriptsize, align=center},
			legend style={draw=none, fill=none, font=\scriptsize,
				at={(0.5,-0.42)}, anchor=north},
			legend cell align=left,
		]
			\nextgroupplot[
				xlabel={first code number $z_{1}$\\(a) the code of $21$ of the $366$ days},
				ylabel={second code number $z_{2}$}, enlargelimits=0.14]
			\addplot[only marks, mark=square, mark size=1.8pt, black,
				restrict expr to domain={\thisrow{warm}}{0:0}]
				table[col sep=comma, x=z1, y=z2]{encoder_code_plot.csv};
			\addlegendentry{cold day (open squares)}
			\addplot[only marks, mark=*, mark size=1.5pt, black,
				restrict expr to domain={\thisrow{warm}}{1:1}]
				table[col sep=comma, x=z1, y=z2]{encoder_code_plot.csv};
			\addlegendentry{warm day (filled circles)}
			\nextgroupplot[
				xlabel={code length $\featuredim'$\\(b) error of two encoders},
				ylabel={reconstruction error}, xtick={1,...,8}, ymin=0]
			\addplot[black, thick]
				table[col sep=comma, x=dprime, y=pca]{encoder_errors.csv};
			\addlegendentry{PCA encoder (solid curve)}
			\addplot[black, thick, dashed]
				table[col sep=comma, x=dprime, y=random]{encoder_errors.csv};
			\addlegendentry{random encoder (dashed curve)}
		\end{groupplot}
		\end{tikzpicture}
		\caption{The PCA encoder of the $366$ days of $2024$ at
			Krems, each day a data point with eight scaled
			measurements. (a) The two code numbers of every $18$th day,
			$21$ in all, drawn as a cold day (open squares) or a warm day
			(filled circles) by whether the maximum temperature lies below
			or above its median. (b) The reconstruction error against
			the code length $\featuredim'$ for the PCA encoder (solid
			curve) and for a random encoder with orthonormal rows
			(dashed curve). Data generated by
			\texttt{pythondemos/encoder.py}
			\label{fig_encoder_dict}}
		\end{figure}
		The code length decides what survives. On the days at Krems, the
		reconstruction error of the PCA encoder falls from $4.38$ at
		one number to $2.91$ at two and $0.88$ at four, and at every code
		length it equals the sum of the eigenvalues the code drops
		(Bishop, 2006, Sect.~12.1.2, Eq.~(12.18)). A random encoder
		with orthonormal rows leaves $5.71$ at two numbers, so which
		encoder is used matters beside how long its code is. Losing
		something is not inevitable: the token-to-vector encoder shortens a
		one-hot vector of length $|\vocabulary|$ to $\featuredim'$
		numbers and still loses nothing, since distinct tokens carry
		distinct rows of $\mE$ and the nearest row therefore recovers the
		token. What training keeps is usually the encoder
		alone: the decoder is dropped and the code serves as the
		feature transformation of a later learning task, which
		makes an encoder a feature learning and a dimensionality reduction method
		(Bengio et al., 2013).
\par
		The code of two numbers leaves a reconstruction error of $2.91$,
		and that loss is not spread evenly over the eight measurements.
		Of the unit sample variance of each measurement, the code
		keeps $96$ percent of the mean temperature and $3$ percent of the
		wind speed. What the dropped eigenvectors carried is beyond
		every decoder, not only the transpose $\mW^{\top}$.
		Of the $80$ pairs of days whose codes lie closer than $0.1$
		together, 14 February and 22 November 2024 differ most in wind
		speed, $1.0$ against $4.5$ meters per second, and the decoder
		returns $1.76$ meters per second for both. Whether the loss matters
		is decided by the later learning task and not by the
		reconstruction error alone. A day counts as windy when its mean wind
		speed lies above the median of the $366$ values, which holds
		for $130$ of them. A linear classifier learned by least squares has an
		accuracy of $98$ percent from all eight scaled measurements
		and $66$ percent from the two code numbers, against the $64$ percent
		reached by calling every day calm. One new feature chosen from
		the labels instead of from the sample covariance matrix, the
		linear discriminant analysis (LDA) direction, reaches $98$ percent as well. A code learned to
		reconstruct well is therefore not a code learned to predict well,
		which is the difference LDA draws against PCA. An
		encoder is also fixed once learned, and the dataset it was
		learned from decides how well it codes a later day. The PCA
		encoder of the $183$ days of January to March and of October to
		December leaves $1.93$ on those days and $10.87$ on the other
		$183$, against $2.48$ for the encoder of the whole year. What the
		code buys is its shortness: fewer numbers per data point to
		store and to hand to a later learning task, and at
		$\featuredim' = 2$ a plot of the dataset on two axes.

\subsection{Cross References}
decoder, autoencoder, VAE, feature learning, feature transformation, dimensionality reduction, PCA, LDA, embedding, token, latent variable, ERM, squared error loss

\subsection{References}
\begin{enumerate}
  \item Bengio et al. (2013). Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8), pp.~1798--1828.
  \item Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media, New York, NY, USA.
  \item Goodfellow et al. (2016). Deep Learning. MIT Press, Cambridge, MA, USA.
  \item Hotelling (1933). Analysis of a Complex of Statistical Variables into Principal Components. Journal of Educational Psychology, 24(6), pp.~417--441.
  \item Vaswani et al. (2017). Attention is all you need. In: Adv. Neural Inf. Process. Syst., pp.~5998--6008.
\end{enumerate}

\end{document}
