\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb}
\usepackage{algorithm,algpseudocode}
\usepackage{bm}
\usepackage{hyperref}
\usepackage{float}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{xcolor}
\definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0}
\definecolor{aaltoBlue}{RGB}{0, 51, 102}
\definecolor{pinegreen}{cmyk}{0.92,0,0.59,0.25}
\definecolor{royalblue}{cmyk}{1,0.50,0,0}
\definecolor{lavander}{cmyk}{0,0.48,0,0}
\definecolor{violet}{cmyk}{0.79,0.88,0,0}
\definecolor{lightblue}{RGB}{173, 216, 230}
\usepackage{tikz}
\usepackage{pgfplots}
\usetikzlibrary{calc,arrows.meta,positioning,fit,shapes.geometric,shapes.misc,intersections,patterns,decorations.pathreplacing,backgrounds,matrix}
\usepgfplotslibrary{fillbetween,groupplots}
\setcounter{secnumdepth}{-\maxdimen}
% Long unbreakable tokens (the ISO/IEC author names in citations, term names
% in the title) otherwise overflow the margin by a few points; a modest
% emergency stretch lets TeX loosen the paragraph instead.
\emergencystretch=2em
\usepackage{mleftright}
% Project macros used by this entry (auto-extracted).
\newcommand{\actfun}{\sigma}
\newcommand{\dataset}{\mathcal{D}}
\newcommand{\defeq}{:=}
\newcommand{\emprisk}[2]{\widehat{L}\big(#1|#2\big)}
\newcommand{\featurevec}{\vx}
\newcommand{\hypothesis}{h}
\newcommand{\mW}[0]{{\bf W}}
\newcommand{\predictedlabel}{\widehat{\truelabel}}
\newcommand{\trainset}{\dataset^{(\rm train)}}
\newcommand{\truelabel}{y}
\newcommand{\va}[0]{{\bf a}}
\newcommand{\valset}{\dataset^{(\rm val)}}
\newcommand{\vb}[0]{{\bf b}}
\newcommand{\vw}[0]{{\bf w}}
\newcommand{\vx}[0]{{\bf x}}
\newcommand{\weights}{\vw}

\begin{document}

\section{deep net}

\subsection*{Author and editors}
Alexander Jung\textsuperscript{*}, Department of Computer Science, Aalto University, Espoo, Finland (ORCID \mbox{\href{https://orcid.org/0000-0001-7538-0990}{0000-0001-7538-0990}}) \\
Editors: Konstantina Olioumtsevits (ORCID \mbox{\href{https://orcid.org/0000-0003-2697-8659}{0000-0003-2697-8659}}), Ekkehard Schnoor (ORCID \mbox{\href{https://orcid.org/0000-0001-8252-1499}{0000-0001-8252-1499}}), Aalto University, Espoo, Finland \\
\textsuperscript{*}Corresponding author: \href{mailto:alex.jung@aalto.fi}{alex.jung@aalto.fi}

\subsection{Synonyms}
deep neural network

\subsection{Abstract}
A deep net is an artificial neural network (ANN) whose hypothesis maps are
		compositions of several layers, each of which applies an
		affine map and then an activation function to every entry of the result.
		The number of layers that carry learnable model parameters
		is the depth of the net, and deep learning is the name of the
		machine learning (ML) methods that use a deep net as their model. The
		layers before the last one compute a
		feature transformation learned from the data, and the
		last layer applies a linear hypothesis to the
		activations they deliver. A deep net therefore makes the
		choice of features part of training instead of a
		design decision taken beforehand. Its model parameters, the
		weights and offset terms of every layer, are learned
		by gradient descent (GD) on the training error, with the gradient computed
		by backpropagation. Depth is a property of the net, not of a
		single hypothesis: adding layers enlarges the
		hypothesis space and lowers the training error, while beyond some
		depth the validation error rises again, an instance of
		overfitting. Depth is therefore a hyperparameter, and a
		feature transformation chosen beforehand can reach a lower
		validation error with far fewer model parameters.

\subsection{Definition}
The daily maximum temperature at Krems an
		der Donau (Austria) rises from January to July and falls again
		towards December. The $365$ days of $2023$ form a training set
		$\trainset$ and the $366$ days of $2024$ a validation set $\valset$,
		with the day of the year as the single feature of a day and
		its maximum temperature as the label. A hypothesis that
		is a linear function of the date can only rise or only fall over
		the year. Its training error, the average squared error loss over
		$\trainset$, is $79.4$ squared degrees, barely below the $83.4$ of
		the constant hypothesis that always predicts the
		sample mean of the labels. A deep net replaces that
		single raw feature by features that it computes
		itself. A hidden layer of eight artificial neurons
		turns the date into eight activations, and a linear
		hypothesis of those eight numbers has a training error of
		$16.9$ and a validation error of $23.7$ on $\valset$
		(Fig.~\ref{fig_deepnet_dict}(a)).
		\begin{figure}[H]
		\centering
		\begin{tikzpicture}
		\begin{groupplot}[
			group style={group size=2 by 1, horizontal sep=2.1cm},
			width=0.40\linewidth, height=0.38\linewidth,
			axis lines=left,
			tick label style={font=\scriptsize},
			label style={font=\scriptsize, align=center},
			legend style={draw=none, fill=none, font=\scriptsize,
				at={(0.5,-0.46)}, anchor=north},
			legend cell align=left,
		]
			\nextgroupplot[
				xlabel={day of the year\\(a) learned hypotheses and $20$ of the $365$ days},
				ylabel={maximum temperature\\in degrees Celsius},
				xmin=0, xmax=370, ymin=-4, ymax=36]
			\addplot[dashed, black] table[col sep=comma, x=doy, y=h1]{deepnet_curves.csv};
			\addlegendentry{depth $1$ (dashed line)}
			\addplot[thick, black] table[col sep=comma, x=doy, y=h2]{deepnet_curves.csv};
			\addlegendentry{depth $2$ (solid curve)}
			\addplot[dotted, black] table[col sep=comma, x=doy, y=h5]{deepnet_curves.csv};
			\addlegendentry{depth $5$ (dotted curve)}
			\addplot[only marks, mark=*, mark size=1.4pt, black] table[col sep=comma, x=doy, y=tmax]{deepnet_days_plot.csv};
			\addlegendentry{day of $2023$ (filled circles)}
			\nextgroupplot[
				xlabel={depth $L$ of the net\\(b) the errors against the depth},
				ylabel={average squared error loss\\in squared degrees},
				xmin=0.6, xmax=5.4, ymin=0, ymax=95, xtick={1,2,3,4,5}]
			\addplot[thick, black, mark=*, mark size=1.6pt] table[col sep=comma, x=depth, y=trainerr]{deepnet_depth.csv};
			\addlegendentry{training error on $\trainset$ (solid curve)}
			\addplot[dashed, black, mark=square, mark size=2pt] table[col sep=comma, x=depth, y=valerr]{deepnet_depth.csv};
			\addlegendentry{validation error on $\valset$ (dashed curve)}
		\end{groupplot}
		\end{tikzpicture}
		\caption{Nets of depth $1$ to $5$ trained on the $365$ days of
			$2023$ at Krems, every hidden layer eight
			artificial neurons wide. (a) The learned
			hypotheses of depth $1$ (dashed line), depth $2$ (solid
			curve), and depth $5$ (dotted curve) over the year, with $20$
			of the $365$ days, every $18$th day in calendar order (filled
			circles). (b) The training error (solid curve, filled circles)
			and the validation error on the $366$ days of $2024$ (dashed curve,
			open squares) against the depth. Data generated by
			\texttt{pythondemos/deepnet.py}
			\label{fig_deepnet_dict}}
		\end{figure}
		A deep net is an artificial neural network (ANN) whose hypothesis maps are
		compositions of $L$ consecutive layers. Layer $\ell$
		reads in the activations $\va^{(\ell-1)}$ of the previous
		layer, applies an affine map to them, and then an
		activation function $\actfun$ to every entry of the result,
		\begin{equation}
			\label{equ_deepnet_layer_dict}
			\va^{(\ell)} = \actfun\big( \mW^{(\ell)} \va^{(\ell-1)}
			+ \vb^{(\ell)} \big) \text{ for } \ell = 1, \ldots, L-1
			\text{,}
		\end{equation}
		starting from the raw feature vector $\va^{(0)} \defeq
		\featurevec$ of a data point. The last layer applies an
		affine map alone and returns the prediction
		\begin{equation}
			\label{equ_deepnet_output_dict}
			\hypothesis^{(\weights)}(\featurevec) = \mW^{(L)}
			\va^{(L-1)} + \vb^{(L)} \text{.}
		\end{equation}
		The number $L$ of layers that carry learnable
		model parameters is the depth of the net, and the term
		\emph{deep learning} comes from it
		(Bishop and Bishop, 2024, Sect.~6.3;
		Goodfellow et al., 2016, p.~164). Deep learning
		covers the machine learning (ML) methods that use a deep net as their
		model. The activation function
		is prescribed rather than learned, most often the rectified linear unit (ReLU)
		$\actfun(z) = \max\{0, z\}$
		(Goodfellow et al., 2016, Sect.~6.3). Without it, every
		layer would be an affine map, the layer recursion
		\eqref{equ_deepnet_layer_dict} and the output map
		\eqref{equ_deepnet_output_dict} would compose to an affine map
		again, and the net would define the same hypothesis space as a
		linear model (Fig.~\ref{fig_deepnet_layers_dict}).
		\begin{figure}[H]
		\centering
		\begin{tikzpicture}[font=\small, >=Latex, node distance=2.6mm,
			box/.style={draw, rounded corners, align=center,
				font=\scriptsize, minimum height=9mm, text width=16mm}]
			\node (x) {$\featurevec$};
			\node[box, right=of x] (af1)
				{affine map\\$\mW^{(1)} \va^{(0)} + \vb^{(1)}$};
			\node[box, right=of af1] (sig) {$\actfun$ on every entry};
			\node[right=of sig] (a1) {$\va^{(1)}$};
			\node[right=of a1] (dots) {$\cdots$};
			\node[right=of dots] (aL) {$\va^{(L-1)}$};
			\node[box, right=of aL] (out)
				{affine map\\$\mW^{(L)} \va^{(L-1)} + \vb^{(L)}$};
			\node[right=of out] (y) {$\predictedlabel$};
			\foreach \a/\b in {x/af1, af1/sig, sig/a1, a1/dots,
				dots/aL, aL/out, out/y}
				\draw[->] (\a) -- (\b);
			\draw[decorate, decoration={brace, mirror, amplitude=4pt}]
				([yshift=-2mm]af1.south west) -- ([yshift=-2mm]aL.south east)
				node[midway, below=5pt, font=\scriptsize, align=center]
				{layers $1$ to $L-1$: a learned
				feature transformation};
			\draw[decorate, decoration={brace, mirror, amplitude=4pt}]
				([yshift=-2mm]out.south west) -- ([yshift=-2mm]out.south east)
				node[midway, below=5pt, font=\scriptsize, align=center]
				{layer $L$: a linear\\hypothesis};
		\end{tikzpicture}
		\caption{A deep net of depth $L$ as a composition of
			layers. Each of the first $L-1$ layers applies an
			affine map to the activations of the previous one and
			then the activation function $\actfun$ to every entry, and the last
			layer applies an affine map alone, so that the
			prediction $\predictedlabel$ is a linear hypothesis
			of the learned features $\va^{(L-1)}$
			\label{fig_deepnet_layers_dict}}
		\end{figure}
		The composition splits at the last layer, and that split
		makes a deep net a feature learning method. The layers $1$ to
		$L-1$ map the raw feature vector $\featurevec$ to the
		activations $\va^{(L-1)}$, a feature transformation
		learned from the data, and the output map
		\eqref{equ_deepnet_output_dict} applies an ordinary linear
		hypothesis to these learned features
		(Goodfellow et al., 2016, Sect.~6.1). A deep net therefore
		makes the choice of features part of training instead
		of a design decision taken beforehand. A deep net whose last hidden
		layer has two artificial neurons turns the date at
		Krems into two learned features. Least squares on those
		two reaches a training error of $14.5$, against the $14.6$ of the net's
		own last layer and the $79.4$ of least squares on the raw
		feature. A learned feature transformation is often
		useful beyond the learning task it was trained for, so the
		layers before the last are kept and only the last
		layer is learned again on a smaller dataset
		(Bishop and Bishop, 2024, Sect.~6.3.4).
\par
		The model parameters $\weights$ collect the weights in
		every matrix $\mW^{(\ell)}$ and the offset terms in every
		vector $\vb^{(\ell)}$, and a choice of $\weights$ selects one
		hypothesis $\hypothesis^{(\weights)}$ of the net. They are
		learned by empirical risk minimization (ERM): gradient descent (GD) with a fixed learning rate lowers
		the training error $\emprisk{\hypothesis^{(\weights)}}{\trainset}$
		by one gradient step after another. Each of them needs the
		gradient of the training error with respect to $\weights$,
		and backpropagation computes it one layer at a time
		(Rumelhart et al., 1986).
\par
		Depth is a property of the net, not of a single hypothesis
		in it, and what a larger depth changes is the size of the
		hypothesis space. Fig.~\ref{fig_deepnet_dict}(b) shows the
		training error at Krems falling from $79.4$ at depth $1$ to $12.0$
		at depth $5$, every hidden layer eight
		artificial neurons wide. The validation error does not follow it
		down. It is smallest at depth $2$, at $23.7$, and rises to $29.5$
		at depth $5$, an instance of overfitting. Depth is therefore
		a hyperparameter, chosen on $\valset$ rather than fixed
		beforehand.
\par
		The depth-$2$ net lowers the validation error to $23.7$ from the $88.2$
		of least squares on the raw feature. A
		feature transformation chosen beforehand reaches a lower
		validation error with a fifth of the model parameters.
		Least squares on the sine and cosine of the date at the annual
		period and at half of it needs five model parameters and has a
		validation error of $22.6$, against the $23.7$ of the net with its $25$.
		Two further costs come with the net. Its training error is not a
		convex function of $\weights$ and has many local minima
		(Goodfellow et al., 2016, Sect.~8.2.2). GD therefore
		returns a different hypothesis from every initialization:
		five initializations of the depth-$2$ net give training errors
		between $16.9$ and $19.1$, and four of the five end above the
		smallest of them. The learned features also carry no reading
		of their own, where the sine and cosine name the annual cycle. An
		explainable artificial intelligence (XAI) method explains the trained net only by an
		approximation that cannot be fully faithful to it
		(Rudin, 2019, Sect.~2). A deep net is the answer where no
		such feature transformation can be written down beforehand: in
		image classification the first hidden layer delivers
		edges, the second corners and contours, and the third parts of
		objects (Goodfellow et al., 2016, Fig.~1.2).

\subsection{Cross References}
ANN, layer, activation, activation function, ReLU, backpropagation, deep learning, feature learning, feature transformation, overfitting, hyperparameter, interpretable ML, CNN, LLM

\subsection{References}
\begin{enumerate}
  \item Bishop and Bishop (2024). Deep Learning: Foundations and Concepts. Springer Nature, Cham, Switzerland.
  \item Goodfellow et al. (2016). Deep Learning. MIT Press, Cambridge, MA, USA.
  \item Rumelhart et al. (1986). Learning Representations by Back-Propagating Errors. Nature, 323, pp.~533--536.
  \item Rudin (2019). Stop explaining black box machine learning models for high-stakes decisions and use interpretable models instead. Nature Mach. Intell., 1(5), pp.~206--215.
\end{enumerate}

\end{document}
