\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb}
\usepackage{algorithm,algpseudocode}
\usepackage{bm}
\usepackage{hyperref}
\usepackage{float}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{xcolor}
\definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0}
\definecolor{aaltoBlue}{RGB}{0, 51, 102}
\definecolor{pinegreen}{cmyk}{0.92,0,0.59,0.25}
\definecolor{royalblue}{cmyk}{1,0.50,0,0}
\definecolor{lavander}{cmyk}{0,0.48,0,0}
\definecolor{violet}{cmyk}{0.79,0.88,0,0}
\definecolor{lightblue}{RGB}{173, 216, 230}
\usepackage{tikz}
\usepackage{pgfplots}
\usetikzlibrary{calc,arrows.meta,positioning,fit,shapes.geometric,shapes.misc,intersections,patterns,decorations.pathreplacing,backgrounds,matrix}
\usepgfplotslibrary{fillbetween,groupplots}
\setcounter{secnumdepth}{-\maxdimen}
% Long unbreakable tokens (the ISO/IEC author names in citations, term names
% in the title) otherwise overflow the margin by a few points; a modest
% emergency stretch lets TeX loosen the paragraph instead.
\emergencystretch=2em
\usepackage{mleftright}
% Project macros used by this entry (auto-extracted).
\newcommand{\bayeshypothesis}{h^{*}}
\newcommand{\dataset}{\mathcal{D}}
\newcommand{\defeq}{:=}
\newcommand{\expect}{\mathbb{E} }
\newcommand{\feature}{x}
\newcommand{\featuredim}{d}
\newcommand{\featuremtx}{\mX}
\newcommand{\featurespace}{\mathcal{X}}
\newcommand{\featurevec}{\vx}
\newcommand{\hypothesis}{h}
\newcommand{\labelvec}{\vy}
\newcommand{\learnthypothesis}{\widehat{\hypothesis}}
\newcommand{\lossfunc}[2]{L\left(#1,#2 \right)}
\newcommand{\mX}[0]{{\bf X}}
\newcommand{\mvnormal}[2]{\mathcal{N}\left(#1,#2\right)}
\newcommand{\normgeneric}[2]{\mleft\lVert #1 \mright\rVert_{#2}}
\newcommand{\pair}[2]{\left( #1,#2 \right)}
\newcommand{\predictedlabel}{\widehat{\truelabel}}
\newcommand{\probdist}{\mathbb{P}}
\newcommand{\reals}{\mathbb{R}}
\newcommand{\sampleidx}{r}
\newcommand{\samplesize}{m}
\newcommand{\trainset}{\dataset^{(\rm train)}}
\newcommand{\truelabel}{y}
\newcommand{\vw}[0]{{\bf w}}
\newcommand{\vx}[0]{{\bf x}}
\newcommand{\vy}[0]{{\bf y}}
\newcommand{\weights}{\vw}

\begin{document}

\section{squared error loss}

\subsection*{Author and editors}
Alexander Jung\textsuperscript{*}, Department of Computer Science, Aalto University, Espoo, Finland (ORCID \mbox{\href{https://orcid.org/0000-0001-7538-0990}{0000-0001-7538-0990}}) \\
Editors: Konstantina Olioumtsevits (ORCID \mbox{\href{https://orcid.org/0000-0003-2697-8659}{0000-0003-2697-8659}}), Ekkehard Schnoor (ORCID \mbox{\href{https://orcid.org/0000-0001-8252-1499}{0000-0001-8252-1499}}), Aalto University, Espoo, Finland \\
\textsuperscript{*}Corresponding author: \href{mailto:alex.jung@aalto.fi}{alex.jung@aalto.fi}

\subsection{Synonyms}
squared loss, quadratic loss

\subsection{Abstract}
The squared error loss of a hypothesis on a
		data point with a numeric label is the square of the
		difference between the label and the prediction that
		the hypothesis delivers for the features of the
		data point. Its average over a training set is the
		training error, also called the mean squared error (MSE), and
		minimizing that average over a linear model is
		least squares, whose solution the normal equations give
		in closed form. Up to an increasing affine transformation, the
		squared error loss is the negative log-likelihood function of the
		label under Gaussian noise around the prediction, so
		least squares is maximum likelihood estimation. The
		hypothesis with the smallest risk under the squared
		error loss is the conditional expectation of the label given
		the features, and the risk of any other
		hypothesis exceeds it by the expected squared difference
		from that conditional expectation. The squared error loss is
		convex and differentiable in the prediction with a
		slope proportional to the error, so a single outlier moves
		the learned hypothesis in proportion to its error, while the
		absolute error loss and the Huber loss bound that displacement.

\subsection{Definition}
A weather service predicts the
		maximum daytime temperature at Krems an der Donau from the morning
		minimum of the same day. Each day is a data point with the
		feature $\feature$, the morning minimum, and the label
		$\truelabel \in \reals$, the maximum. A hypothesis
		$\hypothesis$ delivers the prediction $\predictedlabel =
		\hypothesis(\feature)$, and the maximum measured in the afternoon
		says by how many degrees that prediction was off. For a
		data point with feature vector $\featurevec \in
		\featurespace$ and numeric label $\truelabel \in \reals$,
		consider a hypothesis $\hypothesis: \featurespace
		\rightarrow \reals$. The squared error loss it incurs is the square
		of the difference between the label and its prediction
		(Hastie et al., 2009, Sect.~2.4),
		\begin{equation}
			\label{equ_sqerrloss_def_dict}
			\lossfunc{\pair{\featurevec}{\truelabel}}{\hypothesis} \defeq
			\big(\truelabel - \hypothesis(\featurevec)\big)^{2} \text{.}
		\end{equation}
\par
		On 8 December 2024 the morning minimum at Krems was $3.0$ degrees
		and the maximum $4.6$ degrees. The linear hypothesis
		$\learnthypothesis(\feature) = 9.62 + 1.01 \feature$ learned below
		predicts $12.66$ degrees for that morning, so its squared error
		loss \eqref{equ_sqerrloss_def_dict} on that day is $(4.6 -
		12.66)^{2} \approx 65.0$ squared degrees. The squared error loss is
		measured in the squared unit of the label, vanishes exactly
		when the prediction is correct, and is the same for a
		prediction too high as for one too low by the same amount.
\par
		As a function of the prediction $\predictedlabel$, the
		squared error loss \eqref{equ_sqerrloss_def_dict} is a parabola
		with its minimum at $\predictedlabel = \truelabel$, as
		Fig.~\ref{fig_sqerrloss_curves_dict} shows. It is convex and
		differentiable, with derivative $-2(\truelabel -
		\predictedlabel)$: its slope grows in proportion to the error. For
		a linear model $\hypothesis(\featurevec) = \weights^{\top}
		\featurevec$, the squared error loss is a convex and
		differentiable function of the model parameters
		$\weights$ as well, which is what makes gradient descent (GD) applicable. The
		absolute error loss $|\truelabel - \predictedlabel|$ is convex too
		but non-smooth at $\predictedlabel = \truelabel$, and its
		slope is $\pm 1$ whatever the size of the error. The
		Huber loss with threshold $\delta > 0$ is quadratic for a
		small error and linear for a large one
		(Hastie et al., 2009, Eq.~(10.23)). In terms of the
		error $e \defeq \truelabel - \hypothesis(\featurevec)$, it is
		\begin{equation}
			\label{equ_sqerrloss_huber_dict}
			\begin{cases}
				e^{2} & \text{for } |e| \leq \delta \text{,} \\
				2 \delta |e| - \delta^{2} & \text{otherwise} \text{,}
			\end{cases}
		\end{equation}
		so it agrees with the squared error loss for an error up to
		$\delta$ and its slope is capped at $2 \delta$ beyond
		(Huber, 1992).
		\begin{figure}[H]
		\centering
		\begin{tikzpicture}
		\begin{axis}[
			width=0.72\linewidth, height=0.46\linewidth,
			axis lines=left,
			xlabel={error $e = \truelabel - \hypothesis(\featurevec)$},
			ylabel={loss},
			xmin=-3.2, xmax=3.2, ymin=0, ymax=9.5,
			domain=-3:3, samples=121,
			tick label style={font=\scriptsize},
			label style={font=\scriptsize},
			legend style={draw=none, fill=none, font=\scriptsize,
				at={(0.5,0.97)}, anchor=north},
			legend cell align=left,
		]
			\addplot[black, thick] {x^2};
			\addlegendentry{squared error loss (solid curve)}
			\addplot[black, thick, dashed] {abs(x)};
			\addlegendentry{absolute error loss (dashed curve)}
			\addplot[black, thick, dotted] {ifthenelse(abs(x)<1, x^2, 2*abs(x)-1)};
			\addlegendentry{Huber loss (dotted curve)}
		\end{axis}
		\end{tikzpicture}
		\caption{The squared error loss \eqref{equ_sqerrloss_def_dict}
			(solid curve), the absolute error loss (dashed curve), and the
			Huber loss \eqref{equ_sqerrloss_huber_dict} with
			threshold $\delta = 1$ (dotted curve) as functions of
			the error $e = \truelabel - \hypothesis(\featurevec)$. The three
			agree at an error of $\pm 1$; beyond it, only the squared error
			loss keeps steepening
			\label{fig_sqerrloss_curves_dict}}
		\end{figure}
		The average of the squared error loss over a training set
		$\trainset = \big\{
		\pair{\featurevec^{(\sampleidx)}}{\truelabel^{(\sampleidx)}}
		\big\}_{\sampleidx=1}^{\samplesize}$,
		\begin{equation}
			\label{equ_sqerrloss_trainerr_dict}
			\frac{1}{\samplesize} \sum_{\sampleidx=1}^{\samplesize}
			\big(\truelabel^{(\sampleidx)} -
			\hypothesis(\featurevec^{(\sampleidx)})\big)^{2} \text{,}
		\end{equation}
		is the training error of $\hypothesis$, also called its mean squared error (MSE),
		and empirical risk minimization (ERM) selects from a model the hypothesis with
		the smallest training error. With a linear model, ERM under
		the squared error loss is linear regression by least squares
		(Legendre, 1805). The training error
		\eqref{equ_sqerrloss_trainerr_dict} then reads $(1/\samplesize)
		\normgeneric{\labelvec - \featuremtx \weights}{2}^{2}$, with the
		feature matrix $\featuremtx \in \reals^{\samplesize \times
		\featuredim}$ whose rows are the feature vectors and the
		label vector $\labelvec \in \reals^{\samplesize}$. Its
		gradient vanishes exactly at the solutions of the
		normal equations $\featuremtx^{\top} \featuremtx \weights =
		\featuremtx^{\top} \labelvec$, and when $\featuremtx$ has full
		column rank the unique solution is
		(Bishop, 2006, Sect.~3.1.1)
		\begin{equation}
			\label{equ_sqerrloss_closedform_dict}
			\widehat{\weights} = \big(\featuremtx^{\top}
			\featuremtx\big)^{-1} \featuremtx^{\top} \labelvec \text{.}
		\end{equation}
		The closed form \eqref{equ_sqerrloss_closedform_dict} needs no
		iteration, so there is no update whose fixed point the
		solution would be. When $\featuremtx^{\top} \featuremtx$ is too
		large to invert, GD on the training error
		\eqref{equ_sqerrloss_trainerr_dict} is used instead, and its
		fixed points are again the solutions of the
		normal equations (see linear regression).
\par
		Take as training set every $18$th day of $2024$ at Krems, $21$
		days, with the constant feature $1$ appended to the morning
		minimum so that $\widehat{\weights}$ holds an intercept and a
		slope. The closed form \eqref{equ_sqerrloss_closedform_dict} gives
		the line $\learnthypothesis(\feature) = 9.62 + 1.01 \feature$ of
		Fig.~\ref{fig_sqerrloss_demo_dict}(a), with training error $12.75$
		squared degrees, or $3.57$ degrees after taking the root. The best
		constant prediction under the squared error loss is the
		sample mean of the labels, $17.14$ degrees, and its
		training error is their sample variance, $62.05$: the average
		squared error of a constant $c$ is that variance plus $(c -
		17.14)^{2}$. Under the absolute error loss the best constant is the
		median of the labels instead, $15.80$ degrees.
\par
		The squared error loss is the loss behind maximum likelihood
		estimation when the label is modeled as the prediction
		plus Gaussian noise. Let $\truelabel = \hypothesis(\featurevec) +
		\varepsilon$ with a Gaussian random variable (Gaussian RV) $\varepsilon$ of mean zero
		and variance $\sigma^{2}$, so that $\truelabel$ given
		$\featurevec$ has the probability density function (pdf)
		$\mvnormal{\hypothesis(\featurevec)}{\sigma^{2}}$. The negative
		logarithm of that pdf at the observed label is
		(Bishop, 2006, Sect.~3.1.1, Eqs.~(3.11)--(3.12))
		\begin{equation}
			\label{equ_sqerrloss_nll_dict}
			- \ln \mvnormal{\hypothesis(\featurevec)}{\sigma^{2}}
			(\truelabel) = \frac{1}{2 \sigma^{2}} \big(\truelabel -
			\hypothesis(\featurevec)\big)^{2} + \frac{1}{2} \ln \big(2 \pi
			\sigma^{2}\big) \text{,}
		\end{equation}
		an increasing affine function of the squared error loss
		\eqref{equ_sqerrloss_def_dict}. For data points that are
		realizations of independent and identically distributed (i.i.d.) random variables (RVs), the negative
		log-likelihood function of the training set is the sum of the terms
		\eqref{equ_sqerrloss_nll_dict} over its data points, which is
		$\samplesize / (2 \sigma^{2})$ times the training error
		\eqref{equ_sqerrloss_trainerr_dict} plus a constant. The
		maximum likelihood estimate of $\weights$ is therefore the
		least squares solution \eqref{equ_sqerrloss_closedform_dict},
		whatever $\sigma^{2}$ is, and the maximum likelihood estimate of
		$\sigma^{2}$ is the training error itself, $12.75$ for the $21$
		days. The logistic loss is the negative log-likelihood function for a
		binary label, and Laplacian instead of Gaussian noise gives
		the absolute error loss (Murphy, 2012, Sect.~7.4).
\par
		The risk of a hypothesis is its expected loss
		under a probabilistic model with a joint probability distribution
		$\probdist^{(\featurevec, \truelabel)}$ of the feature vector and
		the label. Under the squared error loss, the risk is
		minimized by the conditional expectation
		$\bayeshypothesis(\featurevec) = \expect\{\truelabel \mid
		\featurevec\}$, the Bayes estimator for this loss
		(Bishop, 2006, Sect.~1.5.5, Eq.~(1.89);
		Hastie et al., 2009, Sect.~2.4, Eq.~(2.13)). The
		risk of any hypothesis $\hypothesis$ splits into two
		parts (Bishop, 2006, Eq.~(1.90)),
		\begin{equation}
			\label{equ_sqerrloss_risk_dict}
			\expect\big\{\big(\truelabel -
			\hypothesis(\featurevec)\big)^{2}\big\} =
			\expect\big\{\big(\bayeshypothesis(\featurevec) -
			\hypothesis(\featurevec)\big)^{2}\big\} +
			\expect\big\{\big(\truelabel -
			\bayeshypothesis(\featurevec)\big)^{2}\big\} \text{,}
		\end{equation}
		the expected squared difference between $\hypothesis$ and the
		conditional expectation, and the expected conditional
		variance of the label, which no hypothesis can
		reduce. For a hypothesis learned from a random
		training set, the first part of the decomposition
		\eqref{equ_sqerrloss_risk_dict} splits further into the squared
		bias and the variance of the learned prediction,
		the decomposition behind the bias--variance trade-off
		(Hastie et al., 2009, Sect.~7.3, Eq.~(7.9)). Under
		the absolute error loss the Bayes estimator is the conditional
		median (Hastie et al., 2009, Eq.~(2.18)). The
		two constant predictions above are the training set
		versions of these two facts, with the average over the $21$ days in
		place of the expectation.
\par
		The growing slope of the parabola has a cost. A data point
		whose label is far from the prediction enters the
		gradient of the training error
		\eqref{equ_sqerrloss_trainerr_dict} in proportion to its error, so
		an outlier pulls the learned hypothesis toward itself
		without bound (Hastie et al., 2009, Sect.~10.6). For
		least squares the pull is explicit. Adding $\Delta$ to the
		label of the $\sampleidx$th training day changes the solution
		\eqref{equ_sqerrloss_closedform_dict} by $(\featuremtx^{\top}
		\featuremtx)^{-1} \featurevec^{(\sampleidx)} \Delta$. The
		prediction for that day changes by $\ell_{\sampleidx}
		\Delta$, where $\ell_{\sampleidx} = (\featurevec^{(\sampleidx)})^{\top}
		(\featuremtx^{\top} \featuremtx)^{-1} \featurevec^{(\sampleidx)}$
		is the leverage of the day.
\par
		Among the $21$ days, the one with the warmest morning, 17 July 2024
		with minimum $19.5$ and maximum $27.8$ degrees, has leverage
		$0.192$. Each degree added to its label moves the
		least squares line at that morning by $0.192$ degrees, which
		is $5.76$ degrees for $\Delta = 30$, and the slope of the line
		rises from $1.01$ to $1.37$ (Fig.~\ref{fig_sqerrloss_demo_dict}(b),
		solid curve). Under the absolute error loss, whose ERM solution is
		least absolute deviation regression, the line moves by at most $1.75$ degrees, and
		not at all once $\Delta$ exceeds $3$ (dashed curve). The
		slope of the absolute error loss is $\pm 1$, so once the corrupted day is
		above the line, its term in the optimality condition no longer
		depends on the size of its error. The Huber loss with $\delta
		= 3$ moves its line by $1.93$ degrees and not at all beyond
		$\Delta = 6$ (dotted curve), since its slope is capped. This
		bounded pull is what robustness to outliers means for
		a loss. The absolute error loss is non-smooth, and the
		Huber loss adds the threshold $\delta$ as a
		hyperparameter; the squared error loss has neither drawback,
		and no such bound.
		\begin{figure}[H]
		\centering
		\begin{tikzpicture}
		\begin{groupplot}[
			group style={group size=2 by 1, horizontal sep=1.7cm},
			width=0.5\linewidth, height=0.44\linewidth,
			axis lines=left,
			tick label style={font=\scriptsize},
			label style={font=\scriptsize, align=center},
			legend style={draw=none, fill=none, font=\scriptsize},
			legend cell align=left,
		]
			\nextgroupplot[
				xlabel={morning minimum temperature\\(a) the $21$ training days and the least squares line},
				ylabel={maximum daytime temperature},
				xmin=-9, xmax=23, ymin=0, ymax=50,
				legend style={at={(0.03,0.97)}, anchor=north west}]
			\addplot[gray!70, dashed, thin, forget plot, -,
				quiver={u=0, v=\thisrow{pred}-\thisrow{tmax}}]
				table[col sep=comma, x=tmin, y=tmax]{sqerrloss_points.csv};
			\addplot[black, thick] table[col sep=comma, x=tmin, y=ls]{sqerrloss_lines.csv};
			\addlegendentry{least squares (solid line)}
			\addplot[only marks, mark=o, mark size=1.6pt, black]
				table[col sep=comma, x=tmin, y=tmax]{sqerrloss_points.csv};
			\addlegendentry{training days (circles)}
			\addplot[only marks, mark=square, mark size=3pt, black, thick,
				restrict expr to domain={\thisrow{corrupted}}{1:1}]
				table[col sep=comma, x=tmin, y=tmax]{sqerrloss_points.csv};
			\addlegendentry{17 July (square)}
			\nextgroupplot[
				xlabel={degrees $\Delta$ added to one label\\(b) displacement of the learned line at that day},
				ylabel={displacement in degrees},
				xmin=0, xmax=30, ymin=0, ymax=8.5,
				legend style={at={(0.03,0.97)}, anchor=north west}]
			\addplot[black, thick] table[col sep=comma, x=delta, y=ls]{sqerrloss_outlier.csv};
			\addlegendentry{squared error loss (solid curve)}
			\addplot[black, thick, dashed] table[col sep=comma, x=delta, y=lad]{sqerrloss_outlier.csv};
			\addlegendentry{absolute error loss (dashed curve)}
			\addplot[black, thick, dotted] table[col sep=comma, x=delta, y=huber]{sqerrloss_outlier.csv};
			\addlegendentry{Huber loss (dotted curve)}
		\end{groupplot}
		\end{tikzpicture}
		\caption{(a) Every $18$th day of $2024$ at Krems, the $21$
			training days (circles), and the line learned by
			least squares (solid line): the squared error loss
			\eqref{equ_sqerrloss_def_dict} of a day is the squared length
			of its dashed segment, and the training error
			\eqref{equ_sqerrloss_trainerr_dict} averages these over the
			$21$ days. The square marks 17 July, whose label is
			corrupted in (b). (b) The displacement of the learned line at
			the morning minimum of that day when $\Delta$ degrees are added
			to its label: the least squares line (solid curve)
			moves by $0.192 \Delta$ without bound, while the line learned
			under the absolute error loss (dashed curve) and under the
			Huber loss with threshold $\delta = 3$ (dotted curve) stop
			moving once the corrupted day lies far enough above them. Data
			generated by \texttt{pythondemos/sqerrloss.py}
			\label{fig_sqerrloss_demo_dict}}
		\end{figure}
		When the label is a vector, the squared error loss is
		the squared Euclidean norm of the difference between label
		and prediction. The clustering error of $k$-means,
		which compares a data point with the cluster centroid
		that stands in for it, and the reconstruction error of principal component analysis (PCA)
		and of an autoencoder are of this form, with the
		feature vector itself in the place of the label.

\subsection{Cross References}
loss, loss function, absolute error loss, Huber loss, logistic loss, $0/1$ loss, training error, MSE, ERM, least squares, linear regression, normal equations, GD, maximum likelihood, Gaussian RV, risk, Bayes estimator, conditional expectation, bias--variance trade-off, outlier, robustness, least absolute deviation regression, Huber regression, clustering error

\subsection{References}
\begin{enumerate}
  \item Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media, New York, NY, USA.
  \item Legendre (1805). Nouvelles méthodes pour la détermination des orbites des comètes. Firmin Didot, Paris.
  \item Murphy (2012). Machine Learning: A Probabilistic Perspective. MIT Press, Cambridge, MA, USA.
  \item Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed., Springer Science+Business Media, New York, NY, USA.
  \item Huber (1992). Robust estimation of a location parameter. In: Breakthroughs in Statistics: Methodology and Distribution, pp.~492--518. Springer-Verlag.
\end{enumerate}

\end{document}
