\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb}
\usepackage{algorithm,algpseudocode}
\usepackage{bm}
\usepackage{hyperref}
\usepackage{float}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{xcolor}
\definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0}
\definecolor{aaltoBlue}{RGB}{0, 51, 102}
\definecolor{pinegreen}{cmyk}{0.92,0,0.59,0.25}
\definecolor{royalblue}{cmyk}{1,0.50,0,0}
\definecolor{lavander}{cmyk}{0,0.48,0,0}
\definecolor{violet}{cmyk}{0.79,0.88,0,0}
\definecolor{lightblue}{RGB}{173, 216, 230}
\usepackage{tikz}
\usepackage{pgfplots}
\usetikzlibrary{calc,arrows.meta,positioning,fit,shapes.geometric,shapes.misc,intersections,patterns,decorations.pathreplacing,backgrounds,matrix}
\usepgfplotslibrary{fillbetween,groupplots}
\setcounter{secnumdepth}{-\maxdimen}
% Long unbreakable tokens (the ISO/IEC author names in citations, term names
% in the title) otherwise overflow the margin by a few points; a modest
% emergency stretch lets TeX loosen the paragraph instead.
\emergencystretch=2em
\usepackage{mleftright}
% Project macros used by this entry (auto-extracted).
\newcommand{\dataset}{\mathcal{D}}
\newcommand{\defeq}{:=}
\newcommand{\emprisk}[2]{\widehat{L}\big(#1|#2\big)}
\newcommand{\expect}{\mathbb{E} }
\newcommand{\featurevec}{\vx}
\newcommand{\hypothesis}{h}
\newcommand{\probdist}{\mathbb{P}}
\newcommand{\risk}[1]{\bar{L} \big( #1 \big) }
\newcommand{\sampleidx}{r}
\newcommand{\samplesize}{m}
\newcommand{\truelabel}{y}
\newcommand{\vx}[0]{{\bf x}}

\begin{document}

\section{mean squared error (MSE)}

\subsection*{Author and editors}
Alexander Jung\textsuperscript{*}, Department of Computer Science, Aalto University, Espoo, Finland (ORCID \mbox{\href{https://orcid.org/0000-0001-7538-0990}{0000-0001-7538-0990}}) \\
Editors: Konstantina Olioumtsevits (ORCID \mbox{\href{https://orcid.org/0000-0003-2697-8659}{0000-0003-2697-8659}}), Ekkehard Schnoor (ORCID \mbox{\href{https://orcid.org/0000-0001-8252-1499}{0000-0001-8252-1499}}), Aalto University, Espoo, Finland \\
\textsuperscript{*}Corresponding author: \href{mailto:alex.jung@aalto.fi}{alex.jung@aalto.fi}



\subsection{Abstract}
The mean squared error (MSE) of a hypothesis is the
		average of its squared error loss, the squared difference between the
		label of a data point and the prediction for it.
		The average is either an expectation under a probabilistic model
		for the data, the independent and identically distributed assumption (i.i.d.\ assumption), which makes the MSE the
		risk under the squared error loss, or the average over the
		data points of a dataset, which makes it the
		empirical risk, the training error or the validation error. The two readings coincide for the
		empirical distribution of the dataset, which assigns the
		same probability to each of its data points. When the
		i.i.d.\ assumption holds, the average over a dataset estimates
		the expectation and improves with the sample size. The root
		MSE, its square root, is measured in the unit of the label.

\subsection{Definition}
Consider a hypothesis
		$\hypothesis$ that predicts the maximum daytime temperature of a day
		at Krems an der Donau (Austria) from its morning minimum. A weather service
		that uses it reports one number for how far the predictions
		are off on average, and the MSE is that number: the average of the
		squared error loss, the squared difference between the label of a
		data point and the prediction for it. The word
		\emph{average} has two readings here, and the MSE is defined in
		both.
\par
		The first reading needs a probabilistic model for how the data
		come about. The independent and identically distributed assumption (i.i.d.\ assumption) is the simplest one: it treats the
		daily weather measurements as realizations of independent and identically distributed (i.i.d.)
		random variables (RVs), so that each day, a data point with
		feature vector $\featurevec$ and label $\truelabel$, is an
		independent draw from one probability distribution $\probdist$ that does not
		change from day to day. Under this model, the MSE of $\hypothesis$
		is the expectation of its squared error loss,
		\begin{equation}
			\label{equ_mse_expectation_dict}
			\risk{\hypothesis} \defeq \expect \big\{ \big(\truelabel -
			\hypothesis(\featurevec)\big)^{2} \big\} \text{,}
		\end{equation}
		which is the risk of $\hypothesis$ under the squared error loss
		(Hastie et al., 2009, Sect.~2.4). The
		expectation \eqref{equ_mse_expectation_dict} is taken over the
		probability distribution $\probdist$ of the data point, which the
		i.i.d.\ assumption postulates but does not specify: in a machine learning (ML)
		application, $\probdist$ is unknown. What can be computed is the MSE on a
		dataset $\dataset = \{(\featurevec^{(1)}, \truelabel^{(1)}),
		\ldots, (\featurevec^{(\samplesize)}, \truelabel^{(\samplesize)})\}$
		of sample size $\samplesize$, the average of the
		squared error loss over its data points,
		\begin{equation}
			\label{equ_mse_average_dict}
			\emprisk{\hypothesis}{\dataset} \defeq \frac{1}{\samplesize}
			\sum_{\sampleidx=1}^{\samplesize} \big(\truelabel^{(\sampleidx)}
			- \hypothesis(\featurevec^{(\sampleidx)})\big)^{2} \text{,}
		\end{equation}
		which is the empirical risk of $\hypothesis$ on $\dataset$ under the
		squared error loss. It is the training error when $\dataset$ is the
		training set and the validation error when $\dataset$ is the
		validation set. For a linear model learned by least squares on the
		$365$ days of $2023$ at Krems, the MSE \eqref{equ_mse_average_dict}
		is $21.1$ on those days and $19.2$ on the $366$ days of $2024$, in
		squared degrees. Its square root, the root MSE of $4.4$ degrees,
		has the unit of the label.
\par
		The two definitions are one definition applied to two
		probability distributions. The empirical distribution
		$\probdist^{(\dataset)}$ of the dataset $\dataset$ assigns the
		probability $1/\samplesize$ to each of its data points.
		The expectation of the squared error loss under
		$\probdist^{(\dataset)}$ is therefore a sum of $\samplesize$ terms
		with weight $1/\samplesize$ each,
		\begin{equation}
			\label{equ_mse_empdist_dict}
			\expect_{(\featurevec, \truelabel) \sim \probdist^{(\dataset)}}
			\big\{ \big(\truelabel - \hypothesis(\featurevec)\big)^{2} \big\}
			= \emprisk{\hypothesis}{\dataset} \text{.}
		\end{equation}
		By the identity \eqref{equ_mse_empdist_dict}, the average
		\eqref{equ_mse_average_dict} is the expectation
		\eqref{equ_mse_expectation_dict} with the
		empirical distribution of $\dataset$ in place of $\probdist$.
		Every statement about the MSE as an expectation then holds for
		the average with $\probdist^{(\dataset)}$ in that role, such as its
		split into the variance of the label around its
		conditional mean and the squared deviation of $\hypothesis$
		from that mean (Bishop, 2006, Sect.~1.5.5).
		Fig.~\ref{fig_mse_dict}(a) draws the empirical distribution of
		the squared errors of the $366$ days of $2024$. Its mean is
		the MSE $19.2$, while its median is $9.1$ and two thirds of
		the days have a loss below the MSE: a few far-off days weigh
		heavily in a squared average. Conversely, when the
		data points of $\dataset$ are realizations of i.i.d.
		RVs with probability distribution $\probdist$, the average
		\eqref{equ_mse_average_dict} is an estimate of the expectation
		\eqref{equ_mse_expectation_dict} that improves with the
		sample size $\samplesize$ by the law of large numbers. Fig.~\ref{fig_mse_dict}(b)
		shows this for the days of $2024$ in a random order: over the first
		$30$ days, the average lies between $12.6$ and $26.7$ in nine of
		ten orders, over the first $180$ days between $17.0$ and $21.3$.
		How far the training error can lie from the risk is the
		subject of generalization.
		\begin{figure}[H]
		\centering
		\begin{tikzpicture}
		\begin{groupplot}[
			group style={group size=2 by 1, horizontal sep=1.7cm},
			width=0.5\linewidth, height=0.44\linewidth,
			axis lines=left,
			tick label style={font=\scriptsize},
			label style={font=\scriptsize, align=center},
			legend style={draw=none, fill=none, font=\scriptsize,
				at={(0.5,-0.42)}, anchor=north},
			legend cell align=left,
		]
			\nextgroupplot[
				xlabel={squared error loss of a day\\(a) empirical distribution of the losses},
				ylabel={fraction of the $366$ days},
				xmin=0, xmax=60, ymin=0, ymax=1.05]
			\addplot[thick, black, const plot]
				table[col sep=comma, x=sqerr, y=cdf]{mse_cdf.csv};
			\addlegendentry{empirical cdf (solid curve)}
			\addplot[dashed, black] coordinates {(19.25,0) (19.25,1)};
			\addlegendentry{MSE (dashed line)}
			\nextgroupplot[
				xlabel={number $\samplesize$ of days averaged\\(b) average over the first $\samplesize$ days},
				ylabel={average squared error loss},
				xmin=0, xmax=366, ymin=0, ymax=42]
			\addplot[thick, black]
				table[col sep=comma, x=m, y=running]{mse_running.csv};
			\addlegendentry{one random order (solid curve)}
			\addplot[dotted, black]
				table[col sep=comma, x=m, y=lo]{mse_running.csv};
			\addlegendentry{percentile band (dotted curves)}
			\addplot[dotted, black, forget plot]
				table[col sep=comma, x=m, y=hi]{mse_running.csv};
			\addplot[dashed, black] coordinates {(0,19.25) (366,19.25)};
			\addlegendentry{MSE over all days (dashed line)}
		\end{groupplot}
		\end{tikzpicture}
		\caption{The squared error loss of a linear model, learned by
			least squares on the days of $2023$ at Krems, on the $366$ days
			of $2024$. (a) The empirical cdf of the $366$ losses (solid
			curve), cut at $60$ squared degrees with $29$ days beyond the
			cut, and their mean, the MSE \eqref{equ_mse_average_dict}
			(dashed line). (b) The average
			of the losses over the first $\samplesize$ days of $2024$ in one
			random order (solid curve), the band between the 5th and the
			95th percentile over $500$ orders, which holds this average for
			nine orders in ten (dotted curves), and the MSE over
			all $366$ days (dashed line). Data generated by
			\texttt{pythondemos/mse.py}
			\label{fig_mse_dict}}
		\end{figure}
		Squaring the deviation is what makes a far-off day weigh heavily,
		and that is the MSE's main limitation. A single mismeasured
		label enters the average in proportion to the square of its
		deviation, so it can dominate the number reported for a whole
		year. The mean absolute error (MAE), the average of the absolute error loss instead of
		the squared error loss, weighs each day in proportion to its
		deviation alone. The difference shows in what each average asks a
		constant prediction to be: the MSE is smallest at the
		mean of the labels, the MAE at their
		median (Hastie et al., 2009, Sect.~10.6). On
		the $366$ days of $2024$ at Krems, these are $17.91$ and $17.60$
		degrees. A decimal point in the wrong place moves one of those
		$366$ labels by $5$, $10$, $20$, or $50$ degrees. The
		mean then moves by $0.014$, $0.027$, $0.055$, and $0.137$
		degrees, the error divided by the sample size. The
		median moves by at most $0.1$ degrees, the step to the
		neighboring label, however large the error becomes. The
		MSE is therefore the wrong summary for a record with
		outliers or a long-tailed error distribution, and the
		MAE or a least absolute deviation regression is the usual answer
		(Hastie et al., 2009, Sect.~10.6). What the MSE
		buys in return is the mean it points at: it is
		differentiable everywhere, which is what lets least squares
		have a closed form.

\subsection{Cross References}
squared error loss, risk, empirical risk, training error, validation error, empirical distribution, expectation, probabilistic model, i.i.d.\ assumption, law of large numbers, generalization, MAE, absolute error loss, outlier, linear regression, loss

\subsection{References}
\begin{enumerate}
  \item Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media, New York, NY, USA.
  \item Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed., Springer Science+Business Media, New York, NY, USA.
\end{enumerate}

\end{document}
