\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb}
\usepackage{algorithm,algpseudocode}
\usepackage{bm}
\usepackage{hyperref}
\usepackage{float}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{xcolor}
\definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0}
\definecolor{aaltoBlue}{RGB}{0, 51, 102}
\definecolor{pinegreen}{cmyk}{0.92,0,0.59,0.25}
\definecolor{royalblue}{cmyk}{1,0.50,0,0}
\definecolor{lavander}{cmyk}{0,0.48,0,0}
\definecolor{violet}{cmyk}{0.79,0.88,0,0}
\definecolor{lightblue}{RGB}{173, 216, 230}
\usepackage{tikz}
\usepackage{pgfplots}
\usetikzlibrary{calc,arrows.meta,positioning,fit,shapes.geometric,shapes.misc,intersections,patterns,decorations.pathreplacing,backgrounds,matrix}
\usepgfplotslibrary{fillbetween,groupplots}
\setcounter{secnumdepth}{-\maxdimen}
% Long unbreakable tokens (the ISO/IEC author names in citations, term names
% in the title) otherwise overflow the margin by a few points; a modest
% emergency stretch lets TeX loosen the paragraph instead.
\emergencystretch=2em
\usepackage{mleftright}
% Project macros used by this entry (auto-extracted).
\newcommand{\defeq}{:=}
\newcommand{\feature}{x}
\newcommand{\featurespace}{\mathcal{X}}
\newcommand{\featurevec}{\vx}
\newcommand{\hypothesis}{h}
\newcommand{\labelprob}{\tilde{\truelabel}}
\newcommand{\learnthypothesis}{\widehat{\hypothesis}}
\newcommand{\lossfunc}[2]{L\left(#1,#2 \right)}
\newcommand{\mI}{\mathbf{I}}
\newcommand{\meanvec}[1]{{\bm \mu}^{(#1)}}
\newcommand{\normgeneric}[2]{\mleft\lVert #1 \mright\rVert_{#2}}
\newcommand{\offset}{b}
\newcommand{\prob}[1]{\mathbb{P}\left(#1\right)}
\newcommand{\probdist}{\mathbb{P}}
\newcommand{\reals}{\mathbb{R}}
\newcommand{\suffstat}{\mathbf{t}}
\newcommand{\truelabel}{y}
\newcommand{\vw}[0]{{\bf w}}
\newcommand{\vx}[0]{{\bf x}}
\newcommand{\weights}{\vw}

\begin{document}

\section{logistic loss}

\subsection*{Author and editors}
Alexander Jung\textsuperscript{*}, Department of Computer Science, Aalto University, Espoo, Finland (ORCID \mbox{\href{https://orcid.org/0000-0001-7538-0990}{0000-0001-7538-0990}}) \\
Editors: Konstantina Olioumtsevits (ORCID \mbox{\href{https://orcid.org/0000-0003-2697-8659}{0000-0003-2697-8659}}), Ekkehard Schnoor (ORCID \mbox{\href{https://orcid.org/0000-0001-8252-1499}{0000-0001-8252-1499}}), Aalto University, Espoo, Finland \\
\textsuperscript{*}Corresponding author: \href{mailto:alex.jung@aalto.fi}{alex.jung@aalto.fi}

\subsection{Synonyms}
log loss, binary cross-entropy (BCE) loss

\subsection{Abstract}
The logistic loss is a convex and smooth
		surrogate for the $0/1$ loss in binary classification. Like the
		hinge loss, it depends on a data point only through the
		location of that data point relative to the
		decision boundary. Unlike the hinge loss it is
		differentiable everywhere, which suits it to
		gradient-based optimization methods such as gradient descent (GD). Running
		empirical risk minimization (ERM) with a linear model and the logistic loss gives
		logistic regression. The logistic loss is the negative logarithm of
		the conditional probability distribution of the label given the
		feature vector, so minimizing it is maximum likelihood
		estimation of the model parameters. A probabilistic model in which
		the features are jointly Gaussian random variables (Gaussian RVs) given the
		label induces exactly that conditional probability distribution, and it is
		not the only one that does.

\subsection{Definition}
A spam filter is a binary classification
		method that learns a hypothesis that maps the features of an email to either \emph{spam} or
		\emph{no spam}. The performance of such a spam filter can be
		measured by counting the fraction of emails that are
		misclassified. This fraction is the average $0/1$ loss.
		However, this average loss is flat almost everywhere as a
		function of the model parameters, so it gives gradient descent (GD) no
		direction to move in. Its shape is the step drawn in
		Fig.~\ref{fig_logloss_dict}: flat on either side of the
		decision boundary and
		with no slope anywhere to follow. Practical
		classification methods therefore use a convex and
		smooth surrogate for the $0/1$ loss, which is less
		flat. One such surrogate is the logistic loss, defined
		next.
\par
		Consider a binary classification problem with feature space
		$\featurespace$ and a real-valued hypothesis
		$\hypothesis: \featurespace \rightarrow \reals$. The two classes
		are encoded below by the label
		$\truelabel \in \{-1, +1\}$. The logistic loss incurred by
		$\hypothesis$ on a data point $(\featurevec, \truelabel)$ is
		(Bishop, 2006)
		\begin{equation}
			\label{equ_log_loss_gls_dict}
			\lossfunc{(\featurevec, \truelabel)}{\hypothesis}
			\defeq \ln\big(1 + \exp(-\truelabel \cdot
			\hypothesis(\featurevec))\big) \text{.}
		\end{equation}
		Like the hinge loss, the logistic loss
		\eqref{equ_log_loss_gls_dict} depends on the data point only
		via the margin $\truelabel \cdot \hypothesis(\featurevec)$.
		Normalizing the logistic loss by $\ln 2$ makes it a
		convex upper bound on the $0/1$ loss, with
		equality at margin $0$ (see Fig.~\ref{fig_logloss_dict}). The
		rescaling is what makes it a bound: at margin $0$ the logistic
		loss itself is $\ln 2 \approx 0.69$, below the value $1$ of
		the $0/1$ loss there. Scaling a surrogate so
		that it passes through $1$ at margin $0$ is the usual convention
		when such loss functions are compared
		(Hastie et al., 2009, Sect.~10.6).
		\begin{figure}[H]
		\begin{center}
		\begin{tikzpicture}[>=Stealth]
			\draw[->, thick] (-2.6,0) -- (3.6,0)
				node[below, font=\small]
				{margin $\truelabel \cdot \hypothesis(\featurevec)$};
			\draw[->, thick] (0,-0.3) -- (0,3.9)
				node[above, font=\small] {loss};
			% rescaled logistic loss log2(1+e^-t): solid black
			\draw[very thick, domain=-2.4:3.4, samples=80]
				plot (\x, {ln(1+exp(-\x))/ln(2)});
			\draw[black!55, thin] (0.95,2.75) -- (-0.30,1.42);
			\node[font=\small, right] at (0.95,2.8)
				{logistic loss $/ \ln 2$};
			% 0/1 loss: dashed gray step
			\draw[very thick, black!55, dashed] (-2.4,1) -- (0,1);
			\draw[very thick, black!55, dashed] (0.02,0) -- (3.4,0);
			\fill[black!55] (0,1) circle (1.8pt);
			\draw[black!55, fill=white] (0,0) circle (1.8pt);
			\node[font=\small, black!55, below left] at (-1.2,0.95)
				{$0/1$ loss};
			\node[below left, font=\small] at (0.0,1) {$1$};
		\end{tikzpicture}
		\end{center}
		\caption{The logistic loss
			$\lossfunc{(\featurevec, \truelabel)}{\hypothesis}$ divided by
			$\ln 2$ (solid curve) and the $0/1$ loss (dashed
			curve) as functions of the
			margin $\truelabel \cdot \hypothesis(\featurevec)$. The
			rescaled logistic loss lies on or above the
			$0/1$ loss for every margin, with equality at margin
			$0$; the rescaling is needed because
			$\ln\big(1 + \exp(0)\big) = \ln 2 < 1$
			\label{fig_logloss_dict}}
		\end{figure}
		Unlike
		the hinge loss, the logistic loss is everywhere
		differentiable. It is therefore well suited for
		gradient-based optimization methods such as GD.
		Empirical risk minimization (ERM) with a linear model and the logistic
		loss yields logistic regression.
\par
		The logistic loss arises naturally from a simple
		probabilistic model in which the data points are independent and identically distributed (i.i.d.)
		random variables (RVs) with a common probability distribution
		$\probdist^{(\featurevec, \truelabel)}$. Conditioned on the
		label $\truelabel$, the feature vector $\featurevec$ is a
		Gaussian random variable (Gaussian RV) with mean vector $\meanvec{-1}$ for
		$\truelabel = -1$ and $\meanvec{+1}$ for $\truelabel = +1$. The
		conditional covariance matrix is the same for both, and fixed to
		$\mI$.
\par
		Under that probabilistic model, the posterior probability
		of $\truelabel = +1$ is the sigmoid function
		$\sigma(\hypothesis(\featurevec))$ with
		$\sigma(z) = 1/(1 + e^{-z})$. The identity holds for the affine
		hypothesis $\hypothesis(\featurevec) = \weights^{\top} \featurevec
		+ \offset$ with the model parameters
		(Bishop, 2006, Sect.~4.2.1, Eqs.~(4.65)--(4.67))
		\[
			\weights = \meanvec{+1} - \meanvec{-1} \text{,} \quad
			\offset = - \tfrac{1}{2} \big(
			\normgeneric{\meanvec{+1}}{2}^{2}
			- \normgeneric{\meanvec{-1}}{2}^{2} \big)
			+ \ln \frac{\prob{\truelabel = +1}}
			{\prob{\truelabel = -1}} \text{.}
		\]
		It is the shared covariance matrix that makes $\hypothesis$ affine: the
		terms quadratic in $\featurevec$ are the same under both
		labels and cancel in the ratio, leaving only the
		inner product with $\weights$. When the two labels are equally
		likely the logarithm vanishes and $\offset$ is the midpoint term
		alone. The probability of $\truelabel = -1$ is then
		$\sigma(-\hypothesis(\featurevec))$,
		because the sigmoid function satisfies $\sigma(-z) = 1 - \sigma(z)$
		(Bishop, 2006, Eq.~4.60). The logistic loss is thus the
		negative logarithm of the conditional probability of the
		label given the feature vector, and minimizing the average
		logistic loss is maximum likelihood estimation of the
		weight vector $\weights$ and the offset $\offset$ of
		$\hypothesis$ under that conditional probability distribution
		(Bishop, 2006, Sect.~4.3.2;
		Murphy, 2012, Sect.~8.3). The Gaussian probabilistic model
		is not the only one that induces this conditional probability distribution. If,
		given the label, the feature vector has a density of the
		form $h(\featurevec) \exp\big( (\weights^{(\truelabel)})^{\top}
		\suffstat(\featurevec) - A(\weights^{(\truelabel)}) \big)$, a member
		of an exponential family whose natural parameter
		$\weights^{(\truelabel)}$ depends on the label while the base
		measure $h$ and the sufficient statistic $\suffstat(\featurevec)$
		do not, then the logarithm of the posterior odds of
		$\truelabel = +1$ is affine in $\suffstat(\featurevec)$, and the
		posterior is again the sigmoid function of an affine
		function (Bishop, 2006, Sect.~4.2.4). Its weight
		vector is the difference of the two natural
		parameters, $\weights = \weights^{(+1)} - \weights^{(-1)}$,
		which for the Gaussian model above, where
		$\suffstat(\featurevec) = \featurevec$ and the natural
		parameter is the mean vector, is the identity
		$\weights = \meanvec{+1} - \meanvec{-1}$ displayed there. Conditionally
		independent binary features, the model behind
		naive Bayes, are another case (Bishop, 2006, Sect.~4.2.3).
\par
		A binary classification problem whose label space is some other
		pair $\{a, b\}$ is covered by an encoding
		$e: \{a, b\} \rightarrow \{-1, +1\}$ with $e(a) = -1$ and
		$e(b) = +1$. The logistic loss
		\eqref{equ_log_loss_gls_dict} is then evaluated at the encoded
		label,
		\[
			\lossfunc{(\featurevec, \truelabel)}{\hypothesis}
			= \ln\big(1 + \exp(-e(\truelabel) \cdot
			\hypothesis(\featurevec))\big) \text{.}
		\]
		The label spaces $\{0, 1\}$ and
		$\{\textit{no spam}, \textit{spam}\}$ are both handled
		this way. Take the label space $\{0, 1\}$, with $e(0) = -1$
		and $e(1) = +1$. The loss can then be written out as
		\[
			- \truelabel \ln \sigma\big(\hypothesis(\featurevec)\big)
			- (1 - \truelabel) \ln \big(1 -
			\sigma(\hypothesis(\featurevec))\big) \text{,}
		\]
		which is the binary cross-entropy (BCE) loss: the
		cross-entropy between the label and the predicted
		probability $\sigma(\hypothesis(\featurevec))$, with the
		label $1$ leaving the second summand zero and the
		label $0$ the first. Summed over a training set it is the
		cross-entropy error function of logistic regression
		(Bishop, 2006, Eq.~4.90).
\par
		Written that way the loss no longer needs the label to
		be one of two values. Let $\labelprob$ be a soft label, a
		number between $0$ and $1$ giving the probability that the
		data point belongs to the class encoded by $1$. Putting
		$\labelprob$ in place
		of $\truelabel$ gives the cross-entropy loss
		\[
			- \labelprob \ln \sigma\big(\hypothesis(\featurevec)\big)
			- (1 - \labelprob) \ln \big(1 -
			\sigma(\hypothesis(\featurevec))\big) \text{,}
		\]
		which reduces to the previous display at
		$\labelprob \in \{0, 1\}$ and carries over to more than two
		classes through the softmax function.
\par
		The value of the logistic loss is harder to interpret than the
		$0/1$ loss, because it depends on the size of the
		margin and not only on its sign. Each hypothesis
		compared below is a linear classifier
		$\hypothesis(\featurevec) = \weights^{\top} \featurevec
		+ \offset$. Take the three
		data points of Fig.~\ref{fig_logloss_margins_dict}, two of
		them with label $+1$ at $(1,1)$ and $(2,3)$ and one with
		label $-1$ at $(3,1)$, and the linear classifier
		$\learnthypothesis^{(1)}(\featurevec) = -0.1 \feature_{1}
		+ 0.05 \feature_{2} + 0.15$. Its margins are $0.1$ on each
		of the three, so it classifies all three correctly and incurs an
		average logistic loss of $0.644$. Multiplying its
		model parameters by $50$ gives $50 \learnthypothesis^{(1)}$,
		which leaves the decision boundary exactly where it was and
		so changes no prediction, yet the average logistic
		loss falls to $0.0067$, a factor of $96$. Two
		hypotheses far apart in logistic loss can therefore
		make exactly the same decisions.
\par
		The logistic loss and the $0/1$ loss can also rank
		two hypotheses in opposite
		orders. A second linear classifier $\learnthypothesis^{(2)}$ has a
		decision boundary parallel to the first but shifted, which
		leaves the
		data point at $(3,1)$ on the wrong side. Its margins
		are $5$, $5$, and $-0.5$, an accuracy of $2/3$ against $1$
		for $\learnthypothesis^{(1)}$, and it still attains the smaller
		average logistic loss, $0.329$ against $0.644$. Its two
		large margins cost $0.0067$ each, so even the single
		negative margin, at $0.974$, totals less than the three
		small margins of $\learnthypothesis^{(1)}$ at $0.644$
		each. Choosing between the two by logistic loss alone
		selects the less accurate one. Only one reading of the number is
		safe, the upper bound above: a small logistic loss forces a
		small $0/1$ loss.
		\begin{figure}[H]
		\begin{center}
		\begin{tikzpicture}
		\begin{axis}[
			width=0.72\linewidth, height=0.52\linewidth,
			axis lines=left,
			xlabel={$\feature_{1}$}, ylabel={$\feature_{2}$},
			xmin=0.2, xmax=3.9, ymin=-0.4, ymax=4.1,
			tick label style={font=\scriptsize},
			label style={font=\scriptsize},
			legend style={draw=none, fill=none, font=\scriptsize,
				at={(0.02,0.98)}, anchor=north west},
			legend cell align=left,
		]
			\addplot[black, thick] table[col sep=comma, x=x1, y=h1]
				{logloss_bound.csv};
			\addlegendentry{decision boundary of
				$\learnthypothesis^{(1)}$ and $50 \learnthypothesis^{(1)}$}
			\addplot[black, thick, dashed] table[col sep=comma, x=x1, y=h2]
				{logloss_bound.csv};
			\addlegendentry{decision boundary of
				$\learnthypothesis^{(2)}$}
			\addplot[only marks, mark=*, mark size=2.6pt, black]
				coordinates {(1,1) (2,3)};
			\addlegendentry{label $\truelabel = +1$}
			\addplot[only marks, mark=square, mark size=3pt, black,
				line width=1pt] coordinates {(3,1)};
			\addlegendentry{label $\truelabel = -1$}
		\end{axis}
		\end{tikzpicture}
		\end{center}
		\caption{The three data points of the example, the two
			with label $\truelabel = +1$ drawn as filled circles and
			the one with label $\truelabel = -1$ as an open square,
			and the decision boundaries of the hypotheses
			compared on them. Scaling $\learnthypothesis^{(1)}$ by $50$
			multiplies its model parameters by $50$ and so leaves the
			decision boundary
			exactly where it was (solid line): every prediction is
			unchanged while the average logistic loss falls. The decision boundary of
			$\learnthypothesis^{(2)}$ (dashed line) is parallel but
			shifted, which puts the data point at $(3,1)$ on the
			wrong side. Data generated by
			\texttt{pythondemos/logloss.py}
			\label{fig_logloss_margins_dict}}
		\end{figure}

\subsection{Cross References}
$0/1$ loss, hinge loss, margin, accuracy, cross-entropy, probabilistic model, sigmoid function, classifier, logistic regression, GD, convex, binary classification

\subsection{References}
\begin{enumerate}
  \item Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media, New York, NY, USA.
  \item Murphy (2012). Machine Learning: A Probabilistic Perspective. MIT Press, Cambridge, MA, USA.
  \item Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed., Springer Science+Business Media, New York, NY, USA.
\end{enumerate}

\end{document}