Dictionary of Applied Machine Learning: 115 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

logistic loss

Updated on 2026-11-11

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also accuracy sigmoid function logistic regression gradient descent convex binary classification

The logistic loss is a convex and smooth surrogate for the $0/1$ loss in binary classification. Like the hinge loss, it depends on a data point only through the location of that data point relative to the decision boundary. Unlike the hinge loss it is differentiable everywhere, which suits it to gradient-based optimization methods such as gradient descent (GD). Running empirical risk minimization (ERM) with a linear model and the logistic loss gives logistic regression. The logistic loss is the negative logarithm of the conditional probability distribution of the label given the feature vector, so minimizing it is maximum likelihood estimation of the model parameters. A probabilistic model in which the features are jointly Gaussian random variables (Gaussian RVs) given the label induces exactly that conditional probability distribution, and it is not the only one that does.

Figure produced by the Python demo of the entry logistic loss
Figure written by the entry's Python demo logloss.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

B-flatA spam filter is a binary classification method that learns a hypothesis that maps the features of an email to either spam or no spam. The performance of such a spam filter can be measured by counting the fraction of emails that are misclassified. This fraction is the average $0/1$ loss. However, this average loss is flat almost everywhere as a function of the model parameters, so it gives gradient descent (GD) no direction to move in. Its shape is the step drawn in Fig. 1: flat on either side of the decision boundary and with no slope anywhere to follow. Practical classification methods therefore use a convex and smooth surrogate for the $0/1$ loss, which is less flat. One such surrogate is the logistic loss, defined next.

B-gdConsider a binary classification problem with feature space $\mathcal{X}$ and a real-valued hypothesis $h: \mathcal{X} \rightarrow \mathbb{R}$. The two classes are encoded below by the label $y \in \{-1, +1\}$. The logistic loss incurred by $h$ on a data point $({\mathbf x}, y)$ is (Bishop, 2006) \begin{equation} \label{equ_log_loss_gls_dict} L\left(({\mathbf x}, y),h \right) := \ln\big(1 + \exp(-y \cdot h({\mathbf x}))\big) \text{.} \end{equation} Like the hinge loss, the logistic loss \(\eqref{equ_log_loss_gls_dict}\) depends on the data point only via the margin $y \cdot h({\mathbf x})$. Normalizing the logistic loss by $\ln 2$ makes it a convex upper bound on the $0/1$ loss, with equality at margin $0$ (see Fig. 1). The rescaling is what makes it a bound: at margin $0$ the logistic loss itself is $\ln 2 \approx 0.69$, below the value $1$ of the $0/1$ loss there. Scaling a surrogate so that it passes through $1$ at margin $0$ is the usual convention when such loss functions are compared (Hastie et al., 2009, Sect. 10.6).

Figure 1 of the entry logloss
Figure 1: The logistic loss $\lossfunc{(\featurevec, \truelabel)}{\hypothesis}$ divided by $\ln 2$ (solid curve) and the $0/1$ loss (dashed curve) as functions of the margin $\truelabel \cdot \hypothesis(\featurevec)$. The rescaled logistic loss lies on or above the $0/1$ loss for every margin, with equality at margin $0$; the rescaling is needed because $\ln\big(1 + \exp(0)\big) = \ln 2 < 1$
Unlike the hinge loss, the logistic loss is everywhere differentiable. It is therefore well suited for gradient-based optimization methods such as GD. Empirical risk minimization (ERM) with a linear model and the logistic loss yields logistic regression.

The logistic loss arises naturally from a simple probabilistic model in which the data points are independent and identically distributed (i.i.d.) random variables (RVs) with a common probability distribution $\mathbb{P}^{({\mathbf x}, y)}$. Conditioned on the label $y$, the feature vector ${\mathbf x}$ is a Gaussian random variable (Gaussian RV) with mean vector ${\boldsymbol \mu}^{(-1)}$ for $y = -1$ and ${\boldsymbol \mu}^{(+1)}$ for $y = +1$. The conditional covariance matrix is the same for both, and fixed to $\mathbf{I}$.

Under that probabilistic model, the posterior probability of $y = +1$ is the sigmoid function $\sigma(h({\mathbf x}))$ with $\sigma(z) = 1/(1 + e^{-z})$. The identity holds for the affine hypothesis $h({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x} + b$ with the model parameters (Bishop, 2006, Sect. 4.2.1, Eqs. (4.65)–(4.67)) \[ {\mathbf w} = {\boldsymbol \mu}^{(+1)} - {\boldsymbol \mu}^{(-1)} \text{,} \quad b = - \tfrac{1}{2} \big( \left\lVert {\boldsymbol \mu}^{(+1)} \right\rVert_{2}^{2} - \left\lVert {\boldsymbol \mu}^{(-1)} \right\rVert_{2}^{2} \big) + \ln \frac{\mathbb{P}\left(y = +1\right)} {\mathbb{P}\left(y = -1\right)} \text{.} \] It is the shared covariance matrix that makes $h$ affine: the terms quadratic in ${\mathbf x}$ are the same under both labels and cancel in the ratio, leaving only the inner product with ${\mathbf w}$. When the two labels are equally likely the logarithm vanishes and $b$ is the midpoint term alone. The probability of $y = -1$ is then $\sigma(-h({\mathbf x}))$, because the sigmoid function satisfies $\sigma(-z) = 1 - \sigma(z)$ (Bishop, 2006, Eq. 4.60). The logistic loss is thus the negative logarithm of the conditional probability of the label given the feature vector, and minimizing the average logistic loss is maximum likelihood estimation of the weight vector ${\mathbf w}$ and the offset $b$ of $h$ under that conditional probability distribution (Bishop, 2006, Sect. 4.3.2; Murphy, 2012, Sect. 8.3). The Gaussian probabilistic model is not the only one that induces this conditional probability distribution. If, given the label, the feature vector has a density of the form $h({\mathbf x}) \exp\big( ({\mathbf w}^{(y)})^{\top} \mathbf{t}({\mathbf x}) - A({\mathbf w}^{(y)}) \big)$, a member of an exponential family whose natural parameter ${\mathbf w}^{(y)}$ depends on the label while the base measure $h$ and the sufficient statistic $\mathbf{t}({\mathbf x})$ do not, then the logarithm of the posterior odds of $y = +1$ is affine in $\mathbf{t}({\mathbf x})$, and the posterior is again the sigmoid function of an affine function (Bishop, 2006, Sect. 4.2.4). Its weight vector is the difference of the two natural parameters, ${\mathbf w} = {\mathbf w}^{(+1)} - {\mathbf w}^{(-1)}$, which for the Gaussian model above, where $\mathbf{t}({\mathbf x}) = {\mathbf x}$ and the natural parameter is the mean vector, is the identity ${\mathbf w} = {\boldsymbol \mu}^{(+1)} - {\boldsymbol \mu}^{(-1)}$ displayed there. Conditionally independent binary features, the model behind naive Bayes, are another case (Bishop, 2006, Sect. 4.2.3).

B-equivA binary classification problem whose label space is some other pair $\{a, b\}$ is covered by an encoding $e: \{a, b\} \rightarrow \{-1, +1\}$ with $e(a) = -1$ and $e(b) = +1$. The logistic loss \(\eqref{equ_log_loss_gls_dict}\) is then evaluated at the encoded label, \[ L\left(({\mathbf x}, y),h \right) = \ln\big(1 + \exp(-e(y) \cdot h({\mathbf x}))\big) \text{.} \] The label spaces $\{0, 1\}$ and $\{\textit{no spam}, \textit{spam}\}$ are both handled this way. Take the label space $\{0, 1\}$, with $e(0) = -1$ and $e(1) = +1$. The loss can then be written out as \[ - y \ln \sigma\big(h({\mathbf x})\big) - (1 - y) \ln \big(1 - \sigma(h({\mathbf x}))\big) \text{,} \] which is the binary cross-entropy (BCE) loss: the cross-entropy between the label and the predicted probability $\sigma(h({\mathbf x}))$, with the label $1$ leaving the second summand zero and the label $0$ the first. Summed over a training set it is the cross-entropy error function of logistic regression (Bishop, 2006, Eq. 4.90).

Written that way the loss no longer needs the label to be one of two values. Let $\tilde{y}$ be a soft label, a number between $0$ and $1$ giving the probability that the data point belongs to the class encoded by $1$. Putting $\tilde{y}$ in place of $y$ gives the cross-entropy loss \[ - \tilde{y} \ln \sigma\big(h({\mathbf x})\big) - (1 - \tilde{y}) \ln \big(1 - \sigma(h({\mathbf x}))\big) \text{,} \] which reduces to the previous display at $\tilde{y} \in \{0, 1\}$ and carries over to more than two classes through the softmax function.

The value of the logistic loss is harder to interpret than the $0/1$ loss, because it depends on the size of the margin and not only on its sign. Each hypothesis compared below is a linear classifier $h({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x} + b$. Take the three data points of Fig. 2, two of them with label $+1$ at $(1,1)$ and $(2,3)$ and one with label $-1$ at $(3,1)$, and the linear classifier $\widehat{h}^{(1)}({\mathbf x}) = -0.1 x_{1} + 0.05 x_{2} + 0.15$. Its margins are $0.1$ on each of the three, so it classifies all three correctly and incurs an average logistic loss of $0.644$. Multiplying its model parameters by $50$ gives $50 \widehat{h}^{(1)}$, which leaves the decision boundary exactly where it was and so changes no prediction, yet the average logistic loss falls to $0.0067$, a factor of $96$. Two hypotheses far apart in logistic loss can therefore make exactly the same decisions.

B-interpThe logistic loss and the $0/1$ loss can also rank two hypotheses in opposite orders. A second linear classifier $\widehat{h}^{(2)}$ has a decision boundary parallel to the first but shifted, which leaves the data point at $(3,1)$ on the wrong side. Its margins are $5$, $5$, and $-0.5$, an accuracy of $2/3$ against $1$ for $\widehat{h}^{(1)}$, and it still attains the smaller average logistic loss, $0.329$ against $0.644$. Its two large margins cost $0.0067$ each, so even the single negative margin, at $0.974$, totals less than the three small margins of $\widehat{h}^{(1)}$ at $0.644$ each. Choosing between the two by logistic loss alone selects the less accurate one. Only one reading of the number is safe, the upper bound above: a small logistic loss forces a small $0/1$ loss.

Figure 2 of the entry logloss
Figure 2: The three data points of the example, the two with label $\truelabel = +1$ drawn as filled circles and the one with label $\truelabel = -1$ as an open square, and the decision boundaries of the hypotheses compared on them. Scaling $\learnthypothesis^{(1)}$ by $50$ multiplies its model parameters by $50$ and so leaves the decision boundary exactly where it was (solid line): every prediction is unchanged while the average logistic loss falls. The decision boundary of $\learnthypothesis^{(2)}$ (dashed line) is parallel but shifted, which puts the data point at $(3,1)$ on the wrong side. Data generated by pythondemos/logloss.py

Synonyms: log loss, binary cross-entropy (BCE) loss.

See also: $\bf 0/1$ loss, hinge loss, margin, accuracy, cross-entropy, probabilistic model, sigmoid function, classifier, logistic regression, gradient descent, convex, binary classification.

References

  1. Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media. doi.org/10.1007/978-0-387-45528-0
  2. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
  3. Murphy (2012). Machine Learning: A Probabilistic Perspective. MIT Press.

Cite this entry

@misc{dictml_logloss,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {logistic loss},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-13},
  url = {https://dictionaryofml.org/terms/logloss.html}
}