# logistic regression

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-11. [HTML](https://dictionaryofml.org/terms/logreg.html) · [typeset PDF](https://dictionaryofml.org/terms/logreg.pdf) · [LaTeX](https://dictionaryofml.org/terms/logreg.tex) · [Python demo](https://dictionaryofml.org/terms/logreg.py) · [notebook](https://dictionaryofml.org/terms/logreg.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Logistic regression learns a linear hypothesis map for binary classification, that is, for data points whose label takes one of two values, $y \in \{-1, 1\}$. The predicted label is the sign of $h({\mathbf x})$, and the absolute value $|h({\mathbf x})|$ measures the confidence in the classification. The parameters are learned by empirical risk minimization with the average logistic loss on the training set. This is equivalent to maximum likelihood estimation under a simple probabilistic model, under which $\sigma(h({\mathbf x}))$ is the probability of the label $1$. When a data point carries this probability as a soft label instead of a class, the logistic loss is replaced by the cross-entropy between soft label and prediction, equivalently their Kullback–Leibler divergence. The resulting objective function is convex and smooth, so variants of gradient descent can be used to minimize it.

## Definition

Consider predicting whether it freezes tomorrow from recent temperature measurements. Records from a weather station turn this question into data: each day is a data point with two numeric features, the maximum temperature of the day and that of the preceding day, collected in the feature vector ${\mathbf x} = (x_{1}, x_{2})^{\top} \in
\mathbb{R}^{2}$. The label $y \in \{-1, 1\}$ is $y = 1$ if the minimum temperature of the following day stays at or above $0^{\circ}$C and $y = -1$ otherwise.

Fig. 1 shows such data points. Logistic regression learns, for such a binary classification task, a linear hypothesis map $h({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x} + b$ with weight vector ${\mathbf w}$ and offset $b$ (Bishop, 2006; Hastie et al., 2009). Despite its name, the method solves a classification problem: the regression refers to fitting the real-valued map $h$, not to a numeric label. The set of feature vectors with $h({\mathbf x}) = 0$ is the decision boundary between the two classes, and ${\mathbf w}$ is its normal vector.

![Figure 1 of the entry logreg](https://dictionaryofml.org/terms/fig/logreg-fig1.svg)

*Figure 1: Data points for the frost forecast at the weather station Krems an der Donau: each day is placed at the maximum temperature of the day ($x_{1}$) and of the preceding day ($x_{2}$). Days followed by a freezing day (minimum temperature below $0^{\circ}$C) are drawn as open triangles; they concentrate where both features are small, but the two classes overlap. Twenty of the 364 days are drawn, every 22nd day without and every 12th day with a freezing successor. The dashed line is the decision boundary of the hypothesis learned from all 364 days and the solid arrow its normal vector ${\mathbf w}$. Data generated by pythondemos/logreg.py*

The prediction, i.e., the classification, is obtained from the sign of $h({\mathbf x})$, which makes the learned $h$ a linear classifier whose decision boundary is the hyperplane ${\mathbf w}^{\top} {\mathbf x} + b = 0$ with normal vector ${\mathbf w}$. The absolute value $|h({\mathbf x})|$ is used as a measure of the confidence in the classification: it grows with the distance of ${\mathbf x}$ from the decision boundary, and a value close to zero indicates low confidence.

The parameters ${\mathbf w}$ and $b$ of logistic regression are learned via empirical risk minimization. The loss it minimizes is the logistic loss. On a data point $({\mathbf x}, y)$, a hypothesis $h$ incurs

$$
L\left(({\mathbf x}, y),h \right) =
\ln\big(1 + \exp(-y \, h({\mathbf x}))\big)
\text{.}
$$

Fig. 2 shows this loss for a data point with $y = 1$ as a function of $h({\mathbf x})$. A correct classification with high confidence incurs a small loss. A wrong classification incurs a loss whose size grows with the confidence in the wrong classification.

![Figure 2 of the entry logreg](https://dictionaryofml.org/terms/fig/logreg-fig2.svg)

*Figure 2: The logistic loss incurred on a data point with label $y = 1$ as a function of the hypothesis value $h({\mathbf x})$. A correct classification, $h({\mathbf x}) > 0$, with high confidence incurs a small loss; the loss grows as $h({\mathbf x})$ becomes negative, that is, as the confidence in a wrong classification increases. The dashed line marks the decision boundary $h({\mathbf x}) = 0$, and the arrows point toward growing confidence in a correct (right) or wrong (left) classification. Data generated by pythondemos/logreg.py*

Averaging this loss over the training set $\mathcal{D}^{(\rm train)} =
\{({\mathbf x}^{(r)},
y^{(r)})\}_{r=1}^{m}$ of $m$ data points, empirical risk minimization delivers

$$
\begin{aligned}
(\widehat{{\mathbf w}}, \widehat{b}) &\in \mathop{\mathrm{arg\,min}}_{{\mathbf w} \in
\mathbb{R}^{d},\, b \in \mathbb{R}} f({\mathbf w}, b), \quad\text{with}\\
f({\mathbf w}, b) &:= \frac{1}{m}
\sum_{r=1}^{m}
\ln\Big(1 + \exp\big(-y^{(r)}
({\mathbf w}^{\top} {\mathbf x}^{(r)} + b)\big)\Big) \text{.}
\end{aligned}
$$

The logistic loss is the negative log-likelihood of the label under a simple probabilistic model for the classification task.

In this probabilistic model, given the label $y$, the feature vector ${\mathbf x}$ is a Gaussian random variable whose mean ${\boldsymbol \mu}^{(y)}$ depends on the label while its covariance matrix ${\mathbf C}$ does not; the two means ${\boldsymbol \mu}^{(-1)}, {\boldsymbol \mu}^{(+1)} \in
\mathbb{R}^{d}$ are the parameters of the model. When both labels are equally probable a priori, the best linear classifier under this model assigns ${\mathbf x}$ to the label whose mean is nearer, with distances measured by ${\mathbf C}^{-1}$, and its decision boundary is the hyperplane through the midpoint of the two means (Bishop, 2006, Sect. 4.2.1). Its weight vector is ${\mathbf w} = {\mathbf C}^{-1}\big({\boldsymbol \mu}^{(+1)} - {\boldsymbol \mu}^{(-1)}\big)$. The factor $\tfrac{1}{2}$ that puts the decision boundary through the midpoint belongs to the offset $b$ and not to ${\mathbf w}$, as does the logarithm of the ratio of the two prior probabilities when they differ: any positive multiple of ${\mathbf w}$ leaves the decision boundary where it is, but only this one reproduces the posterior probability below. Under the model, the posterior probability of $y = 1$ is the sigmoid function $\sigma({\mathbf w}^{\top}
{\mathbf x} + b)$ with $\sigma(z) = 1 / (1 + e^{-z})$ (Bishop, 2006, Sect. 4.2.1, Eqs. (4.65)–(4.67)). Logistic regression estimates ${\mathbf w}$ and $b$ directly, by maximizing the conditional likelihood function of the labels given the feature vectors. This leaves the distribution of the feature vectors unspecified and is a different estimator from maximum likelihood for the means and the covariance matrix of the Gaussian model (Hastie et al., 2009, Sect. 4.4.5). Maximizing the conditional likelihood function coincides with minimizing the average logistic loss $f$ (Murphy, 2012, Sect. 8.3).

The probabilistic model makes the confidence reading of $|h({\mathbf x})|$ above precise: $\sigma(h({\mathbf x}))$ is the probability that the label of a data point with feature vector ${\mathbf x}$ is $y = 1$, and $\sigma(-h({\mathbf x})) = 1 - \sigma(h({\mathbf x}))$ the probability of $y = -1$. A forecast $h({\mathbf x}) = 2$ thus says that $88$ percent of the days with these temperatures are followed by a frost-free day, and a forecast of $0$ that half of them are. Whether these numbers can be trusted is the question of calibration (Dawid, 1982).

In most applications a data point carries only its label, the class that occurred. The logistic loss is the loss for this case. It is the cross-entropy between the probability mass function concentrated on the observed label and the predicted probability mass function $(\sigma(h({\mathbf x})), 1 -
\sigma(h({\mathbf x})))$ (Bishop, 2006, Sect. 4.3.2). Some data points carry more than a class: a soft label $\tilde{y} \in [0, 1]$, the probability of $y = 1$ itself. It may be the frequency of frost among the recorded days with the same temperatures, the share of several annotators who chose the class, or the output of a larger classifier, as in knowledge distillation. For such a data point the logistic loss is replaced by the cross-entropy between the soft label and the prediction,

$$
L\left(({\mathbf x}, \tilde{y}),h \right) =
- \tilde{y} \ln \sigma(h({\mathbf x}))
- (1 - \tilde{y}) \ln \big(1 - \sigma(h({\mathbf x}))\big)
\text{,}
$$

which for $\tilde{y} \in \{0, 1\}$ is the logistic loss again. Learning then minimizes the average of this loss over the training set, as before.

This cross-entropy equals the Kullback–Leibler divergence from the predicted probability mass function to the soft label probability mass function $(\tilde{y}, 1
- \tilde{y})$ plus the entropy of the soft label probability mass function, which does not depend on $h$ (Bishop, 2006, Sect. 1.6.1, Eq. (1.113)). Minimizing either therefore learns the same $h$, and the loss on a data point is smallest exactly when $\sigma(h({\mathbf x})) = \tilde{y}$, when the confidence of the prediction matches the probability of the class. A hard label can never be matched this way, since $\sigma$ never reaches $0$ or $1$. This is why the logistic loss keeps decreasing as $|h({\mathbf x})|$ grows on a correctly classified data point.

With hard or soft labels, the parameters are found by minimizing the average loss $f$. Unlike linear regression with the squared error loss, the minimization admits no closed-form solution (Bishop, 2006, Sect. 4.3.3). As in linear regression, however, the objective function $f$ is convex (Murphy, 2012, Sect. 8.3.1, Eq. (8.7)) and smooth, so gradient descent applies. The parameters are collected as the pair $({\mathbf w}, b)$, and the gradient descent step reads

$$
({\mathbf w}, b)^{(t+1)} = T\big(({\mathbf w}, b)^{(t)}\big)
\text{, with } T({\mathbf w}, b) := ({\mathbf w}, b) - \eta
\nabla f({\mathbf w}, b) \text{, for } t = 0,1,\ldots \text{,}
$$

with learning rate $\eta$. The gradient $\nabla f$ of the average logistic loss has the weight part

$$
\nabla_{{\mathbf w}} f = -\frac{1}{m}
\sum_{r=1}^{m} y^{(r)}
{\mathbf x}^{(r)} \,
\sigma\big(-y^{(r)} ({\mathbf w}^{\top}
{\mathbf x}^{(r)} + b)\big) \text{,}
$$

and analogously for $b$.

The pair $({\mathbf w}, b)$ is a fixed point of the operator $T$ precisely when $\nabla f({\mathbf w}, b) = \mathbf{0}$, which by convexity of $f$ is exactly when $({\mathbf w}, b)$ is a minimizer. It can be shown that $\nabla f$ is Lipschitz continuous with some constant $L$ (Hastie et al., 2009). For a constant learning rate $\eta < 2/L$, every gradient descent step decreases $f$ (Beck, 2017, Lemma 10.4). For the standard choice $\eta = 1/L$ and a minimizer $(\widehat{{\mathbf w}}, \widehat{b})$ of $f$, the gap $f\big(({\mathbf w}, b)^{(t)}\big) - \min f$ at the $t$th iterate is at most $2 L
\left\Vert {({\mathbf w}, b)^{(1)} - (\widehat{{\mathbf w}}, \widehat{b})} \right\Vert^{2}_{2}
/ (t - 1)$ (Bubeck, 2015, Thm. 3.3). The iterates $({\mathbf w}, b)^{(t)}$ themselves need not converge: on a linearly separable training set the minimum is not attained (Hastie et al., 2009, Sect. 4.4.5), and the norm of the iterates grows without bound while the training error tends to zero. Adding the penalty term $\alpha \left\Vert {{\mathbf w}} \right\Vert^{2}_{2}$, with a positive $\alpha$, to $f$ removes this singularity (Bishop, 2006, Sect. 4.3.2). The penalized objective function is strictly convex in $({\mathbf w}, b)$: its second derivative along a direction $({\mathbf u}, c) \neq (\mathbf{0}, 0)$ is

$$
\frac{1}{m} \sum_{r=1}^{m}
s^{(r)} \big({\mathbf u}^{\top}{\mathbf x}^{(r)} + c\big)^{2}
+ 2\alpha \left\Vert {{\mathbf u}} \right\Vert^{2}_{2} \text{,}
$$

with $s^{(r)} = \sigma\big(h({\mathbf x}^{(r)})\big)\,
\sigma\big(-h({\mathbf x}^{(r)})\big) > 0$, and this vanishes only for ${\mathbf u} = \mathbf{0}$ and $c = 0$. When both labels occur in the training set, the penalized objective function also grows without bound along every unbounded sequence of pairs $({\mathbf w}, b)$, so it has a unique minimizer. Its gradient is Lipschitz continuous with constant $L + 2\alpha$. For a learning rate $\eta < 2/(L + 2\alpha)$ the values of the gradient descent iterates again converge to the minimum, the iterates stay bounded, and they approach the unique minimizer.

In practice, plain gradient descent is seldom used to minimize $f$: the smooth convex objective function is minimized faster by Newton's method, equivalently iteratively reweighted least squares (Hastie et al., 2009, Sect. 4.4.1, Eq. (4.28)), by the quasi-Newton solver L-BFGS (Liu and Nocedal, 1989), and, for a large training set, by coordinate descent (Wright, 2015).

What these solvers minimize is the average logistic loss, but that is not the only loss with which a linear classifier can be learned. In principle, the squared error loss serves too: with the label values encoded as the real numbers $+1$ and $-1$, the squared error $\big(y - ({\mathbf w}^{\top}{\mathbf x} + b)\big)^{2}$ is well defined. Minimizing the squared error loss, however, has a flaw: it penalizes a prediction ${\mathbf w}^{\top}{\mathbf x} + b$ that is large, positive, and correct, while the logistic loss keeps decreasing as this margin grows (Fig. 3). The hinge loss of the support vector machine shares this behavior on a correctly classified data point but stays flat once the margin exceeds one.

![Figure 3 of the entry logreg](https://dictionaryofml.org/terms/fig/logreg-fig3.svg)

*Figure 3: The logistic loss, squared error loss and hinge loss incurred on a data point with label $y = 1$ as a function of the hypothesis value $h({\mathbf x})$. The squared error loss turns upward and penalizes a confident correct prediction, whereas the logistic loss keeps decreasing; the hinge loss stays flat once the margin exceeds one. The dashed line marks the decision boundary $h({\mathbf x}) = 0$, and the arrows point toward growing confidence in a correct (right) or wrong (left) classification. Data generated by pythondemos/logreg.py*

For more than two label values, logistic regression extends in two ways. A simple reduction trains one binary classification logistic regression per label, each separating that label from the rest, and predicts with the most confident of them; this is one-vs-rest classification. An alternative replaces the sigmoid function by the softmax function, which models the probabilities of all labels jointly, so that they sum to one and no label is left without one; fitting it by maximum likelihood yields multinomial logistic regression (Bishop, 2006, Sect. 4.3.4).

**See also:** [classification](https://dictionaryofml.org/terms/classification.html), [binary classification](https://dictionaryofml.org/terms/binclass.html), [linear classifier](https://dictionaryofml.org/terms/linclass.html), [decision boundary](https://dictionaryofml.org/terms/decisionboundary.html), [sigmoid function](https://dictionaryofml.org/terms/sigmoid.html), [logistic loss](https://dictionaryofml.org/terms/logloss.html), squared error loss, hinge loss, [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [gradient descent](https://dictionaryofml.org/terms/gd.html), maximum likelihood, [linear regression](https://dictionaryofml.org/terms/linreg.html), softmax function, [regularization](https://dictionaryofml.org/terms/regularization.html), calibration, soft label, cross-entropy, Kullback–Leibler divergence.

## References

- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Murphy (2012). Machine Learning: A Probabilistic Perspective. *MIT Press*.
- Dawid (1982). The Well-Calibrated Bayesian. *Journal of the American Statistical Association*.
- Beck (2017). First-Order Methods in Optimization. *SIAM-Society for Industrial and Applied Mathematics*.
- Bubeck (2015). Convex Optimization: Algorithms and Complexity. *Found. Trends Mach. Learn.*. <https://doi.org/10.1561/2200000050>
- Liu and Nocedal (1989). On the Limited Memory Method for Large Scale Optimization. *Mathematical Programming B*.
- Wright (2015). Coordinate Descent Algorithms. *Mathematical Programming*. <https://doi.org/10.1007/s10107-015-0892-3>

## Cite this entry

```bibtex
@misc{dictml_logreg,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {logistic regression},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/logreg.html}
}
```