Dictionary of Applied Machine Learning

logistic regression

Updated on 2026-09-04

Typeset PDF version — the authoritative form of this entry

Python demo — a script that recomputes what this entry states and prints one line per check

Logistic regression learns a linear hypothesis map $\hypothesis(\featurevec) = \weights^{\top} \featurevec$ for a binary classification problem with label $\truelabel \in \{-1, 1\}$. The sign of $\hypothesis(\featurevec)$ is the predicted label, and the sigmoid function transform $\sigma(\hypothesis(\featurevec))$ estimates the probability of the label $\truelabel = 1$. The parameters are learned by empirical risk minimization (ERM) with the average logistic loss on the training set, which is equivalent to maximum likelihood estimation under the sigmoid function probability model. The resulting objective is convex and smooth, so gradient descent (GD) converges to a minimizer whenever one exists; on a linearly separable training set a penalty term restores a unique minimizer. Thresholding the learned hypothesis yields a linear classifier whose decision boundary is a hyperplane.

Definition

An email service must decide, for every incoming message, whether to move it to the spam folder. Features extracted from the message — word frequencies, sender information — form its feature vector $\featurevec \in \reals^{\featuredim}$, and the label $\truelabel \in \{-1, 1\}$ records whether the message is spam. Logistic regression learns, for such a binary classification problem, a linear hypothesis map $\hypothesis(\featurevec) = \weights^{\top} \featurevec$ (Bishop, 2006; Hastie et al., 2009). Despite its name, the method solves a classification problem — the regression in the name refers to fitting the real-valued map $\hypothesis$, not to a numeric label.

B-gdThe value $\hypothesis(\featurevec)$ is read in two ways. Its sign is the predicted label, which makes the learned $\hypothesis$ a linear classifier whose decision boundary is the hyperplane $\weights^{\top} \featurevec = 0$. Its sigmoid function transform $\sigma(\hypothesis(\featurevec))$ is an estimate of the probability of the label $\truelabel = 1$ given the feature vector. The quality of a candidate $\weights$ is measured by the average logistic loss on the training set $\trainset = \{(\featurevec^{(\sampleidx)}, \truelabel^{(\sampleidx)})\}_{\sampleidx=1}^{\samplesize}$ of $\samplesize$ data points, and empirical risk minimization (ERM) delivers \[ \widehat{\weights} \in \argmin_{\weights \in \reals^{\featuredim}} f(\weights) \text{, with } f(\weights) \defeq \frac{1}{\samplesize} \sum_{\sampleidx=1}^{\samplesize} \log\Big(1 + \exp\big(-\truelabel^{(\sampleidx)} \weights^{\top} \featurevec^{(\sampleidx)}\big)\Big) \text{.} \] Minimizing $f$ is equivalent to maximum likelihood estimation under the model that assigns the label $\truelabel = 1$ with probability $\sigma(\weights^{\top} \featurevec)$ (Bishop, 2006). Fig. 1 shows a learned probability curve for a training set with a single feature.

Figure 1 of the entry logreg
Figure 1: Logistic regression on a training set with a single feature. Data points with label $\truelabel = 1$ are drawn as filled circles at height one, those with $\truelabel = -1$ as open squares at height zero. The curve is the sigmoid function transform $\sigma(\widehat{\weights}^{\top}\featurevec)$ of the learned hypothesis: it estimates the probability of the label $\truelabel = 1$ and crosses $1/2$ at the decision boundary. Data generated by pythondemos/logreg.py
Unlike linear regression with the squared error loss, the minimization admits no closed-form solution, but $f$ is convex and smooth, so gradient descent (GD) applies. The iteration is \[ \weights^{(\iteridx+1)} = T\big(\weights^{(\iteridx)}\big) \text{, with } T(\weights) \defeq \weights - \lrate \nabla f(\weights) \text{,} \] with learning rate $\lrate$ and the gradient $\nabla f(\weights)$. A vector $\weights$ is a fixed point of the operator $T$ exactly when $\nabla f(\weights) = \mathbf{0}$, which by convexity of $f$ is exactly when $\weights$ is a minimizer. For a sufficiently small learning rate, no update increases $f$, and the iterates converge to a minimizer whenever one exists (Hastie et al., 2009). On a linearly separable training set no minimizer exists — the norm of the iterates grows without bound while the training error tends to zero — and adding a penalty term to $f$, one of the three routes that regularization distinguishes, restores a unique minimizer.

Applying linear regression with the squared error loss to the binary labels is a near miss: the squared error loss penalizes a prediction $\weights^{\top}\featurevec$ that is large, positive, and correct, while the logistic loss decreases as this margin grows. For more than two label values, replacing the sigmoid function by the softmax function yields multinomial logistic regression (Bishop, 2006).

See also: classification, binary classification, linear classifier, decision boundary, sigmoid function, logistic loss, empirical risk minimization, gradient descent, maximum likelihood, linear regression, softmax function, regularization.

References

  1. Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media. doi.org/10.1007/978-0-387-45528-0
  2. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7

Cite this entry

@misc{dictml_logreg,
  author = {Jung, Alexander and Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {logistic regression},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-11},
  url = {https://dictionaryofml.org/terms/logreg.html}
}