Dictionary of Applied Machine Learning: 115 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

binary classification

Updated on 2026-11-08

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also classification decision boundary logistic loss logistic regression accuracy confusion matrix baseline

Binary classification refers to machine learning (ML) applications with a binary label space. The goal of binary classification is to assign each data point to one of two classes, each class represented by one of the two label values. Strictly speaking, a binary classifier is a hypothesis map from the feature space to a binary label space. However, it is customary to construct such a classifier by comparing the value of a real-valued hypothesis map against a threshold. Linear classifiers are obtained from the sign of a linear hypothesis map. A binary classifier partitions the feature space into two decision regions separated by a decision boundary, which is a hyperplane for a linear classifier. The natural loss function is the $0/1$ loss, whose average over a dataset is $1$ minus the accuracy. However, this loss function is hard to optimize for training methods, and its average on imbalanced data is dominated by the majority label value, so it does not indicate which of the two kinds of mistake a classifier makes. Practical methods therefore often use other loss functions for model training and validation.

Figure produced by the Python demo of the entry binary classification
Figure written by the entry's Python demo binclass.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

A frost warning answers one question: will the temperature in the coming night fall below $0$ degrees Celsius? The answer is yes or no, nothing in between. An email is spam or not. A credit application defaults or not. A biopsy image is malignant or benign. Binary classification is the machine learning (ML) problem behind all of these questions: predict, from the features of a data point, which of two values its label, a number $y$, takes.

Binary classification is classification with a label space $\mathcal{Y}$ that holds two elements. Each data point carries one label, and that label takes one of the two values. The two values are usually written (or encoded) as $\mathcal{Y} = \{-1, +1\}$ or $\mathcal{Y} = \{0, 1\}$. The choice of the two label values (the encoding) is a convention and irrelevant for the basic properties of binary classification methods. However, mathematical expressions about them often assume a particular encoding, such as $\{-1, +1\}$, and need to be modified accordingly if a different encoding, such as $\{0, 1\}$, is used.

Strictly speaking, a binary classifier is a hypothesis map that takes values in a binary label space $\mathcal{Y}$. It is usually built from a real-valued hypothesis map $h: \mathcal{X} \rightarrow \mathbb{R}$ by comparing its value against a threshold. For the label encoding $\mathcal{Y} = \{-1, +1\}$ and the threshold $0$, a data point is then classified by the sign, $\widehat{y} = \operatorname{sign}(h({\mathbf x}))$. The classifier partitions the feature space into two decision regions, one per label value. They are separated by the decision boundary, the set of ${\mathbf x}$ with $h({\mathbf x}) = 0$. The product $y \cdot h({\mathbf x})$ is the margin of $h$ on the data point $({\mathbf x}, y)$. It is positive exactly when the prediction is right.

B-regionsFor a linear classifier the real-valued hypothesis is $h({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x} + w_{0}$ with model parameters ${\mathbf w}$ and $w_{0}$, so the decision boundary is the hyperplane ${\mathbf w}^{\top} {\mathbf x} + w_{0} = 0$, a straight line when there are two features (Bishop, 2006, Sect. 4.1.1). The predicted label value is the side of that hyperplane on which ${\mathbf x}$ lies, and the distance of ${\mathbf x}$ from the hyperplane is $|h({\mathbf x})| / \Vert {{\mathbf w}} \Vert$. A data point that lies on the side of the other label value is misclassified. Its $0/1$ loss is $1$ (Fig. 1).

Figure 1 of the entry binclass
Figure 1: Binary classification of data points with two features using a linear classifier. The decision boundary $\hypothesis(\featurevec) = 0$ is a hyperplane, here a straight line, and it cuts the feature space into the two decision regions of the label values $\truelabel = +1$ (filled circles, where $\hypothesis(\featurevec) > 0$) and $\truelabel = -1$ (open squares, where $\hypothesis(\featurevec) < 0$). The model parameters are scaled so that $\norm{\weights} = 1$, and the double arrow shows the margin $\truelabel \cdot \hypothesis(\featurevec)$ of one data point, which is then its distance from the line. One of the $8$ data points is misclassified: it incurs a $0/1$ loss of $1$, and the average $0/1$ loss is $1/8$
The shape of the decision regions depends on the model, and so does how closely they follow the training set. Fig. 2 shows three classifiers learned from the same $30$ days of weather recordings. The linear classifier divides the feature space into two half-planes and misclassifies three of the $30$ days. The decision tree compares one feature with a threshold at each of its splits, so its decision regions are unions of axis-parallel rectangles (Hastie et al., 2009, Sect. 9.2.1). Grown until every leaf holds a single label value, it predicts the label value of every day correctly. The artificial neural network (ANN) has a curve as decision boundary and also predicts every day correctly. On $170$ further days, the linear classifier is the most accurate of the three. The tree and the ANN have adapted their decision regions to single days of the training set, which is overfitting.
Figure 2 of the entry binclass
Figure 2: Decision regions of three binary classifiers learned from the same $30$ data points. Each data point represents the weather conditions around a specific day: its features are the minimum and the maximum temperature of the day in degrees Celsius. Filled circles are days followed by frost ($\truelabel = +1$), open squares the other days ($\truelabel = -1$). The shaded area is the decision region where the classifier predicts $+1$. (a) A linear classifier, whose regions are two half-planes. (b) A decision tree grown until every leaf holds a single label value, whose regions are unions of axis-parallel rectangles. (c) An ANN with one hidden layer of $32$ units, whose decision boundary is a curve. Data generated by pythondemos/binclass.py
The natural loss function for binary classification is the $0/1$ loss. It is $1$ for a wrong label value and $0$ for a right one. Its average over a dataset is therefore the fraction of misclassified data points, i.e., $1$ minus the accuracy. Empirical risk minimization (ERM) with the $0/1$ loss results in computationally hard optimization problems (Shalev-Shwartz and Ben-David, 2014, Ch. 12). Practical methods therefore minimize a convex surrogate of the $0/1$ loss (Shalev-Shwartz and Ben-David, 2014, Sect. 12.3). The hinge loss gives the support vector machine (SVM), and the logistic loss gives logistic regression. Fig. 3 shows the three loss functions as functions of the margin.
Figure 3 of the entry binclass
Figure 3: The three loss functions of binary classification (with label space $\labelspace = \{-1, +1\}$) as functions of the margin $\truelabel \cdot \hypothesis(\featurevec)$: the $0/1$ loss (solid curve; at margin $0$ it takes the value $1$, marked by a filled circle, and not the value $0$, marked by an open circle), the hinge loss (dashed curve), and the logistic loss (dotted curve). Both surrogates are convex and decreasing functions of the margin, so a more negative margin costs more, while the $0/1$ loss only jumps once, at margin $0$
The average $0/1$ loss is not only challenging as an objective for training but can also be misleading. In particular, the average $0/1$ loss (and the associated accuracy) does not indicate which of the two mistakes a classifier makes. The two kinds of mistake, a wrong $+1$ and a wrong $-1$, usually have different costs. On imbalanced data, the constant baseline that always returns the majority label value already reaches a high accuracy. Both are reasons to report the confusion matrix, and the precision, recall, and F$_1$ score computed from it, beside the accuracy.

In the frost warning of the opening, the features of a data point are the minimum and the maximum air temperature of one day. Its label is $y = +1$ if the minimum temperature of the following night falls below $0$ degrees Celsius, and $y = -1$ otherwise. A wrong $-1$ leaves a fruit grower's blossoms unprotected in a frost. A wrong $+1$ only costs one night of covering or heating them. The two mistakes are therefore not interchangeable, and the classifier is chosen so that wrong $-1$ predictions are rare.

Synonyms: two-class classification.

See also: classification, multi-label classification, classifier, decision boundary, $\bf 0/1$ loss, logistic loss, logistic regression, accuracy, confusion matrix, imbalanced data, baseline.

References

  1. Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media. doi.org/10.1007/978-0-387-45528-0
  2. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
  3. Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. Cambridge Univ. Press. doi.org/10.1017/cbo9781107298019

Cite this entry

@misc{dictml_binclass,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {binary classification},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-13},
  url = {https://dictionaryofml.org/terms/binclass.html}
}