Dictionary of Applied Machine Learning · binary classification

binary classification — Python demo

Updated on 2026-11-03

Numerical companion to the entry binary classification: it recomputes what the entry states and prints one line per check

Backs the entry's claims numerically: the average zero-one loss of a classifier is 1 minus its accuracy; the decision boundary of a linear classifier is a straight line that cuts the feature space into the two decision regions; a decision tree and an ANN learned from the same days cut the feature space differently (axis-parallel rectangles, a curved boundary); the zero-one loss is not convex in the model parameters, while the logistic loss (divided by log 2) and the hinge loss are convex and upper bound it; on imbalanced data the constant answer already has a high accuracy, so the confusion matrix with precision and recall says more than the accuracy alone. Self-contained (numpy and matplotlib only), fixed seed.

Run it without installing anything:
uv run https://dictionaryofml.org/terms/binclass.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download binclass.py into a folder and run uv run binclass.py there. With NumPy and Matplotlib already installed, python3 binclass.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download binclass.py · Notebook · Open in Colab

Before running a downloaded copy, scan it: uvx bandit -q -t B102,B307,B602,B605 binclass.py (B102,B307,B602,B605 are the exec, eval and shell checks of bandit) reports nothing: the script runs no shell command and evaluates no generated code.

The script, block by block

One cell per block of the script: the code, and what that code printed when it last ran here

setup

"""
binclass.py — numerical companion to the glossary entry 'binary
classification'.

Purpose
-------
Backs the entry's claims numerically: the average zero-one loss of a
classifier is 1 minus its accuracy; the decision boundary of a linear
classifier is a straight line that cuts the feature space into the two
decision regions; a decision tree and an ANN learned from the same days
cut the feature space differently (axis-parallel rectangles, a curved
boundary); the zero-one loss is not convex in the model parameters,
while the logistic loss (divided by log 2) and the hinge loss are convex
and upper bound it; on imbalanced data the constant answer already has a
high accuracy, so the confusion matrix with precision and recall says
more than the accuracy alone.  Self-contained (numpy and matplotlib
only), fixed seed.

Setup
-----
The frost warning of the entry: for m = 200 days, two features (the
minimum and the maximum air temperature of the day, in degrees Celsius)
and the label y = +1 if the minimum temperature of the following night
falls below 0 degrees Celsius, y = -1 otherwise.  The labels are
imbalanced: about 20 percent of the days are followed by frost.  A linear
classifier h(x) = w^T x + w_0 is obtained by logistic regression, i.e.,
by minimizing the average logistic loss (3000 small steps against its
slope).

Blocks
------
[B-zeroone]   The average zero-one loss of the learned linear classifier
              equals 1 minus its accuracy on the m days.
[B-boundary]  Every day on the +1 side of the line w^T x + w_0 = 0 is
              predicted +1 and every day on the other side -1: the
              decision regions are the two half-planes.
[B-regions]   A linear classifier, a decision tree (each split compares
              one feature with a threshold, grown until every leaf is
              pure) and an ANN with one hidden layer of 32 units are
              learned from the same 30 of the m days.  On a grid over
              the feature space, the tree's +1 region is a union of
              axis-parallel rectangles (its boundary runs only along
              grid lines), while the ANN's boundary is a curve.  The
              tree and the ANN reproduce the label of every one of the
              30 days, the linear classifier does not, and on the other
              170 days the linear classifier is the most accurate of
              the three: the tree and the ANN overfit.
[B-convex]    Along a straight line through the model parameters of the
              linear classifier, the average zero-one loss is piecewise
              constant and not convex (it violates the midpoint
              inequality), while the average logistic loss / log 2 and
              the average hinge loss satisfy it and lie on or above the
              zero-one loss at every point.
[B-imbalance] The constant answer "no frost" reaches an accuracy near
              0.8 with recall 0; the learned linear classifier's
              confusion matrix gives a recall below its accuracy, a gap
              the accuracy alone hides.

Outputs
-------
binclass_data.csv    : x1, x2, y of the m days, the linear classifier's
                       predicted label, and sub = 1 for the 30 days the
                       three classifiers of [B-regions] are learned from.
binclass_regions.csv : grid over the feature space with the label
                       predicted by the linear classifier (lin), the
                       decision tree (tree) and the ANN (ann) learned
                       from the 30 days; 60 columns per row, rows
                       ordered by x2.
binclass_line.csv    : t, average zero-one, logistic / log 2 and hinge
                       loss along the line of model parameters
                       w(t) = w_hat + t * d.
binclass_cm.csv      : the confusion matrix of the linear classifier and
                       of the constant answer, with accuracy, precision,
                       recall.
binclass.png         : matplotlib preview of four panels (checking only).
"""

# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///

import numpy as np
import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt

from pathlib import Path

OUT_DIR = Path(__file__).parent

report = []


def check(name, ok):
    report.append((name, bool(ok)))
    print(f"  [{'ok' if ok else 'FAIL'}] {name}")


rng = np.random.default_rng(0)
LOG2 = np.log(2.0)

m = 200
tmin = rng.uniform(-4.0, 14.0, m)                     # minimum of the day
tmax = tmin + rng.uniform(2.0, 13.0, m)               # maximum of the day
X = np.c_[tmin, tmax]
# the following night cools more after a day with a large temperature range
next_min = 0.8 * tmin - 0.45 * (tmax - tmin) + 3.5 + 1.8 * rng.standard_normal(m)
y = np.where(next_min < 0.0, 1.0, -1.0)
Xb = np.c_[X, np.ones(m)]                             # (x1, x2, 1): w and w_0 in one


def avg_losses(w):
    margin = y * (Xb @ w)
    return (float(np.mean(margin <= 0)),
            float(np.mean(np.log1p(np.exp(-margin))) / LOG2),
            float(np.mean(np.maximum(0.0, 1.0 - margin))))


# logistic regression: minimize the average logistic loss, step by step
w = np.zeros(3)
for _ in range(3000):
    margin = y * (Xb @ w)
    slope = -(Xb * (y / (1.0 + np.exp(margin)))[:, None]).mean(axis=0)
    w = w - 0.01 * slope
y_hat = np.where(Xb @ w > 0, 1.0, -1.0)

B-zeroone

The average zero-one loss of the learned linear classifier equals 1 minus its accuracy on the m days.

zo = float(np.mean(y_hat != y))
acc = float(np.mean(y_hat == y))
check(f"[B-zeroone]   average zero-one loss {zo:.3f} = 1 - accuracy "
      f"{1 - acc:.3f}", abs(zo - (1 - acc)) < 1e-12)
  [ok] [B-zeroone]   average zero-one loss 0.060 = 1 - accuracy 0.060

B-boundary

Every day on the +1 side of the line w^T x + w_0 = 0 is predicted +1 and every day on the other side -1: the decision regions are the two half-planes.

side = np.sign(Xb @ w)
check(f"[B-boundary]  predicted label equals the side of the line "
      f"w^T x + w_0 = 0 for all {m} days; {int(np.sum(y_hat == 1))} days "
      f"on the +1 side", np.array_equal(side, y_hat))
  [ok] [B-boundary]  predicted label equals the side of the line w^T x + w_0 = 0 for all 200 days; 41 days on the +1 side

B-regions

A linear classifier, a decision tree (each split compares one feature with a threshold, grown until every leaf is pure) and an ANN with one hidden layer of 32 units are learned from the same 30 of the m days. On a grid over the feature space, the tree's +1 region is a union of axis-parallel rectangles (its boundary runs only along grid lines), while the ANN's boundary is a curve. The tree and the ANN reproduce the label of every one of the 30 days, the linear classifier does not, and on the other 170 days the linear classifier is the most accurate of the three: the tree and the ANN overfit.

n_sub = 30
sub = np.zeros(m, dtype=bool); sub[:n_sub] = True   # the first 30 days
Xs_, ys_ = X[sub], y[sub]


def gini(labels):
    if len(labels) == 0:
        return 0.0
    p = np.mean(labels == 1)
    return 2.0 * p * (1.0 - p)


def grow_tree(Xs, ys):
    """A node is a leaf (label) or (feature, threshold, left, right);
    splitting stops only when a leaf holds one label."""
    majority = 1.0 if np.mean(ys == 1) >= 0.5 else -1.0
    if gini(ys) == 0.0:
        return majority
    best = (gini(ys), None, None)
    for j in range(2):
        vals = np.unique(Xs[:, j])
        for thr in (vals[:-1] + vals[1:]) / 2:
            left = Xs[:, j] <= thr
            score = (left.mean() * gini(ys[left])
                     + (~left).mean() * gini(ys[~left]))
            if score < best[0] - 1e-12:
                best = (score, j, thr)
    if best[1] is None:
        return majority
    j, thr = best[1], best[2]
    left = Xs[:, j] <= thr
    return (j, thr, grow_tree(Xs[left], ys[left]), grow_tree(Xs[~left], ys[~left]))


def tree_predict(node, Xs):
    if not isinstance(node, tuple):
        return np.full(len(Xs), node)
    j, thr, left, right = node
    out = np.empty(len(Xs))
    mask = Xs[:, j] <= thr
    out[mask] = tree_predict(left, Xs[mask])
    out[~mask] = tree_predict(right, Xs[~mask])
    return out


def tree_depth(node):
    return 0 if not isinstance(node, tuple) else 1 + max(tree_depth(node[2]), tree_depth(node[3]))


# the linear classifier of the 30 days, by logistic regression as above
Xb_s = np.c_[Xs_, np.ones(n_sub)]
w_s = np.zeros(3)
for _ in range(3000):
    margin = ys_ * (Xb_s @ w_s)
    slope = -(Xb_s * (ys_ / (1.0 + np.exp(margin)))[:, None]).mean(axis=0)
    w_s = w_s - 0.01 * slope

tree = grow_tree(Xs_, ys_)

# an ANN with one hidden layer of 32 tanh units; inputs in tens of degrees
# so that the first weights start small; trained like the linear classifier
n_hidden = 32
W1 = 0.5 * rng.standard_normal((2, n_hidden)); b1 = np.zeros(n_hidden)
W2 = 0.5 * rng.standard_normal(n_hidden); b2 = 0.0


def ann_score(Xs, W1, b1, W2, b2):
    H = np.tanh((Xs / 10.0) @ W1 + b1)
    return H @ W2 + b2, H


for _ in range(20000):
    s_, H = ann_score(Xs_, W1, b1, W2, b2)
    g = -ys_ / (1.0 + np.exp(ys_ * s_)) / n_sub          # d(avg logistic loss)/ds
    gW2 = H.T @ g; gb2 = g.sum()
    gH = np.outer(g, W2) * (1.0 - H ** 2)
    gW1 = (Xs_ / 10.0).T @ gH; gb1 = gH.sum(axis=0)
    W1 -= 0.5 * gW1; b1 -= 0.5 * gb1; W2 -= 0.5 * gW2; b2 -= 0.5 * gb2


def predict_all(Xq):
    lin = np.where(np.c_[Xq, np.ones(len(Xq))] @ w_s > 0, 1, -1)
    tr = tree_predict(tree, Xq).astype(int)
    an = np.where(ann_score(Xq, W1, b1, W2, b2)[0] > 0, 1, -1)
    return lin, tr, an


g1 = np.linspace(-4.0, 14.0, 60); g2 = np.linspace(-2.0, 27.0, 60)
G1, G2 = np.meshgrid(g1, g2)                          # row i: x2 = g2[i]
G = np.c_[G1.ravel(), G2.ravel()]
r_lin, r_tree, r_ann = predict_all(G)
R_tree = r_tree.reshape(G1.shape); R_ann = r_ann.reshape(G1.shape)
# the tree's boundary runs along grid lines: a cell differs from its
# neighbour only across a horizontal or vertical line, so every row and
# every column of the grid changes label at most (number of splits) times
n_splits = 2 ** tree_depth(tree) - 1
tree_changes = max(int((np.diff(R_tree, axis=1) != 0).sum(axis=1).max()),
                   int((np.diff(R_tree, axis=0) != 0).sum(axis=0).max()))
acc_in = [float(np.mean(p == ys_)) for p in predict_all(Xs_)]
acc_out = [float(np.mean(p == y[~sub])) for p in predict_all(X[~sub])]
check(f"[B-regions]   accuracy on the 30 days: linear {acc_in[0]:.2f}, tree "
      f"{acc_in[1]:.2f}, ANN {acc_in[2]:.2f}; on the other {m - n_sub} days: "
      f"linear {acc_out[0]:.2f}, tree {acc_out[1]:.2f}, ANN {acc_out[2]:.2f}; "
      f"the tree has depth {tree_depth(tree)} and its label changes at most "
      f"{tree_changes} times along any grid row or column",
      acc_in[1] == 1.0 and acc_in[2] == 1.0 and acc_in[0] < 1.0
      and acc_out[0] > max(acc_out[1], acc_out[2]) and tree_changes <= n_splits)
  [ok] [B-regions]   accuracy on the 30 days: linear 0.90, tree 1.00, ANN 1.00; on the other 170 days: linear 0.93, tree 0.84, ANN 0.92; the tree has depth 3 and its label changes at most 4 times along any grid row or column

B-convex

Along a straight line through the model parameters of the linear classifier, the average zero-one loss is piecewise constant and not convex (it violates the midpoint inequality), while the average logistic loss / log 2 and the average hinge loss satisfy it and lie on or above the zero-one loss at every point.

d = np.array([1.0, -1.5, 0.4]); d /= np.linalg.norm(d)
ts = np.linspace(-3.0, 3.0, 121)
line = np.array([avg_losses(w + 0.1 * t * d) for t in ts])
zo_l, lg_l, hg_l = line[:, 0], line[:, 1], line[:, 2]


def midpoint_violations(v):
    # f((a+b)/2) <= (f(a)+f(b))/2 for all grid pairs a < b of equal spacing
    n = len(v); bad = 0; tot = 0
    for i in range(n):
        for j in range(i + 2, n, 2):
            k = (i + j) // 2; tot += 1
            if v[k] > (v[i] + v[j]) / 2 + 1e-9:
                bad += 1
    return bad, tot


zo_bad, tot = midpoint_violations(zo_l)
lg_bad, _ = midpoint_violations(lg_l)
hg_bad, _ = midpoint_violations(hg_l)
levels = len(np.unique(np.round(zo_l, 6)))
check(f"[B-convex]    along the line of model parameters the zero-one loss takes "
      f"{levels} values and violates the midpoint inequality {zo_bad} of "
      f"{tot} times; logistic/log 2: {lg_bad}, hinge: {hg_bad}; both "
      f"surrogates >= zero-one loss everywhere",
      zo_bad > 0 and lg_bad == 0 and hg_bad == 0
      and (lg_l >= zo_l - 1e-12).all() and (hg_l >= zo_l - 1e-12).all())
  [ok] [B-convex]    along the line of model parameters the zero-one loss takes 19 values and violates the midpoint inequality 745 of 3600 times; logistic/log 2: 0, hinge: 0; both surrogates >= zero-one loss everywhere

B-imbalance

The constant answer "no frost" reaches an accuracy near 0.8 with recall 0; the learned linear classifier's confusion matrix gives a recall below its accuracy, a gap the accuracy alone hides.

def confusion(pred):
    tp = int(np.sum((pred == 1) & (y == 1))); fn = int(np.sum((pred == -1) & (y == 1)))
    fp = int(np.sum((pred == 1) & (y == -1))); tn = int(np.sum((pred == -1) & (y == -1)))
    accuracy = (tp + tn) / m
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    return tp, fn, fp, tn, accuracy, precision, recall


cm_learned = confusion(y_hat)
cm_const = confusion(-np.ones(m))
frac_pos = float(np.mean(y == 1))
check(f"[B-imbalance] {frac_pos:.2f} of the days are followed by frost; the "
      f"constant answer -1 has accuracy {cm_const[4]:.2f} and recall "
      f"{cm_const[6]:.2f}; the linear classifier has accuracy "
      f"{cm_learned[4]:.2f}, precision {cm_learned[5]:.2f}, recall "
      f"{cm_learned[6]:.2f}",
      0.15 <= frac_pos <= 0.25 and cm_const[4] >= 0.75 and cm_const[6] == 0
      and 0.5 < cm_learned[6] < cm_learned[4])

# ---------------------------------------------------------------- CSV
with open(OUT_DIR / "binclass_data.csv", "w") as fh:
    fh.write("x1,x2,y,yhat,sub\n")
    for a, b, c, e, f in zip(X[:, 0], X[:, 1], y, y_hat, sub):
        fh.write(f"{a:.3f},{b:.3f},{int(c)},{int(e)},{int(f)}\n")
with open(OUT_DIR / "binclass_regions.csv", "w") as fh:
    fh.write("x1,x2,lin,tree,ann\n")
    for a, b, c, e, f in zip(G[:, 0], G[:, 1], r_lin, r_tree, r_ann):
        fh.write(f"{a:.3f},{b:.3f},{c},{e},{f}\n")
with open(OUT_DIR / "binclass_line.csv", "w") as fh:
    fh.write("t,zeroone,logistic_rescaled,hinge\n")
    for t, a, b, c in zip(ts, zo_l, lg_l, hg_l):
        fh.write(f"{t:.3f},{a:.4f},{b:.4f},{c:.4f}\n")
with open(OUT_DIR / "binclass_cm.csv", "w") as fh:
    fh.write("classifier,tp,fn,fp,tn,accuracy,precision,recall\n")
    for name, cm in (("linear", cm_learned), ("constant_no", cm_const)):
        fh.write(f"{name},{cm[0]},{cm[1]},{cm[2]},{cm[3]},{cm[4]:.4f},"
                 f"{cm[5]:.4f},{cm[6]:.4f}\n")

# -------------------------------------------------------------- preview
fig, axes = plt.subplots(2, 2, figsize=(9.6, 8.0))
(ax1, ax2), (ax3, ax4) = axes
pos, neg = y == 1, y == -1


def points(ax, which):
    ax.plot(X[pos & which, 0], X[pos & which, 1], "ko", ms=3.5, label="$y = +1$ (frost)")
    ax.plot(X[neg & which, 0], X[neg & which, 1], "ks", mfc="none", ms=3.5, label="$y = -1$ (no frost)")
    ax.set_xlim(g1[0], g1[-1]); ax.set_ylim(g2[0], g2[-1])
    ax.set_xlabel("minimum temperature $x_1$ (degrees Celsius)")
    ax.set_ylabel("maximum temperature $x_2$ (degrees Celsius)")


xx = np.linspace(g1[0], g1[-1], 50)
ax1.plot(xx, -(w[0] * xx + w[2]) / w[1], "k-", lw=1.4, label="decision boundary")
points(ax1, np.ones(m, dtype=bool))
ax1.set_xlabel("minimum temperature $x_1$ (degrees Celsius)")
ax1.set_ylabel("maximum temperature $x_2$ (degrees Celsius)")
ax1.set_title("linear classifier of all 200 days: decision boundary")
ax1.legend(frameon=False, fontsize=7, loc="upper left")
for ax, R, name in ((ax3, R_tree, "decision tree"), (ax4, R_ann, "ANN")):
    ax.contourf(G1, G2, R, levels=[-2, 0, 2], colors=["white", "0.75"])
    points(ax, sub)
    ax.set_title(f"{name} learned from 30 days: +1 region shaded")
ax2.plot(ts, zo_l, "k-", lw=1.4, label="average zero-one loss")
ax2.plot(ts, lg_l, "k--", lw=1.2, label="average logistic loss / log 2")
ax2.plot(ts, hg_l, "k:", lw=1.6, label="average hinge loss")
ax2.set_xlabel("position $t$ along a line of model parameters")
ax2.set_ylabel("average loss over the m days")
ax2.set_title("zero-one loss is not convex, its surrogates are")
ax2.legend(frameon=False, fontsize=8)
fig.tight_layout()
fig.savefig(OUT_DIR / "binclass.png", dpi=110)

n_ok = sum(ok for _, ok in report)
print(f"\n{n_ok}/{len(report)} checks pass")
print(f"wrote {OUT_DIR / 'binclass_data.csv'}, {OUT_DIR / 'binclass_regions.csv'}, "
      f"{OUT_DIR / 'binclass_line.csv'}, {OUT_DIR / 'binclass_cm.csv'}, "
      f"{OUT_DIR / 'binclass.png'}")
if n_ok != len(report):
    raise SystemExit(1)
  [ok] [B-imbalance] 0.20 of the days are followed by frost; the constant answer -1 has accuracy 0.81 and recall 0.00; the linear classifier has accuracy 0.94, precision 0.83, recall 0.87

5/5 checks pass
wrote /Users/junga1/dictionaryappliedml/pythondemos/binclass_data.csv, /Users/junga1/dictionaryappliedml/pythondemos/binclass_regions.csv, /Users/junga1/dictionaryappliedml/pythondemos/binclass_line.csv, /Users/junga1/dictionaryappliedml/pythondemos/binclass_cm.csv, /Users/junga1/dictionaryappliedml/pythondemos/binclass.png
Preview figure produced by binclass.py
The preview figure the block B-imbalance writes when the script runs