Dictionary of Applied Machine Learning · logistic loss

logistic loss — Python demo

Updated on 2026-11-03

Numerical companion to the entry logistic loss: it recomputes what the entry states and prints one line per check

Backs the entry's claims about the logistic loss numerically: divided by log 2 it is an upper bound on the zero-one loss with equality at margin 0; it is convex and differentiable everywhere while the hinge loss has a kink at margin 1; the average zero-one loss is flat almost everywhere in the model parameters, so GD gets no direction from it, whereas GD on the average logistic loss decreases it at every step and thereby also drives the zero-one loss down. It also backs the entry's caveat on reading the number itself: the logistic loss of a hypothesis is not determined by the decisions it makes, and ranking two hypotheses by it can invert the ranking by accuracy. Self-contained (numpy and matplotlib only), fixed seed.

Run it without installing anything:
uv run https://dictionaryofml.org/terms/logloss.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download logloss.py into a folder and run uv run logloss.py there. With NumPy and Matplotlib already installed, python3 logloss.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download logloss.py · Notebook · Open in Colab

Before running a downloaded copy, scan it: uvx bandit -q -t B102,B307,B602,B605 logloss.py (B102,B307,B602,B605 are the exec, eval and shell checks of bandit) reports nothing: the script runs no shell command and evaluates no generated code.

The script, block by block

One cell per block of the script: the code, and what that code printed when it last ran here

setup

"""
logloss.py — numerical companion to the glossary entry 'logistic loss'.

Purpose
-------
Backs the entry's claims about the logistic loss numerically: divided by
log 2 it is an upper bound on the zero-one loss with equality at margin 0;
it is convex and differentiable everywhere while the hinge loss has a kink
at margin 1; the average zero-one loss is flat almost everywhere in the
model parameters, so GD gets no direction from it, whereas GD on the
average logistic loss decreases it at every step and thereby also drives
the zero-one loss down.  It also backs the entry's caveat on reading the
number itself: the logistic loss of a hypothesis is not determined by the
decisions it makes, and ranking two hypotheses by it can invert the
ranking by accuracy.  Self-contained (numpy and matplotlib only), fixed
seed.

Setup
-----
A spam filter as binary classification: m = 200 emails, each with two
features (a count of suspicious words and a count of links, scaled to
[0, 1]) and a label y in {-1, +1}.  The linear hypothesis h(x) = w^T x + b
is trained by ERM with the logistic loss, using GD with 300 updates that
subtract 0.5 times the gradient of the average logistic loss.

Blocks
------
[B-bound] On a grid of margins, log(1 + exp(-margin)) / log 2 is at least
          the zero-one loss everywhere, equal to it at margin 0 and
          strictly above it elsewhere.
[B-kink]  The hinge loss has different one-sided slopes at margin 1 (-1 and
          0); the logistic loss has the same one-sided slopes at every
          margin (differentiable everywhere).
[B-flat]  Starting from random model parameters, 200 small random changes
          of the parameters leave the average zero-one loss unchanged in
          at least 95 percent of the cases and the average logistic loss
          unchanged in none: the zero-one loss is flat almost everywhere
          and gives GD no direction.
[B-gd]    GD on the average logistic loss decreases it at every one of the
          300 updates; at the end the average zero-one loss is below 0.1
          and lies below the rescaled logistic loss, as the bound promises.
[B-equiv] The margin form with labels in {-1,+1} and the binary
          cross-entropy form with labels in {0,1} agree to within
          rounding error over a grid of hypothesis values, for both labels.
[B-interp] Three hypotheses on three data points, given by their margins.
          Scaling a hypothesis by 50 leaves every decision unchanged and
          divides its average logistic loss by 96, so the number does not
          read off the decisions.  Worse, the hypothesis with the smaller
          average logistic loss is the one with the lower accuracy: 0.329
          at accuracy 2/3 against 0.644 at accuracy 1.

Outputs
-------
logloss_margin.csv : margin, logistic loss divided by log 2, zero-one loss
                     and hinge loss on a grid of margins.
logloss_gd.csv     : update index, average logistic loss divided by log 2
                     and average zero-one loss along the GD run.
logloss_interp.csv : hypothesis, accuracy and average logistic loss for
                     the three hypotheses of [B-interp].
logloss.png        : matplotlib preview of both panels (checking only).
"""

# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///

import numpy as np
import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt

from pathlib import Path

OUT_DIR = Path(__file__).parent

report = []


def check(name, ok):
    report.append((name, bool(ok)))
    print(f"  [{'ok' if ok else 'FAIL'}] {name}")


rng = np.random.default_rng(0)
LOG2 = np.log(2.0)


def logistic(margin):
    return np.log1p(np.exp(-margin))


def zero_one(margin):
    return (margin <= 0).astype(float)


def hinge(margin):
    return np.maximum(0.0, 1.0 - margin)

B-bound

On a grid of margins, log(1 + exp(-margin)) / log 2 is at least the zero-one loss everywhere, equal to it at margin 0 and strictly above it elsewhere.

mg = np.linspace(-4.0, 4.0, 161)
rescaled = logistic(mg) / LOG2
gap = rescaled - zero_one(mg)
at_zero = abs(logistic(0.0) / LOG2 - 1.0)
check(f"[B-bound] logistic loss / log 2 >= zero-one loss on the grid "
      f"(smallest gap {gap.min():.2e}), equality at margin 0 "
      f"(|difference| {at_zero:.1e})",
      gap.min() >= -1e-12 and at_zero < 1e-12
      and (gap[np.abs(mg) > 1e-9] > 0).all())
  [ok] [B-bound] logistic loss / log 2 >= zero-one loss on the grid (smallest gap 0.00e+00), equality at margin 0 (|difference| 0.0e+00)

B-kink

The hinge loss has different one-sided slopes at margin 1 (-1 and 0); the logistic loss has the same one-sided slopes at every margin (differentiable everywhere).

eps = 1e-6
hinge_left = (hinge(1.0) - hinge(1.0 - eps)) / eps
hinge_right = (hinge(1.0 + eps) - hinge(1.0)) / eps
log_slopes = [((logistic(t) - logistic(t - eps)) / eps,
               (logistic(t + eps) - logistic(t)) / eps)
              for t in (-2.0, 0.0, 1.0, 2.0)]
max_jump = max(abs(a - b) for a, b in log_slopes)
check(f"[B-kink]  hinge loss slopes at margin 1: {hinge_left:+.2f} (left) "
      f"vs {hinge_right:+.2f} (right); logistic loss: largest one-sided "
      f"slope difference {max_jump:.1e}",
      abs(hinge_left + 1.0) < 1e-4 and abs(hinge_right) < 1e-4
      and max_jump < 1e-4)
  [ok] [B-kink]  hinge loss slopes at margin 1: -1.00 (left) vs +0.00 (right); logistic loss: largest one-sided slope difference 2.5e-07

B-flat

Starting from random model parameters, 200 small random changes of the parameters leave the average zero-one loss unchanged in at least 95 percent of the cases and the average logistic loss unchanged in none: the zero-one loss is flat almost everywhere and gives GD no direction.

m = 200
X = rng.uniform(0.0, 1.0, (m, 2))
y = np.where(0.8 * X[:, 0] + 1.1 * X[:, 1] - 0.9
             + 0.08 * rng.standard_normal(m) > 0, 1.0, -1.0)
Xb = np.c_[X, np.ones(m)]                      # (x1, x2, 1): w and b in one


def avg_zero_one(w):
    return float(np.mean(zero_one(y * (Xb @ w))))


def avg_logistic(w):
    return float(np.mean(logistic(y * (Xb @ w))))


w_rand = rng.standard_normal(3)
same_zo = sum(avg_zero_one(w_rand + 1e-4 * rng.standard_normal(3))
              == avg_zero_one(w_rand) for _ in range(200))
same_lg = sum(avg_logistic(w_rand + 1e-4 * rng.standard_normal(3))
              == avg_logistic(w_rand) for _ in range(200))
check(f"[B-flat]  small random changes of the model parameters leave the "
      f"average zero-one loss unchanged in {same_zo}/200 cases, the "
      f"average logistic loss in {same_lg}/200",
      same_zo >= 190 and same_lg == 0)
  [ok] [B-flat]  small random changes of the model parameters leave the average zero-one loss unchanged in 200/200 cases, the average logistic loss in 0/200

B-gd

GD on the average logistic loss decreases it at every one of the 300 updates; at the end the average zero-one loss is below 0.1 and lies below the rescaled logistic loss, as the bound promises.

w = np.zeros(3)
n_updates = 300
trace_lg, trace_zo = [], []
for t in range(n_updates):
    margin = y * (Xb @ w)
    grad = -(Xb * (y / (1.0 + np.exp(margin)))[:, None]).mean(axis=0)
    w = w - 0.5 * grad
    trace_lg.append(avg_logistic(w) / LOG2)
    trace_zo.append(avg_zero_one(w))
decreasing = all(b < a for a, b in zip(trace_lg[:-1], trace_lg[1:]))
check(f"[B-gd]    GD lowers the average logistic loss at every update "
      f"({trace_lg[0]:.3f} -> {trace_lg[-1]:.3f}); final average zero-one "
      f"loss {trace_zo[-1]:.3f} <= rescaled logistic loss {trace_lg[-1]:.3f}",
      decreasing and trace_zo[-1] < 0.1 and trace_zo[-1] <= trace_lg[-1])
  [ok] [B-gd]    GD lowers the average logistic loss at every update (0.986 -> 0.451); final average zero-one loss 0.070 <= rescaled logistic loss 0.451

B-equiv

The margin form with labels in {-1,+1} and the binary cross-entropy form with labels in {0,1} agree to within rounding error over a grid of hypothesis values, for both labels.

# The margin form with labels in {-1,+1} and the binary cross-entropy form
# with labels in {0,1} are the same loss: check them against each other on
# a grid of hypothesis values, for both labels.
def sigmoid(v):
    return 1.0 / (1.0 + np.exp(-v))


hv = np.linspace(-6.0, 6.0, 1201)
gap = 0.0
for y01, ypm in ((1.0, 1.0), (0.0, -1.0)):
    margin_form = logistic(ypm * hv)
    xent_form = -(y01 * np.log(sigmoid(hv))
                  + (1.0 - y01) * np.log(1.0 - sigmoid(hv)))
    gap = max(gap, float(np.abs(margin_form - xent_form).max()))
check(f"[B-equiv] the margin form and the binary cross-entropy form agree "
      f"for both labels over 1201 hypothesis values (largest gap {gap:.1e})",
      gap < 1e-12)
  [ok] [B-equiv] the margin form and the binary cross-entropy form agree for both labels over 1201 hypothesis values (largest gap 5.5e-14)
    h1               w=(-0.100,+0.050) b=+0.150  margins [0.1 0.1 0.1]  accuracy 1.00, average logistic loss 0.6444
    h1 scaled by 50  w=(-5.000,+2.500) b=+7.500  margins [5. 5. 5.]  accuracy 1.00, average logistic loss 0.0067
    h2               w=(-2.250,+1.125) b=+6.125  margins [ 5.   5.  -0.5]  accuracy 0.67, average logistic loss 0.3292

B-interp

Three hypotheses on three data points, given by their margins. Scaling a hypothesis by 50 leaves every decision unchanged and divides its average logistic loss by 96, so the number does not read off the decisions. Worse, the hypothesis with the smaller average logistic loss is the one with the lower accuracy: 0.329 at accuracy 2/3 against 0.644 at accuracy 1.

# A toy data set in the plane: three data points, two of them labelled +1.
# Three linear hypotheses h(x) = w^T x + b are compared on it.  Accuracy sees
# only the sign of the margin; the logistic loss sees its size as well.
X_TOY = np.array([[1.0, 1.0], [2.0, 3.0], [3.0, 1.0]])
Y_TOY = np.array([1.0, 1.0, -1.0])


def margins(w, b):
    return Y_TOY * (X_TOY @ w + b)


def hypothesis_for(target):
    """The affine hypothesis attaining the three given margins: three data
    points and three parameters, so the system is square."""
    A = np.column_stack([X_TOY, np.ones(len(X_TOY))]) * Y_TOY[:, None]
    c = np.linalg.solve(A, np.asarray(target, dtype=float))
    return c[:2], float(c[2])


w1, b1 = hypothesis_for([0.1, 0.1, 0.1])        # correct on all three, barely
w2, b2 = hypothesis_for([5.0, 5.0, -0.5])       # wrong on the third, confident
HYPS = {"h1": (w1, b1), "h1 scaled by 50": (50 * w1, 50 * b1), "h2": (w2, b2)}
acc, avg = {}, {}
for name, (w, b) in HYPS.items():
    mar = margins(w, b)
    acc[name] = float(np.mean(mar > 0))
    avg[name] = float(np.mean(logistic(mar)))
    print(f"    {name:16s} w=({w[0]:+.3f},{w[1]:+.3f}) b={b:+.3f}  margins "
          f"{np.round(mar, 2)}  accuracy {acc[name]:.2f}, average logistic "
          f"loss {avg[name]:.4f}")
ratio = avg["h1"] / avg["h1 scaled by 50"]
check(f"[B-interp] scaling h1 by 50 leaves every decision unchanged yet "
      f"divides its average logistic loss by {ratio:.0f}",
      acc["h1"] == acc["h1 scaled by 50"] == 1.0 and ratio > 50)
_grid = np.column_stack(np.meshgrid(np.linspace(-1, 5, 100),
                                    np.linspace(-1, 5, 100))).reshape(2, -1).T
_s1 = _grid @ w1 + b1
_s50 = _grid @ (50 * w1) + 50 * b1
# A grid point lying on the decision boundary has no decision to agree on:
# its score is zero up to rounding, and which side of zero it lands on is
# set by the summation order inside the matrix product. Comparing np.sign
# exactly therefore failed under a numpy built against a different BLAS
# (2026-11-03, moving the demos to uv), while the claim it is testing --
# that scaling the weights moves no decision -- held throughout. The
# comparison now skips the boundary itself and says how many points that
# leaves.
_off = np.abs(_s1) > 1e-9
check(f"[B-interp] scaling leaves the decision boundary where it was: the "
      f"two hypotheses agree on all {int(_off.sum())} of {_off.size} grid "
      f"points that do not lie on the boundary",
      np.array_equal(np.sign(_s1[_off]), np.sign(_s50[_off])))
check(f"[B-interp] the smaller average logistic loss belongs to the less "
      f"accurate hypothesis: {avg['h2']:.3f} at accuracy {acc['h2']:.2f} "
      f"against {avg['h1']:.3f} at accuracy {acc['h1']:.2f}",
      avg["h2"] < avg["h1"] and acc["h2"] < acc["h1"])

# ---------------------------------------------------------------- CSV
with open(OUT_DIR / "logloss_interp.csv", "w") as fh:
    fh.write("hypothesis,accuracy,avg_logistic\n")
    for name in HYPS:
        fh.write(f"{name},{acc[name]:.4f},{avg[name]:.4f}\n")
with open(OUT_DIR / "logloss_toy.csv", "w") as fh:
    fh.write("x1,x2,label\n")
    for (a, b_), lab in zip(X_TOY, Y_TOY):
        fh.write(f"{a:.3f},{b_:.3f},{lab:+.0f}\n")
with open(OUT_DIR / "logloss_bound.csv", "w") as fh:
    fh.write("x1,h1,h2\n")                       # the two decision boundaries
    for g in np.linspace(0.0, 4.0, 41):
        fh.write(f"{g:.3f},{(-w1[0]*g - b1)/w1[1]:.4f},"
                 f"{(-w2[0]*g - b2)/w2[1]:.4f}\n")
with open(OUT_DIR / "logloss_margin.csv", "w") as fh:
    fh.write("margin,logistic_rescaled,zeroone,hinge\n")
    for a, b, c, d in zip(mg, rescaled, zero_one(mg), hinge(mg)):
        fh.write(f"{a:.3f},{b:.4f},{c:.1f},{d:.4f}\n")
with open(OUT_DIR / "logloss_gd.csv", "w") as fh:
    fh.write("update,logistic_rescaled,zeroone\n")
    for t, (a, b) in enumerate(zip(trace_lg, trace_zo), start=1):
        fh.write(f"{t},{a:.4f},{b:.4f}\n")

# -------------------------------------------------------------- preview
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.2, 3.8))
ax1.plot(mg, rescaled, "k-", lw=1.4, label="logistic loss / log 2")
ax1.plot(mg, zero_one(mg), "k--", lw=1.2, label="zero-one loss")
ax1.plot(mg, hinge(mg), "k:", lw=1.6, label="hinge loss")
ax1.set_xlabel("margin $y \\cdot h(\\mathbf{x})$")
ax1.set_ylabel("loss")
ax1.set_ylim(-0.1, 4.2)
ax1.set_title("three losses as functions of the margin")
ax1.legend(frameon=False, fontsize=8)
ax2.plot(range(1, n_updates + 1), trace_lg, "k-", lw=1.4,
         label="average logistic loss / log 2")
ax2.plot(range(1, n_updates + 1), trace_zo, "k--", lw=1.2,
         label="average zero-one loss")
ax2.set_xlabel("GD update")
ax2.set_ylabel("average loss over the m emails")
ax2.set_title("GD on the logistic loss also lowers the zero-one loss")
ax2.legend(frameon=False, fontsize=8)
fig.tight_layout()
fig.savefig(OUT_DIR / "logloss.png", dpi=110)

n_ok = sum(ok for _, ok in report)
print(f"\n{n_ok}/{len(report)} checks pass")
print(f"wrote {OUT_DIR / 'logloss_margin.csv'}, {OUT_DIR / 'logloss_gd.csv'}, "
      f"{OUT_DIR / 'logloss.png'}")
if n_ok != len(report):
    raise SystemExit(1)
  [ok] [B-interp] scaling h1 by 50 leaves every decision unchanged yet divides its average logistic loss by 96
  [ok] [B-interp] scaling leaves the decision boundary where it was: the two hypotheses agree on all 9950 of 10000 grid points that do not lie on the boundary
  [ok] [B-interp] the smaller average logistic loss belongs to the less accurate hypothesis: 0.329 at accuracy 0.67 against 0.644 at accuracy 1.00

8/8 checks pass
wrote /Users/junga1/dictionaryappliedml/pythondemos/logloss_margin.csv, /Users/junga1/dictionaryappliedml/pythondemos/logloss_gd.csv, /Users/junga1/dictionaryappliedml/pythondemos/logloss.png
Preview figure produced by logloss.py
The preview figure the block B-interp writes when the script runs