{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "sigmoid.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# sigmoid function \u2014 Python demo\n\nNumerical companion to the entry [sigmoid function](https://dictionaryofml.org/terms/sigmoid.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nChecks the entry's claims about the sigmoid function sigma(z) = 1 / (1 + exp(-z)): its range, limits and symmetry; the closed form of its derivative, its maximum 1/4 and its vanishing for large |z|; and the log-odds reading, where sigma of a linear function of the feature vector is exactly the posterior probability of the label when the feature vectors of the two classes are Gaussian with one shared covariance matrix. Self-contained (numpy/matplotlib only), fixed seed.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/sigmoid.py`](https://dictionaryofml.org/terms/sigmoid.py); CC BY 4.0."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"sigmoid.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"\nsigmoid.py \u2014 numerical companion to the glossary entry 'sigmoid function'.\n\nPurpose\n-------\nChecks the entry's claims about the sigmoid function sigma(z) =\n1 / (1 + exp(-z)): its range, limits and symmetry; the closed form of its\nderivative, its maximum 1/4 and its vanishing for large |z|; and the\nlog-odds reading, where sigma of a linear function of the feature vector\nis exactly the posterior probability of the label when the feature\nvectors of the two classes are Gaussian with one shared covariance\nmatrix.  Self-contained (numpy/matplotlib only), fixed seed.\n\nBlocks\n------\n[B-shape]   sigma maps the real line into (0, 1), tends to 0 and 1 at the\n            two ends, passes through 1/2 at z = 0, is strictly increasing,\n            and satisfies sigma(-z) = 1 - sigma(z).\n[B-deriv]   sigma'(z) = sigma(z)(1 - sigma(z)) matches the slope of sigma\n            measured between nearby points; it is at most 1/4, at z = 0,\n            and at |z| = 10 it is below 1e-4: a neuron with a score of\n            that size passes almost no gradient back.\n[B-logodds] Two classes with Gaussian feature vectors sharing one\n            covariance matrix: the posterior probability of the label 1\n            equals sigma(a) for the log-odds a, a is a linear function\n            w^T x + b of the feature vector, and log(sigma / (1 - sigma))\n            recovers a.\n\nOutputs\n-------\nsigmoid_curve.csv     : z, sigma(z), sigma'(z) on a grid\nsigmoid_posterior.csv : feature vectors on a line through the feature\n                        space, their log-odds, the posterior probability\n                        of label 1 and sigma of the linear function\nsigmoid.png           : matplotlib preview (checking only)\n\"\"\"\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nfrom pathlib import Path\n\nOUT_DIR = Path(__file__).parent\n\nreport = []\n\n\ndef check(name, ok):\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")\n\n\ndef sigma(z):\n    return 1.0 / (1.0 + np.exp(-z))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-shape]** sigma maps the real line into (0, 1), tends to 0 and 1 at the two ends, passes through 1/2 at z = 0, is strictly increasing, and satisfies sigma(-z) = 1 - sigma(z)."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "z = np.linspace(-12.0, 12.0, 2401)\ns = sigma(z)\ncheck(f\"[B-shape]   sigma takes values in (0, 1): on [-12, 12] between {s.min():.2e} and \"\n      f\"{s.max():.6f}\", 0.0 < s.min() and s.max() < 1.0)\ncheck(f\"[B-shape]   sigma(-40) = {sigma(-40.0):.1e} and 1 - sigma(40) = {1 - sigma(40.0):.1e}: \"\n      \"the two limits 0 and 1\", sigma(-40.0) < 1e-15 and 1 - sigma(40.0) < 1e-15)\ncheck(\"[B-shape]   sigma(0) = 1/2 and sigma is strictly increasing\",\n      sigma(0.0) == 0.5 and bool(np.all(np.diff(s) > 0)))\ncheck(f\"[B-shape]   sigma(-z) = 1 - sigma(z), to within {np.max(np.abs(sigma(-z) - (1 - s))):.1e}\",\n      np.max(np.abs(sigma(-z) - (1 - s))) < 1e-15)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-deriv]** sigma'(z) = sigma(z)(1 - sigma(z)) matches the slope of sigma measured between nearby points; it is at most 1/4, at z = 0, and at |z| = 10 it is below 1e-4: a neuron with a score of that size passes almost no gradient back."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "ds = s * (1.0 - s)\nslope = (sigma(z + 1e-5) - sigma(z - 1e-5)) / 2e-5\ncheck(f\"[B-deriv]   sigma'(z) = sigma(z)(1 - sigma(z)) matches the measured slope to within \"\n      f\"{np.max(np.abs(ds - slope)):.1e}\", np.max(np.abs(ds - slope)) < 1e-8)\ncheck(f\"[B-deriv]   the derivative is at most 1/4, reached at z = {z[np.argmax(ds)]:.2f}\",\n      abs(ds.max() - 0.25) < 1e-12 and z[np.argmax(ds)] == 0.0)\ncheck(f\"[B-deriv]   at |z| = 10 the derivative is {sigma(10.0) * (1 - sigma(10.0)):.1e}: almost \"\n      \"no gradient passes back\", sigma(10.0) * (1 - sigma(10.0)) < 1e-4)\nwith open(OUT_DIR / \"sigmoid_curve.csv\", \"w\") as fh:\n    fh.write(\"z,sigma,derivative\\n\")\n    for zi, si, di in zip(z[::20], s[::20], ds[::20]):\n        fh.write(f\"{zi:.2f},{si:.6f},{di:.6f}\\n\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-logodds]** Two classes with Gaussian feature vectors sharing one covariance matrix: the posterior probability of the label 1 equals sigma(a) for the log-odds a, a is a linear function w^T x + b of the feature vector, and log(sigma / (1 - sigma)) recovers a."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# two classes in the plane, Gaussian feature vectors with one shared\n# covariance matrix; the posterior probability of label 1 follows from the\n# two densities and the class probabilities\nrng = np.random.default_rng(0)\nmu1, mu0 = np.array([1.5, 0.5]), np.array([-1.0, -0.5])\nC = np.array([[1.0, 0.4], [0.4, 0.8]]); Cinv = np.linalg.inv(C)\np1 = 0.4                                            # probability of the label 1\n\n\ndef density(x, mu):\n    d = x - mu\n    return np.exp(-0.5 * np.einsum(\"ij,jk,ik->i\", d, Cinv, d)) / (2 * np.pi * np.sqrt(np.linalg.det(C)))\n\n\nw = Cinv @ (mu1 - mu0)\nb = -0.5 * mu1 @ Cinv @ mu1 + 0.5 * mu0 @ Cinv @ mu0 + np.log(p1 / (1 - p1))\nxs = rng.standard_normal((2000, 2)) * 1.5\nlogodds = np.log(density(xs, mu1) * p1) - np.log(density(xs, mu0) * (1 - p1))\nposterior = density(xs, mu1) * p1 / (density(xs, mu1) * p1 + density(xs, mu0) * (1 - p1))\ncheck(f\"[B-logodds] the posterior probability of the label 1 equals sigma(a) for the log-odds a, \"\n      f\"to within {np.max(np.abs(posterior - sigma(logodds))):.1e}\",\n      np.max(np.abs(posterior - sigma(logodds))) < 1e-12)\ncheck(f\"[B-logodds] with one shared covariance matrix the log-odds is linear, a = w^T x + b with \"\n      f\"w = ({w[0]:.2f}, {w[1]:.2f}), b = {b:.2f}, to within \"\n      f\"{np.max(np.abs(logodds - (xs @ w + b))):.1e}\", np.max(np.abs(logodds - (xs @ w + b))) < 1e-10)\ncheck(\"[B-logodds] log(sigma / (1 - sigma)) recovers the log-odds\",\n      np.max(np.abs(np.log(posterior / (1 - posterior)) - logodds)) < 1e-8)\nt = np.linspace(-3.0, 3.0, 121)\nline = np.outer(t, (mu1 - mu0) / np.linalg.norm(mu1 - mu0)) + (mu1 + mu0) / 2\npost_line = density(line, mu1) * p1 / (density(line, mu1) * p1 + density(line, mu0) * (1 - p1))\nwith open(OUT_DIR / \"sigmoid_posterior.csv\", \"w\") as fh:\n    fh.write(\"t,x1,x2,logodds,posterior,sigma_linear\\n\")\n    for ti, xi, lo, po in zip(t, line, line @ w + b, post_line):\n        fh.write(f\"{ti:.2f},{xi[0]:.4f},{xi[1]:.4f},{lo:.4f},{po:.6f},{sigma(lo):.6f}\\n\")\n\n# -------------------------------------------------------------- preview\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.5, 3.6))\nax1.plot(z, s, \"k-\", label=r\"$\\sigma(z)$\")\nax1.plot(z, ds, \"k--\", label=r\"$\\sigma'(z) = \\sigma(z)(1-\\sigma(z))$\")\nax1.axhline(0.25, color=\"0.6\", lw=0.8, ls=\":\", label=\"1/4\")\nax1.set_xlabel(\"score $z$\"); ax1.set_ylabel(\"value\"); ax1.set_title(\"the sigmoid function and its derivative\")\nax1.legend(frameon=False, fontsize=8)\nax2.plot(t, post_line, \"k-\", lw=2.5, alpha=0.4, label=\"posterior probability of label 1\")\nax2.plot(t, sigma(line @ w + b), \"k--\", label=r\"$\\sigma(\\mathbf{w}^\\top\\mathbf{x}+b)$\")\nax2.set_xlabel(\"position along the line between the class means\"); ax2.set_ylabel(\"probability\")\nax2.set_title(\"two Gaussian classes, one covariance matrix\"); ax2.legend(frameon=False, fontsize=8)\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"sigmoid.png\", dpi=110)\n\nn_ok = sum(ok for _, ok in report)\nprint(f\"\\n{n_ok}/{len(report)} checks pass\")\nprint(f\"wrote {OUT_DIR / 'sigmoid_curve.csv'}, {OUT_DIR / 'sigmoid_posterior.csv'}, {OUT_DIR / 'sigmoid.png'}\")\nif n_ok != len(report):\n    raise SystemExit(1)"
  }
 ]
}