Dictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

stochastic gradient descent (SGD)

Updated on 2026-11-03

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also gradient descent gradient empirical risk minimization

Stochastic gradient descent (SGD) is a variant of gradient descent (GD) in which the gradient of the objective function is replaced by a computationally cheaper stochastic approximation. Its main application in machine learning (ML) is empirical risk minimization (ERM) on a training set that is large or stored in a distributed database: the gradient of the empirical risk requires an average over the entire training set. SGD approximates this average by the average over a randomly drawn batch of data points. The batch size trades the computational cost of a single update against the accuracy of the gradient approximation.

Figure produced by the Python demo of the entry stochastic gradient descent (SGD)
Figure written by the entry's Python demo stochGD.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

B-weatherConsider the training set of Fig. 1: twelve weather stations, each contributing one data point with the minimum temperature of a day as feature and its maximum temperature as label, stored on two servers, one of GeoSphere Austria and one of the Finnish Meteorological Institute (FMI). The training set holds $m$ data points. Linear regression learns a linear hypothesis from it, so that its predictions match the labels. That amounts to finding the minimum of a differentiable convex objective function, the empirical risk \[ f({\mathbf w}) := \frac{1}{m} \sum_{r=1}^{m} L\left({\mathbf z}^{(r)},{\mathbf w} \right) \text{.} \] The training set is $\mathcal{D}^{(\rm train)} = \big\{ {\mathbf z}^{(1)}, \ldots, {\mathbf z}^{(m)} \big\}$, and each data point ${\mathbf z}^{(r)}$ is a feature vector together with its label. Here, $L\left({\mathbf z}^{(r)},{\mathbf w} \right)$ is the loss that the hypothesis with model parameters ${\mathbf w}$ incurs on ${\mathbf z}^{(r)}$.

B-approxGradient descent (GD) searches for a minimum of a differentiable convex function by repeating the gradient step \[ {\mathbf w}^{(t+1)} = {\mathbf w}^{(t)} - \eta^{(t)} \nabla f({\mathbf w}^{(t)}) \text{,} \] with a step size $\eta^{(t)} > 0$ that is small enough for each step to decrease $f$ (see GD). The gradient of the above objective function is $\nabla f({\mathbf w}) = \frac{1}{m} \sum_{r=1}^{m} \nabla_{{\mathbf w}} L\left({\mathbf z}^{(r)},{\mathbf w} \right)$, an average with one term per data point. Its exact evaluation therefore requires a pass over the entire training set, which is computationally expensive when $m$ is large. SGD instead uses the cheaper estimate \[ g({\mathbf w}) := \frac{1}{B} \sum_{r \in \mathcal{B}} \nabla_{{\mathbf w}} L\left({\mathbf z}^{(r)},{\mathbf w} \right) \text{,} \] the average of the same terms over a batch $\mathcal{B} \subseteq \{1, \ldots, m\}$ of $B = |\mathcal{B}|$ indices chosen uniformly at random (see Fig. 1).

Figure 1 of the entry stochGD
Figure 1: A training set of $\samplesize = 12$ data points: each is the daily minimum temperature (feature $\feature$) and daily maximum temperature (label $\truelabel$) observed on 1 July 2026 at a different weather station, stored on two servers, at Geosphere.at (filled circles) and FMI.fi (filled squares). The dotted vertical segments are the prediction errors of a linear hypothesis map $\hypothesis$ (solid curve). A GD update of the model parameters $\weights$ uses the prediction errors of all $\samplesize$ data points; an SGD update uses only those of a randomly chosen batch $\batch$ (inside red rings). Data generated by pythondemos/stochGD.py, which downloads the observations from the open data services of GeoSphere Austria and the FMI
Each SGD iteration generates a new batch $\mathcal{B}^{(t)}$ and performs the GD update with the stochastic estimate in place of the gradient (see Fig. 2). The update reads \[ {\mathbf w}^{(t+1)} = {\mathbf w}^{(t)} - \eta^{(t)} \frac{1}{B} \sum_{r \in \mathcal{B}^{(t)}} \nabla_{{\mathbf w}} L\left({\mathbf z}^{(r)},{\mathbf w}^{(t)} \right) \text{,} \] with a step size $\eta^{(t)} > 0$. For a uniformly drawn batch, the expectation of the update is the GD step ${\mathbf w} \mapsto {\mathbf w} - \eta^{(t)} \nabla f({\mathbf w})$. The fixed points of that step are the points satisfying the zero-gradient condition $\nabla f({\mathbf w}) = \mathbf{0}$. One classical set of sufficient conditions for the convergence of such random iterations to a minimum of $f$ comes from stochastic approximation (Robbins and Monro, 1951). The step sizes must diminish such that $\sum_{t=1}^{\infty} \eta^{(t)} = \infty$ while $\sum_{t=1}^{\infty} \big(\eta^{(t)}\big)^{2} < \infty$ (Bertsekas, 2016, Sect. 2.4.1).
Figure 2 of the entry stochGD
Figure 2: SGD as GD on an approximation of the objective function: the empirical risk, the average loss over all $\samplesize$ data points of the training set (solid curve), and the average over one randomly drawn batch $\batch$ (dashed curve). The gradient of the dashed curve is the stochastic estimate that SGD uses in place of the gradient of the solid one
The batch size $B$ is the parameter of SGD that sets the cost and the noise of one update: $B = m$ recovers GD, while $B = 1$ updates with the gradient of a single data point and yields the noisiest estimate. SGD with $1 < B < m$ is mini-batch SGD (Bottou, 1999). Each update costs $B$ gradient evaluations instead of $m$. That lower cost is what makes empirical risk minimization (ERM) on a large training set feasible, which is the main application of SGD in machine learning (ML). The update is noisier than a gradient step, and its variance shrinks as the batch grows (Bottou, 2010). A larger batch draws the dashed curve of Fig. 2 closer to the solid one. At $B = m$, the two coincide; at $B = 1$, they lie furthest apart.

When the training set is spread over several servers, as in Fig. 1, a batch can be drawn from one server at a time, so no server needs the whole training set.

See also: gradient descent, gradient-based method, gradient, empirical risk minimization, batch, step size, objective function, zero-gradient condition, online gradient descent.

References

  1. Robbins and Monro (1951). A Stochastic Approximation Method. The Annals of Mathematical Statistics. doi.org/10.1214/aoms/1177729586
  2. Bertsekas (2016). Nonlinear Programming. Athena Scientific.
  3. Bottou (1999). On-line learning and stochastic approximations. On-Line Learning in Neural Networks.
  4. Bottou (2010). Large-scale machine learning with stochastic gradient descent. Proceedings of COMPSTAT'2010. doi.org/10.1007/978-3-7908-2604-3_16

Cite this entry

@misc{dictml_stochGD,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {stochastic gradient descent (SGD)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-04},
  url = {https://dictionaryofml.org/terms/stochGD.html}
}