stochastic gradient descent

Typeset PDF version — the authoritative form of this entry

Stochastic gradient descent (SGD) is a variant of gradient descent (GD) in which the gradient of the objective function is replaced by a computationally cheaper stochastic approximation. Its main application in machine learning (ML) is empirical risk minimization (ERM) on a training set that is large or stored in a distributed database: the gradient of the empirical risk is a sum of one gradient per data point, so its exact evaluation requires a pass over the entire training set. SGD approximates this sum by the sum over a randomly chosen subset of data points, referred to as a batch. The batch size is an important parameter of SGD: it trades the computational cost of a single update against the accuracy of the gradient approximation.

Definition

SGD is obtained from gradient descent (GD) by replacing the gradient of the objective function with a stochastic approximation. A main application of SGD is to train a parameterized model via empirical risk minimization (ERM) on a training set $\dataset$ that is either large or not readily available (e.g., when data points are stored in a database distributed globally). To evaluate the gradient of the empirical risk (as a function of the model parameters $\weights$), it is necessary to compute a sum $\sum_{\sampleidx=1}^{\samplesize} \nabla_{\weights} \lossfunc{\datapoint^{(\sampleidx)}}{\weights}$ over all data points in the training set. A stochastic approximation to the gradient is obtained by replacing the sum $\sum_{\sampleidx=1}^{\samplesize} \nabla_{\weights} \lossfunc{\datapoint^{(\sampleidx)}}{\weights}$ with a sum $\sum_{\sampleidx \in \batch} \nabla_{\weights} \lossfunc{\datapoint^{(\sampleidx)}}{\weights}$ over a randomly chosen subset $\batch \subseteq \{1, \,\ldots, \,\samplesize\}$ (see Fig. 1). These randomly chosen data points are often referred to as a batch. The batch size $|\batch|$ is an important parameter of SGD. SGD with $|\batch|> 1$ is referred to as mini-batch SGD (Bottou, 1999).

Figure 1 of the entry stochGD — Figure 1: SGD for ERM approximates the gradient by replacing the sum $\sum_{\sampleidx=1}^{\samplesize} \nabla_{\weights} \lossfunc{\datapoint^{(\sampleidx)}}{\weights}$ over all data points in the training set (indexed by $\sampleidx=1, \,\ldots, \,\samplesize$) with a sum over a randomly chosen subset $\batch \subseteq \{1, \,\ldots, \,\samplesize\}$

See also: GD, gradient, objective function, stochastic, model, ERM, training set, data point, empirical risk, function, model parameter, batch, parameter.

References

Bottou (1999). On-line learning and stochastic approximations. On-Line Learning in Neural Networks.

Cite this entry

@misc{dictml_stochGD,
  author = {Jung, Alexander},
  title = {stochastic gradient descent},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-08-06},
  url = {https://dictionaryofml.org/terms/stochGD.html}
}