Dictionary of Applied Machine Learning

gradient

Updated on 2026-09-17

Typeset PDF version — the authoritative form of this entry

Python demo — a script that recomputes what this entry states and prints one line per check

The gradient of a real-valued function is a vector that determines the local linear approximation of the function. The entries of the gradient are the partial derivatives of the function. Their existence does not by itself make the function differentiable: the vector they form need not give a local linear approximation. For a convex function, it does. Geometrically, a nonzero gradient is orthogonal to the level sets and points in the direction of steepest ascent. In machine learning (ML), gradients of the empirical risk minimization (ERM) objective function drive gradient descent (GD) methods and are computed for deep networks by backpropagation.

Definition

B-partialsTraining a machine learning (ML) model adjusts its model parameters $\weights$ to reduce a loss $f(\weights)$, which requires knowing how $f$ changes as $\weights$ moves. The gradient of a real-valued function $f: \reals^{\featuredim} \rightarrow \reals: \weights \mapsto f(\weights)$ answers this by determining a local linear approximation of $f$. Formally, the gradient of $f$ at a point $\weights' \in \reals^{\featuredim}$ is a vector $\vg \in \reals^{\featuredim}$ such that \begin{equation}\label{eq_gradient_def} \lim_{\weights \rightarrow \weights'} \frac{f(\weights) - \big( f(\weights') + \vg^{\top} (\weights - \weights') \big)}{\normgeneric{\weights - \weights'}{2}} = 0 \text{.} \end{equation} If such a vector exists, it is unique, the function $f$ is differentiable at $\weights'$, and the vector is denoted by $\nabla f(\weights')$ or $\nabla f(\weights) \big|_{\weights'}$ (Rudin, 1976, Ch. 9). The function $f(\weights') + \big(\nabla f(\weights')\big)^{\top} (\weights - \weights')$ is the local linear approximation of $f$ at $\weights'$ (see Fig. 1).

Figure 1 of the entry gradient
Figure 1: A differentiable function $f: \reals \rightarrow \reals$ (thick curve) and its local linear approximation at a point $\weights'$ (thin straight line). The dashed slope triangle shows the displacement $\weights - \weights'$ (horizontal leg) and the resulting change $\big(\nabla f(\weights')\big)^{\top} (\weights - \weights')$ (vertical leg) of the local linear approximation
The entries of the gradient are the partial derivatives of $f$, $\nabla f(\weights) = \big( {\partial f}/{\partial \weight_{1}}, \ldots, {\partial f}/{\partial \weight_{\featuredim}} \big)^{\top}$; the partial derivative ${\partial f}/{\partial \weight_{\featureidx}}$ is the rate of change of $f$ when only the $\featureidx$-th entry of $\weights$ varies. In general, the existence of all partial derivatives of $f$ at a point does not guarantee that the gradient exists there (Rudin, 1976, Ch. 9). Additional assumptions on $f$ restore the implication; convexity is one sufficient condition. If a convex function $f: \reals^{\featuredim} \rightarrow \reals$ has partial derivatives $\partial f / \partial \weight_{\featureidx}$, for $\featureidx = 1, \ldots, \featuredim$, at a point $\weights'$, then $f$ is differentiable at $\weights'$, and the gradient $\nabla f(\weights')$ is the vector of these partial derivatives (Rockafellar, 1970, Sect. 25).

B-hilbertThe definition carries over to a real-valued function $f: \hilbertspace \rightarrow \reals$ on a real Hilbert space $\hilbertspace$ (a Hilbert space over $\reals$; the complex case needs a separate treatment of the sesquilinear inner product): the inner product $\innerprod{\vg}{\weights - \weights'}$ of $\hilbertspace$ replaces the term $\vg^{\top} (\weights - \weights')$ in the defining limit \(\eqref{eq_gradient_def}\), and the norm of $\hilbertspace$ replaces the Euclidean norm $\normgeneric{\weights - \weights'}{2}$ (Bauschke and Combettes, 2011).

B-backpropThe gradient has a geometric interpretation. At every point where $f$ is differentiable and $\nabla f \neq \mathbf{0}$, the gradient is orthogonal to the level set of $f$ through that point — the set $\{\weights : f(\weights) = c\}$ of points where $f$ takes a common value $c$ — and it points in the direction of steepest ascent of $f$. The negative gradient $-\nabla f(\weights')$ points in the direction of steepest descent, and gradient descent (GD) repeatedly steps along it (see Fig. 2; Boyd and Vandenberghe, 2004, Sect. 9.4). At a local minimum of a differentiable function $f: \reals^{\featuredim} \rightarrow \reals$, there cannot be any direction of descent and consequently the gradient must vanish (see zero-gradient condition).

Figure 2 of the entry gradient
Figure 2: Level sets $f(\weights) = c$ (the curves along which $f$ equals a constant $c$) of a differentiable function $f$ of two variables $\weight_{1}, \weight_{2}$ (axes, lower left), with a minimum at $\widehat{\weights}$. The gradient $\nabla f(\weights')$ (solid arrow) is orthogonal to the level set through $\weights'$ (the curve through $\weights'$) and points in the direction of steepest ascent. The negative gradient $-\nabla f(\weights')$ (dashed arrow) points in the direction of steepest descent, toward smaller values of $f$. Data generated by pythondemos/gradient.py
Gradients are instrumental for the training of ML models. They guide the update of model parameters to minimize the incurred loss on a training set $\trainset = \big\{ \big(\featurevec^{(\sampleidx)}, \truelabel^{(\sampleidx)}\big) \big\}_{\sampleidx=1}^{\samplesize}$, which is the goal of empirical risk minimization (ERM). As a case in point, linear regression minimizes the objective function defined by \[ f(\weights) \defeq \frac{1}{\samplesize} \sum_{\sampleidx=1}^{\samplesize} \big( \truelabel^{(\sampleidx)} - \weights^{\top} \featurevec^{(\sampleidx)} \big)^{2} \text{.} \] This function is convex and differentiable, with gradient \[ \nabla f(\weights) = - \frac{2}{\samplesize} \sum_{\sampleidx=1}^{\samplesize} \big( \truelabel^{(\sampleidx)} - \weights^{\top} \featurevec^{(\sampleidx)} \big) \featurevec^{(\sampleidx)} \text{.} \] For a deep artificial neural network (ANN) whose activation functions, loss, and every other operation it is composed of are differentiable, the ERM objective function is also differentiable. The gradient of this objective function can be computed by backpropagation (Goodfellow et al., 2016, Sect. 6.5; Rumelhart et al., 1986).

See also: function, vector, differentiable, partial derivative, gradient descent, zero-gradient condition, convex, Hilbert space.

References

  1. Rudin (1976). Principles of Mathematical Analysis. McGraw-Hill.
  2. Rockafellar (1970). Convex Analysis. Princeton Univ. Press. doi.org/10.1515/9781400873173
  3. Bauschke and Combettes (2011). Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer Science+Business Media. doi.org/10.1007/978-1-4419-9467-7
  4. Boyd and Vandenberghe (2004). Convex Optimization. Cambridge Univ. Press. doi.org/10.1017/CBO9780511804441
  5. Goodfellow et al. (2016). Deep Learning. MIT Press.
  6. Rumelhart et al. (1986). Learning Representations by Back-Propagating Errors. Nature. doi.org/10.1038/323533a0

Cite this entry

@misc{dictml_gradient,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {gradient},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-21},
  url = {https://dictionaryofml.org/terms/gradient.html}
}