Dictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

gradient

Updated on 2026-11-04

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also gradient descent convex Hilbert space

The gradient of a real-valued function is a vector that determines the local linear approximation of the function. The entries of the gradient are the partial derivatives of the function. Their existence does not by itself make the function differentiable: the vector they form need not give a local linear approximation. For a convex function, the existence of all partial derivatives at a point already makes it differentiable there. Geometrically, a nonzero gradient is orthogonal to the level sets and points in the direction of steepest ascent. In machine learning (ML), gradients of the empirical risk minimization (ERM) objective function define the update of gradient descent (GD) methods and are computed for deep networks by backpropagation.

Figure produced by the Python demo of the entry gradient
Figure written by the entry's Python demo gradient.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

B-partialsA linear model for the maximum temperature at Krems an der Donau has two model parameters, and changing either of them changes the average squared error loss on the training set. Training a machine learning (ML) model adjusts its model parameters ${\mathbf w}$ to reduce a loss $f({\mathbf w})$, which requires knowing how $f$ changes as ${\mathbf w}$ moves. The gradient of a real-valued function $f: \mathbb{R}^{d} \rightarrow \mathbb{R}: {\mathbf w} \mapsto f({\mathbf w})$ answers this by determining a local linear approximation of $f$. Formally, the gradient of $f$ at a point ${\mathbf w}' \in \mathbb{R}^{d}$ is a vector ${\mathbf g} \in \mathbb{R}^{d}$ such that \begin{equation}\label{eq_gradient_def} \lim_{{\mathbf w} \rightarrow {\mathbf w}'} \frac{f({\mathbf w}) - \big( f({\mathbf w}') + {\mathbf g}^{\top} ({\mathbf w} - {\mathbf w}') \big)}{\left\lVert {\mathbf w} - {\mathbf w}' \right\rVert_{2}} = 0 \text{.} \end{equation} If such a vector exists, it is unique, the function $f$ is differentiable at ${\mathbf w}'$, and the vector is denoted by $\nabla f({\mathbf w}')$ or $\nabla f({\mathbf w}) \big|_{{\mathbf w}'}$ (Rudin, 1976, Def. 9.11 and Thm. 9.12). The function $f({\mathbf w}') + \big(\nabla f({\mathbf w}')\big)^{\top} ({\mathbf w} - {\mathbf w}')$ is the local linear approximation of $f$ at ${\mathbf w}'$ (see Fig. 1).

Figure 1 of the entry gradient
Figure 1: A differentiable function $f: \reals \rightarrow \reals$ (thick curve) and its local linear approximation at a point $\weights'$ (thin straight line). The dashed slope triangle shows the displacement $\weights - \weights'$ (horizontal leg) and the resulting change $\big(\nabla f(\weights')\big)^{\top} (\weights - \weights')$ (vertical leg) of the local linear approximation
The entries of the gradient are the partial derivatives of $f$, $\nabla f({\mathbf w}) = \big( {\partial f}/{\partial w_{1}}, \ldots, {\partial f}/{\partial w_{d}} \big)^{\top}$; the partial derivative ${\partial f}/{\partial w_{j}}$ is the rate of change of $f$ when only the $j$-th entry of ${\mathbf w}$ varies. The existence of all partial derivatives of $f$ at a point does not guarantee that the gradient exists there (Rudin, 1976, Def. 9.16 and Thm. 9.21). Additional assumptions on $f$ restore the implication; convexity is one sufficient condition. If a convex function $f: \mathbb{R}^{d} \rightarrow \mathbb{R}$ has partial derivatives $\partial f / \partial w_{j}$, for $j = 1, \ldots, d$, at a point ${\mathbf w}'$, then $f$ is differentiable at ${\mathbf w}'$, and the gradient $\nabla f({\mathbf w}')$ is the vector of these partial derivatives (Rockafellar, 1970, Sect. 25).

B-backpropAt every point where $f$ is differentiable and $\nabla f \neq \mathbf{0}$, the gradient is orthogonal to the level set of $f$ through that point, the set $\{{\mathbf w} : f({\mathbf w}) = c\}$ of points where $f$ takes a common value $c$. It also points in the direction of steepest ascent of $f$. The negative gradient $-\nabla f({\mathbf w}')$ points in the direction of steepest descent, and gradient descent (GD) repeatedly steps along it (see Fig. 2; Boyd and Vandenberghe, 2004, Sect. 9.4.1). At a local minimum of a differentiable function $f: \mathbb{R}^{d} \rightarrow \mathbb{R}$, there cannot be any direction of descent and consequently the gradient must vanish (see zero-gradient condition).

Figure 2 of the entry gradient
Figure 2: Level sets $f(\weights) = c$ (the curves along which $f$ equals a constant $c$) of a differentiable function $f$ of two variables $\weight_{1}, \weight_{2}$ (axes, lower left), with a minimum at $\widehat{\weights}$. The gradient $\nabla f(\weights')$ (solid arrow) is orthogonal to the level set through $\weights'$ (the curve through $\weights'$) and points in the direction of steepest ascent. The negative gradient $-\nabla f(\weights')$ (dashed arrow) points in the direction of steepest descent, toward smaller values of $f$. Data generated by pythondemos/gradient.py
Gradients are used in the training of ML models. They determine the update of model parameters that minimizes the incurred loss on a training set $\mathcal{D}^{(\rm train)} = \big\{ \big({\mathbf x}^{(r)}, y^{(r)}\big) \big\}_{r=1}^{m}$, which is the goal of empirical risk minimization (ERM). For example, linear regression minimizes the objective function defined by \[ f({\mathbf w}) := \frac{1}{m} \sum_{r=1}^{m} \big( y^{(r)} - {\mathbf w}^{\top} {\mathbf x}^{(r)} \big)^{2} \text{.} \] This function is convex and differentiable, with gradient \[ \nabla f({\mathbf w}) = - \frac{2}{m} \sum_{r=1}^{m} \big( y^{(r)} - {\mathbf w}^{\top} {\mathbf x}^{(r)} \big) {\mathbf x}^{(r)} \text{.} \] For a deep artificial neural network (ANN) whose activation functions, loss, and every other operation it is composed of are differentiable, the ERM objective function is also differentiable. The gradient of this objective function can be computed by backpropagation (Goodfellow et al., 2016, Sect. 6.5; Rumelhart et al., 1986).

The definition carries over to a real-valued function $f: \mathcal{H} \rightarrow \mathbb{R}$ on a real Hilbert space $\mathcal{H}$, a Hilbert space over $\mathbb{R}$. In the defining limit \(\eqref{eq_gradient_def}\), the inner product $\langle {\mathbf g}, {\mathbf w} - {\mathbf w}' \rangle$ of $\mathcal{H}$ replaces the term ${\mathbf g}^{\top} ({\mathbf w} - {\mathbf w}')$. The norm of $\mathcal{H}$ replaces the Euclidean norm $\left\lVert {\mathbf w} - {\mathbf w}' \right\rVert_{2}$ (Bauschke and Combettes, 2011, Remark 2.44 and Def. 2.45).

See also: function, vector, differentiable, partial derivative, gradient descent, zero-gradient condition, convex, Hilbert space.

References

  1. Rudin (1976). Principles of Mathematical Analysis. McGraw-Hill.
  2. Rockafellar (1970). Convex Analysis. Princeton Univ. Press. doi.org/10.1515/9781400873173
  3. Boyd and Vandenberghe (2004). Convex Optimization. Cambridge Univ. Press. doi.org/10.1017/CBO9780511804441
  4. Goodfellow et al. (2016). Deep Learning. MIT Press.
  5. Rumelhart et al. (1986). Learning Representations by Back-Propagating Errors. Nature. doi.org/10.1038/323533a0
  6. Bauschke and Combettes (2011). Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer Science+Business Media. doi.org/10.1007/978-1-4419-9467-7

Cite this entry

@misc{dictml_gradient,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {gradient},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-04},
  url = {https://dictionaryofml.org/terms/gradient.html}
}